Skip to content

[improve][doc] Add chunking documentation for DLQ/RLQ and add integration tests#1511

Open
geniusjoe wants to merge 1 commit into
apache:masterfrom
geniusjoe:dev/EnableChunking-comment
Open

[improve][doc] Add chunking documentation for DLQ/RLQ and add integration tests#1511
geniusjoe wants to merge 1 commit into
apache:masterfrom
geniusjoe:dev/EnableChunking-comment

Conversation

@geniusjoe

Copy link
Copy Markdown
Contributor

Master Issue: apache/pulsar#21048

Related Issue: #805

Motivation

In the Java SDK (PR apache/pulsar#21048), the DLQ/RLQ producers are hardcoded with
enableBatching(false).enableChunking(true) to ensure large chunked messages can be forwarded to DLQ/RLQ
topics successfully.

However, in the Go SDK, the DLQ/RLQ producers are created using DLQPolicy.ProducerOptions, which defaults
to a zero-value ProducerOptions{} (i.e., EnableChunking: false, DisableBatching: false). This means
that if a producer sends a chunked message larger than the broker's maxMessageSize, and the consumer is
configured with DLQ/RLQ, the DLQ/RLQ producer will fail to forward the message because it cannot chunk it.

We considered automatically enabling chunking in the DLQ/RLQ producers when the user has not explicitly
configured ProducerOptions. However, since EnableChunking and DisableBatching are both bool fields
with a zero value of false, there is no way to distinguish between:

  • The user intentionally setting them to false (meaning "I don't want chunking")
  • The user not configuring them at all (meaning "I didn't think about it")

Automatically overriding these values would break backward compatibility for users who explicitly set
EnableChunking: false. Therefore, instead of changing the default behavior, we add documentation to
guide users to enable chunking in DLQPolicy.ProducerOptions when using chunked messages with DLQ/RLQ.

Modifications

  • Added a note to the EnableChunking field documentation in ProducerOptions (producer.go) explaining
    that when chunking is enabled and the consumer uses DLQ/RLQ, it is recommended to also enable chunking
    in DLQPolicy.ProducerOptions.
  • Added a note to the DLQPolicy.ProducerOptions field documentation (consumer.go) explaining that if
    the messages being consumed were produced with chunking enabled, it is recommended to also enable chunking
    here. This helps in decoupled producer/consumer scenarios where the consumer developer may not be aware
    of the producer's chunking configuration.
  • Added TestChunkReconsumeLater integration test to verify chunked messages (5MB, exceeding broker
    maxMessageSize) can be sent to the retry topic via ReconsumeLater and routed to DLQ after exceeding
    max retries, when DLQPolicy.ProducerOptions has chunking enabled.
  • Added TestChunkDLQWithNack integration test to verify chunked messages (5MB) are routed to DLQ
    after exceeding max redelivery count via Nack, when DLQPolicy.ProducerOptions has chunking enabled.

Verifying this change

This change added tests and can be verified as follows:

  • Added TestChunkReconsumeLater: sends a 5MB chunked message, verifies it flows through retry topic
    via ReconsumeLater and eventually lands in DLQ after exceeding MaxDeliveries.
  • Added TestChunkDLQWithNack: sends a 5MB chunked message, verifies it is routed to DLQ after
    exceeding max redelivery count via Nack, with original properties preserved.

Both tests use payloads larger than broker maxMessageSize (~1MB) to ensure that DLQ/RLQ producers
must have chunking enabled to successfully forward messages. Without chunking enabled in
DLQPolicy.ProducerOptions, these tests would timeout.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API: no
  • The schema: no
  • The default values of configurations: no
  • The wire protocol: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? GoDocs
  • If a feature is not applicable for documentation, explain why?
    This PR adds documentation (GoDoc comments) to guide users on how to properly configure DLQ/RLQ
    when using chunked messages.

@geniusjoe geniusjoe changed the title [improve] Add chunking documentation for DLQ/RLQ and add integration tests [improve][doc] Add chunking documentation for DLQ/RLQ and add integration tests Jun 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant