[improve][doc] Add chunking documentation for DLQ/RLQ and add integration tests#1511
Open
geniusjoe wants to merge 1 commit into
Open
[improve][doc] Add chunking documentation for DLQ/RLQ and add integration tests#1511geniusjoe wants to merge 1 commit into
geniusjoe wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Master Issue: apache/pulsar#21048
Related Issue: #805
Motivation
In the Java SDK (PR apache/pulsar#21048), the DLQ/RLQ producers are hardcoded with
enableBatching(false).enableChunking(true)to ensure large chunked messages can be forwarded to DLQ/RLQtopics successfully.
However, in the Go SDK, the DLQ/RLQ producers are created using
DLQPolicy.ProducerOptions, which defaultsto a zero-value
ProducerOptions{}(i.e.,EnableChunking: false,DisableBatching: false). This meansthat if a producer sends a chunked message larger than the broker's
maxMessageSize, and the consumer isconfigured with DLQ/RLQ, the DLQ/RLQ producer will fail to forward the message because it cannot chunk it.
We considered automatically enabling chunking in the DLQ/RLQ producers when the user has not explicitly
configured
ProducerOptions. However, sinceEnableChunkingandDisableBatchingare bothboolfieldswith a zero value of
false, there is no way to distinguish between:false(meaning "I don't want chunking")Automatically overriding these values would break backward compatibility for users who explicitly set
EnableChunking: false. Therefore, instead of changing the default behavior, we add documentation toguide users to enable chunking in
DLQPolicy.ProducerOptionswhen using chunked messages with DLQ/RLQ.Modifications
EnableChunkingfield documentation inProducerOptions(producer.go) explainingthat when chunking is enabled and the consumer uses DLQ/RLQ, it is recommended to also enable chunking
in
DLQPolicy.ProducerOptions.DLQPolicy.ProducerOptionsfield documentation (consumer.go) explaining that ifthe messages being consumed were produced with chunking enabled, it is recommended to also enable chunking
here. This helps in decoupled producer/consumer scenarios where the consumer developer may not be aware
of the producer's chunking configuration.
TestChunkReconsumeLaterintegration test to verify chunked messages (5MB, exceeding brokermaxMessageSize) can be sent to the retry topic via
ReconsumeLaterand routed to DLQ after exceedingmax retries, when
DLQPolicy.ProducerOptionshas chunking enabled.TestChunkDLQWithNackintegration test to verify chunked messages (5MB) are routed to DLQafter exceeding max redelivery count via
Nack, whenDLQPolicy.ProducerOptionshas chunking enabled.Verifying this change
This change added tests and can be verified as follows:
TestChunkReconsumeLater: sends a 5MB chunked message, verifies it flows through retry topicvia
ReconsumeLaterand eventually lands in DLQ after exceedingMaxDeliveries.TestChunkDLQWithNack: sends a 5MB chunked message, verifies it is routed to DLQ afterexceeding max redelivery count via
Nack, with original properties preserved.Both tests use payloads larger than broker
maxMessageSize(~1MB) to ensure that DLQ/RLQ producersmust have chunking enabled to successfully forward messages. Without chunking enabled in
DLQPolicy.ProducerOptions, these tests would timeout.Does this pull request potentially affect one of the following parts:
Documentation
This PR adds documentation (GoDoc comments) to guide users on how to properly configure DLQ/RLQ
when using chunked messages.