From dfae79dfb93a6f0b040f1b38b8a0c563b2583e43 Mon Sep 17 00:00:00 2001 From: Chris Constable Date: Mon, 9 Nov 2026 15:46:55 -0700 Subject: [PATCH] update external storage spec with more detail about encoding/decoding references. --- features/external_storage/spec.md | 73 +++++++++++++++++++------------ 1 file changed, 46 insertions(+), 27 deletions(-) diff --git a/features/external_storage/spec.md b/features/external_storage/spec.md index 8630e80b..13f3d0ee 100644 --- a/features/external_storage/spec.md +++ b/features/external_storage/spec.md @@ -1,6 +1,6 @@ # External Storage Specification -Last updated: 2026-10-15 +Last updated: 2026-11-09 ## Motivation @@ -87,12 +87,15 @@ offloaded bytes. It is not part of the exchange contract: a producer that omits produces a reference that any SDK can read, and a consumer must not depend on it being present. -The payload that finally goes into the request also carries the metadata item -`__temporal_external_storage_reference`, with the value `true`. The SDK attaches this item after the -codec has encoded the reference, so it sits on the outermost payload where a reader can see -it without decoding anything. When no codec is configured the reference is itself the -outermost payload, and it carries the item just the same. This follows the same convention as -the `__temporal_system_payload` item that marks a system Nexus envelope. +`__temporal_external_storage_reference` is a metadata item, with the value `true`, that +identifies a payload as a reference. It exists because the payload codec also encodes the +reference, and after that the encoding and message type above are no longer visible, so a +reader could not otherwise tell a reference from any other payload. The SDK attaches this item +after the codec has encoded the reference, so it sits on the outermost payload, the one that +goes into the request, where a reader can see it without decoding anything. When no codec is +configured the reference is itself the outermost payload, and it carries the item just the +same. This follows the same convention as the `__temporal_system_payload` item that marks a +system Nexus envelope. A payload therefore needs retrieval when either of two conditions holds, and both are readable from metadata alone: @@ -103,7 +106,8 @@ readable from metadata alone: Neither half of the second condition is sufficient on its own, because `json/protobuf` is an ordinary encoding used for any protobuf payload. The second condition exists for a reference -that no codec encoded, and for a history written before the marker existed. +that no codec encoded, and for a history written before `__temporal_external_storage_reference` +existed. Earlier prereleases used an encoding of `json/external-storage-reference` with a different body. That encoding is being removed before general availability. An SDK must not produce it, @@ -141,7 +145,7 @@ sequenceDiagram SDK->>+Codec: encode(the new references only) Codec-->>-SDK: encoded references - Note over SDK: mark each encoded reference
and restore its recorded size + Note over SDK: add __temporal_external_storage_reference
to each encoded reference
and restore its recorded size SDK->>-Server: outgoing request ``` @@ -177,8 +181,8 @@ held had the payload stayed inline. ### Retrieval The retrieval path runs whenever an SDK decodes payloads it received from the server. It is the -inverse of the offload path, but it is not symmetric with it, because the marker lets the SDK find the references -without decoding anything first. +inverse of the offload path, but it is not symmetric with it, because `__temporal_external_storage_reference` +lets the SDK find the references without decoding anything first. ```mermaid sequenceDiagram @@ -191,14 +195,14 @@ sequenceDiagram actor App as Application Server->>+SDK: incoming response - Note over SDK: find the marked payloads
from metadata alone + Note over SDK: find the references
from metadata alone - SDK->>+Store: retrieve(the marked payloads) - Store->>+Codec: decode(a marked payload) + SDK->>+Store: retrieve(the references) + Store->>+Codec: decode(an encoded reference) Codec-->>-Store: the reference Store->>+Driver: retrieve(context, claims) Driver-->>-Store: stored payloads, still encoded - Store-->>-SDK: each marked payload replaced by its stored payload + Store-->>-SDK: each reference replaced by its stored payload SDK->>+Codec: decode(all payloads) Codec-->>-SDK: decoded payloads @@ -214,9 +218,19 @@ The steps are: 1. Find the payloads that need retrieval, using the two metadata conditions in the wire format section. This reads metadata only. No payload is decoded, and a payload that needs no retrieval is not touched. -2. For each of those payloads, obtain the reference. A payload carrying the marker was - encoded by the codec, so decode that one payload to get the reference. A payload that - matches on encoding and message type is already the reference. +2. For each of those payloads, obtain the reference. A payload that matches on encoding and + message type is already the reference. A payload carrying + `__temporal_external_storage_reference` was encoded by the codec, so decode that one + payload, as it appears in history, to get the reference. Then check that the result is a + reference, by the encoding and message type condition. If it is not, fail with + `TMPRL1106`. This is what happens when the codec that wrote the reference is not the codec + configured to read it. When no codec is configured, a payload carrying + `__temporal_external_storage_reference` is already the reference, and it must still pass + the same check. + + `__temporal_external_storage_reference` and `external_payloads` stay on the payload the + codec decodes. A codec must already tolerate them, because a codec server receives these + payloads exactly as they appear in history. 3. Read the driver name and the claim data. Group the claims by driver name and look up each driver by that name. A reference naming a driver that is not registered is an error. 4. Call each driver once with its group of claims. A driver returns one payload for each @@ -239,7 +253,8 @@ has already decoded. Step 1 reads metadata and nothing else. An SDK that applies its codec downstream of this point, per data-converter call, cannot decode every payload here to look for references: the payloads would reach the data converter already decoded and be decoded again. Detecting from -metadata, and decoding only a marked payload, leaves every other payload untouched. +metadata, and decoding only a payload carrying `__temporal_external_storage_reference`, leaves +every other payload untouched. The codec still runs the same number of times as an unconditional decode would have given it: once for a payload that was never offloaded, twice for one that was. @@ -252,9 +267,10 @@ An SDK that finds a payload needing retrieval and has no external storage config fail with `TMPRL1105`. This applies to every SDK, including one with no external storage support at all; the alternative is handing a reference to application code as though it were the application's own data. Such an SDK should say so in the error rather than telling the -user to configure something it does not have. Detecting this needs no codec and no driver. +user to configure something it does not have. Detecting this needs no codec and no driver, so +an SDK checks for it before decoding any reference. -A history written before the marker existed still reads: its references were never +A history written before `__temporal_external_storage_reference` existed still reads: its references were never codec-encoded, so step 1 finds them by the second metadata condition, already in plain form. #### When to retrieve @@ -636,6 +652,9 @@ Three channels identify the failure, and two of them are the same in every SDK: operator alerts on. - A reference found with no external storage configured is `TMPRL1105`. This is where the difference between a misconfiguration and a driver outage is carried. +- A payload carrying `__temporal_external_storage_reference` that the configured codec does + not decode to a reference is `TMPRL1106`. + This is a codec misconfiguration, not a driver outage. - The task failure cause, where the path has one, is `WORKFLOW_TASK_FAILED_CAUSE_EXTERNAL_STORAGE_FAILURE` for a workflow task and `ACTIVITY_TASK_FAILED_CAUSE_EXTERNAL_STORAGE_FAILURE` for an activity task, rather than a @@ -645,8 +664,8 @@ Three channels identify the failure, and two of them are the same in every SDK: The type on the failure itself is not standardized, and an SDK need not use a separate type for each kind of storage failure. -`TMPRL1105` stays retryable: the customer fixes it by configuring external storage, so the -worker retries until the unit of work exhausts its attempts or its timeout. The cost is that +`TMPRL1105` and `TMPRL1106` stay retryable: the customer fixes them by configuring external +storage or the right codec, so the worker retries until the unit of work exhausts its attempts or its timeout. The cost is that a misconfigured worker stalls quietly, which is why the observability below is required and not optional. @@ -752,15 +771,15 @@ compare names instead, rather than letting the panic escape. Go applies the payload codec per data-converter call, inside workflow execution, rather than as one pass over a message. The external storage layer must therefore be able to invoke the -codec itself: to decode a marked payload on the inbound path, and to encode a new reference -on the outbound path. +codec itself: to decode an encoded reference on the inbound path, and to encode a new +reference on the outbound path. ### Java Java applies the payload codec per data-converter call, inside the data converter the user supplies, rather than as one pass over a message. The external storage layer must therefore -be able to invoke the codec itself: to decode a marked payload on the inbound path, and to -encode a new reference on the outbound path. +be able to invoke the codec itself: to decode an encoded reference on the inbound path, and +to encode a new reference on the outbound path. ### Python