Skip to content

feat(services): support if_match conditional writes for azblob and gcs#7890

Open
shanielh wants to merge 1 commit into
apache:mainfrom
shanielh:feat/conditional-writes-gcs-azblob
Open

feat(services): support if_match conditional writes for azblob and gcs#7890
shanielh wants to merge 1 commit into
apache:mainfrom
shanielh:feat/conditional-writes-gcs-azblob

Conversation

@shanielh

@shanielh shanielh commented Jul 8, 2026

Copy link
Copy Markdown

Which issue does this PR close?

Closes #7889

Rationale for this change

Conditional-write support (if_match / if_none_match) was inconsistent across object-storage services: S3 supported if_match but not arbitrary-ETag if_none_match (a genuine AWS API limitation — S3's x-amz-if-none-match only accepts the literal *), Azure Blob supported if_none_match but not if_match, and GCS supported neither for writes. This PR closes the two gaps that are actually fixable given each provider's API (Azure's if_match, GCS's generation-based conditional writes), while leaving S3 as-is since its gap is not implementable through the S3 API.

What changes are included in this PR?

  • Azure Blob: added write_with_if_match capability and wired the If-Match header into azblob_put_blob_request, mirroring the existing if_none_match handling.
  • GCS: added write_with_if_match and write_with_if_none_match capabilities. GCS's JSON API has no ETag-based conditional write mechanism, only generation-number query params, so these are implemented on top of ifGenerationMatch / ifGenerationNotMatch. The object generation number is exposed via the existing Metadata::version() field (already populated by GCS's stat/read/list paths), so no new Metadata fields were needed. Non-numeric if_match/if_none_match values return an Unsupported error.
  • Capability docs: documented in core/core/src/types/capability.rs that GCS repurposes if_match/if_none_match to carry a generation number rather than a literal ETag; added a corresponding note to GCS's docs.md.
  • Behavior tests: updated the shared test_write_with_if_match / test_write_with_if_none_match tests (core/tests/behavior/async_write.rs) to use the object's generation (Metadata::version()) as the match token for GCS and its ETag for all other backends, since enabling the new capability flags makes these shared tests exercise GCS for the first time.
  • No changes to S3.

Are there any user-facing changes?

Yes:

  • Azure Blob users can now pass if_match to write_with, matching the behavior already available for if_none_match.
  • GCS users can now pass if_match / if_none_match to write_with for optimistic-concurrency writes, but must pass the object's generation number (from Metadata::version()), not a literal ETag — this is called out in the capability doc comments and GCS's docs.md.

AI Usage Statement

This PR was developed with Claude Code (Anthropic), model Claude Sonnet 5, including codebase investigation, implementation, and verification (cargo fmt, cargo clippy -D warnings, cargo check --all-features, doc tests). Live behavior tests against real GCS/Azure backends were not run — no credentials were available in the development environment.

@shanielh shanielh requested a review from Xuanwo as a code owner July 8, 2026 13:58
@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. releases-note/feat The PR implements a new feature or has a title that begins with "feat" labels Jul 8, 2026
Azure Blob's Put Blob API supports If-Match with an arbitrary ETag,
so wire it in alongside the existing If-None-Match support.

GCS's JSON API has no ETag-based conditional write mechanism, only
generation-number query params. Implement write_with_if_match and
write_with_if_none_match on top of ifGenerationMatch/ifGenerationNotMatch,
reusing the object generation already exposed via Metadata::version().

S3 is unchanged: its If-None-Match header only accepts the literal
wildcard, which is already covered by if_not_exists().
@shanielh shanielh force-pushed the feat/conditional-writes-gcs-azblob branch from e5dd147 to 253431f Compare July 9, 2026 05:46

@Xuanwo Xuanwo left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for working on this.

The azure part seems good to me, but I strongly againest the gcs change.

pub write_with_cache_control: bool,
/// Indicates if conditional write operations using If-Match are supported.
///
/// The value passed to `if_match` is normally a literal ETag. GCS is an exception:

@Xuanwo Xuanwo Jul 9, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we don't do this. It's very clear that if-match accepts an etag.


Think about this, we now have a user want to support all of s3, azblob and gcs.

The user is using APIs like:

let expected_etag = op.stat(pathx).await?.etag();
let _ = op.write_with(pathx, data).if_match(expected_etag).await?;

Then this user will go crazy to find that it doesn't work on gcs, because if_match always didn't match the give etag. After reading docs, they realized that they need to do:

let expected_etag = if op.scheme() == "gcs" {
    op.stat(pathx).await?.version()
} else {
    op.stat(pathx).await?.etag()
};

let _ = op.write_with(pathx, data).if_match(expected_etag).await?;

I don't think they will love opendal again.

/// Indicates if conditional write operations using If-None-Match are supported.
///
/// The value passed to `if_none_match` is normally a literal ETag. GCS is an exception:
/// it has no ETag-based conditional write API, so it repurposes this field to carry

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same.

write_with_content_type: true,
write_with_content_encoding: true,
write_with_user_metadata: true,
write_with_if_match: true,

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't declare a feature that services didn't support.

@shanielh

shanielh commented Jul 9, 2026

Copy link
Copy Markdown
Author

@Xuanwo maybe we should add a new capability for write if version match? I believe the write if none is fine for GCS.

If so. I'll be happy to implement it for GCS.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat" size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

services: inconsistent conditional write (if_match/if_none_match) support across s3, gcs, azblob

2 participants