Glue DataLakeCatalog INSERT fails with Metadata changed during write operation #2453

Type of problem

Bug report - something's broken

Describe the situation

An INSERT into an Iceberg table through a Glue DataLakeCatalog fails before any row is written. ClickHouse retries the write after a metadata conflict, re-reads the table metadata, and then stops because the schema id no longer matches the one the insert started with.

Observed on the amd_release build of PR #2289 commit 1050cd4aa3ce5a842a40978afc5990c0055b4503, against LocalStack Glue. The same INSERT against a REST catalog on that binary succeeds.

This issue:

• Returns code 48 NOT_IMPLEMENTED on the release build
• Fails on the INSERT. No ALTER runs before it
• The REST catalog accepts the same insert

How to reproduce the behavior

Environment

• Version: 26.6.4.20001.altinityantalya
• Build type: release (build_amd_release)
• Binary: https://altinity-build-artifacts.s3.amazonaws.com/PRs/2289/1050cd4aa3ce5a842a40978afc5990c0055b4503/build_amd_release/clickhouse
• Catalog: LocalStack Glue (http://localstack:4566, region us-east-1). Table files are in MinIO at http://minio:9000/warehouse. Glue credentials used by ClickHouse are the MinIO root user admin / password.

Steps

1. An unpartitioned Iceberg table namespace.events already exists in Glue. It has no partition spec and no sort order. Its schema is:

2. Attach that catalog from ClickHouse:

SET allow_experimental_database_glue_catalog = 1;

CREATE DATABASE datalake
ENGINE = DataLakeCatalog('http://localstack:4566')
SETTINGS
    catalog_type = 'glue',
    storage_endpoint = 'http://minio:9000/warehouse',
    region = 'us-east-1',
    aws_access_key_id = 'admin',
    aws_secret_access_key = 'password';

3. Insert ten rows in one statement. One row has this shape; the other nine use the same column types:

SET allow_insert_into_iceberg = 1;

INSERT INTO datalake.`namespace.events` VALUES (
    true,
    toInt32(48945440),
    toInt64(0),
    toFloat32(3.4028235e38),
    toFloat64(-2.690978632545191e38),
    toDecimal32(2849446.23, 2),
    '1900-01-01',
    toTime64('23:59:59.999999', 6),
    toDateTime64('1900-01-01 00:00:00.000000', 6),
    toDateTime64('1970-01-01 00:00:00.000000', 6, 'UTC'),
    '',
    '',
    'aaaaa',
    '481782c1-84a2-a731-c002-d3fb91ed2876',
    true,
    toInt64(4500638031972640604),
    toDecimal32(-4037558.56, 2),
    'example',
    [toInt32(-2147483648)],
    ['a'],
    [[toInt32(-2147483648)]],
    map('key', toInt64(1)),
    tuple(toInt32(1), 'v')
);

Expected behavior

The insert commits a new data snapshot and leaves the schema id unchanged. That is what this INSERT does against a REST catalog on the same binary.

Actual behavior

On release builds

The insert fails. No row is written.

Code: 48. DB::Exception: Received from localhost:9000. DB::Exception: Metadata changed during write operation, try again: While executing WaitForAsyncInsert. (NOT_IMPLEMENTED)

Root cause analysis

Iceberg::write treats a metadata conflict as retryable. On that retry it re-reads the latest metadata with ignore_explicit_metadata_file_path = true. If current-schema-id differs from the schema id captured at the start of the insert, it throws NOT_IMPLEMENTED instead of continuing the retry (src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergWrites.cpp, the check next to Metadata changed during write operation).

Glue is not a transactional catalog. ICatalog::isTransactional() defaults to false, and RestCatalog overrides it to true. For a non-transactional catalog, ClickHouse writes vN.metadata.json itself and then asks the catalog to store that path. GlueCatalog::updateMetadata calls UpdateTable and throws DATALAKE_DATABASE_ERROR when Glue rejects the call, so a false return from the write path is writeMetadataFileAndVersionHint failing. The insert then re-reads metadata and hits the schema-id check above.

A schema-committing ALTER on the same Glue setup fails separately, after 100 retries: #2454. Reads, and ALTER statements that do not publish a new schema, succeed. That includes rejecting DROP of a partition or sort column, and a MODIFY COLUMN that records nothing because the type is unchanged.

Additional context

Build

• Branch: 2091-support-firstafter-keywords-for-alter-table-modify-column-in-iceberg-tables (PR #2289)
• Commit: 1050cd4aa3ce5a842a40978afc5990c0055b4503
• Catalog: Glue via LocalStack. The same INSERT against the REST catalog on this binary succeeds.

Related

• #2454 — Glue ALTER that must publish a new schema returns code 290 after 100 retries.
• Closed Glue ALTER issues #2083, #2084, and #2085 describe different symptoms.