Glue DataLakeCatalog ALTER schema commit fails after 100 retries

[DimensionWieldr]

Type of problem

Bug report - something's broken

Describe the situation

An Iceberg ALTER that has to publish a new schema through a Glue DataLakeCatalog never commits. ClickHouse retries the commit 100 times and then returns LIMIT_EXCEEDED. The same statements commit on the REST catalog.

Reproduced on the latest Antalya release, 26.6.4.20001.altinityantalya (v26.6.4.20001.altinityantalya), against LocalStack Glue.

Two statements fail the same way:

• ADD COLUMN extra Nullable(String) FIRST on a table (name, value)
• MODIFY COLUMN id Nullable(Int64) AFTER id, which widens int to long and leaves id where it is

Statements that do not publish a schema succeed on that same Glue database: rejecting DROP of a partition or sort column, rejecting a type change that both sides map to the same Iceberg type, and MODIFY COLUMN id Nullable(Int32) AFTER id when id is already Nullable(Int32).

This issue:

• Returns code 290 LIMIT_EXCEEDED on the release build
• Hits both ADD COLUMN ... FIRST and a type-changing MODIFY COLUMN
• The REST catalog commits both statements

---

How to reproduce the behavior

Environment

• Version: 26.6.4.20001.altinityantalya (Antalya release, git 2073b1f88dec885444fac97edb8179ad1753c1ad)
• Build type: release
• Image: altinity/clickhouse-server:26.6.4.20001.altinityantalya
• Catalog: LocalStack Glue (http://localstack:4566, region us-east-1). Table files are in MinIO at http://minio:9000/warehouse. Glue credentials used by ClickHouse are the MinIO root user admin / password.

Steps

1. Two unpartitioned Iceberg tables already exist in Glue, each with one data row and no sort order.

namespace.placed has fields name (optional string) and value (optional long), and the row ('Alice', 20).

namespace.ids has fields name (optional string) and id (optional int), and the row ('Alice', 1).

2. Attach the catalog:

SET allow_experimental_database_glue_catalog = 1;

CREATE DATABASE datalake
ENGINE = DataLakeCatalog('http://localstack:4566')
SETTINGS
    catalog_type = 'glue',
    storage_endpoint = 'http://minio:9000/warehouse',
    region = 'us-east-1',
    aws_access_key_id = 'admin',
    aws_secret_access_key = 'password';

3. Add a column at the front of namespace.placed:

SET allow_insert_into_iceberg = 1;

ALTER TABLE datalake.`namespace.placed`
    ADD COLUMN extra Nullable(String) FIRST;

4. On namespace.ids, restate the current type and then widen it. The first statement succeeds and does not change the schema id. The second fails:

ALTER TABLE datalake.`namespace.ids`
    MODIFY COLUMN id Nullable(Int32) AFTER id;

ALTER TABLE datalake.`namespace.ids`
    MODIFY COLUMN id Nullable(Int64) AFTER id;

---

Expected behavior

Both statements commit a new schema.

extra is added to namespace.placed.

id on namespace.ids becomes Iceberg long (Nullable(Int64) in ClickHouse) and stays after name. The row is still Alice, 1.

Both statements commit against a REST catalog.

---

Actual behavior

On release builds

Each schema-publishing statement fails after 100 attempts.

Code: 290. DB::Exception: Received from localhost:9000. DB::Exception: ALTER TABLE commit did not succeed after 100 retries (concurrent modification or catalog rejection). (LIMIT_EXCEEDED)

ADD COLUMN extra FIRST fails on that first statement. On namespace.ids, MODIFY COLUMN id Nullable(Int32) AFTER id succeeds and leaves the schema id unchanged. The widening MODIFY then returns the error above. id stays Nullable(Int32).

---

Root cause analysis

changeSchema in src/Storages/ObjectStorage/DataLakes/Iceberg/Mutations.cpp retries up to MAX_TRANSACTION_RETRIES (100). For a catalog with isTransactional() == false, ClickHouse writes the new metadata file and then calls ICatalog::updateSchema. Glue uses that default. REST overrides isTransactional() to true, sends the schema with assert-current-schema-id, and lets the catalog write the metadata file.

GlueCatalog::updateSchema copies the new columns into a Glue UpdateTable request and stores metadata_location. UpdateTable failure throws DATALAKE_DATABASE_ERROR immediately (Can not update metadata in glue catalog ...). It does not return false. A run of 100 retries therefore means writeMetadataFileAndVersionHint returned false on every attempt. The Glue UpdateTable call was not the call that returned code 290.

ICatalog::updateSchema documents this split. new_metadata_path is for non-transactional catalogs such as Glue. previous_schema_id is the optimistic check used by transactional catalogs such as Iceberg REST. Glue has no compare-and-swap on the schema id.

On this release, generateAddColumnMetadata and generateModifyColumnMetadata are called without the FIRST / AFTER arguments. The statements still enter this commit loop, and that loop is what returns code 290.

---

Additional context

Release

• Tag: v26.6.4.20001.altinityantalya
• Commit: 2073b1f88dec885444fac97edb8179ad1753c1ad
• Catalog: Glue via LocalStack. The same statements against the REST catalog succeed.

Related

• A Glue INSERT that fails with code 48, Metadata changed during write operation, was seen on a later build and is tracked separately: #2453. That insert succeeds on this release.
• #2090 was the REST form of a catalog commit that failed after the metadata file was written. It is closed. This failure is the Glue commit path, and the error text is the 100-retry LIMIT_EXCEEDED.
• #2215 is the REST commit-unknown retry for a nested ADD COLUMN. That one returns Column already exists. It is a different bug.