Support aggregate function states in Parquet and Iceberg by zvonand · Pull Request #2301 · Altinity/ClickHouse

zvonand · 2026-09-02T06:53:48Z

Closes #2206

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support aggregate function states in Parquet and Iceberg

CI/CD Options

Exclude tests:

Regression jobs to run:

zvonand · 2026-09-02T06:54:03Z

@codex review

chatgpt-codex-connector · 2026-09-02T06:54:23Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-02T07:16:26.909845Z	`090c16a`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

github-actions · 2026-09-02T06:55:07Z

Workflow [PR], commit [e44403c]

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 090c16ac1f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

zvonand · 2026-09-03T16:09:21Z

@blau-ai

Reuse `ISerialization` for aggregate state conversion, trim nonessential comments, and consolidate overlapping tests while preserving distinct format and Iceberg coverage. PR: #2301

Parquet and Iceberg have no aggregate-state type, so neither can describe an `AggregateFunction` or `SimpleAggregateFunction` column with its schema alone. This adds a ClickHouse-only annotation next to the data - the `clickhouse.column_types` key of a parquet file's footer, and a `clickhouse.type` key on an Iceberg schema field - naming the ClickHouse type so it can be rebuilt on read. An `AggregateFunction` state is stored as an opaque `BYTE_ARRAY` / `binary` holding the bytes the `-State` combinator produces; a `SimpleAggregateFunction(f, T)` is stored as an ordinary value of `T`. Other query engines ignore the key and see a plain binary or `T`-typed column. The recorded name always carries the state version, which pins the serialized layout: `getName` drops a zero version, so a versioned function pinned to version 0 would otherwise be rebuilt with the default version and its bytes misread. `getNameForAnnotation` keeps it. Two experimental settings gate the feature, both off by default: `allow_experimental_aggregate_function_states_in_parquet` for writing such a column and for reconstructing one during parquet schema inference, and `allow_experimental_aggregate_function_states_in_iceberg` for `CREATE TABLE` and for honouring a `clickhouse.type` key that names an `AggregateFunction`. Honouring the annotation lets the file or the table metadata, rather than the query, choose the deserializer the stored bytes are handed to, so a refused annotation is rejected rather than read as `String` - reading states as strings would be a wrong result, not an error. `SimpleAggregateFunction` needs no opt-in on read, holding ordinary values with no state deserializer involved. The annotation is checked against the type the schema derives on its own before it is honoured, so a stale or crafted one cannot re-type a column to anything of the same nesting shape. The parquet reader checks strictly, requiring a type its own writer maps to what the file holds; Iceberg checks the nesting structure, which is all its type system allows. The Iceberg gate is read from the context of the query that parses the schema rather than from one captured at `ATTACH`, so a reading query can opt in at all; `IcebergSchemaProcessor::addIcebergTableSchema` publishes a schema-id only once every field has parsed and drops what it wrote on failure, so a refused read can be retried with the setting on. The metadata prefetcher asks for the schema-id with `SchemaParsing::Skip`, since it has no query to carry the gate and must not publish a schema that later queries would be served. The parquet schema cache is keyed by both settings, so a schema inferred with a gate open is not served to a query that has it closed. `ALTER TABLE ... EXPORT PART` and `EXPORT PARTITION` from an `AggregatingMergeTree` table into an Iceberg table carry both gates: `EXPORT PARTITION` records them in its ZooKeeper manifest, so every replica executing the task applies the values the `ALTER` gave instead of its own profile, and the destination is resolved under those settings. A manifest without the fields, written before they existed, reads them as closed. Iceberg records no bounds for aggregate states, whose extremes cannot be compared, and parquet min/max statistics over serialized states are never used for pruning. Adds `04673_parquet_aggregate_function_state`, integration tests for the export paths and for round-tripping states through Spark, and unit tests for the annotation matching, the name parsing, the schema processor and the metadata generator. Documents the feature in `Parquet.md` and `iceberg.md`. PR: #2301

…ion-states-in-parquet-iceberg

Reuse `removeLowCardinalityAndNullable`, inline the one-use schema helper, and remove redundant footer state and comments. Share metadata-test lookup code while preserving all regression scenarios. Validated with a Debug build, 103 targeted unit tests, and the Parquet aggregate-state regression matching its reference output. Related: #2301

…et-iceberg' of github.com:Altinity/ClickHouse into feature/antalya-26.6/aggregate-function-states-in-parquet-iceberg

…ion-states-in-parquet-iceberg

arthurpassos

I have reviewed the docs and the parquet implementation, looks ok so far. I'll review the iceberg impl and the tests tomorrow. Few comments below

arthurpassos · 2026-10-01T19:43:45Z

 namespace parq = parquet::format;

+/// Maps top-level columns to ClickHouse types not represented by the Parquet schema.
+constexpr const char * clickhouse_column_types_key = "clickhouse.column_types";


hm.. it would be funny if upstream one day decided to use the very same kvp for a different implementation, but I suppose it is extremely unlikely

well, yes, there is always a chance of something like this. but I do not see any other option actually.

I think I just take this PR into upstream later, there is (mostly) nothing antalya-specific in it

arthurpassos · 2026-10-01T19:45:20Z

+            Poco::JSON::Parser parser;
+            const auto object = parser.parse(kv.value).extract<Poco::JSON::Object::Ptr>();
+            for (const auto & name : object->getNames())
+                clickhouse_column_type_names[name] = object->getValue<String>(name);


A json key value pair inside a parquet key value parquet, that's funny, but probably the best way

arthurpassos · 2026-10-01T19:47:03Z

    std::unordered_map<String, GeoColumnMetadata> geo_columns;

+    /// Type names from the `clickhouse.column_types` footer metadata.
+    std::unordered_map<String, String> clickhouse_column_type_names;


I would make the name a bit more descriptive by adding the custom keyword

agree. claude suggested annotated_column_type_names, which IMO looks even better

arthurpassos · 2026-10-01T19:52:31Z


+DataTypePtr SchemaConverter::resolveAnnotatedType(const String & column_name, const String & type_name) const
+{
+    ASTPtr ast;


Instead of parsing text into an AST, ins't it easier to simply do the below? Ofc it assumes type_name is the "full type name" the data type factory requires

auto data_type = DataTypeFactory::instance().get(type_name)); if (WhichDataType(data_type).isAggregateFunction() && !options.format.parquet.allow_aggregate_function_states) throw

Ok, I suppose it is not that simple because it might be a nested data type. In that case you'd need something similar to what you've implemented in needsClickHouseTypeAnnotation

arthurpassos · 2026-10-01T19:56:04Z

+            {
+                annotated_type = resolveAnnotatedType(col.name, it->second);
+
+                if (!annotatedTypeMatchesDerived(annotated_type, col.output_type, /*strict=*/ true))


I feel like annotatedTypeMatchesDerived should be called from inside the resolveAnnotatedType, but it is up to you

arthurpassos

Reviewed the tests, looks ok. Pending part is the iceberg implementation

`SchemaConverter::resolveAnnotatedType` now takes the type derived from the parquet schema and verifies it against the annotated type with `annotatedTypeMatchesDerived`, so it only returns a validated type. Related: #2301 (comment)

…egate_function_states_in_open_formats` Replace `allow_experimental_aggregate_function_states_in_parquet` and `allow_experimental_aggregate_function_states_in_iceberg` with a single setting. Both gated the same trust decision: honouring a ClickHouse type recorded in the data, which chooses the deserializer for the stored bytes. Each check keeps its current behavior. The export task and manifest now carry one flag. Related: #2301 (comment)

zvonand added the antalya-26.6 label Sep 2, 2026

chatgpt-codex-connector Bot reviewed Sep 2, 2026

View reviewed changes

Comment thread src/Processors/Formats/Impl/Parquet/Write.cpp Outdated

Comment thread src/Processors/Formats/Impl/Parquet/SchemaConverter.cpp Outdated

Comment thread src/Storages/ObjectStorage/DataLakes/Iceberg/DataFileStatistics.cpp

This comment was marked as outdated.

Sign in to view

zvonand force-pushed the feature/antalya-26.6/aggregate-function-states-in-parquet-iceberg branch 2 times, most recently from d5099b8 to ea636f0 Compare September 3, 2026 10:46

This comment was marked as outdated.

Sign in to view

CarlosFelipeOR mentioned this pull request Sep 14, 2026

CAS: object-store connections exhaust the container's ephemeral ports, fencing out the mount lease #2243

Closed

zvonand force-pushed the feature/antalya-26.6/aggregate-function-states-in-parquet-iceberg branch from de5caea to 322a47b Compare September 24, 2026 11:23

zvonand added the port-antalya PRs to be ported to all new Antalya releases label Sep 28, 2026

Merge branch 'antalya-26.6' into feature/antalya-26.6/aggregate-funct…

d7ac016

…ion-states-in-parquet-iceberg

svb-alt requested a review from arthurpassos September 28, 2026 16:19

zvonand added 3 commits September 30, 2026 14:04

Merge branch 'feature/antalya-26.6/aggregate-function-states-in-parqu…

632a76d

…et-iceberg' of github.com:Altinity/ClickHouse into feature/antalya-26.6/aggregate-function-states-in-parquet-iceberg

Merge branch 'antalya-26.6' into feature/antalya-26.6/aggregate-funct…

08eedde

…ion-states-in-parquet-iceberg

arthurpassos reviewed Oct 1, 2026

View reviewed changes

arthurpassos reviewed Oct 2, 2026

View reviewed changes

Comment thread tests/integration/test_export_merge_tree_part_to_iceberg/test.py Outdated

Comment thread tests/queries/0_stateless/02902_topKGeneric_deserialization_memory.sql Outdated

zvonand added 5 commits October 2, 2026 15:28

remove dumb test change

12b3b95

small improvements

542bf18

rename

0d68264

Conversation

zvonand commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

zvonand commented Sep 2, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

github-actions Bot commented Sep 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

zvonand commented Sep 3, 2026

Uh oh!

This comment was marked as outdated.

arthurpassos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arthurpassos Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zvonand Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

zvonand Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

arthurpassos Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

zvonand Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos Oct 1, 2026

Choose a reason for hiding this comment

Uh oh!

arthurpassos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

zvonand commented Sep 2, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 2, 2026 •

edited

Loading

github-actions Bot commented Sep 2, 2026 •

edited

Loading

arthurpassos Oct 1, 2026 •

edited

Loading

zvonand Oct 2, 2026 •

edited

Loading