From 62fabfd1faebff6775118a6ae7cc86237b73de72 Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Thu, 17 Sep 2026 15:15:02 +0300 Subject: [PATCH 01/22] Update contrib/datasketches-cpp to 5.2.0 and backport the HLL union fix Backport of ClickHouse/ClickHouse@a067567ebb179fcf59cd20ef4dbb003c6625d6c7, re-created here rather than cherry-picked so that it carries a sign-off. The submodule tracked `apache/datasketches-cpp` directly, pinned to `76edd74f` (2024-05-16), an upstream development commit. It moves onto a `ClickHouse/`-prefixed branch of our fork, the way `docs/development/contrib` asks: ClickHouse/datasketches-cpp, branch ClickHouse/5.2.0 de8553ba 5.2.0, the newest upstream release (2025-01-15) 23bd9b07 backport of apache/datasketches-cpp#512 apache/datasketches-cpp#512 is the HyperLogLog union fix. No upstream release carries it, so it is cherry-picked onto the `5.2.0` tag there. `uniqApacheHLL`, added next, needs it: without it a merged HLL sketch reports a wrong estimate and serializes a state that other DataSketches implementations read differently. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: UnamedRus --- .gitmodules | 2 +- contrib/datasketches-cpp | 2 +- ...5055_uniq_theta_compressed_state.reference | 1 + .../05055_uniq_theta_compressed_state.sql | 22 +++++++++++++++++++ 4 files changed, 25 insertions(+), 2 deletions(-) create mode 100644 tests/queries/0_stateless/05055_uniq_theta_compressed_state.reference create mode 100644 tests/queries/0_stateless/05055_uniq_theta_compressed_state.sql diff --git a/.gitmodules b/.gitmodules index 6acf8b86ad42..343e61166d95 100644 --- a/.gitmodules +++ b/.gitmodules @@ -154,7 +154,7 @@ url = https://github.com/ClickHouse/NuRaft [submodule "contrib/datasketches-cpp"] path = contrib/datasketches-cpp - url = https://github.com/apache/datasketches-cpp + url = https://github.com/ClickHouse/datasketches-cpp [submodule "contrib/yaml-cpp"] path = contrib/yaml-cpp url = https://github.com/ClickHouse/yaml-cpp diff --git a/contrib/datasketches-cpp b/contrib/datasketches-cpp index 76edd74f5db2..23bd9b070ddf 160000 --- a/contrib/datasketches-cpp +++ b/contrib/datasketches-cpp @@ -1 +1 @@ -Subproject commit 76edd74f5db286b672c170a8ded4ce39b3a8800f +Subproject commit 23bd9b070ddf7f0dcf32c87d6752ce535374697b diff --git a/tests/queries/0_stateless/05055_uniq_theta_compressed_state.reference b/tests/queries/0_stateless/05055_uniq_theta_compressed_state.reference new file mode 100644 index 000000000000..5a931d5663aa --- /dev/null +++ b/tests/queries/0_stateless/05055_uniq_theta_compressed_state.reference @@ -0,0 +1 @@ +16 16 30 diff --git a/tests/queries/0_stateless/05055_uniq_theta_compressed_state.sql b/tests/queries/0_stateless/05055_uniq_theta_compressed_state.sql new file mode 100644 index 000000000000..9668fc48c74e --- /dev/null +++ b/tests/queries/0_stateless/05055_uniq_theta_compressed_state.sql @@ -0,0 +1,22 @@ +-- Tags: no-fasttest +-- - no-fasttest -- compiled w/o datasketches + +-- Apache DataSketches can serialize a Theta sketch in a compressed form (serialization +-- version 4) that packs the retained values at a variable number of bits per entry. +-- ClickHouse always writes the uncompressed form, but it has to read the compressed one, +-- because an `AggregateFunction(uniqTheta, ...)` state can come from another +-- implementation of the library. Two of the unpacking routines decoded it incorrectly, +-- and the result was a silently too low estimate rather than an error. + +-- Both states below are canonical compressed encodings of a sketch that retains 16 values +-- with theta = 1, so the estimate is exact. The first packs its values at 33 bits per +-- entry and the second at 35; the two widths are decoded by separate routines. The two +-- sketches have exactly two values in common, so their union retains 30. + +WITH + CAST(unhex('4B01040321011ACC9310000001F4000000FA0000007D0000003E8000001F4000000FA0000007D180000000000001F4000000FA0000007D0000003E8000001F4000000FA0000007D0000003E8') AS AggregateFunction(uniqTheta, UInt64)) AS state_33_bits, + CAST(unhex('4F01040323011ACC93100000007D0000000FA1000000004000000000000007D0000000FA0000001F40000003E80000007D0000000FA1000000000000003E80000007D0000000FA0000001F40000003E8') AS AggregateFunction(uniqTheta, UInt64)) AS state_35_bits +SELECT + finalizeAggregation(state_33_bits), + finalizeAggregation(state_35_bits), + finalizeAggregation(uniqThetaUnion(state_33_bits, state_35_bits)); From 8cd058668b70adfb39dca33309142019d4905ae0 Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Thu, 17 Sep 2026 15:15:03 +0300 Subject: [PATCH 02/22] Add uniqApacheHLL: Apache DataSketches HLL as a native aggregate function Ported from `vk/uniq-apache-hll` (github.com/UnamedRus/ClickHouse), which develops the function against `master`. `uniqApacheHLL` counts distinct values into an Apache DataSketches HLL sketch, and its `-State` serializes that sketch in the DataSketches format, so states can be exchanged with Java, Python and C++ services through the standard `-State`/`-Merge` combinators. Only the argument types those libraries hash the same way are accepted - integers of at most 64 bits, Enum8/16, BFloat16, Float32/64, String, FixedString, UUID, IPv4, IPv6, Date, Date32, DateTime and DateTime64 - so no state can be built here that an external consumer cannot reproduce. Two deviations from the branch this is taken from, both forced by the age of this base: - the state overrides `merge`, not `mergeImpl`. `IAggregateFunction::merge` is still the pure virtual here; the split into a non-virtual `merge` plus a `mergeImpl` override came later. - `introduced_in` says 26.6 rather than 26.9, this being the release it ships in. NOT BUILT OR TESTED on this base - only ported and checked by inspection against the 26.6 headers. CI is the first real build. Co-Authored-By: Claude Opus 5 (1M context) Signed-off-by: UnamedRus --- contrib/datasketches-cpp-cmake/CMakeLists.txt | 1 + .../AggregateFunctionUniqApacheHLL.cpp | 150 +++++++++ .../AggregateFunctionUniqApacheHLL.h | 295 ++++++++++++++++++ .../registerAggregateFunctions.cpp | 6 + .../04327_uniq_apache_hll.reference | 19 ++ .../0_stateless/04327_uniq_apache_hll.sql | 60 ++++ .../05026_uniq_apache_hll_interop.reference | 23 ++ .../05026_uniq_apache_hll_interop.sql | 95 ++++++ ...5027_uniq_apache_hll_cross_scale.reference | 11 + .../05027_uniq_apache_hll_cross_scale.sql | 47 +++ ...8_uniq_apache_hll_argument_types.reference | 26 ++ .../05028_uniq_apache_hll_argument_types.sql | 54 ++++ ..._uniq_apache_hll_corrupted_state.reference | 3 + .../05136_uniq_apache_hll_corrupted_state.sh | 37 +++ 14 files changed, 827 insertions(+) create mode 100644 src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp create mode 100644 src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h create mode 100644 tests/queries/0_stateless/04327_uniq_apache_hll.reference create mode 100644 tests/queries/0_stateless/04327_uniq_apache_hll.sql create mode 100644 tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference create mode 100644 tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql create mode 100644 tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.reference create mode 100644 tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.sql create mode 100644 tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.reference create mode 100644 tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.sql create mode 100644 tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.reference create mode 100755 tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.sh diff --git a/contrib/datasketches-cpp-cmake/CMakeLists.txt b/contrib/datasketches-cpp-cmake/CMakeLists.txt index 497d6956d0ef..ff4199c17430 100644 --- a/contrib/datasketches-cpp-cmake/CMakeLists.txt +++ b/contrib/datasketches-cpp-cmake/CMakeLists.txt @@ -10,6 +10,7 @@ add_library(_datasketches INTERFACE) target_include_directories(_datasketches SYSTEM BEFORE INTERFACE "${ClickHouse_SOURCE_DIR}/contrib/datasketches-cpp/common/include" "${ClickHouse_SOURCE_DIR}/contrib/datasketches-cpp/count/include" + "${ClickHouse_SOURCE_DIR}/contrib/datasketches-cpp/hll/include" "${ClickHouse_SOURCE_DIR}/contrib/datasketches-cpp/theta/include") add_library(ch_contrib::datasketches ALIAS _datasketches) diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp new file mode 100644 index 000000000000..a9425146ea95 --- /dev/null +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp @@ -0,0 +1,150 @@ +#include +#include +#include + +#if USE_DATASKETCHES + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int ARGUMENT_OUT_OF_BOUND; + extern const int BAD_ARGUMENTS; + extern const int ILLEGAL_TYPE_OF_ARGUMENT; + extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH; +} + +AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( + const std::string & name, const DataTypes & argument_types, const Array & params, const Settings *) +{ + uint8_t lg_config_k = 12; + datasketches::target_hll_type target_type = datasketches::HLL_4; + + if (params.size() > 2) + throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, + "Aggregate function {} accepts at most two parameters (lg_k, type).", name); + + if (!params.empty()) + { + const UInt64 lg_k_param = applyVisitor(FieldVisitorConvertToNumber(), params[0]); + if (lg_k_param < 4 || lg_k_param > 21) + throw Exception(ErrorCodes::ARGUMENT_OUT_OF_BOUND, + "Parameter lg_k for aggregate function {} is out of range: [4, 21].", name); + lg_config_k = static_cast(lg_k_param); + } + + if (params.size() == 2) + { + if (params[1].getType() != Field::Types::String) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Parameter type for aggregate function {} must be a string.", name); + + const String type_param = params[1].safeGet(); + if (type_param == "HLL_4") + target_type = datasketches::HLL_4; + else if (type_param == "HLL_6") + target_type = datasketches::HLL_6; + else if (type_param == "HLL_8") + target_type = datasketches::HLL_8; + else + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Parameter type for aggregate function {} must be one of 'HLL_4', 'HLL_6', 'HLL_8'.", name); + } + + /// Only the types a sketch hashes the same way in every language are accepted, so that no state + /// is built here that an external consumer cannot reproduce. Anything else - a decimal, a wide + /// integer, an array, a tuple, several arguments - would first need a hash only ClickHouse has. + if (argument_types.size() != 1) + throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, + "Aggregate function {} requires exactly one argument, passed {}. Several arguments would " + "have to be hashed to one value by ClickHouse first, which no producer outside ClickHouse " + "can reproduce. To count distinct combinations of several columns, use uniq, uniqCombined " + "or uniqHLL12.", + name, argument_types.size()); + + const IDataType & argument_type = *argument_types[0]; + WhichDataType which(argument_type); + + /// Backed by a decimal, but accepted unlike one: it holds the epoch time a caller elsewhere + /// passes to `update(long)`. + if (which.isDateTime64()) + return std::make_shared>(lg_config_k, target_type, argument_types, params); + + /// `createWithNumericType` also covers the 128 and 256 bit integers, for which no byte order is + /// agreed on, so they have to be refused before it is reached. + if (!which.isInt128() && !which.isInt256() && !which.isUInt128() && !which.isUInt256()) + { + AggregateFunctionPtr res(createWithNumericType( + argument_type, lg_config_k, target_type, argument_types, params)); + if (res) + return res; + } + + if (which.isDate()) + return std::make_shared>(lg_config_k, target_type, argument_types, params); + if (which.isDate32()) + return std::make_shared>(lg_config_k, target_type, argument_types, params); + if (which.isDateTime()) + return std::make_shared>(lg_config_k, target_type, argument_types, params); + if (which.isStringOrFixedString()) + return std::make_shared>(lg_config_k, target_type, argument_types, params); + if (which.isUUID()) + return std::make_shared>(lg_config_k, target_type, argument_types, params); + if (which.isIPv4()) + return std::make_shared>(lg_config_k, target_type, argument_types, params); + if (which.isIPv6()) + return std::make_shared>(lg_config_k, target_type, argument_types, params); + + throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, + "Aggregate function {} does not support the argument type {}: only the types an Apache " + "DataSketches HLL sketch hashes the same way outside ClickHouse are supported: integers of " + "at most 64 bits, Enum8, Enum16, BFloat16, Float32, Float64, String, FixedString, UUID, " + "IPv4, IPv6, Date, Date32, DateTime and DateTime64. To count distinct values of any other " + "type, use uniq, uniqCombined or uniqHLL12.", + name, argument_type.getName()); +} + +void registerAggregateFunctionUniqApacheHLL(AggregateFunctionFactory & factory); +void registerAggregateFunctionUniqApacheHLL(AggregateFunctionFactory & factory) +{ + FunctionDocumentation::Description description = R"( +Calculates the approximate number of different argument values using an [Apache DataSketches](https://datasketches.apache.org/docs/HLL/HllSketches.html) HyperLogLog sketch. + +The serialized state produced by the `-State` combinator carries the sketch in the Apache DataSketches HLL format, framed with a varint length prefix, so sketches can be exchanged with external services (Java, Python, C++) using the standard `-State`/`-Merge` combinators. For example, a sketch built by an upstream service can be merged with `uniqApacheHLLMerge`, and a sketch built in ClickHouse can be exported with `uniqApacheHLLState`. + +Only the types that a sketch hashes the same way in every implementation are accepted, so that every state ClickHouse produces can be reproduced elsewhere. Integers of at most 64 bits are hashed as their 8-byte representation (matching DataSketches `update(long)`), floating-point values as an IEEE-754 double, strings as their raw bytes, `UUID`s as their canonical 16 bytes and `IPv6` addresses in network order. + +`Date`, `Date32`, `DateTime` and `DateTime64` are hashed as the integer they hold, which for `DateTime64(3)` is the epoch milliseconds a caller elsewhere would pass to `update(long)`. The unit belongs to the column type rather than to the value, so both sides have to agree on it. + +Every other type is rejected, because a sketch over it would have to be built from a hash that only ClickHouse can compute: the 128 and 256 bit integers have no agreed byte order across implementations, decimals have no representation any DataSketches binding accepts, and an array or a tuple has none either. The same goes for a call with more than one argument. Use `uniq`, `uniqCombined` or `uniqHLL12` to count distinct values of those. + +An estimate obtained by merging sketches is not the same number as one computed in a single pass over the same values, even though both are derived from identical registers: DataSketches reports the HIP estimator for a sketch that has only been updated and the composite estimator for one produced by a union. The result therefore depends on how the aggregation was partitioned across threads, parts and shards, and is slightly less accurate once any merge has taken place. + +The resolution of a merged sketch is the smallest `lg_k` among its inputs, not the `lg_k` named by the type. Merging a sketch that was built with a lower `lg_k` - for example one produced by another service - permanently lowers the resolution of both the estimate and the state written back. + )"; + FunctionDocumentation::Syntax syntax = "uniqApacheHLL([lg_k, [type]])(x)"; + FunctionDocumentation::Arguments arguments = { + {"x", "Column to compute the number of distinct values of.", {"(U)Int8/16/32/64", "Enum8", "Enum16", "BFloat16", "Float32", "Float64", "String", "FixedString", "UUID", "IPv4", "IPv6", "Date", "Date32", "DateTime", "DateTime64"}}, + }; + FunctionDocumentation::Parameters parameters = { + {"lg_k", "Optional. Log-base-2 of the number of buckets, in range [4, 21]. Higher means better accuracy and more memory. Default: 12.", {"UInt8"}}, + {"type", "Optional. Storage format of the sketch: 'HLL_4', 'HLL_6', or 'HLL_8'. Default: 'HLL_4'.", {"String"}}, + }; + FunctionDocumentation::ReturnedValue returned_value = {"Returns the approximate number of distinct values.", {"UInt64"}}; + FunctionDocumentation::Examples examples = { + {"Basic usage", "SELECT uniqApacheHLL(number) FROM numbers(1000)", "1000"}, + {"With parameters", "SELECT uniqApacheHLL(14, 'HLL_8')(number) FROM numbers(1000)", "1000"}, + }; + FunctionDocumentation::IntroducedIn introduced_in = {26, 6}; + FunctionDocumentation::Category category = FunctionDocumentation::Category::AggregateFunction; + FunctionDocumentation documentation = {description, syntax, arguments, parameters, returned_value, examples, introduced_in, category}; + + AggregateFunctionProperties properties = { .returns_default_when_only_null = true, .is_order_dependent = false }; + + factory.registerFunction("uniqApacheHLL", {createAggregateFunctionUniqApacheHLL, documentation, properties}); +} + +} + +#endif diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h new file mode 100644 index 000000000000..b223440e1e1b --- /dev/null +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -0,0 +1,295 @@ +#pragma once + +#include "config.h" + +#if USE_DATASKETCHES + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +#include +#include +#include +#include + +namespace DB +{ + +namespace ErrorCodes +{ + extern const int CORRUPTED_DATA; +} + + +/** An Apache DataSketches HLL sketch as an aggregate function state, serialized in the + * DataSketches HLL format so that `-State`/`-Merge` interoperate with external services. + * + * `lg_config_k` and the target type are owned by the aggregate function and passed in, so the + * state stores the sketch alone. + */ +class HllSketchData : private boost::noncopyable +{ +private: + /// Used for insertions. + std::unique_ptr sk_update; + /// Used for merging. + std::unique_ptr sk_union; + + datasketches::hll_sketch * getSkUpdate(uint8_t lg_config_k, datasketches::target_hll_type tgt_type) + { + if (!sk_update) + sk_update = std::make_unique(lg_config_k, tgt_type); + return sk_update.get(); + } + + datasketches::hll_union * getSkUnion(uint8_t lg_config_k) + { + if (!sk_union) + { + /// `hll_union` takes `lg_max_k` in [7, 21] while a sketch may use [4, 21], hence the + /// floor of 7. It does not inflate the result: `get_result` downsamples to the smallest + /// `lg_config_k` the union has seen. So the resolution of a merged state is the minimum + /// over its inputs, not the `lg_config_k` of the type, and merging a coarser sketch from + /// elsewhere lowers both the estimate and the state written back. Merged from empty + /// states only there is no minimum, and it serializes as `lg_config_k = 7`. + sk_union = std::make_unique(std::max(lg_config_k, 7)); + } + return sk_union.get(); + } + + /// Fold a sketch updated after the union was allocated into it, so merges see one state. + void foldUpdateIntoUnionIfNeeded() + { + if (sk_union && sk_update) + { + sk_union->update(*sk_update); + sk_update.reset(nullptr); + } + } + +public: + HllSketchData() = default; + ~HllSketchData() = default; + + template + void insert(T value, uint8_t lg_config_k, datasketches::target_hll_type tgt_type) + { + getSkUpdate(lg_config_k, tgt_type)->update(value); + foldUpdateIntoUnionIfNeeded(); + } + + void insertData(const char * data, size_t size, uint8_t lg_config_k, datasketches::target_hll_type tgt_type) + { + getSkUpdate(lg_config_k, tgt_type)->update(static_cast(data), size); + foldUpdateIntoUnionIfNeeded(); + } + + UInt64 size(datasketches::target_hll_type tgt_type) const + { + /// Round rather than truncate: `get_estimate` returns a `double`, and `999.9999` for an + /// exactly-known cardinality must not become `999`. + if (sk_union) + return static_cast(std::llround(sk_union->get_result(tgt_type).get_estimate())); + if (sk_update) + return static_cast(std::llround(sk_update->get_estimate())); + return 0; + } + + void merge(const HllSketchData & rhs, uint8_t lg_config_k, datasketches::target_hll_type tgt_type) + { + datasketches::hll_union * u = getSkUnion(lg_config_k); + + if (sk_update) + { + u->update(*sk_update); + sk_update.reset(nullptr); + } + + if (rhs.sk_update) + u->update(*rhs.sk_update); + else if (rhs.sk_union) + u->update(rhs.sk_union->get_result(tgt_type)); + } + + /// You can only call this for an empty object. + void read(ReadBuffer & in, uint8_t lg_config_k) + { + datasketches::hll_sketch::vector_bytes bytes; + readVectorBinary(bytes, in); + if (bytes.empty()) + return; + + try + { + auto sk = datasketches::hll_sketch::deserialize(bytes.data(), bytes.size()); + getSkUnion(lg_config_k)->update(std::move(sk)); + } + catch (const DB::Exception &) + { + throw; + } + catch (const std::bad_alloc &) + { + /// Memory pressure, not corrupted data. + throw; + } + catch (const std::exception & e) + { + /// `datasketches` reports malformed input as `std::invalid_argument` / `std::out_of_range`. + /// Not being `DB::Exception`, those escape `SerializationAggregateFunction`'s + /// `catch (...)` and abort as a logical error, so translate them here. + throw Exception(ErrorCodes::CORRUPTED_DATA, "Cannot deserialize HLL sketch state: {}", e.what()); + } + } + + void write(WriteBuffer & out, datasketches::target_hll_type tgt_type) const + { + if (sk_update) + { + auto bytes = sk_update->serialize_compact(); + writeVectorBinary(bytes, out); + } + else if (sk_union) + { + auto bytes = sk_union->get_result(tgt_type).serialize_compact(); + writeVectorBinary(bytes, out); + } + else + { + datasketches::hll_sketch::vector_bytes bytes; + writeVectorBinary(bytes, out); + } + } +}; + + +/** `uniqApacheHLL` over a single column of a type the sketch can hash directly. + * + * The value takes one of the three shapes the DataSketches API accepts - an 8-byte integer, an + * IEEE-754 double, or raw bytes - so that an external producer reaches the same sketch. + * + * `lg_config_k` and the target type live here rather than in the state, which is what lets states + * of different parameterisations share one binary representation. + */ +template +class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper> +{ + using Base = IAggregateFunctionDataHelper>; + + uint8_t lg_config_k; + datasketches::target_hll_type target_type; + +public: + AggregateFunctionUniqApacheHLL( + uint8_t lg_config_k_, + datasketches::target_hll_type target_type_, + const DataTypes & argument_types_, + const Array & params_) + : Base(argument_types_, params_, std::make_shared()) + , lg_config_k(lg_config_k_) + , target_type(target_type_) + { + } + + String getName() const override { return "uniqApacheHLL"; } + + bool allocatesMemoryInArena() const override { return false; } + + void add(AggregateDataPtr __restrict place, const IColumn ** columns, size_t row_num, Arena *) const override + { + auto & data = this->data(place); + + if constexpr (std::is_same_v) + { + const auto value = columns[0]->getDataAt(row_num); + data.insertData(value.data(), value.size(), lg_config_k, target_type); + } + else + { + const auto & value = assert_cast &>(*columns[0]).getData()[row_num]; + + if constexpr (std::is_same_v) + { + /// ClickHouse holds a UUID as two 64-bit halves in host order, so its bytes in + /// memory are not the canonical 16 an external producer works from. + const UInt64 halves[2] = { + std::byteswap(UUIDHelpers::getHighBytes(value)), + std::byteswap(UUIDHelpers::getLowBytes(value)), + }; + data.insertData(reinterpret_cast(halves), sizeof(halves), lg_config_k, target_type); + } + else if constexpr (std::is_same_v) + /// Already held in network order, which is the canonical form. + data.insertData(reinterpret_cast(&value), sizeof(value), lg_config_k, target_type); + else if constexpr (is_decimal) + /// `DateTime64(3)` holds the epoch milliseconds a caller elsewhere passes to + /// `update(long)`. The scale belongs to the type, so both sides must agree on it. + data.insert(static_cast(value.value), lg_config_k, target_type); + else if constexpr (std::is_same_v) + data.insert(static_cast(value.toUnderType()), lg_config_k, target_type); + else if constexpr (std::is_same_v || std::is_floating_point_v) + data.insert(static_cast(value), lg_config_k, target_type); + else if constexpr (std::is_signed_v) + data.insert(static_cast(value), lg_config_k, target_type); + else + data.insert(static_cast(value), lg_config_k, target_type); + } + } + /// A serialized sketch describes its own configuration, so states of different parameterisations + /// are interchangeable. Merging across them takes the resolution of the coarsest input. + bool haveSameStateRepresentationImpl(const IAggregateFunction & rhs) const override + { + return getName() == rhs.getName() && this->haveEqualArgumentTypes(rhs); + } + + void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena *) const override + { + this->data(place).merge(this->data(rhs), lg_config_k, target_type); + } + + void serialize(ConstAggregateDataPtr __restrict place, WriteBuffer & buf, std::optional /* version */) const override + { + this->data(place).write(buf, target_type); + } + + void deserialize(AggregateDataPtr __restrict place, ReadBuffer & buf, std::optional /* version */, Arena *) const override + { + this->data(place).read(buf, lg_config_k); + } + + void insertResultInto(AggregateDataPtr __restrict place, IColumn & to, Arena *) const override + { + assert_cast(to).getData().push_back(this->data(place).size(target_type)); + } + +}; + + +/// `uniqApacheHLL([lg_k, [type]])(x)`, with `lg_k` in [4, 21] (default 12) and `type` one of +/// 'HLL_4', 'HLL_6', 'HLL_8' (default 'HLL_4'). +AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( + const std::string & name, const DataTypes & argument_types, const Array & params, const Settings *); + +} + +#endif diff --git a/src/AggregateFunctions/registerAggregateFunctions.cpp b/src/AggregateFunctions/registerAggregateFunctions.cpp index 46997750a403..670180d87a46 100644 --- a/src/AggregateFunctions/registerAggregateFunctions.cpp +++ b/src/AggregateFunctions/registerAggregateFunctions.cpp @@ -61,6 +61,9 @@ void registerAggregateFunctionSumMap(AggregateFunctionFactory &); void registerAggregateFunctionsUniq(AggregateFunctionFactory &); void registerAggregateFunctionUniqCombined(AggregateFunctionFactory &); void registerAggregateFunctionUniqUpTo(AggregateFunctionFactory &); +#if USE_DATASKETCHES +void registerAggregateFunctionUniqApacheHLL(AggregateFunctionFactory &); +#endif void registerAggregateFunctionTopK(AggregateFunctionFactory &); void registerAggregateFunctionsBitwise(AggregateFunctionFactory &); void registerAggregateFunctionsBitmap(AggregateFunctionFactory &); @@ -174,6 +177,9 @@ void registerAggregateFunctions() registerAggregateFunctionsUniq(factory); registerAggregateFunctionUniqCombined(factory); registerAggregateFunctionUniqUpTo(factory); +#if USE_DATASKETCHES + registerAggregateFunctionUniqApacheHLL(factory); +#endif registerAggregateFunctionTopK(factory); registerAggregateFunctionsBitwise(factory); registerAggregateFunctionCramersV(factory); diff --git a/tests/queries/0_stateless/04327_uniq_apache_hll.reference b/tests/queries/0_stateless/04327_uniq_apache_hll.reference new file mode 100644 index 000000000000..f2059d5f29fa --- /dev/null +++ b/tests/queries/0_stateless/04327_uniq_apache_hll.reference @@ -0,0 +1,19 @@ +accuracy +1 +1 +1 +1 +empty and single +0 +1 +argument types +1 +1 +1 +1 +state/merge roundtrip is native +1 +1 +1 +AggregateFunction(uniqApacheHLL(14, \'HLL_8\'), UInt64) +parameter validation diff --git a/tests/queries/0_stateless/04327_uniq_apache_hll.sql b/tests/queries/0_stateless/04327_uniq_apache_hll.sql new file mode 100644 index 000000000000..584ba4c067c2 --- /dev/null +++ b/tests/queries/0_stateless/04327_uniq_apache_hll.sql @@ -0,0 +1,60 @@ +-- Tags: no-fasttest +-- ^ DataSketches is not built in fast-test builds. + +SELECT 'accuracy'; +-- HLL is approximate but deterministic; assert the estimate is within the expected error band. +SELECT abs(toInt64(uniqApacheHLL(number)) - 1000) < 30 FROM numbers(1000); +SELECT abs(toInt64(uniqApacheHLL(number)) - 100000) < 3000 FROM numbers(100000); +-- Higher lg_k -> better accuracy. +SELECT abs(toInt64(uniqApacheHLL(14)(number)) - 100000) < 1500 FROM numbers(100000); +-- Storage type does not change the estimate. +SELECT uniqApacheHLL(12, 'HLL_4')(number) = uniqApacheHLL(12, 'HLL_8')(number) FROM numbers(1000); + +SELECT 'empty and single'; +SELECT uniqApacheHLL(number) FROM numbers(0); +SELECT uniqApacheHLL(number) FROM numbers(1); + +SELECT 'argument types'; +SELECT abs(toInt64(uniqApacheHLL(toInt32(number))) - 500) < 20 FROM numbers(500); +SELECT abs(toInt64(uniqApacheHLL(toFloat64(number))) - 500) < 20 FROM numbers(500); +SELECT abs(toInt64(uniqApacheHLL(toString(number))) - 500) < 20 FROM numbers(500); +SELECT abs(toInt64(uniqApacheHLL(toDate('2020-01-01') + number)) - 500) < 20 FROM numbers(500); + +SELECT 'state/merge roundtrip is native'; +-- Merging per-group `-State` values covers exactly the same set as a direct aggregate, so the two +-- estimates agree to within the error of the sketch. They are not required to be equal: DataSketches +-- reports the HIP estimate for a sketch that has only been updated and the composite estimate for +-- one that came out of a union, so a merged result is not the same number as a directly built one +-- even though both are computed from identical registers. +SELECT + abs(toInt64(uniqApacheHLLMerge(s)) - toInt64((SELECT uniqApacheHLL(number) FROM numbers(100000)))) < 3000 +FROM +( + SELECT uniqApacheHLLState(number) AS s + FROM numbers(100000) + GROUP BY number % 17 +); + +-- Merging is independent of how the input was partitioned: the registers of the union do not depend +-- on the grouping, and both sides come out of a union, so two different partitionings of the same +-- set agree exactly. +SELECT + (SELECT uniqApacheHLLMerge(s) FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17)) + = (SELECT uniqApacheHLLMerge(s) FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 13)); + +-- The same holds for a single state. +SELECT + abs(toInt64(uniqApacheHLLMerge(s)) - toInt64((SELECT uniqApacheHLL(number) FROM numbers(1000)))) < 30 +FROM +( + SELECT uniqApacheHLLState(number) AS s FROM numbers(1000) +); + +-- The state type carries the sketch parameters. +SELECT toTypeName(uniqApacheHLLState(14, 'HLL_8')(number)) FROM numbers(1); + +SELECT 'parameter validation'; +SELECT uniqApacheHLL(3)(number) FROM numbers(1); -- { serverError ARGUMENT_OUT_OF_BOUND } +SELECT uniqApacheHLL(22)(number) FROM numbers(1); -- { serverError ARGUMENT_OUT_OF_BOUND } +SELECT uniqApacheHLL(12, 'HLL_9')(number) FROM numbers(1); -- { serverError BAD_ARGUMENTS } +SELECT uniqApacheHLL(12, 'HLL_4', 1)(number) FROM numbers(1); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } diff --git a/tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference b/tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference new file mode 100644 index 000000000000..ac07e5cd8a1c --- /dev/null +++ b/tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference @@ -0,0 +1,23 @@ +import sketches built outside ClickHouse +5 +5 +8 +8 +3 +84 +5 +0 +types wider than the 8 bytes update(long) takes +1 +5 +1 +5 +DateTime64 is hashed as the epoch time it counts +1 +export sketches for consumption outside ClickHouse +1 +1 +1 +states survive a write/read cycle through storage +20 +1 diff --git a/tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql b/tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql new file mode 100644 index 000000000000..890d74aa89b0 --- /dev/null +++ b/tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql @@ -0,0 +1,95 @@ +-- Tags: no-fasttest +-- ^ DataSketches is not built in fast-test builds. + +-- The state of `uniqApacheHLL` carries an Apache DataSketches HLL sketch in its native serialized +-- form, framed with the varint length prefix that `writeVectorBinary` adds. Every blob below was +-- produced by the Apache DataSketches C++ library directly rather than by ClickHouse, so these +-- queries pin the wire contract with external producers in both directions. A change that breaks +-- interoperability has to change one of these literals. + +SELECT 'import sketches built outside ClickHouse'; + +-- lg_k = 12, HLL_4, `hll_sketch::update(uint64_t)` over 0..4. Coupon list mode. +SELECT finalizeAggregation(CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)')); + +-- The same sketch reached through `-Merge`. +SELECT uniqApacheHLLMerge(s) FROM (SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s); + +-- Two external sketches over 0..4 and 3..7 union to 8 distinct values. +SELECT uniqApacheHLLMerge(s) FROM +( + SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s + UNION ALL + SELECT CAST(unhex('1C0201070C030805007581660781BC5D067B65E608FC2D420AC1E91705'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s +); + +-- An external sketch merged with one built by ClickHouse: the two hash identically, so 0..4 and +-- 3..7 overlap on 3 and 4 and the union is 8. +SELECT uniqApacheHLLMerge(s) FROM +( + SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s + UNION ALL + SELECT uniqApacheHLLState(number) AS s FROM numbers(3, 5) +); + +-- lg_k = 12, HLL_4, raw bytes of 'alpha', 'beta', 'gamma'. +SELECT finalizeAggregation(CAST(unhex('140201070C03080300BD3A090A8E5A62115168C90A'), 'AggregateFunction(uniqApacheHLL, UInt64)')); + +-- lg_k = 4, HLL_4, 100 distinct values. Dense HLL mode rather than a coupon list. +SELECT finalizeAggregation(CAST(unhex('300A0107040008020215EB1DC787F15440000000000000FB3F000000000000000003000000000000000251214121031025'), 'AggregateFunction(uniqApacheHLL(4), UInt64)')); + +-- lg_k = 14, HLL_8: the non-default parameters are carried by the state type. +SELECT finalizeAggregation(CAST(unhex('1C0201070E03080508CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL(14, \'HLL_8\'), UInt64)')); + +-- An externally produced empty sketch. +SELECT finalizeAggregation(CAST(unhex('080201070C030C0000'), 'AggregateFunction(uniqApacheHLL, UInt64)')); + +SELECT 'types wider than the 8 bytes update(long) takes'; + +-- lg_k = 12, HLL_4 over the canonical 16 bytes of 00000000-0000-0000-0000-00000000000{0..4}. ClickHouse +-- holds a UUID as two 64-bit halves in host order, so hashing its bytes in memory would not agree with +-- an external producer working from the textual form; the canonical order is hashed instead. +SELECT hex(toString(uniqApacheHLLState(toUUID(concat('00000000-0000-0000-0000-00000000000', toString(number)))))) + = '1C0201070C0308050050C94D05854BD10ADB8CBD053C56FB07F8FDB206' +FROM numbers(5) SETTINGS max_threads = 1; +SELECT finalizeAggregation(CAST(unhex('1C0201070C0308050050C94D05854BD10ADB8CBD053C56FB07F8FDB206'), 'AggregateFunction(uniqApacheHLL, UUID)')); + +-- The same for 2001:db8::1 .. ::5, which ClickHouse already holds in network order. +SELECT hex(toString(uniqApacheHLLState(toIPv6(concat('2001:db8::', hex(number + 1)))))) + = '1C0201070C0308050018216E09FAB4750F52D5BB07DFBDE30A79BC9D0B' +FROM numbers(5) SETTINGS max_threads = 1; +SELECT finalizeAggregation(CAST(unhex('1C0201070C0308050018216E09FAB4750F52D5BB07DFBDE30A79BC9D0B'), 'AggregateFunction(uniqApacheHLL, IPv6)')); + +SELECT 'DateTime64 is hashed as the epoch time it counts'; + +-- A `DateTime64(3)` is hashed as its epoch milliseconds, which is the `long` a caller elsewhere +-- passes for the same instant. These five are 2020-01-01 00:00:00.000 and the next four seconds. +SELECT hex(toString(uniqApacheHLLState(toDateTime64('2020-01-01 00:00:00.000', 3, 'UTC') + number))) + = '1C0201070C0308050001D4B019CBDD6F1059310D0E833938083897B304' +FROM numbers(5) SETTINGS max_threads = 1; + +SELECT 'export sketches for consumption outside ClickHouse'; + +-- `max_threads` is pinned because a state that was merged from several partial states may lay its +-- coupons out in a different order than a state built by a single thread. +SELECT hex(toString(uniqApacheHLLState(number))) = '1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06' FROM numbers(5) SETTINGS max_threads = 1; +SELECT hex(toString(uniqApacheHLLState(14, 'HLL_8')(number))) = '1C0201070E03080508CBD7C2042BF2FB06862FF90D7581660781BC5D06' FROM numbers(5) SETTINGS max_threads = 1; + +-- Importing an external sketch and exporting it again must reproduce it byte for byte. +SELECT hex(toString(uniqApacheHLLMergeState(s))) = '1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06' +FROM (SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s) +SETTINGS max_threads = 1; + +SELECT 'states survive a write/read cycle through storage'; + +DROP TABLE IF EXISTS hll_interop_states; +CREATE TABLE hll_interop_states (k UInt8, s AggregateFunction(uniqApacheHLL, UInt64)) ENGINE = AggregatingMergeTree ORDER BY k; +INSERT INTO hll_interop_states SELECT number % 4 AS k, uniqApacheHLLState(number) FROM numbers(20) GROUP BY k; +OPTIMIZE TABLE hll_interop_states FINAL; +-- Small enough to stay in coupon mode, so the union is exact regardless of how it was partitioned. +SELECT uniqApacheHLLMerge(s) FROM hll_interop_states; +-- A state read back from disk is still a valid sketch for an external consumer. +SELECT countDistinct(hex(toString(s))) = 4 FROM hll_interop_states; +DROP TABLE hll_interop_states; + +-- Malformed states are rejected as `CORRUPTED_DATA`; see `05136_uniq_apache_hll_corrupted_state.sh`. diff --git a/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.reference b/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.reference new file mode 100644 index 000000000000..2fc0b7c7d810 --- /dev/null +++ b/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.reference @@ -0,0 +1,11 @@ +the two types remain distinct +AggregateFunction(uniqApacheHLL, UInt64) +AggregateFunction(uniqApacheHLL(8), UInt64) +merging states built with a different lg_k +1 +1 +relabelling with CAST does not rescale +1 1 +a state of one lg_k can be stored in a column declared with another +20 +only the parameters are interchangeable diff --git a/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.sql b/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.sql new file mode 100644 index 000000000000..43588e9d133a --- /dev/null +++ b/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.sql @@ -0,0 +1,47 @@ +-- Tags: no-fasttest +-- ^ DataSketches is not built in fast-test builds. + +-- `lg_k` and the sketch type configure the sketch but are held by the aggregate function rather than +-- by its state, whose layout is the same whatever they are, and a serialized sketch records its own +-- `lg_k`. `uniqApacheHLL` therefore reports states of different parameterisations as having one +-- binary representation, which lets `-Merge` read a state built with one `lg_k` under a function +-- declared with another, and lets `CAST` relabel a column between the two types. + +SELECT 'the two types remain distinct'; +SELECT toTypeName(uniqApacheHLLState(number)) FROM numbers(1); +SELECT toTypeName(uniqApacheHLLState(8)(number)) FROM numbers(1); + +SELECT 'merging states built with a different lg_k'; +-- Downsampling during a union produces the same registers as building at the target `lg_k` from the +-- start, so rescaling 17 states of `lg_k` 12 agrees exactly with merging 17 built at 8. Both sides +-- come out of a union and so use the same estimator. +SELECT + (SELECT uniqApacheHLLMerge(8)(s) FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17)) + = (SELECT uniqApacheHLLMerge(8)(s) FROM (SELECT uniqApacheHLLState(8)(number) AS s FROM numbers(100000) GROUP BY number % 17)); + +-- Merging is lossy in one direction only: the union takes the resolution of its coarsest input, so a +-- rescaled state is much smaller than the states it was built from. +SELECT + length(toString(uniqApacheHLLMergeState(8)(s))) < length(toString(uniqApacheHLLMergeState(s))) / 8 +FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17); + +SELECT 'relabelling with CAST does not rescale'; +-- `CAST` between the two types re-associates the column with the other function without touching its +-- data, so the sketch keeps the resolution it was built with and the estimate does not change. +SELECT + finalizeAggregation(CAST(s, 'AggregateFunction(uniqApacheHLL(8), UInt64)')) = finalizeAggregation(s), + length(toString(CAST(s, 'AggregateFunction(uniqApacheHLL(8), UInt64)'))) = length(toString(s)) +FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(1000)); + +SELECT 'a state of one lg_k can be stored in a column declared with another'; +DROP TABLE IF EXISTS hll_cross_scale; +CREATE TABLE hll_cross_scale (s AggregateFunction(uniqApacheHLL, UInt64)) ENGINE = MergeTree ORDER BY tuple(); +INSERT INTO hll_cross_scale SELECT uniqApacheHLLState(8)(number) FROM numbers(20); +SELECT uniqApacheHLLMerge(s) FROM hll_cross_scale; +DROP TABLE hll_cross_scale; + +SELECT 'only the parameters are interchangeable'; +-- The argument types and the function itself must still match: `haveEqualArgumentTypes` and the name +-- comparison keep these apart even though the parameters no longer do. +SELECT CAST(uniqApacheHLLState(toString(number)), 'AggregateFunction(uniqApacheHLL, UInt64)') FROM numbers(10); -- { serverError CANNOT_CONVERT_TYPE } +SELECT CAST(uniqThetaState(number), 'AggregateFunction(uniqApacheHLL, UInt64)') FROM numbers(10); -- { serverError CANNOT_CONVERT_TYPE } diff --git a/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.reference b/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.reference new file mode 100644 index 000000000000..6478f4dcb46e --- /dev/null +++ b/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.reference @@ -0,0 +1,26 @@ +types the sketch hashes directly +20 +20 +20 +20 +20 +20 +20 +20 +20 +20 +20 +20 +20 +20 +3 +the wrappers of an accepted type are accepted +20 +20 +a DateTime64 is the integer a caller elsewhere would hash +1 +types no other implementation can reproduce are rejected +one argument only +the parameters still apply +20 +AggregateFunction(uniqApacheHLL(8, \'HLL_6\'), UInt64) diff --git a/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.sql b/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.sql new file mode 100644 index 000000000000..6b133e97f64c --- /dev/null +++ b/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.sql @@ -0,0 +1,54 @@ +-- Tags: no-fasttest +-- ^ DataSketches is not built in fast-test builds. + +-- `uniqApacheHLL` accepts only the types a sketch hashes the same way in every implementation, as +-- an 8-byte integer, an IEEE-754 double or their raw bytes. Anything else would have to be hashed +-- to a single value by ClickHouse first, which no producer outside ClickHouse could reproduce. +-- All counts below are small enough for the sketch to stay in coupon mode, where it is exact. + +SELECT 'types the sketch hashes directly'; +SELECT uniqApacheHLL(toUInt64(number)) FROM numbers(20); +SELECT uniqApacheHLL(toInt32(number)) FROM numbers(20); +SELECT uniqApacheHLL(toBFloat16(number)) FROM numbers(20); +SELECT uniqApacheHLL(toFloat32(number)) FROM numbers(20); +SELECT uniqApacheHLL(toFloat64(number)) FROM numbers(20); +SELECT uniqApacheHLL(toString(number)) FROM numbers(20); +SELECT uniqApacheHLL(toFixedString(toString(number), 8)) FROM numbers(20); +SELECT uniqApacheHLL(toDate('2020-01-01') + number) FROM numbers(20); +SELECT uniqApacheHLL(toDate32('2020-01-01') + number) FROM numbers(20); +SELECT uniqApacheHLL(toDateTime('2020-01-01 00:00:00') + number) FROM numbers(20); +SELECT uniqApacheHLL(toDateTime64('2020-01-01 00:00:00.000', 3) + number) FROM numbers(20); +SELECT uniqApacheHLL(reinterpretAsUUID(toUInt128(number))) FROM numbers(20); +SELECT uniqApacheHLL(toIPv4('1.2.3.0') + number) FROM numbers(20); +SELECT uniqApacheHLL(toIPv6(concat('2001:db8::', hex(number + 1)))) FROM numbers(20); +SELECT uniqApacheHLL(CAST(number % 3, 'Enum8(\'a\' = 0, \'b\' = 1, \'c\' = 2)')) FROM numbers(20); + +SELECT 'the wrappers of an accepted type are accepted'; +SELECT uniqApacheHLL(toNullable(number)) FROM numbers(20); +SELECT uniqApacheHLL(toLowCardinality(toString(number))) FROM numbers(20); + +SELECT 'a DateTime64 is the integer a caller elsewhere would hash'; +-- `DateTime64(3)` counts epoch milliseconds, so it produces the sketch of those milliseconds. +SELECT hex(toString(uniqApacheHLLState(toDateTime64('2020-01-01 00:00:00.000', 3, 'UTC') + number))) + = hex(toString(uniqApacheHLLState(toInt64(1577836800000) + number * 1000))) +FROM numbers(5) SETTINGS max_threads = 1; + +SELECT 'types no other implementation can reproduce are rejected'; +-- No byte order for the wide integers is agreed on across implementations. +SELECT uniqApacheHLL(toInt128(number)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } +SELECT uniqApacheHLL(toUInt256(number)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } +-- No DataSketches binding accepts a decimal, and the scale is not part of the sketch. +SELECT uniqApacheHLL(toDecimal64(number, 2)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } +SELECT uniqApacheHLL(toDecimal128(number, 4)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } +-- A composite value would have to be hashed by ClickHouse first. +SELECT uniqApacheHLL(materialize([number])) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } +SELECT uniqApacheHLL((number, number + 1)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } + +SELECT 'one argument only'; +SELECT uniqApacheHLL(number, number + 1) FROM numbers(20); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } +SELECT uniqApacheHLL(toString(number), number) FROM numbers(20); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } +SELECT uniqApacheHLL() FROM numbers(1); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } + +SELECT 'the parameters still apply'; +SELECT uniqApacheHLL(8)(reinterpretAsUUID(toUInt128(number))) FROM numbers(20); +SELECT toTypeName(uniqApacheHLLState(8, 'HLL_6')(number)) FROM numbers(1); diff --git a/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.reference b/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.reference new file mode 100644 index 000000000000..f70b0368a421 --- /dev/null +++ b/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.reference @@ -0,0 +1,3 @@ +OK unknown type +OK bad payload +OK rowbinary diff --git a/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.sh b/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.sh new file mode 100755 index 000000000000..18ad78792ac1 --- /dev/null +++ b/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.sh @@ -0,0 +1,37 @@ +#!/usr/bin/env bash +# Tags: no-fasttest +# no-fasttest -- compiled w/o datasketches + +# A `uniqApacheHLL` state that is not a sketch must be rejected as `CORRUPTED_DATA`. +# +# `datasketches` reports a payload it cannot parse as `std::invalid_argument` or +# `std::out_of_range`, neither of which is a `DB::Exception`, so without the +# translation in `HllSketchData::read` they would escape +# `SerializationAggregateFunction`'s `catch (...)` and be reported as a logical error. +# +# This is a shell test rather than a `.sql` one with a `serverError` hint because the +# client prints an extra exception with a stack trace to stderr for this class of error, +# which a `.sql` test counts as a failure. The same applies to `uniqTheta`, see +# `04307_uniqTheta_corrupted_state_106259.sh`. + +CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) +# shellcheck source=../shell_config.sh +. "$CUR_DIR"/../shell_config.sh + +# A well-formed varint length prefix followed by eight bytes that are not a sketch: +# the first byte of the payload is the preamble length, which no HLL sketch uses. +$CLICKHOUSE_CLIENT --query \ + "SELECT finalizeAggregation(CAST(unhex('08FFFFFFFFFFFFFFFF'), 'AggregateFunction(uniqApacheHLL, UInt64)'))" 2>&1 \ + | grep -q -F 'CORRUPTED_DATA' && echo 'OK unknown type' || echo 'FAIL unknown type' + +$CLICKHOUSE_CLIENT --query \ + "SELECT finalizeAggregation(CAST(unhex('0801020304050607FF'), 'AggregateFunction(uniqApacheHLL, UInt64)'))" 2>&1 \ + | grep -q -F 'CORRUPTED_DATA' && echo 'OK bad payload' || echo 'FAIL bad payload' + +# The `RowBinary` path reaches `HllSketchData::read` through `deserializeBinary`: +# the leading 0x03 claims a three-byte payload, shorter than any HLL sketch. +printf '\x03\x03\x03\x30\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' \ + | $CLICKHOUSE_LOCAL --input-format=RowBinary \ + --structure='x AggregateFunction(uniqApacheHLL, IPv6)' \ + --query='SELECT x FROM table' 2>&1 \ + | grep -q -F 'CORRUPTED_DATA' && echo 'OK rowbinary' || echo 'FAIL rowbinary' From 9c9c7fbb49f510a5ecd72c41c14ab5952d1d4d58 Mon Sep 17 00:00:00 2001 From: Andrey Zvonov <32552679+zvonand@users.noreply.github.com> Date: Tue, 29 Sep 2026 21:26:57 +0200 Subject: [PATCH 03/22] Simplify `uniqApacheHLL` implementation and tests Reuse `assertUnary`, consolidate state handling, and shorten repetitive comments and documentation. Merge overlapping HLL tests while retaining interoperability, cross-resolution, corruption, and dependency coverage. Use the canonical `Enum` documentation type to avoid a logical-error exception when reading `system.functions`. Validation: rebuilt ClickHouse and passed all four remaining PR tests with the pinned DataSketches revision; no tests skipped. Related: https://github.com/Altinity/ClickHouse/pull/2398 --- .../AggregateFunctionUniqApacheHLL.cpp | 45 ++++------- .../AggregateFunctionUniqApacheHLL.h | 80 ++++--------------- .../04327_uniq_apache_hll.reference | 29 ++++--- .../0_stateless/04327_uniq_apache_hll.sql | 65 ++++++++------- .../05026_uniq_apache_hll_interop.reference | 14 ++-- .../05026_uniq_apache_hll_interop.sql | 65 +++++++-------- ...5027_uniq_apache_hll_cross_scale.reference | 11 --- .../05027_uniq_apache_hll_cross_scale.sql | 47 ----------- ...8_uniq_apache_hll_argument_types.reference | 26 ------ .../05028_uniq_apache_hll_argument_types.sql | 54 ------------- .../05055_uniq_theta_compressed_state.sql | 14 +--- ..._uniq_apache_hll_corrupted_state.reference | 1 - .../05136_uniq_apache_hll_corrupted_state.sh | 22 +---- 13 files changed, 130 insertions(+), 343 deletions(-) delete mode 100644 tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.reference delete mode 100644 tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.sql delete mode 100644 tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.reference delete mode 100644 tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.sql diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp index a9425146ea95..8da3fc42092a 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp @@ -1,5 +1,6 @@ #include #include +#include #include #if USE_DATASKETCHES @@ -15,7 +16,7 @@ namespace ErrorCodes extern const int NUMBER_OF_ARGUMENTS_DOESNT_MATCH; } -AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( +static AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( const std::string & name, const DataTypes & argument_types, const Array & params, const Settings *) { uint8_t lg_config_k = 12; @@ -52,27 +53,16 @@ AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( "Parameter type for aggregate function {} must be one of 'HLL_4', 'HLL_6', 'HLL_8'.", name); } - /// Only the types a sketch hashes the same way in every language are accepted, so that no state - /// is built here that an external consumer cannot reproduce. Anything else - a decimal, a wide - /// integer, an array, a tuple, several arguments - would first need a hash only ClickHouse has. - if (argument_types.size() != 1) - throw Exception(ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH, - "Aggregate function {} requires exactly one argument, passed {}. Several arguments would " - "have to be hashed to one value by ClickHouse first, which no producer outside ClickHouse " - "can reproduce. To count distinct combinations of several columns, use uniq, uniqCombined " - "or uniqHLL12.", - name, argument_types.size()); + assertUnary(name, argument_types); const IDataType & argument_type = *argument_types[0]; WhichDataType which(argument_type); - /// Backed by a decimal, but accepted unlike one: it holds the epoch time a caller elsewhere - /// passes to `update(long)`. + /// Unlike other decimals, `DateTime64` has an interoperable representation as epoch ticks. if (which.isDateTime64()) return std::make_shared>(lg_config_k, target_type, argument_types, params); - /// `createWithNumericType` also covers the 128 and 256 bit integers, for which no byte order is - /// agreed on, so they have to be refused before it is reached. + /// Exclude wide integers: DataSketches has no portable representation for them. if (!which.isInt128() && !which.isInt256() && !which.isUInt128() && !which.isUInt256()) { AggregateFunctionPtr res(createWithNumericType( @@ -97,12 +87,8 @@ AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( return std::make_shared>(lg_config_k, target_type, argument_types, params); throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, - "Aggregate function {} does not support the argument type {}: only the types an Apache " - "DataSketches HLL sketch hashes the same way outside ClickHouse are supported: integers of " - "at most 64 bits, Enum8, Enum16, BFloat16, Float32, Float64, String, FixedString, UUID, " - "IPv4, IPv6, Date, Date32, DateTime and DateTime64. To count distinct values of any other " - "type, use uniq, uniqCombined or uniqHLL12.", - name, argument_type.getName()); + "Illegal type {} of argument for aggregate function {}. Use uniq, uniqCombined or uniqHLL12 for unsupported types.", + argument_type.getName(), name); } void registerAggregateFunctionUniqApacheHLL(AggregateFunctionFactory & factory); @@ -111,21 +97,18 @@ void registerAggregateFunctionUniqApacheHLL(AggregateFunctionFactory & factory) FunctionDocumentation::Description description = R"( Calculates the approximate number of different argument values using an [Apache DataSketches](https://datasketches.apache.org/docs/HLL/HllSketches.html) HyperLogLog sketch. -The serialized state produced by the `-State` combinator carries the sketch in the Apache DataSketches HLL format, framed with a varint length prefix, so sketches can be exchanged with external services (Java, Python, C++) using the standard `-State`/`-Merge` combinators. For example, a sketch built by an upstream service can be merged with `uniqApacheHLLMerge`, and a sketch built in ClickHouse can be exported with `uniqApacheHLLState`. - -Only the types that a sketch hashes the same way in every implementation are accepted, so that every state ClickHouse produces can be reproduced elsewhere. Integers of at most 64 bits are hashed as their 8-byte representation (matching DataSketches `update(long)`), floating-point values as an IEEE-754 double, strings as their raw bytes, `UUID`s as their canonical 16 bytes and `IPv6` addresses in network order. - -`Date`, `Date32`, `DateTime` and `DateTime64` are hashed as the integer they hold, which for `DateTime64(3)` is the epoch milliseconds a caller elsewhere would pass to `update(long)`. The unit belongs to the column type rather than to the value, so both sides have to agree on it. - -Every other type is rejected, because a sketch over it would have to be built from a hash that only ClickHouse can compute: the 128 and 256 bit integers have no agreed byte order across implementations, decimals have no representation any DataSketches binding accepts, and an array or a tuple has none either. The same goes for a call with more than one argument. Use `uniq`, `uniqCombined` or `uniqHLL12` to count distinct values of those. +The `-State` and `-Merge` combinators exchange Apache DataSketches HLL sketches with a varint length prefix. -An estimate obtained by merging sketches is not the same number as one computed in a single pass over the same values, even though both are derived from identical registers: DataSketches reports the HIP estimator for a sketch that has only been updated and the composite estimator for one produced by a union. The result therefore depends on how the aggregation was partitioned across threads, parts and shards, and is slightly less accurate once any merge has taken place. +For interoperability, integers of at most 64 bits are hashed as 8-byte integers, floating-point values as IEEE-754 doubles, strings as raw bytes, `UUID` values as canonical 16 bytes, and `IPv6` addresses in network order. +`Date`, `Date32`, `DateTime` and `DateTime64` use their underlying integer values. External producers must use the same unit; for example, `DateTime64(3)` uses epoch milliseconds. +Unsupported types and multiple arguments are rejected. Use `uniq`, `uniqCombined` or `uniqHLL12` for those inputs. -The resolution of a merged sketch is the smallest `lg_k` among its inputs, not the `lg_k` named by the type. Merging a sketch that was built with a lower `lg_k` - for example one produced by another service - permanently lowers the resolution of both the estimate and the state written back. +Merging can switch from the HIP estimator to the less accurate composite estimator, so results can depend on partitioning across threads, parts and shards. +Merging a lower-resolution sketch permanently lowers the result's resolution, regardless of the declared `lg_k`. )"; FunctionDocumentation::Syntax syntax = "uniqApacheHLL([lg_k, [type]])(x)"; FunctionDocumentation::Arguments arguments = { - {"x", "Column to compute the number of distinct values of.", {"(U)Int8/16/32/64", "Enum8", "Enum16", "BFloat16", "Float32", "Float64", "String", "FixedString", "UUID", "IPv4", "IPv6", "Date", "Date32", "DateTime", "DateTime64"}}, + {"x", "Column to compute the number of distinct values of.", {"(U)Int8/16/32/64", "Enum", "BFloat16", "Float32", "Float64", "String", "FixedString", "UUID", "IPv4", "IPv6", "Date", "Date32", "DateTime", "DateTime64"}}, }; FunctionDocumentation::Parameters parameters = { {"lg_k", "Optional. Log-base-2 of the number of buckets, in range [4, 21]. Higher means better accuracy and more memory. Default: 12.", {"UInt8"}}, diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index b223440e1e1b..718398fe97ae 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -25,7 +25,6 @@ #include #include -#include #include #include @@ -41,18 +40,11 @@ namespace ErrorCodes } -/** An Apache DataSketches HLL sketch as an aggregate function state, serialized in the - * DataSketches HLL format so that `-State`/`-Merge` interoperate with external services. - * - * `lg_config_k` and the target type are owned by the aggregate function and passed in, so the - * state stores the sketch alone. - */ -class HllSketchData : private boost::noncopyable +/// Keeps insertion and union states separate to preserve the estimator used before merging. +class HllSketchData { private: - /// Used for insertions. std::unique_ptr sk_update; - /// Used for merging. std::unique_ptr sk_union; datasketches::hll_sketch * getSkUpdate(uint8_t lg_config_k, datasketches::target_hll_type tgt_type) @@ -66,31 +58,22 @@ class HllSketchData : private boost::noncopyable { if (!sk_union) { - /// `hll_union` takes `lg_max_k` in [7, 21] while a sketch may use [4, 21], hence the - /// floor of 7. It does not inflate the result: `get_result` downsamples to the smallest - /// `lg_config_k` the union has seen. So the resolution of a merged state is the minimum - /// over its inputs, not the `lg_config_k` of the type, and merging a coarser sketch from - /// elsewhere lowers both the estimate and the state written back. Merged from empty - /// states only there is no minimum, and it serializes as `lg_config_k = 7`. + /// `hll_union` requires at least 7, but preserves the lower resolution of its inputs. sk_union = std::make_unique(std::max(lg_config_k, 7)); } return sk_union.get(); } - /// Fold a sketch updated after the union was allocated into it, so merges see one state. void foldUpdateIntoUnionIfNeeded() { if (sk_union && sk_update) { sk_union->update(*sk_update); - sk_update.reset(nullptr); + sk_update.reset(); } } public: - HllSketchData() = default; - ~HllSketchData() = default; - template void insert(T value, uint8_t lg_config_k, datasketches::target_hll_type tgt_type) { @@ -106,8 +89,7 @@ class HllSketchData : private boost::noncopyable UInt64 size(datasketches::target_hll_type tgt_type) const { - /// Round rather than truncate: `get_estimate` returns a `double`, and `999.9999` for an - /// exactly-known cardinality must not become `999`. + /// Rounding preserves exact cardinalities despite floating-point error. if (sk_union) return static_cast(std::llround(sk_union->get_result(tgt_type).get_estimate())); if (sk_update) @@ -119,11 +101,7 @@ class HllSketchData : private boost::noncopyable { datasketches::hll_union * u = getSkUnion(lg_config_k); - if (sk_update) - { - u->update(*sk_update); - sk_update.reset(nullptr); - } + foldUpdateIntoUnionIfNeeded(); if (rhs.sk_update) u->update(*rhs.sk_update); @@ -150,47 +128,27 @@ class HllSketchData : private boost::noncopyable } catch (const std::bad_alloc &) { - /// Memory pressure, not corrupted data. throw; } catch (const std::exception & e) { - /// `datasketches` reports malformed input as `std::invalid_argument` / `std::out_of_range`. - /// Not being `DB::Exception`, those escape `SerializationAggregateFunction`'s - /// `catch (...)` and abort as a logical error, so translate them here. + /// Translate malformed input to avoid a logical exception in `SerializationAggregateFunction`. throw Exception(ErrorCodes::CORRUPTED_DATA, "Cannot deserialize HLL sketch state: {}", e.what()); } } void write(WriteBuffer & out, datasketches::target_hll_type tgt_type) const { + datasketches::hll_sketch::vector_bytes bytes; if (sk_update) - { - auto bytes = sk_update->serialize_compact(); - writeVectorBinary(bytes, out); - } + bytes = sk_update->serialize_compact(); else if (sk_union) - { - auto bytes = sk_union->get_result(tgt_type).serialize_compact(); - writeVectorBinary(bytes, out); - } - else - { - datasketches::hll_sketch::vector_bytes bytes; - writeVectorBinary(bytes, out); - } + bytes = sk_union->get_result(tgt_type).serialize_compact(); + writeVectorBinary(bytes, out); } }; -/** `uniqApacheHLL` over a single column of a type the sketch can hash directly. - * - * The value takes one of the three shapes the DataSketches API accepts - an 8-byte integer, an - * IEEE-754 double, or raw bytes - so that an external producer reaches the same sketch. - * - * `lg_config_k` and the target type live here rather than in the state, which is what lets states - * of different parameterisations share one binary representation. - */ template class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper> { @@ -230,8 +188,7 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper if constexpr (std::is_same_v) { - /// ClickHouse holds a UUID as two 64-bit halves in host order, so its bytes in - /// memory are not the canonical 16 an external producer works from. + /// Convert the two host-order halves of a `UUID` to canonical bytes. const UInt64 halves[2] = { std::byteswap(UUIDHelpers::getHighBytes(value)), std::byteswap(UUIDHelpers::getLowBytes(value)), @@ -242,8 +199,7 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper /// Already held in network order, which is the canonical form. data.insertData(reinterpret_cast(&value), sizeof(value), lg_config_k, target_type); else if constexpr (is_decimal) - /// `DateTime64(3)` holds the epoch milliseconds a caller elsewhere passes to - /// `update(long)`. The scale belongs to the type, so both sides must agree on it. + /// Hash `DateTime64` as epoch ticks; external producers must use the same scale. data.insert(static_cast(value.value), lg_config_k, target_type); else if constexpr (std::is_same_v) data.insert(static_cast(value.toUnderType()), lg_config_k, target_type); @@ -255,8 +211,8 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper data.insert(static_cast(value), lg_config_k, target_type); } } - /// A serialized sketch describes its own configuration, so states of different parameterisations - /// are interchangeable. Merging across them takes the resolution of the coarsest input. + + /// Serialized sketches carry their configuration, so parameters need not match. bool haveSameStateRepresentationImpl(const IAggregateFunction & rhs) const override { return getName() == rhs.getName() && this->haveEqualArgumentTypes(rhs); @@ -284,12 +240,6 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper }; - -/// `uniqApacheHLL([lg_k, [type]])(x)`, with `lg_k` in [4, 21] (default 12) and `type` one of -/// 'HLL_4', 'HLL_6', 'HLL_8' (default 'HLL_4'). -AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( - const std::string & name, const DataTypes & argument_types, const Array & params, const Settings *); - } #endif diff --git a/tests/queries/0_stateless/04327_uniq_apache_hll.reference b/tests/queries/0_stateless/04327_uniq_apache_hll.reference index f2059d5f29fa..ef814b89b951 100644 --- a/tests/queries/0_stateless/04327_uniq_apache_hll.reference +++ b/tests/queries/0_stateless/04327_uniq_apache_hll.reference @@ -1,19 +1,30 @@ accuracy 1 1 -1 -1 +1 1 empty and single 0 1 -argument types -1 -1 -1 -1 -state/merge roundtrip is native -1 +state and merge 1 1 AggregateFunction(uniqApacheHLL(14, \'HLL_8\'), UInt64) parameter validation +argument types +20 +20 +20 +20 +20 +20 +20 +20 +20 +20 +20 +3 +nullable and low cardinality +20 +20 +unsupported types +one argument only diff --git a/tests/queries/0_stateless/04327_uniq_apache_hll.sql b/tests/queries/0_stateless/04327_uniq_apache_hll.sql index 584ba4c067c2..1de290b47236 100644 --- a/tests/queries/0_stateless/04327_uniq_apache_hll.sql +++ b/tests/queries/0_stateless/04327_uniq_apache_hll.sql @@ -2,30 +2,20 @@ -- ^ DataSketches is not built in fast-test builds. SELECT 'accuracy'; --- HLL is approximate but deterministic; assert the estimate is within the expected error band. -SELECT abs(toInt64(uniqApacheHLL(number)) - 1000) < 30 FROM numbers(1000); SELECT abs(toInt64(uniqApacheHLL(number)) - 100000) < 3000 FROM numbers(100000); --- Higher lg_k -> better accuracy. SELECT abs(toInt64(uniqApacheHLL(14)(number)) - 100000) < 1500 FROM numbers(100000); -- Storage type does not change the estimate. -SELECT uniqApacheHLL(12, 'HLL_4')(number) = uniqApacheHLL(12, 'HLL_8')(number) FROM numbers(1000); +SELECT + uniqApacheHLL(12, 'HLL_4')(number) = uniqApacheHLL(12, 'HLL_6')(number), + uniqApacheHLL(12, 'HLL_4')(number) = uniqApacheHLL(12, 'HLL_8')(number) +FROM numbers(1000); SELECT 'empty and single'; SELECT uniqApacheHLL(number) FROM numbers(0); SELECT uniqApacheHLL(number) FROM numbers(1); -SELECT 'argument types'; -SELECT abs(toInt64(uniqApacheHLL(toInt32(number))) - 500) < 20 FROM numbers(500); -SELECT abs(toInt64(uniqApacheHLL(toFloat64(number))) - 500) < 20 FROM numbers(500); -SELECT abs(toInt64(uniqApacheHLL(toString(number))) - 500) < 20 FROM numbers(500); -SELECT abs(toInt64(uniqApacheHLL(toDate('2020-01-01') + number)) - 500) < 20 FROM numbers(500); - -SELECT 'state/merge roundtrip is native'; --- Merging per-group `-State` values covers exactly the same set as a direct aggregate, so the two --- estimates agree to within the error of the sketch. They are not required to be equal: DataSketches --- reports the HIP estimate for a sketch that has only been updated and the composite estimate for --- one that came out of a union, so a merged result is not the same number as a directly built one --- even though both are computed from identical registers. +SELECT 'state and merge'; +-- Direct and merged sketches may use different estimators. SELECT abs(toInt64(uniqApacheHLLMerge(s)) - toInt64((SELECT uniqApacheHLL(number) FROM numbers(100000)))) < 3000 FROM @@ -35,22 +25,11 @@ FROM GROUP BY number % 17 ); --- Merging is independent of how the input was partitioned: the registers of the union do not depend --- on the grouping, and both sides come out of a union, so two different partitionings of the same --- set agree exactly. +-- Different partitions must produce the same union estimate. SELECT (SELECT uniqApacheHLLMerge(s) FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17)) = (SELECT uniqApacheHLLMerge(s) FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 13)); --- The same holds for a single state. -SELECT - abs(toInt64(uniqApacheHLLMerge(s)) - toInt64((SELECT uniqApacheHLL(number) FROM numbers(1000)))) < 30 -FROM -( - SELECT uniqApacheHLLState(number) AS s FROM numbers(1000) -); - --- The state type carries the sketch parameters. SELECT toTypeName(uniqApacheHLLState(14, 'HLL_8')(number)) FROM numbers(1); SELECT 'parameter validation'; @@ -58,3 +37,33 @@ SELECT uniqApacheHLL(3)(number) FROM numbers(1); -- { serverError ARGUMENT_OUT_O SELECT uniqApacheHLL(22)(number) FROM numbers(1); -- { serverError ARGUMENT_OUT_OF_BOUND } SELECT uniqApacheHLL(12, 'HLL_9')(number) FROM numbers(1); -- { serverError BAD_ARGUMENTS } SELECT uniqApacheHLL(12, 'HLL_4', 1)(number) FROM numbers(1); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } + +SELECT 'argument types'; +-- Small counts stay in coupon mode, where the estimate is exact. +SELECT uniqApacheHLL(toUInt64(number)) FROM numbers(20); +SELECT uniqApacheHLL(toInt32(number)) FROM numbers(20); +SELECT uniqApacheHLL(toBFloat16(number)) FROM numbers(20); +SELECT uniqApacheHLL(toFloat32(number)) FROM numbers(20); +SELECT uniqApacheHLL(toFloat64(number)) FROM numbers(20); +SELECT uniqApacheHLL(toString(number)) FROM numbers(20); +SELECT uniqApacheHLL(toFixedString(toString(number), 8)) FROM numbers(20); +SELECT uniqApacheHLL(toDate('2020-01-01') + number) FROM numbers(20); +SELECT uniqApacheHLL(toDate32('2020-01-01') + number) FROM numbers(20); +SELECT uniqApacheHLL(toDateTime('2020-01-01 00:00:00') + number) FROM numbers(20); +SELECT uniqApacheHLL(toIPv4('1.2.3.0') + number) FROM numbers(20); +SELECT uniqApacheHLL(CAST(number % 3, 'Enum8(\'a\' = 0, \'b\' = 1, \'c\' = 2)')) FROM numbers(20); + +SELECT 'nullable and low cardinality'; +SELECT uniqApacheHLL(toNullable(number)) FROM numbers(20); +SELECT uniqApacheHLL(toLowCardinality(toString(number))) FROM numbers(20); + +SELECT 'unsupported types'; +SELECT uniqApacheHLL(toInt128(number)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } +SELECT uniqApacheHLL(toUInt256(number)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } +SELECT uniqApacheHLL(toDecimal64(number, 2)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } +SELECT uniqApacheHLL(materialize([number])) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } +SELECT uniqApacheHLL((number, number + 1)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } + +SELECT 'one argument only'; +SELECT uniqApacheHLL(number, number + 1) FROM numbers(20); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } +SELECT uniqApacheHLL() FROM numbers(1); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } diff --git a/tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference b/tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference index ac07e5cd8a1c..70c97654e417 100644 --- a/tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference +++ b/tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference @@ -1,23 +1,23 @@ import sketches built outside ClickHouse 5 -5 -8 -8 -3 +11 84 5 0 types wider than the 8 bytes update(long) takes 1 -5 1 -5 DateTime64 is hashed as the epoch time it counts 1 export sketches for consumption outside ClickHouse 1 1 1 -states survive a write/read cycle through storage +store and merge states with a different lg_k 20 +merging states built with a different lg_k +1 1 +relabelling with CAST does not rescale +1 1 +only the parameters are interchangeable diff --git a/tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql b/tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql index 890d74aa89b0..f29084519fac 100644 --- a/tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql +++ b/tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql @@ -1,40 +1,24 @@ -- Tags: no-fasttest -- ^ DataSketches is not built in fast-test builds. --- The state of `uniqApacheHLL` carries an Apache DataSketches HLL sketch in its native serialized --- form, framed with the varint length prefix that `writeVectorBinary` adds. Every blob below was --- produced by the Apache DataSketches C++ library directly rather than by ClickHouse, so these --- queries pin the wire contract with external producers in both directions. A change that breaks --- interoperability has to change one of these literals. +-- Fixtures were generated by Apache DataSketches C++, with a varint length prefix. +-- These checks pin compatibility with external producers and consumers. SELECT 'import sketches built outside ClickHouse'; --- lg_k = 12, HLL_4, `hll_sketch::update(uint64_t)` over 0..4. Coupon list mode. +-- lg_k = 12, HLL_4, `hll_sketch::update` on `uint64_t` values over 0..4. Coupon list mode. SELECT finalizeAggregation(CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)')); --- The same sketch reached through `-Merge`. -SELECT uniqApacheHLLMerge(s) FROM (SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s); - --- Two external sketches over 0..4 and 3..7 union to 8 distinct values. +-- External sketches over 0..4 and 3..7 overlap with ClickHouse values 6..10. SELECT uniqApacheHLLMerge(s) FROM ( SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s UNION ALL SELECT CAST(unhex('1C0201070C030805007581660781BC5D067B65E608FC2D420AC1E91705'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s -); - --- An external sketch merged with one built by ClickHouse: the two hash identically, so 0..4 and --- 3..7 overlap on 3 and 4 and the union is 8. -SELECT uniqApacheHLLMerge(s) FROM -( - SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s UNION ALL - SELECT uniqApacheHLLState(number) AS s FROM numbers(3, 5) + SELECT uniqApacheHLLState(number) AS s FROM numbers(6, 5) ); --- lg_k = 12, HLL_4, raw bytes of 'alpha', 'beta', 'gamma'. -SELECT finalizeAggregation(CAST(unhex('140201070C03080300BD3A090A8E5A62115168C90A'), 'AggregateFunction(uniqApacheHLL, UInt64)')); - -- lg_k = 4, HLL_4, 100 distinct values. Dense HLL mode rather than a coupon list. SELECT finalizeAggregation(CAST(unhex('300A0107040008020215EB1DC787F15440000000000000FB3F000000000000000003000000000000000251214121031025'), 'AggregateFunction(uniqApacheHLL(4), UInt64)')); @@ -46,32 +30,26 @@ SELECT finalizeAggregation(CAST(unhex('080201070C030C0000'), 'AggregateFunction( SELECT 'types wider than the 8 bytes update(long) takes'; --- lg_k = 12, HLL_4 over the canonical 16 bytes of 00000000-0000-0000-0000-00000000000{0..4}. ClickHouse --- holds a UUID as two 64-bit halves in host order, so hashing its bytes in memory would not agree with --- an external producer working from the textual form; the canonical order is hashed instead. +-- Canonical bytes of `UUID` values 00000000-0000-0000-0000-00000000000{0..4}. SELECT hex(toString(uniqApacheHLLState(toUUID(concat('00000000-0000-0000-0000-00000000000', toString(number)))))) = '1C0201070C0308050050C94D05854BD10ADB8CBD053C56FB07F8FDB206' FROM numbers(5) SETTINGS max_threads = 1; -SELECT finalizeAggregation(CAST(unhex('1C0201070C0308050050C94D05854BD10ADB8CBD053C56FB07F8FDB206'), 'AggregateFunction(uniqApacheHLL, UUID)')); -- The same for 2001:db8::1 .. ::5, which ClickHouse already holds in network order. SELECT hex(toString(uniqApacheHLLState(toIPv6(concat('2001:db8::', hex(number + 1)))))) = '1C0201070C0308050018216E09FAB4750F52D5BB07DFBDE30A79BC9D0B' FROM numbers(5) SETTINGS max_threads = 1; -SELECT finalizeAggregation(CAST(unhex('1C0201070C0308050018216E09FAB4750F52D5BB07DFBDE30A79BC9D0B'), 'AggregateFunction(uniqApacheHLL, IPv6)')); SELECT 'DateTime64 is hashed as the epoch time it counts'; --- A `DateTime64(3)` is hashed as its epoch milliseconds, which is the `long` a caller elsewhere --- passes for the same instant. These five are 2020-01-01 00:00:00.000 and the next four seconds. +-- Epoch milliseconds for 2020-01-01 00:00:00.000 UTC and the next four seconds. SELECT hex(toString(uniqApacheHLLState(toDateTime64('2020-01-01 00:00:00.000', 3, 'UTC') + number))) = '1C0201070C0308050001D4B019CBDD6F1059310D0E833938083897B304' FROM numbers(5) SETTINGS max_threads = 1; SELECT 'export sketches for consumption outside ClickHouse'; --- `max_threads` is pinned because a state that was merged from several partial states may lay its --- coupons out in a different order than a state built by a single thread. +-- Use one thread to keep coupon ordering deterministic. SELECT hex(toString(uniqApacheHLLState(number))) = '1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06' FROM numbers(5) SETTINGS max_threads = 1; SELECT hex(toString(uniqApacheHLLState(14, 'HLL_8')(number))) = '1C0201070E03080508CBD7C2042BF2FB06862FF90D7581660781BC5D06' FROM numbers(5) SETTINGS max_threads = 1; @@ -80,16 +58,33 @@ SELECT hex(toString(uniqApacheHLLMergeState(s))) = '1C0201070C03080500CBD7C2042B FROM (SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s) SETTINGS max_threads = 1; -SELECT 'states survive a write/read cycle through storage'; +SELECT 'store and merge states with a different lg_k'; DROP TABLE IF EXISTS hll_interop_states; CREATE TABLE hll_interop_states (k UInt8, s AggregateFunction(uniqApacheHLL, UInt64)) ENGINE = AggregatingMergeTree ORDER BY k; -INSERT INTO hll_interop_states SELECT number % 4 AS k, uniqApacheHLLState(number) FROM numbers(20) GROUP BY k; +INSERT INTO hll_interop_states SELECT number % 4 AS k, uniqApacheHLLState(8)(number) FROM numbers(20) GROUP BY k; OPTIMIZE TABLE hll_interop_states FINAL; -- Small enough to stay in coupon mode, so the union is exact regardless of how it was partitioned. SELECT uniqApacheHLLMerge(s) FROM hll_interop_states; --- A state read back from disk is still a valid sketch for an external consumer. -SELECT countDistinct(hex(toString(s))) = 4 FROM hll_interop_states; DROP TABLE hll_interop_states; --- Malformed states are rejected as `CORRUPTED_DATA`; see `05136_uniq_apache_hll_corrupted_state.sh`. +SELECT 'merging states built with a different lg_k'; +-- Downsampling and building at the lower resolution must produce the same union estimate. +SELECT + (SELECT uniqApacheHLLMerge(8)(s) FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17)) + = (SELECT uniqApacheHLLMerge(8)(s) FROM (SELECT uniqApacheHLLState(8)(number) AS s FROM numbers(100000) GROUP BY number % 17)); + +-- Downsampling must also reduce the serialized state size. +SELECT + length(toString(uniqApacheHLLMergeState(8)(s))) < length(toString(uniqApacheHLLMergeState(s))) / 8 +FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17); + +SELECT 'relabelling with CAST does not rescale'; +SELECT + finalizeAggregation(CAST(s, 'AggregateFunction(uniqApacheHLL(8), UInt64)')) = finalizeAggregation(s), + length(toString(CAST(s, 'AggregateFunction(uniqApacheHLL(8), UInt64)'))) = length(toString(s)) +FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(1000)); + +SELECT 'only the parameters are interchangeable'; +SELECT CAST(uniqApacheHLLState(toString(number)), 'AggregateFunction(uniqApacheHLL, UInt64)') FROM numbers(10); -- { serverError CANNOT_CONVERT_TYPE } +SELECT CAST(uniqThetaState(number), 'AggregateFunction(uniqApacheHLL, UInt64)') FROM numbers(10); -- { serverError CANNOT_CONVERT_TYPE } diff --git a/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.reference b/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.reference deleted file mode 100644 index 2fc0b7c7d810..000000000000 --- a/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.reference +++ /dev/null @@ -1,11 +0,0 @@ -the two types remain distinct -AggregateFunction(uniqApacheHLL, UInt64) -AggregateFunction(uniqApacheHLL(8), UInt64) -merging states built with a different lg_k -1 -1 -relabelling with CAST does not rescale -1 1 -a state of one lg_k can be stored in a column declared with another -20 -only the parameters are interchangeable diff --git a/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.sql b/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.sql deleted file mode 100644 index 43588e9d133a..000000000000 --- a/tests/queries/0_stateless/05027_uniq_apache_hll_cross_scale.sql +++ /dev/null @@ -1,47 +0,0 @@ --- Tags: no-fasttest --- ^ DataSketches is not built in fast-test builds. - --- `lg_k` and the sketch type configure the sketch but are held by the aggregate function rather than --- by its state, whose layout is the same whatever they are, and a serialized sketch records its own --- `lg_k`. `uniqApacheHLL` therefore reports states of different parameterisations as having one --- binary representation, which lets `-Merge` read a state built with one `lg_k` under a function --- declared with another, and lets `CAST` relabel a column between the two types. - -SELECT 'the two types remain distinct'; -SELECT toTypeName(uniqApacheHLLState(number)) FROM numbers(1); -SELECT toTypeName(uniqApacheHLLState(8)(number)) FROM numbers(1); - -SELECT 'merging states built with a different lg_k'; --- Downsampling during a union produces the same registers as building at the target `lg_k` from the --- start, so rescaling 17 states of `lg_k` 12 agrees exactly with merging 17 built at 8. Both sides --- come out of a union and so use the same estimator. -SELECT - (SELECT uniqApacheHLLMerge(8)(s) FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17)) - = (SELECT uniqApacheHLLMerge(8)(s) FROM (SELECT uniqApacheHLLState(8)(number) AS s FROM numbers(100000) GROUP BY number % 17)); - --- Merging is lossy in one direction only: the union takes the resolution of its coarsest input, so a --- rescaled state is much smaller than the states it was built from. -SELECT - length(toString(uniqApacheHLLMergeState(8)(s))) < length(toString(uniqApacheHLLMergeState(s))) / 8 -FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17); - -SELECT 'relabelling with CAST does not rescale'; --- `CAST` between the two types re-associates the column with the other function without touching its --- data, so the sketch keeps the resolution it was built with and the estimate does not change. -SELECT - finalizeAggregation(CAST(s, 'AggregateFunction(uniqApacheHLL(8), UInt64)')) = finalizeAggregation(s), - length(toString(CAST(s, 'AggregateFunction(uniqApacheHLL(8), UInt64)'))) = length(toString(s)) -FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(1000)); - -SELECT 'a state of one lg_k can be stored in a column declared with another'; -DROP TABLE IF EXISTS hll_cross_scale; -CREATE TABLE hll_cross_scale (s AggregateFunction(uniqApacheHLL, UInt64)) ENGINE = MergeTree ORDER BY tuple(); -INSERT INTO hll_cross_scale SELECT uniqApacheHLLState(8)(number) FROM numbers(20); -SELECT uniqApacheHLLMerge(s) FROM hll_cross_scale; -DROP TABLE hll_cross_scale; - -SELECT 'only the parameters are interchangeable'; --- The argument types and the function itself must still match: `haveEqualArgumentTypes` and the name --- comparison keep these apart even though the parameters no longer do. -SELECT CAST(uniqApacheHLLState(toString(number)), 'AggregateFunction(uniqApacheHLL, UInt64)') FROM numbers(10); -- { serverError CANNOT_CONVERT_TYPE } -SELECT CAST(uniqThetaState(number), 'AggregateFunction(uniqApacheHLL, UInt64)') FROM numbers(10); -- { serverError CANNOT_CONVERT_TYPE } diff --git a/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.reference b/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.reference deleted file mode 100644 index 6478f4dcb46e..000000000000 --- a/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.reference +++ /dev/null @@ -1,26 +0,0 @@ -types the sketch hashes directly -20 -20 -20 -20 -20 -20 -20 -20 -20 -20 -20 -20 -20 -20 -3 -the wrappers of an accepted type are accepted -20 -20 -a DateTime64 is the integer a caller elsewhere would hash -1 -types no other implementation can reproduce are rejected -one argument only -the parameters still apply -20 -AggregateFunction(uniqApacheHLL(8, \'HLL_6\'), UInt64) diff --git a/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.sql b/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.sql deleted file mode 100644 index 6b133e97f64c..000000000000 --- a/tests/queries/0_stateless/05028_uniq_apache_hll_argument_types.sql +++ /dev/null @@ -1,54 +0,0 @@ --- Tags: no-fasttest --- ^ DataSketches is not built in fast-test builds. - --- `uniqApacheHLL` accepts only the types a sketch hashes the same way in every implementation, as --- an 8-byte integer, an IEEE-754 double or their raw bytes. Anything else would have to be hashed --- to a single value by ClickHouse first, which no producer outside ClickHouse could reproduce. --- All counts below are small enough for the sketch to stay in coupon mode, where it is exact. - -SELECT 'types the sketch hashes directly'; -SELECT uniqApacheHLL(toUInt64(number)) FROM numbers(20); -SELECT uniqApacheHLL(toInt32(number)) FROM numbers(20); -SELECT uniqApacheHLL(toBFloat16(number)) FROM numbers(20); -SELECT uniqApacheHLL(toFloat32(number)) FROM numbers(20); -SELECT uniqApacheHLL(toFloat64(number)) FROM numbers(20); -SELECT uniqApacheHLL(toString(number)) FROM numbers(20); -SELECT uniqApacheHLL(toFixedString(toString(number), 8)) FROM numbers(20); -SELECT uniqApacheHLL(toDate('2020-01-01') + number) FROM numbers(20); -SELECT uniqApacheHLL(toDate32('2020-01-01') + number) FROM numbers(20); -SELECT uniqApacheHLL(toDateTime('2020-01-01 00:00:00') + number) FROM numbers(20); -SELECT uniqApacheHLL(toDateTime64('2020-01-01 00:00:00.000', 3) + number) FROM numbers(20); -SELECT uniqApacheHLL(reinterpretAsUUID(toUInt128(number))) FROM numbers(20); -SELECT uniqApacheHLL(toIPv4('1.2.3.0') + number) FROM numbers(20); -SELECT uniqApacheHLL(toIPv6(concat('2001:db8::', hex(number + 1)))) FROM numbers(20); -SELECT uniqApacheHLL(CAST(number % 3, 'Enum8(\'a\' = 0, \'b\' = 1, \'c\' = 2)')) FROM numbers(20); - -SELECT 'the wrappers of an accepted type are accepted'; -SELECT uniqApacheHLL(toNullable(number)) FROM numbers(20); -SELECT uniqApacheHLL(toLowCardinality(toString(number))) FROM numbers(20); - -SELECT 'a DateTime64 is the integer a caller elsewhere would hash'; --- `DateTime64(3)` counts epoch milliseconds, so it produces the sketch of those milliseconds. -SELECT hex(toString(uniqApacheHLLState(toDateTime64('2020-01-01 00:00:00.000', 3, 'UTC') + number))) - = hex(toString(uniqApacheHLLState(toInt64(1577836800000) + number * 1000))) -FROM numbers(5) SETTINGS max_threads = 1; - -SELECT 'types no other implementation can reproduce are rejected'; --- No byte order for the wide integers is agreed on across implementations. -SELECT uniqApacheHLL(toInt128(number)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } -SELECT uniqApacheHLL(toUInt256(number)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } --- No DataSketches binding accepts a decimal, and the scale is not part of the sketch. -SELECT uniqApacheHLL(toDecimal64(number, 2)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } -SELECT uniqApacheHLL(toDecimal128(number, 4)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } --- A composite value would have to be hashed by ClickHouse first. -SELECT uniqApacheHLL(materialize([number])) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } -SELECT uniqApacheHLL((number, number + 1)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } - -SELECT 'one argument only'; -SELECT uniqApacheHLL(number, number + 1) FROM numbers(20); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } -SELECT uniqApacheHLL(toString(number), number) FROM numbers(20); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } -SELECT uniqApacheHLL() FROM numbers(1); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } - -SELECT 'the parameters still apply'; -SELECT uniqApacheHLL(8)(reinterpretAsUUID(toUInt128(number))) FROM numbers(20); -SELECT toTypeName(uniqApacheHLLState(8, 'HLL_6')(number)) FROM numbers(1); diff --git a/tests/queries/0_stateless/05055_uniq_theta_compressed_state.sql b/tests/queries/0_stateless/05055_uniq_theta_compressed_state.sql index 9668fc48c74e..848cfc356e9f 100644 --- a/tests/queries/0_stateless/05055_uniq_theta_compressed_state.sql +++ b/tests/queries/0_stateless/05055_uniq_theta_compressed_state.sql @@ -1,17 +1,9 @@ -- Tags: no-fasttest -- - no-fasttest -- compiled w/o datasketches --- Apache DataSketches can serialize a Theta sketch in a compressed form (serialization --- version 4) that packs the retained values at a variable number of bits per entry. --- ClickHouse always writes the uncompressed form, but it has to read the compressed one, --- because an `AggregateFunction(uniqTheta, ...)` state can come from another --- implementation of the library. Two of the unpacking routines decoded it incorrectly, --- and the result was a silently too low estimate rather than an error. - --- Both states below are canonical compressed encodings of a sketch that retains 16 values --- with theta = 1, so the estimate is exact. The first packs its values at 33 bits per --- entry and the second at 35; the two widths are decoded by separate routines. The two --- sketches have exactly two values in common, so their union retains 30. +-- Compressed Theta states (serialization version 4) exercise separate 33- and 35-bit +-- unpacking routines that previously underestimated cardinality. Each retains 16 values +-- with theta = 1; two shared values give a union of 30. WITH CAST(unhex('4B01040321011ACC9310000001F4000000FA0000007D0000003E8000001F4000000FA0000007D180000000000001F4000000FA0000007D0000003E8000001F4000000FA0000007D0000003E8') AS AggregateFunction(uniqTheta, UInt64)) AS state_33_bits, diff --git a/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.reference b/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.reference index f70b0368a421..d7c73d3f18be 100644 --- a/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.reference +++ b/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.reference @@ -1,3 +1,2 @@ OK unknown type -OK bad payload OK rowbinary diff --git a/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.sh b/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.sh index 18ad78792ac1..80c16c7e17f6 100755 --- a/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.sh +++ b/tests/queries/0_stateless/05136_uniq_apache_hll_corrupted_state.sh @@ -2,34 +2,20 @@ # Tags: no-fasttest # no-fasttest -- compiled w/o datasketches -# A `uniqApacheHLL` state that is not a sketch must be rejected as `CORRUPTED_DATA`. -# -# `datasketches` reports a payload it cannot parse as `std::invalid_argument` or -# `std::out_of_range`, neither of which is a `DB::Exception`, so without the -# translation in `HllSketchData::read` they would escape -# `SerializationAggregateFunction`'s `catch (...)` and be reported as a logical error. -# -# This is a shell test rather than a `.sql` one with a `serverError` hint because the -# client prints an extra exception with a stack trace to stderr for this class of error, -# which a `.sql` test counts as a failure. The same applies to `uniqTheta`, see +# Malformed states must raise `CORRUPTED_DATA`, not a logical exception. +# A shell test tolerates the client's extra stack trace, as in # `04307_uniqTheta_corrupted_state_106259.sh`. CUR_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd) # shellcheck source=../shell_config.sh . "$CUR_DIR"/../shell_config.sh -# A well-formed varint length prefix followed by eight bytes that are not a sketch: -# the first byte of the payload is the preamble length, which no HLL sketch uses. +# Valid length prefix, invalid sketch preamble. $CLICKHOUSE_CLIENT --query \ "SELECT finalizeAggregation(CAST(unhex('08FFFFFFFFFFFFFFFF'), 'AggregateFunction(uniqApacheHLL, UInt64)'))" 2>&1 \ | grep -q -F 'CORRUPTED_DATA' && echo 'OK unknown type' || echo 'FAIL unknown type' -$CLICKHOUSE_CLIENT --query \ - "SELECT finalizeAggregation(CAST(unhex('0801020304050607FF'), 'AggregateFunction(uniqApacheHLL, UInt64)'))" 2>&1 \ - | grep -q -F 'CORRUPTED_DATA' && echo 'OK bad payload' || echo 'FAIL bad payload' - -# The `RowBinary` path reaches `HllSketchData::read` through `deserializeBinary`: -# the leading 0x03 claims a three-byte payload, shorter than any HLL sketch. +# The `RowBinary` payload is shorter than any valid HLL sketch. printf '\x03\x03\x03\x30\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' \ | $CLICKHOUSE_LOCAL --input-format=RowBinary \ --structure='x AggregateFunction(uniqApacheHLL, IPv6)' \ From 562352fabff053515ba52306a096db61a726dbf6 Mon Sep 17 00:00:00 2001 From: Andrey Zvonov <32552679+zvonand@users.noreply.github.com> Date: Wed, 30 Sep 2026 13:53:03 +0200 Subject: [PATCH 04/22] Fix `uniqApacheHLL` merge resolution below 7 and empty state serialization - Construct `hll_union` with the requested `lg_k` instead of flooring it at 7, so `uniqApacheHLLMerge(4..6)` produces sketches with the requested `lg_k`. `hll_union` accepts `lg_max_k` in [4, 21]. - Serialize an untouched state as the library's compact empty sketch instead of a zero-length vector, so external DataSketches consumers can read it. Related: https://github.com/Altinity/ClickHouse/pull/2398#discussion_r4142688238 Related: https://github.com/Altinity/ClickHouse/pull/2398#discussion_r4142688245 --- .../AggregateFunctionUniqApacheHLL.h | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index 718398fe97ae..a6325e9f0ca8 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -26,7 +26,6 @@ #include -#include #include #include #include @@ -57,10 +56,7 @@ class HllSketchData datasketches::hll_union * getSkUnion(uint8_t lg_config_k) { if (!sk_union) - { - /// `hll_union` requires at least 7, but preserves the lower resolution of its inputs. - sk_union = std::make_unique(std::max(lg_config_k, 7)); - } + sk_union = std::make_unique(lg_config_k); return sk_union.get(); } @@ -137,13 +133,15 @@ class HllSketchData } } - void write(WriteBuffer & out, datasketches::target_hll_type tgt_type) const + void write(WriteBuffer & out, uint8_t lg_config_k, datasketches::target_hll_type tgt_type) const { datasketches::hll_sketch::vector_bytes bytes; if (sk_update) bytes = sk_update->serialize_compact(); else if (sk_union) bytes = sk_union->get_result(tgt_type).serialize_compact(); + else + bytes = datasketches::hll_sketch(lg_config_k, tgt_type).serialize_compact(); writeVectorBinary(bytes, out); } }; @@ -225,7 +223,7 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper void serialize(ConstAggregateDataPtr __restrict place, WriteBuffer & buf, std::optional /* version */) const override { - this->data(place).write(buf, target_type); + this->data(place).write(buf, lg_config_k, target_type); } void deserialize(AggregateDataPtr __restrict place, ReadBuffer & buf, std::optional /* version */, Arena *) const override From 81fb83c627ce91272e5ae780bb8b7a663e41e25b Mon Sep 17 00:00:00 2001 From: Andrey Zvonov <32552679+zvonand@users.noreply.github.com> Date: Thu, 1 Oct 2026 11:12:51 +0200 Subject: [PATCH 05/22] Fix `uniqApacheHLL` on an all-NULL argument and merge its tests For `Nullable(Nothing)` the factory creates the nested function before the `Null` combinator replaces it with `nothing`, so `uniqApacheHLL(NULL)` threw `ILLEGAL_TYPE_OF_ARGUMENT` instead of returning 0 like `uniq` and `uniqTheta`. Merge `05026_uniq_apache_hll_interop` into `04327_uniq_apache_hll` and add the all-NULL cases there. `05136_uniq_apache_hll_corrupted_state.sh` stays a shell test because `CORRUPTED_DATA` is always logged with a stack trace to stderr. https://github.com/Altinity/ClickHouse/pull/2398#issuecomment-5925895693 --- .../AggregateFunctionUniqApacheHLL.cpp | 4 + .../04327_uniq_apache_hll.reference | 27 ++++++ .../0_stateless/04327_uniq_apache_hll.sql | 93 +++++++++++++++++++ .../05026_uniq_apache_hll_interop.reference | 23 ----- .../05026_uniq_apache_hll_interop.sql | 90 ------------------ 5 files changed, 124 insertions(+), 113 deletions(-) delete mode 100644 tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference delete mode 100644 tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp index 8da3fc42092a..21fb8363e155 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp @@ -86,6 +86,10 @@ static AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( if (which.isIPv6()) return std::make_shared>(lg_config_k, target_type, argument_types, params); + /// For `Nullable(Nothing)` the `Null` combinator replaces this function with `nothing`, but it must be created first. + if (argument_type.onlyNull()) + return std::make_shared>(lg_config_k, target_type, argument_types, params); + throw Exception(ErrorCodes::ILLEGAL_TYPE_OF_ARGUMENT, "Illegal type {} of argument for aggregate function {}. Use uniq, uniqCombined or uniqHLL12 for unsupported types.", argument_type.getName(), name); diff --git a/tests/queries/0_stateless/04327_uniq_apache_hll.reference b/tests/queries/0_stateless/04327_uniq_apache_hll.reference index ef814b89b951..3939bfc12e81 100644 --- a/tests/queries/0_stateless/04327_uniq_apache_hll.reference +++ b/tests/queries/0_stateless/04327_uniq_apache_hll.reference @@ -26,5 +26,32 @@ argument types nullable and low cardinality 20 20 +0 +0 +0 +UInt64 0 unsupported types one argument only +import sketches built outside ClickHouse +5 +11 +84 +5 +0 +types wider than the 8 bytes update(long) takes +1 +1 +DateTime64 is hashed as the epoch time it counts +1 +export sketches for consumption outside ClickHouse +1 +1 +1 +store and merge states with a different lg_k +20 +merging states built with a different lg_k +1 +1 +relabelling with CAST does not rescale +1 1 +only the parameters are interchangeable diff --git a/tests/queries/0_stateless/04327_uniq_apache_hll.sql b/tests/queries/0_stateless/04327_uniq_apache_hll.sql index 1de290b47236..4d2a4bf5d477 100644 --- a/tests/queries/0_stateless/04327_uniq_apache_hll.sql +++ b/tests/queries/0_stateless/04327_uniq_apache_hll.sql @@ -56,6 +56,11 @@ SELECT uniqApacheHLL(CAST(number % 3, 'Enum8(\'a\' = 0, \'b\' = 1, \'c\' = 2)')) SELECT 'nullable and low cardinality'; SELECT uniqApacheHLL(toNullable(number)) FROM numbers(20); SELECT uniqApacheHLL(toLowCardinality(toString(number))) FROM numbers(20); +-- All-NULL argument returns 0, as `uniq` does. +SELECT uniqApacheHLL(NULL); +SELECT uniqApacheHLL(14, 'HLL_8')(NULL); +SELECT uniqApacheHLL(NULL) FROM numbers(10); +SELECT toTypeName(uniqApacheHLLState(NULL)), uniqApacheHLLState(NULL); SELECT 'unsupported types'; SELECT uniqApacheHLL(toInt128(number)) FROM numbers(20); -- { serverError ILLEGAL_TYPE_OF_ARGUMENT } @@ -67,3 +72,91 @@ SELECT uniqApacheHLL((number, number + 1)) FROM numbers(20); -- { serverError IL SELECT 'one argument only'; SELECT uniqApacheHLL(number, number + 1) FROM numbers(20); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } SELECT uniqApacheHLL() FROM numbers(1); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } + +-- Fixtures were generated by Apache DataSketches C++, with a varint length prefix. +-- These checks pin compatibility with external producers and consumers. + +SELECT 'import sketches built outside ClickHouse'; + +-- lg_k = 12, HLL_4, `hll_sketch::update` on `uint64_t` values over 0..4. Coupon list mode. +SELECT finalizeAggregation(CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)')); + +-- External sketches over 0..4 and 3..7 overlap with ClickHouse values 6..10. +SELECT uniqApacheHLLMerge(s) FROM +( + SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s + UNION ALL + SELECT CAST(unhex('1C0201070C030805007581660781BC5D067B65E608FC2D420AC1E91705'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s + UNION ALL + SELECT uniqApacheHLLState(number) AS s FROM numbers(6, 5) +); + +-- lg_k = 4, HLL_4, 100 distinct values. Dense HLL mode rather than a coupon list. +SELECT finalizeAggregation(CAST(unhex('300A0107040008020215EB1DC787F15440000000000000FB3F000000000000000003000000000000000251214121031025'), 'AggregateFunction(uniqApacheHLL(4), UInt64)')); + +-- lg_k = 14, HLL_8: the non-default parameters are carried by the state type. +SELECT finalizeAggregation(CAST(unhex('1C0201070E03080508CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL(14, \'HLL_8\'), UInt64)')); + +-- An externally produced empty sketch. +SELECT finalizeAggregation(CAST(unhex('080201070C030C0000'), 'AggregateFunction(uniqApacheHLL, UInt64)')); + +SELECT 'types wider than the 8 bytes update(long) takes'; + +-- Canonical bytes of `UUID` values 00000000-0000-0000-0000-00000000000{0..4}. +SELECT hex(toString(uniqApacheHLLState(toUUID(concat('00000000-0000-0000-0000-00000000000', toString(number)))))) + = '1C0201070C0308050050C94D05854BD10ADB8CBD053C56FB07F8FDB206' +FROM numbers(5) SETTINGS max_threads = 1; + +-- The same for 2001:db8::1 .. ::5, which ClickHouse already holds in network order. +SELECT hex(toString(uniqApacheHLLState(toIPv6(concat('2001:db8::', hex(number + 1)))))) + = '1C0201070C0308050018216E09FAB4750F52D5BB07DFBDE30A79BC9D0B' +FROM numbers(5) SETTINGS max_threads = 1; + +SELECT 'DateTime64 is hashed as the epoch time it counts'; + +-- Epoch milliseconds for 2020-01-01 00:00:00.000 UTC and the next four seconds. +SELECT hex(toString(uniqApacheHLLState(toDateTime64('2020-01-01 00:00:00.000', 3, 'UTC') + number))) + = '1C0201070C0308050001D4B019CBDD6F1059310D0E833938083897B304' +FROM numbers(5) SETTINGS max_threads = 1; + +SELECT 'export sketches for consumption outside ClickHouse'; + +-- Use one thread to keep coupon ordering deterministic. +SELECT hex(toString(uniqApacheHLLState(number))) = '1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06' FROM numbers(5) SETTINGS max_threads = 1; +SELECT hex(toString(uniqApacheHLLState(14, 'HLL_8')(number))) = '1C0201070E03080508CBD7C2042BF2FB06862FF90D7581660781BC5D06' FROM numbers(5) SETTINGS max_threads = 1; + +-- Importing an external sketch and exporting it again must reproduce it byte for byte. +SELECT hex(toString(uniqApacheHLLMergeState(s))) = '1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06' +FROM (SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s) +SETTINGS max_threads = 1; + +SELECT 'store and merge states with a different lg_k'; + +DROP TABLE IF EXISTS hll_interop_states; +CREATE TABLE hll_interop_states (k UInt8, s AggregateFunction(uniqApacheHLL, UInt64)) ENGINE = AggregatingMergeTree ORDER BY k; +INSERT INTO hll_interop_states SELECT number % 4 AS k, uniqApacheHLLState(8)(number) FROM numbers(20) GROUP BY k; +OPTIMIZE TABLE hll_interop_states FINAL; +-- Small enough to stay in coupon mode, so the union is exact regardless of how it was partitioned. +SELECT uniqApacheHLLMerge(s) FROM hll_interop_states; +DROP TABLE hll_interop_states; + +SELECT 'merging states built with a different lg_k'; +-- Downsampling and building at the lower resolution must produce the same union estimate. +SELECT + (SELECT uniqApacheHLLMerge(8)(s) FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17)) + = (SELECT uniqApacheHLLMerge(8)(s) FROM (SELECT uniqApacheHLLState(8)(number) AS s FROM numbers(100000) GROUP BY number % 17)); + +-- Downsampling must also reduce the serialized state size. +SELECT + length(toString(uniqApacheHLLMergeState(8)(s))) < length(toString(uniqApacheHLLMergeState(s))) / 8 +FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17); + +SELECT 'relabelling with CAST does not rescale'; +SELECT + finalizeAggregation(CAST(s, 'AggregateFunction(uniqApacheHLL(8), UInt64)')) = finalizeAggregation(s), + length(toString(CAST(s, 'AggregateFunction(uniqApacheHLL(8), UInt64)'))) = length(toString(s)) +FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(1000)); + +SELECT 'only the parameters are interchangeable'; +SELECT CAST(uniqApacheHLLState(toString(number)), 'AggregateFunction(uniqApacheHLL, UInt64)') FROM numbers(10); -- { serverError CANNOT_CONVERT_TYPE } +SELECT CAST(uniqThetaState(number), 'AggregateFunction(uniqApacheHLL, UInt64)') FROM numbers(10); -- { serverError CANNOT_CONVERT_TYPE } diff --git a/tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference b/tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference deleted file mode 100644 index 70c97654e417..000000000000 --- a/tests/queries/0_stateless/05026_uniq_apache_hll_interop.reference +++ /dev/null @@ -1,23 +0,0 @@ -import sketches built outside ClickHouse -5 -11 -84 -5 -0 -types wider than the 8 bytes update(long) takes -1 -1 -DateTime64 is hashed as the epoch time it counts -1 -export sketches for consumption outside ClickHouse -1 -1 -1 -store and merge states with a different lg_k -20 -merging states built with a different lg_k -1 -1 -relabelling with CAST does not rescale -1 1 -only the parameters are interchangeable diff --git a/tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql b/tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql deleted file mode 100644 index f29084519fac..000000000000 --- a/tests/queries/0_stateless/05026_uniq_apache_hll_interop.sql +++ /dev/null @@ -1,90 +0,0 @@ --- Tags: no-fasttest --- ^ DataSketches is not built in fast-test builds. - --- Fixtures were generated by Apache DataSketches C++, with a varint length prefix. --- These checks pin compatibility with external producers and consumers. - -SELECT 'import sketches built outside ClickHouse'; - --- lg_k = 12, HLL_4, `hll_sketch::update` on `uint64_t` values over 0..4. Coupon list mode. -SELECT finalizeAggregation(CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)')); - --- External sketches over 0..4 and 3..7 overlap with ClickHouse values 6..10. -SELECT uniqApacheHLLMerge(s) FROM -( - SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s - UNION ALL - SELECT CAST(unhex('1C0201070C030805007581660781BC5D067B65E608FC2D420AC1E91705'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s - UNION ALL - SELECT uniqApacheHLLState(number) AS s FROM numbers(6, 5) -); - --- lg_k = 4, HLL_4, 100 distinct values. Dense HLL mode rather than a coupon list. -SELECT finalizeAggregation(CAST(unhex('300A0107040008020215EB1DC787F15440000000000000FB3F000000000000000003000000000000000251214121031025'), 'AggregateFunction(uniqApacheHLL(4), UInt64)')); - --- lg_k = 14, HLL_8: the non-default parameters are carried by the state type. -SELECT finalizeAggregation(CAST(unhex('1C0201070E03080508CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL(14, \'HLL_8\'), UInt64)')); - --- An externally produced empty sketch. -SELECT finalizeAggregation(CAST(unhex('080201070C030C0000'), 'AggregateFunction(uniqApacheHLL, UInt64)')); - -SELECT 'types wider than the 8 bytes update(long) takes'; - --- Canonical bytes of `UUID` values 00000000-0000-0000-0000-00000000000{0..4}. -SELECT hex(toString(uniqApacheHLLState(toUUID(concat('00000000-0000-0000-0000-00000000000', toString(number)))))) - = '1C0201070C0308050050C94D05854BD10ADB8CBD053C56FB07F8FDB206' -FROM numbers(5) SETTINGS max_threads = 1; - --- The same for 2001:db8::1 .. ::5, which ClickHouse already holds in network order. -SELECT hex(toString(uniqApacheHLLState(toIPv6(concat('2001:db8::', hex(number + 1)))))) - = '1C0201070C0308050018216E09FAB4750F52D5BB07DFBDE30A79BC9D0B' -FROM numbers(5) SETTINGS max_threads = 1; - -SELECT 'DateTime64 is hashed as the epoch time it counts'; - --- Epoch milliseconds for 2020-01-01 00:00:00.000 UTC and the next four seconds. -SELECT hex(toString(uniqApacheHLLState(toDateTime64('2020-01-01 00:00:00.000', 3, 'UTC') + number))) - = '1C0201070C0308050001D4B019CBDD6F1059310D0E833938083897B304' -FROM numbers(5) SETTINGS max_threads = 1; - -SELECT 'export sketches for consumption outside ClickHouse'; - --- Use one thread to keep coupon ordering deterministic. -SELECT hex(toString(uniqApacheHLLState(number))) = '1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06' FROM numbers(5) SETTINGS max_threads = 1; -SELECT hex(toString(uniqApacheHLLState(14, 'HLL_8')(number))) = '1C0201070E03080508CBD7C2042BF2FB06862FF90D7581660781BC5D06' FROM numbers(5) SETTINGS max_threads = 1; - --- Importing an external sketch and exporting it again must reproduce it byte for byte. -SELECT hex(toString(uniqApacheHLLMergeState(s))) = '1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06' -FROM (SELECT CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, UInt64)') AS s) -SETTINGS max_threads = 1; - -SELECT 'store and merge states with a different lg_k'; - -DROP TABLE IF EXISTS hll_interop_states; -CREATE TABLE hll_interop_states (k UInt8, s AggregateFunction(uniqApacheHLL, UInt64)) ENGINE = AggregatingMergeTree ORDER BY k; -INSERT INTO hll_interop_states SELECT number % 4 AS k, uniqApacheHLLState(8)(number) FROM numbers(20) GROUP BY k; -OPTIMIZE TABLE hll_interop_states FINAL; --- Small enough to stay in coupon mode, so the union is exact regardless of how it was partitioned. -SELECT uniqApacheHLLMerge(s) FROM hll_interop_states; -DROP TABLE hll_interop_states; - -SELECT 'merging states built with a different lg_k'; --- Downsampling and building at the lower resolution must produce the same union estimate. -SELECT - (SELECT uniqApacheHLLMerge(8)(s) FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17)) - = (SELECT uniqApacheHLLMerge(8)(s) FROM (SELECT uniqApacheHLLState(8)(number) AS s FROM numbers(100000) GROUP BY number % 17)); - --- Downsampling must also reduce the serialized state size. -SELECT - length(toString(uniqApacheHLLMergeState(8)(s))) < length(toString(uniqApacheHLLMergeState(s))) / 8 -FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(100000) GROUP BY number % 17); - -SELECT 'relabelling with CAST does not rescale'; -SELECT - finalizeAggregation(CAST(s, 'AggregateFunction(uniqApacheHLL(8), UInt64)')) = finalizeAggregation(s), - length(toString(CAST(s, 'AggregateFunction(uniqApacheHLL(8), UInt64)'))) = length(toString(s)) -FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(1000)); - -SELECT 'only the parameters are interchangeable'; -SELECT CAST(uniqApacheHLLState(toString(number)), 'AggregateFunction(uniqApacheHLL, UInt64)') FROM numbers(10); -- { serverError CANNOT_CONVERT_TYPE } -SELECT CAST(uniqThetaState(number), 'AggregateFunction(uniqApacheHLL, UInt64)') FROM numbers(10); -- { serverError CANNOT_CONVERT_TYPE } From 00dbe28f1cb00a2cdf371cdf7cf3244674926373 Mon Sep 17 00:00:00 2001 From: Julian Huang Date: Fri, 2 Oct 2026 14:11:25 -0400 Subject: [PATCH 06/22] ci: run aggregate functions regression From a626ce89892e86effccc2ec82ea6daa721918d01 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 3 Oct 2026 05:07:16 +0000 Subject: [PATCH 07/22] Defer folding in `uniqApacheHLL` and add missing includes Once a union existed, every inserted row created a sketch, merged it into the union and freed it. Keep inserting into the update sketch and fold it into the union only when the state is merged, read or serialized. Merging a state that holds both sketches now takes both without modifying it. Include ``, `` and `` for `std::byteswap`, `std::bad_alloc` and `std::exception`. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Jim6F2cKTTpxvY82Kq8a28 --- .../AggregateFunctionUniqApacheHLL.h | 22 +++++++++++++------ 1 file changed, 15 insertions(+), 7 deletions(-) diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index a6325e9f0ca8..e4e0c3d3978f 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -26,8 +26,11 @@ #include +#include #include +#include #include +#include #include namespace DB @@ -43,8 +46,9 @@ namespace ErrorCodes class HllSketchData { private: - std::unique_ptr sk_update; - std::unique_ptr sk_union; + /// Folding is deferred until the state is read, so it can happen in const methods. + mutable std::unique_ptr sk_update; + mutable std::unique_ptr sk_union; datasketches::hll_sketch * getSkUpdate(uint8_t lg_config_k, datasketches::target_hll_type tgt_type) { @@ -60,7 +64,8 @@ class HllSketchData return sk_union.get(); } - void foldUpdateIntoUnionIfNeeded() + /// Inserts keep going into `sk_update` even when `sk_union` exists, to avoid creating and merging a sketch per row. + void foldUpdateIntoUnionIfNeeded() const { if (sk_union && sk_update) { @@ -74,17 +79,17 @@ class HllSketchData void insert(T value, uint8_t lg_config_k, datasketches::target_hll_type tgt_type) { getSkUpdate(lg_config_k, tgt_type)->update(value); - foldUpdateIntoUnionIfNeeded(); } void insertData(const char * data, size_t size, uint8_t lg_config_k, datasketches::target_hll_type tgt_type) { getSkUpdate(lg_config_k, tgt_type)->update(static_cast(data), size); - foldUpdateIntoUnionIfNeeded(); } UInt64 size(datasketches::target_hll_type tgt_type) const { + foldUpdateIntoUnionIfNeeded(); + /// Rounding preserves exact cardinalities despite floating-point error. if (sk_union) return static_cast(std::llround(sk_union->get_result(tgt_type).get_estimate())); @@ -99,10 +104,11 @@ class HllSketchData foldUpdateIntoUnionIfNeeded(); + /// `rhs` may hold both sketches; take both without modifying it. + if (rhs.sk_union) + u->update(rhs.sk_union->get_result(tgt_type)); if (rhs.sk_update) u->update(*rhs.sk_update); - else if (rhs.sk_union) - u->update(rhs.sk_union->get_result(tgt_type)); } /// You can only call this for an empty object. @@ -135,6 +141,8 @@ class HllSketchData void write(WriteBuffer & out, uint8_t lg_config_k, datasketches::target_hll_type tgt_type) const { + foldUpdateIntoUnionIfNeeded(); + datasketches::hll_sketch::vector_bytes bytes; if (sk_update) bytes = sk_update->serialize_compact(); From 516fd39ff9923aa5c51d4f0fcf72a4dc6705c6f3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 3 Oct 2026 05:09:12 +0000 Subject: [PATCH 08/22] Make `uniqApacheHLL` reject non-integer `lg_k` and skip merging empty states A fractional or negative `lg_k` was silently truncated or wrapped; it now fails with `BAD_ARGUMENTS`. Merging an empty state no longer allocates a union. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Jim6F2cKTTpxvY82Kq8a28 --- src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp | 8 ++++++-- src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h | 3 +++ tests/queries/0_stateless/04327_uniq_apache_hll.sql | 2 ++ 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp index 21fb8363e155..ada8f1096a5a 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp @@ -1,7 +1,6 @@ #include #include #include -#include #if USE_DATASKETCHES @@ -28,7 +27,12 @@ static AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( if (!params.empty()) { - const UInt64 lg_k_param = applyVisitor(FieldVisitorConvertToNumber(), params[0]); + const Field::Types::Which lg_k_type = params[0].getType(); + if (lg_k_type != Field::Types::UInt64 && !(lg_k_type == Field::Types::Int64 && params[0].safeGet() >= 0)) + throw Exception(ErrorCodes::BAD_ARGUMENTS, + "Parameter lg_k for aggregate function {} must be an unsigned integer.", name); + + const UInt64 lg_k_param = lg_k_type == Field::Types::Int64 ? static_cast(params[0].safeGet()) : params[0].safeGet(); if (lg_k_param < 4 || lg_k_param > 21) throw Exception(ErrorCodes::ARGUMENT_OUT_OF_BOUND, "Parameter lg_k for aggregate function {} is out of range: [4, 21].", name); diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index e4e0c3d3978f..3b91512b25fe 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -100,6 +100,9 @@ class HllSketchData void merge(const HllSketchData & rhs, uint8_t lg_config_k, datasketches::target_hll_type tgt_type) { + if (!rhs.sk_update && !rhs.sk_union) + return; + datasketches::hll_union * u = getSkUnion(lg_config_k); foldUpdateIntoUnionIfNeeded(); diff --git a/tests/queries/0_stateless/04327_uniq_apache_hll.sql b/tests/queries/0_stateless/04327_uniq_apache_hll.sql index 4d2a4bf5d477..5a64f954eb88 100644 --- a/tests/queries/0_stateless/04327_uniq_apache_hll.sql +++ b/tests/queries/0_stateless/04327_uniq_apache_hll.sql @@ -35,6 +35,8 @@ SELECT toTypeName(uniqApacheHLLState(14, 'HLL_8')(number)) FROM numbers(1); SELECT 'parameter validation'; SELECT uniqApacheHLL(3)(number) FROM numbers(1); -- { serverError ARGUMENT_OUT_OF_BOUND } SELECT uniqApacheHLL(22)(number) FROM numbers(1); -- { serverError ARGUMENT_OUT_OF_BOUND } +SELECT uniqApacheHLL(12.5)(number) FROM numbers(1); -- { serverError BAD_ARGUMENTS } +SELECT uniqApacheHLL(-1)(number) FROM numbers(1); -- { serverError BAD_ARGUMENTS } SELECT uniqApacheHLL(12, 'HLL_9')(number) FROM numbers(1); -- { serverError BAD_ARGUMENTS } SELECT uniqApacheHLL(12, 'HLL_4', 1)(number) FROM numbers(1); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } From 5cc6aaa7c1185b23560f19ea6fe53e877b49adcd Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 3 Oct 2026 05:27:11 +0000 Subject: [PATCH 09/22] Remove stray blank line in `AggregateFunctionUniqApacheHLL` Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Jim6F2cKTTpxvY82Kq8a28 --- src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h | 1 - 1 file changed, 1 deletion(-) diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index 3b91512b25fe..250417cd754a 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -246,7 +246,6 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper { assert_cast(to).getData().push_back(this->data(place).size(target_type)); } - }; } From 2440dbeaa7f8fca632278df0b922f415f81b91ca Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Mon, 5 Oct 2026 12:02:22 +0900 Subject: [PATCH 10/22] Use `FieldVisitorConvertToNumber` for the `uniqApacheHLL` parameter `lg_k` Return to the conversion the other `uniq*` functions use. Non-finite, out of range and non-numeric values already fail with `CANNOT_CONVERT_TYPE`, and a negative integer wraps and is rejected by the `[4, 21]` range check with `ARGUMENT_OUT_OF_BOUND`. Only an in-range fraction such as `12.5` is truncated. Drop the `12.5` test and expect `ARGUMENT_OUT_OF_BOUND` for `-1`. Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 --- src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp | 8 ++------ tests/queries/0_stateless/04327_uniq_apache_hll.sql | 3 +-- 2 files changed, 3 insertions(+), 8 deletions(-) diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp index ada8f1096a5a..21fb8363e155 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp @@ -1,6 +1,7 @@ #include #include #include +#include #if USE_DATASKETCHES @@ -27,12 +28,7 @@ static AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( if (!params.empty()) { - const Field::Types::Which lg_k_type = params[0].getType(); - if (lg_k_type != Field::Types::UInt64 && !(lg_k_type == Field::Types::Int64 && params[0].safeGet() >= 0)) - throw Exception(ErrorCodes::BAD_ARGUMENTS, - "Parameter lg_k for aggregate function {} must be an unsigned integer.", name); - - const UInt64 lg_k_param = lg_k_type == Field::Types::Int64 ? static_cast(params[0].safeGet()) : params[0].safeGet(); + const UInt64 lg_k_param = applyVisitor(FieldVisitorConvertToNumber(), params[0]); if (lg_k_param < 4 || lg_k_param > 21) throw Exception(ErrorCodes::ARGUMENT_OUT_OF_BOUND, "Parameter lg_k for aggregate function {} is out of range: [4, 21].", name); diff --git a/tests/queries/0_stateless/04327_uniq_apache_hll.sql b/tests/queries/0_stateless/04327_uniq_apache_hll.sql index 5a64f954eb88..cf1111734501 100644 --- a/tests/queries/0_stateless/04327_uniq_apache_hll.sql +++ b/tests/queries/0_stateless/04327_uniq_apache_hll.sql @@ -35,8 +35,7 @@ SELECT toTypeName(uniqApacheHLLState(14, 'HLL_8')(number)) FROM numbers(1); SELECT 'parameter validation'; SELECT uniqApacheHLL(3)(number) FROM numbers(1); -- { serverError ARGUMENT_OUT_OF_BOUND } SELECT uniqApacheHLL(22)(number) FROM numbers(1); -- { serverError ARGUMENT_OUT_OF_BOUND } -SELECT uniqApacheHLL(12.5)(number) FROM numbers(1); -- { serverError BAD_ARGUMENTS } -SELECT uniqApacheHLL(-1)(number) FROM numbers(1); -- { serverError BAD_ARGUMENTS } +SELECT uniqApacheHLL(-1)(number) FROM numbers(1); -- { serverError ARGUMENT_OUT_OF_BOUND } SELECT uniqApacheHLL(12, 'HLL_9')(number) FROM numbers(1); -- { serverError BAD_ARGUMENTS } SELECT uniqApacheHLL(12, 'HLL_4', 1)(number) FROM numbers(1); -- { serverError NUMBER_OF_ARGUMENTS_DOESNT_MATCH } From 12c65dda8740251446e5839bd2c7d2edaeb66308 Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Mon, 5 Oct 2026 12:12:27 +0900 Subject: [PATCH 11/22] Include `config.h` explicitly where `USE_DATASKETCHES` is used for registration `registerAggregateFunctions.cpp` only got `USE_DATASKETCHES` transitively through `IColumn.h`. If that include chain changed, `#if USE_DATASKETCHES` would become false without any build error and `uniqApacheHLL` and `uniqTheta` would stop being registered. Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 --- src/AggregateFunctions/registerAggregateFunctions.cpp | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/AggregateFunctions/registerAggregateFunctions.cpp b/src/AggregateFunctions/registerAggregateFunctions.cpp index 670180d87a46..22291d140905 100644 --- a/src/AggregateFunctions/registerAggregateFunctions.cpp +++ b/src/AggregateFunctions/registerAggregateFunctions.cpp @@ -1,5 +1,7 @@ #include +#include "config.h" + #include #include From b403cd03a8a09071f698c23b4365508feac3bb39 Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Mon, 5 Oct 2026 12:12:27 +0900 Subject: [PATCH 12/22] Update `contrib/datasketches-cpp` to include the compact theta preamble bounds check Moves the submodule from `23bd9b07` to `4e917778`, the tip of `ClickHouse/datasketches-cpp` branch `ClickHouse/5.2.0`. The new commit checks the buffer size before `compact_theta_sketch_parser::parse` reads the `num_entries` and `theta` fields, which an AST fuzzer found to read out of bounds on truncated `uniqTheta` states: https://github.com/ClickHouse/ClickHouse/issues/119595 It does not touch the HLL code used by `uniqApacheHLL`. Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 --- contrib/datasketches-cpp | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contrib/datasketches-cpp b/contrib/datasketches-cpp index 23bd9b070ddf..4e91777839ba 160000 --- a/contrib/datasketches-cpp +++ b/contrib/datasketches-cpp @@ -1 +1 @@ -Subproject commit 23bd9b070ddf7f0dcf32c87d6752ce535374697b +Subproject commit 4e91777839ba01c43f0da7d049f15478301726d3 From 254c630a861a2f7735a4c5ebc2dc8f9ac4493630 Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Mon, 5 Oct 2026 18:29:20 +0900 Subject: [PATCH 13/22] Merge `rhs` sketches before folding the pending one in `uniqApacheHLL` In `merge`, take `rhs.sk_update` and `rhs.sk_union` first and fold this state's own pending `sk_update` into the union last. When `rhs` has a lower resolution than the declared `lg_k`, the union is then created at that resolution directly, instead of being built at `lg_k` and downsampled by `hll_union::update`. The merged result is the same either way. Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 --- .../AggregateFunctionUniqApacheHLL.h | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index 250417cd754a..cd5f23dfd7a8 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -105,13 +105,15 @@ class HllSketchData datasketches::hll_union * u = getSkUnion(lg_config_k); - foldUpdateIntoUnionIfNeeded(); - /// `rhs` may hold both sketches; take both without modifying it. - if (rhs.sk_union) - u->update(rhs.sk_union->get_result(tgt_type)); if (rhs.sk_update) u->update(*rhs.sk_update); + if (rhs.sk_union) + u->update(rhs.sk_union->get_result(tgt_type)); + + /// Fold our own pending sketch last: a lower-resolution `rhs` sketch is then taken first, + /// instead of a union at the declared `lg_k` being built and downsampled afterwards. + foldUpdateIntoUnionIfNeeded(); } /// You can only call this for an empty object. From 31e9e7ae07b276e1465e1f6376a32bc4b6b05ade Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Mon, 5 Oct 2026 18:50:41 +0900 Subject: [PATCH 14/22] Add a test pinning how `uniqApacheHLL` hashes every argument type on border values `04327_uniq_apache_hll` pins byte-exact states only for `UInt64`, `UUID`, `IPv6` and `DateTime64(3)`. Everything else is checked by cardinality alone, so a change in how `Float64`, `IPv4`, `Date` or `String` values are hashed would break interoperability with other DataSketches implementations without failing any test. The new test feeds each supported type its border values, such as the minimum and maximum of every integer width, values at and above the sign bit of the unsigned types, `-0`, `NaN`, infinities and denormals, `FixedString` padding, negative `Date32` and `DateTime64`, and compares the serialized state with the bytes written by the Apache DataSketches C++ library. It also pins the empty sketch for several `lg_k` and storage types, the list, set and dense modes, and that types share the hash of their underlying value. Integers are widened by value to 64 bits, as Java `update(long)` and Python `update(int)` do. The C++ overloads for narrow unsigned types sign-extend instead, so a `UInt32` of `4294967295` hashes differently from `Int32` `-1`; the test pins that. The expected output was generated by the C++ library and cross-checked against two fixtures already in `04327_uniq_apache_hll`. It has not been run against a server. Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 --- ...137_uniq_apache_hll_hash_borders.reference | 65 ++++++++++++++ .../05137_uniq_apache_hll_hash_borders.sql | 84 +++++++++++++++++++ 2 files changed, 149 insertions(+) create mode 100644 tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.reference create mode 100644 tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.sql diff --git a/tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.reference b/tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.reference new file mode 100644 index 000000000000..97f02e1a3b1a --- /dev/null +++ b/tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.reference @@ -0,0 +1,65 @@ +integers +Int8 1C0201070C030805001FABB112D964E70BCBD7C2042BF2FB062F537613 +UInt8 180201070C03080400CBD7C2042F53761320A9190E34F7530A +Int16 180201070C0308040026276604D964E70BCBD7C204A82F2604 +UInt16 180201070C03080400CBD7C204A82F26040066F004710BB407 +Int32 180201070C0308040047E5A708D964E70BCBD7C2043F53B008 +UInt32 180201070C03080400CBD7C2043F53B008D8458307B7943A08 +Int64 180201070C030804007519C20CD964E70BCBD7C204FFBAB40D +UInt64 180201070C03080400CBD7C204FFBAB40D7519C20CD964E70B +floats +Float64 240201070C03080700CBD7C20422822A14EB31BD18289EDB04B1A3460D2BF2FB068467FB11 +Float32 2C0301070C050800010800000022822A14289EDB04CBD7C204ECC8DD04AFD68204B1A3460DD894520CEB31BD18 +BFloat16 180201070C0308040022822A148C904A04731C001BB859430B +strings +String 200201070C030806002F823B04218D870411EF66057D7DF6064F461A042E01AA06 +FixedString(5) 140201070C03080300D8B13F07D82DDD0773B92C06 +dates and times +Date 180201070C03080400CBD7C204A82F26040066F004710BB407 +Date32 1C0201070C03080500015F2707D964E70BCBD7C2042BF2FB06272CF70A +DateTime 180201070C03080400CBD7C2043F53B008D8458307B7943A08 +DateTime64(0) 1C0201070C030805007BA2BA05D964E70BCBD7C2042BF2FB068425C90A +DateTime64(3) 1C0201070C0308050040A80204D964E70BCBD7C2042BF2FB06FB95071A +DateTime64(6) 1C0201070C03080500B89F8606D964E70BCBD7C2042BF2FB06F79DA104 +network and identifiers +IPv4 1C0201070C03080500CBD7C2043F53B008D8458307F5682308B7943A08 +IPv6 180201070C0308040050C94D05854BD10A18216E096C95BA0E +UUID 140201070C0308030050C94D05C08DEA136C95BA0E +Enum8 140201070C030803001FABB112CBD7C2042F537613 +Enum16 140201070C0308030026276604CBD7C204A82F2604 +sketch modes and parameters +empty, lg_k 12, HLL_4 080201070C030C0000 +empty, lg_k 4, HLL_4 0802010704030C0000 +empty, lg_k 21, HLL_8 0802010715030C0008 +empty, lg_k 12, HLL_6 080201070C030C0004 +list, 8 values 2C0301070C050800010800000081BC5D06862FF90DCBD7C204C1E9170575816607FC2D420A7B65E6082BF2FB06 +set, 9 values 300301070C050800010900000081BC5D06862FF90DCBD7C204C1E91705D216730775816607FC2D420A7B65E6082BF2FB06 +hll, lg_k 4, HLL_4, 100 values 300A0107040008020215EB1DC787F15440000000000000FB3F000000000000000003000000000000000251214121031025 +hll, lg_k 4, HLL_6, 100 values 350A0107040008000615EB1DC787F15440000000000000FB3F0000000000000000000000000000000084301C033118035108C2701000 +hll, lg_k 4, HLL_8, 100 values 380A0107040008000A15EB1DC787F15440000000000000FB3F0000000000000000000000000000000004020307030403060304050202030704 +merging empty states gives an empty sketch +080201070C030C0000 +integers are widened by value +Int8 -1 = Int64 -1 1 +Int32 min = Int64 min 1 +UInt8 200 = UInt64 200 1 +UInt8 200 = Int16 200 1 +UInt32 max != Int32 -1 1 +UInt64 max = Int64 -1 1 +floating point values are canonicalized +-0 = 0 1 +Float32 1.5 = Float64 1.5 1 +Float32 0.1 widened 1 +Float32 0.1 != Float64 0.1 1 +inf != -inf 1 +NaN payloads 1 +types share the hash of their underlying value +Enum8 max = Int8 127 1 +Date32 -1 = Int32 -1 1 +Date 65535 = UInt16 65535 1 +DateTime 2^31 = UInt32 2^31 1 +IPv4 max = UInt32 max 1 +DateTime64(3) -1 = Int64 -1 1 +String abc = FixedString(3) abc 1 +String abc != FixedString(5) abc 1 +LowCardinality(String) = String 1 diff --git a/tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.sql b/tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.sql new file mode 100644 index 000000000000..19bce1c67c0a --- /dev/null +++ b/tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.sql @@ -0,0 +1,84 @@ +-- Tags: no-fasttest +-- ^ DataSketches is not built in fast-test builds. + +-- Pins how every supported argument type is hashed, on the border values of its range. +-- Each expected state was produced by the Apache DataSketches C++ library: integers are +-- widened by value to 64 bits (as Java `update(long)` and Python `update(int)` do), +-- floats are widened to `double` with `-0.0` and `NaN` canonicalized, and strings and +-- fixed-size types are hashed as raw bytes. A state fed with up to eight values stays +-- a coupon list, so the bytes depend only on the values and their order. + +SELECT 'integers'; +SELECT 'Int8', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['-128', '-1', '0', '1', '127'], 'Array(Int8)')) AS x) SETTINGS max_threads = 1; +SELECT 'UInt8', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['0', '127', '128', '255'], 'Array(UInt8)')) AS x) SETTINGS max_threads = 1; +SELECT 'Int16', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['-32768', '-1', '0', '32767'], 'Array(Int16)')) AS x) SETTINGS max_threads = 1; +SELECT 'UInt16', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['0', '32767', '32768', '65535'], 'Array(UInt16)')) AS x) SETTINGS max_threads = 1; +SELECT 'Int32', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['-2147483648', '-1', '0', '2147483647'], 'Array(Int32)')) AS x) SETTINGS max_threads = 1; +SELECT 'UInt32', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['0', '2147483647', '2147483648', '4294967295'], 'Array(UInt32)')) AS x) SETTINGS max_threads = 1; +SELECT 'Int64', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['-9223372036854775808', '-1', '0', '9223372036854775807'], 'Array(Int64)')) AS x) SETTINGS max_threads = 1; +SELECT 'UInt64', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['0', '9223372036854775807', '9223372036854775808', '18446744073709551615'], 'Array(UInt64)')) AS x) SETTINGS max_threads = 1; + +SELECT 'floats'; +SELECT 'Float64', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['0', '-0', '1.5', 'nan', 'inf', '-inf', '5e-324', '1.7976931348623157e308'], 'Array(Float64)')) AS x) SETTINGS max_threads = 1; +SELECT 'Float32', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['0', '-0', '0.1', '1.5', 'nan', 'inf', '-inf', '3.4028235e38', '1e-45'], 'Array(Float32)')) AS x) SETTINGS max_threads = 1; +SELECT 'BFloat16', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin([toBFloat16(1.5), toBFloat16(-2.0), toBFloat16(0.5), toBFloat16(256.0)]) AS x) SETTINGS max_threads = 1; + +SELECT 'strings'; +SELECT 'String', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(['a', 'abc', 'привет', 'a\0b', ' ', '\xFF']) AS x) SETTINGS max_threads = 1; +SELECT 'FixedString(5)', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['a', 'abc', 'abcde'], 'Array(FixedString(5))')) AS x) SETTINGS max_threads = 1; + +SELECT 'dates and times'; +SELECT 'Date', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(arrayMap(v -> toDate(v), [0, 32767, 32768, 65535])) AS x) SETTINGS max_threads = 1; +SELECT 'Date32', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(arrayMap(v -> toDate32(v), [-25567, -1, 0, 1, 120529])) AS x) SETTINGS max_threads = 1; +SELECT 'DateTime', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(arrayMap(v -> toDateTime(v, 'UTC'), [0, 2147483647, 2147483648, 4294967295])) AS x) SETTINGS max_threads = 1; +SELECT 'DateTime64(0)', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(arrayMap(v -> toDateTime64(v, 0, 'UTC'), CAST(['-2208988799', '-1', '0', '1', '1577836800'], 'Array(Int64)'))) AS x) SETTINGS max_threads = 1; +SELECT 'DateTime64(3)', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(arrayMap(v -> fromUnixTimestamp64Milli(v, 'UTC'), CAST(['-2208988799999', '-1', '0', '1', '1577836800123'], 'Array(Int64)'))) AS x) SETTINGS max_threads = 1; +SELECT 'DateTime64(6)', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(arrayMap(v -> fromUnixTimestamp64Micro(v, 'UTC'), CAST(['-2208988799999999', '-1', '0', '1', '1577836800123456'], 'Array(Int64)'))) AS x) SETTINGS max_threads = 1; + +SELECT 'network and identifiers'; +SELECT 'IPv4', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['0.0.0.0', '127.255.255.255', '128.0.0.0', '192.168.1.1', '255.255.255.255'], 'Array(IPv4)')) AS x) SETTINGS max_threads = 1; +SELECT 'IPv6', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['::', '::1', '2001:db8::1', 'ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff'], 'Array(IPv6)')) AS x) SETTINGS max_threads = 1; +SELECT 'UUID', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['00000000-0000-0000-0000-000000000000', '01234567-89ab-cdef-0123-456789abcdef', 'ffffffff-ffff-ffff-ffff-ffffffffffff'], 'Array(UUID)')) AS x) SETTINGS max_threads = 1; +SELECT 'Enum8', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['min', 'zero', 'max'], 'Array(Enum8(\'min\' = -128, \'zero\' = 0, \'max\' = 127))')) AS x) SETTINGS max_threads = 1; +SELECT 'Enum16', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['min', 'zero', 'max'], 'Array(Enum16(\'min\' = -32768, \'zero\' = 0, \'max\' = 32767))')) AS x) SETTINGS max_threads = 1; + +SELECT 'sketch modes and parameters'; +SELECT 'empty, lg_k 12, HLL_4', hex(toString(uniqApacheHLLState(number))) FROM numbers(0) SETTINGS max_threads = 1; +SELECT 'empty, lg_k 4, HLL_4', hex(toString(uniqApacheHLLState(4, 'HLL_4')(number))) FROM numbers(0) SETTINGS max_threads = 1; +SELECT 'empty, lg_k 21, HLL_8', hex(toString(uniqApacheHLLState(21, 'HLL_8')(number))) FROM numbers(0) SETTINGS max_threads = 1; +SELECT 'empty, lg_k 12, HLL_6', hex(toString(uniqApacheHLLState(12, 'HLL_6')(number))) FROM numbers(0) SETTINGS max_threads = 1; +SELECT 'list, 8 values', hex(toString(uniqApacheHLLState(number))) FROM numbers(8) SETTINGS max_threads = 1; +SELECT 'set, 9 values', hex(toString(uniqApacheHLLState(number))) FROM numbers(9) SETTINGS max_threads = 1; +SELECT 'hll, lg_k 4, HLL_4, 100 values', hex(toString(uniqApacheHLLState(4, 'HLL_4')(number))) FROM numbers(100) SETTINGS max_threads = 1; +SELECT 'hll, lg_k 4, HLL_6, 100 values', hex(toString(uniqApacheHLLState(4, 'HLL_6')(number))) FROM numbers(100) SETTINGS max_threads = 1; +SELECT 'hll, lg_k 4, HLL_8, 100 values', hex(toString(uniqApacheHLLState(4, 'HLL_8')(number))) FROM numbers(100) SETTINGS max_threads = 1; + +SELECT 'merging empty states gives an empty sketch'; +SELECT hex(toString(uniqApacheHLLMergeState(s))) FROM (SELECT uniqApacheHLLState(number) AS s FROM numbers(0)); + +SELECT 'integers are widened by value'; +SELECT 'Int8 -1 = Int64 -1', hex(toString(uniqApacheHLLState(toInt8(-1)))) = hex(toString(uniqApacheHLLState(toInt64(-1)))); +SELECT 'Int32 min = Int64 min', hex(toString(uniqApacheHLLState(toInt32(-2147483648)))) = hex(toString(uniqApacheHLLState(toInt64(-2147483648)))); +SELECT 'UInt8 200 = UInt64 200', hex(toString(uniqApacheHLLState(toUInt8(200)))) = hex(toString(uniqApacheHLLState(toUInt64(200)))); +SELECT 'UInt8 200 = Int16 200', hex(toString(uniqApacheHLLState(toUInt8(200)))) = hex(toString(uniqApacheHLLState(toInt16(200)))); +SELECT 'UInt32 max != Int32 -1', hex(toString(uniqApacheHLLState(toUInt32(4294967295)))) != hex(toString(uniqApacheHLLState(toInt32(-1)))); +SELECT 'UInt64 max = Int64 -1', hex(toString(uniqApacheHLLState(toUInt64(18446744073709551615)))) = hex(toString(uniqApacheHLLState(toInt64(-1)))); + +SELECT 'floating point values are canonicalized'; +SELECT '-0 = 0', hex(toString(uniqApacheHLLState(toFloat64('-0')))) = hex(toString(uniqApacheHLLState(toFloat64('0')))); +SELECT 'Float32 1.5 = Float64 1.5', hex(toString(uniqApacheHLLState(toFloat32(1.5)))) = hex(toString(uniqApacheHLLState(toFloat64(1.5)))); +SELECT 'Float32 0.1 widened', hex(toString(uniqApacheHLLState(toFloat32(0.1)))) = hex(toString(uniqApacheHLLState(toFloat64(toFloat32(0.1))))); +SELECT 'Float32 0.1 != Float64 0.1', hex(toString(uniqApacheHLLState(toFloat32(0.1)))) != hex(toString(uniqApacheHLLState(toFloat64(0.1)))); +SELECT 'inf != -inf', hex(toString(uniqApacheHLLState(toFloat64('inf')))) != hex(toString(uniqApacheHLLState(toFloat64('-inf')))); +SELECT 'NaN payloads', uniqApacheHLL(x) FROM (SELECT arrayJoin([reinterpretAsFloat64(unhex('000000000000F87F')), reinterpretAsFloat64(unhex('010000000000F87F')), reinterpretAsFloat64(unhex('000000000000F8FF'))]) AS x); + +SELECT 'types share the hash of their underlying value'; +SELECT 'Enum8 max = Int8 127', hex(toString(uniqApacheHLLState(CAST('max', 'Enum8(\'max\' = 127)')))) = hex(toString(uniqApacheHLLState(toInt8(127)))); +SELECT 'Date32 -1 = Int32 -1', hex(toString(uniqApacheHLLState(toDate32(toInt32(-1))))) = hex(toString(uniqApacheHLLState(toInt32(-1)))); +SELECT 'Date 65535 = UInt16 65535', hex(toString(uniqApacheHLLState(toDate(65535)))) = hex(toString(uniqApacheHLLState(toUInt16(65535)))); +SELECT 'DateTime 2^31 = UInt32 2^31', hex(toString(uniqApacheHLLState(toDateTime(toUInt32(2147483648), 'UTC')))) = hex(toString(uniqApacheHLLState(toUInt32(2147483648)))); +SELECT 'IPv4 max = UInt32 max', hex(toString(uniqApacheHLLState(toIPv4('255.255.255.255')))) = hex(toString(uniqApacheHLLState(toUInt32(4294967295)))); +SELECT 'DateTime64(3) -1 = Int64 -1', hex(toString(uniqApacheHLLState(fromUnixTimestamp64Milli(toInt64(-1), 'UTC')))) = hex(toString(uniqApacheHLLState(toInt64(-1)))); +SELECT 'String abc = FixedString(3) abc', hex(toString(uniqApacheHLLState('abc'))) = hex(toString(uniqApacheHLLState(toFixedString('abc', 3)))); +SELECT 'String abc != FixedString(5) abc', hex(toString(uniqApacheHLLState('abc'))) != hex(toString(uniqApacheHLLState(toFixedString('abc', 5)))); +SELECT 'LowCardinality(String) = String', hex(toString(uniqApacheHLLState(toLowCardinality('abc')))) = hex(toString(uniqApacheHLLState('abc'))); From 8d494605d2f45d2c65ee59a84a4391b3715ed976 Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Mon, 5 Oct 2026 18:55:06 +0900 Subject: [PATCH 15/22] Ignore empty strings in `uniqApacheHLL` and pin its `NULL` handling The Apache DataSketches implementations skip empty strings: Java `update(String)` and `update(byte[])`, the C++ `update(std::string)` (and so Python `update(str)`), and Spark's `hll_sketch_agg`, which also ignores `NULL`. `uniqApacheHLL` counted `''` as a value, so a sketch of data that contains empty strings differed from one built outside ClickHouse by one value. Skip them in `add`, which also makes the state of empty strings only the empty sketch. `NULL` values were already ignored by the `Null` combinator. Because the function has `returns_default_when_only_null`, that combinator writes its flag byte for every state, even one built from `NULL` values only, so the state of a `Nullable` argument is `0x01` followed by the ordinary sketch. Document it and pin it, together with import of an external sketch into a `Nullable` state type and merging of `Nullable` states, in a new test. Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 --- .../AggregateFunctionUniqApacheHLL.cpp | 3 ++ .../AggregateFunctionUniqApacheHLL.h | 6 +++ ...8_uniq_apache_hll_null_and_empty.reference | 18 ++++++++ .../05138_uniq_apache_hll_null_and_empty.sql | 41 +++++++++++++++++++ 4 files changed, 68 insertions(+) create mode 100644 tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.reference create mode 100644 tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.sql diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp index 21fb8363e155..72e55c032748 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp @@ -107,6 +107,9 @@ For interoperability, integers of at most 64 bits are hashed as 8-byte integers, `Date`, `Date32`, `DateTime` and `DateTime64` use their underlying integer values. External producers must use the same unit; for example, `DateTime64(3)` uses epoch milliseconds. Unsupported types and multiple arguments are rejected. Use `uniq`, `uniqCombined` or `uniqHLL12` for those inputs. +`NULL` values and empty strings are ignored, as in the Java, Python and C++ implementations and in Spark; unlike `uniq`, an empty string is not counted as a value. +For a `Nullable` argument the state is the sketch preceded by a single `0x01` byte, which is how the `Null` combinator marks its state, so an external consumer has to skip that byte. + Merging can switch from the HIP estimator to the less accurate composite estimator, so results can depend on partitioning across threads, parts and shards. Merging a lower-resolution sketch permanently lowers the result's resolution, regardless of the declared `lg_k`. )"; diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index cd5f23dfd7a8..4d49ea0e7a86 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -191,6 +191,12 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper if constexpr (std::is_same_v) { const auto value = columns[0]->getDataAt(row_num); + + /// Other DataSketches implementations ignore empty strings, and the sketch of a mixed + /// input must be the same as theirs. + if (value.size() == 0) + return; + data.insertData(value.data(), value.size(), lg_config_k, target_type); } else diff --git a/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.reference b/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.reference new file mode 100644 index 000000000000..1c5efbbc4df3 --- /dev/null +++ b/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.reference @@ -0,0 +1,18 @@ +empty strings +0 +080201070C030C0000 +0C0201070C030801002F823B04 +1 +1 +NULL values +2 +1 +0 +0 +the state of a Nullable argument is the sketch behind a 0x01 byte +011C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06 +01080201070C030C0000 +01080201070C030C0000 +01080201070C030C0000 +5 +3 diff --git a/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.sql b/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.sql new file mode 100644 index 000000000000..da38d5687a13 --- /dev/null +++ b/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.sql @@ -0,0 +1,41 @@ +-- Tags: no-fasttest +-- ^ DataSketches is not built in fast-test builds. + +-- `NULL` values and empty strings are ignored by the Apache DataSketches implementations in Java, Python +-- and C++, and by Spark's `hll_sketch_agg`, so a sketch built here must ignore them too. +-- Unlike `uniq`, an empty string is therefore not counted as a value. + +SELECT 'empty strings'; +SELECT uniqApacheHLL(x) FROM (SELECT arrayJoin(['', '', '']) AS x); +-- Nothing but empty strings leaves the empty sketch. +SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(['', '']) AS x); +-- The state of 'a' written by the DataSketches C++ library. +SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(['', 'a', '']) AS x) SETTINGS max_threads = 1; +SELECT + (SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(['a', '', 'b']) AS x)) + = (SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(['a', 'b']) AS x)) +SETTINGS max_threads = 1; +-- A fixed-size string of zero bytes is not empty. +SELECT uniqApacheHLL(toFixedString('', 3)); + +SELECT 'NULL values'; +SELECT uniqApacheHLL(x) FROM (SELECT arrayJoin([NULL, 1, NULL, 2, 2]) AS x); +SELECT uniqApacheHLL(x) FROM (SELECT arrayJoin(['', NULL, 'a']) AS x); +SELECT uniqApacheHLL(CAST(NULL, 'Nullable(UInt64)')) FROM numbers(3); +SELECT uniqApacheHLL(toNullable(number)) FROM numbers(0); + +SELECT 'the state of a Nullable argument is the sketch behind a 0x01 byte'; +-- The `Null` combinator always writes its flag, even when every row was `NULL`. +SELECT hex(toString(uniqApacheHLLState(toNullable(number)))) FROM numbers(5) SETTINGS max_threads = 1; +SELECT hex(toString(uniqApacheHLLState(toNullable(number)))) FROM numbers(0); +SELECT hex(toString(uniqApacheHLLState(CAST(NULL, 'Nullable(UInt64)')))) FROM numbers(3); +SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(['', NULL]) AS x); +-- Importing an external sketch into a Nullable state type needs the same byte. +SELECT finalizeAggregation(CAST(unhex('011C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, Nullable(UInt64))')); +-- Merging Nullable states, including one built from `NULL` values only. +SELECT uniqApacheHLLMerge(s) FROM +( + SELECT uniqApacheHLLState(toNullable(number)) AS s FROM numbers(3) + UNION ALL + SELECT uniqApacheHLLState(CAST(NULL, 'Nullable(UInt64)')) AS s FROM numbers(2) +); From 30ad260b002913e3ee0787aa7acecac035ca74b3 Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Mon, 5 Oct 2026 19:01:56 +0900 Subject: [PATCH 16/22] Keep the `uniqApacheHLL` state of a `Nullable` argument a bare DataSketches sketch `uniqApacheHLL` has `returns_default_when_only_null`, so the generic `Null` combinator wrote its flag byte (always `1`) before the sketch of every `Nullable` argument. The state of `Nullable` columns, the common case, was then not a DataSketches sketch, and an external consumer would have had to skip the byte, or an importer to add it. Give the function its own null adapter, `AggregateFunctionNullUnary`, as `sumCount` and `intervalLengthSum` do. `NULL` rows are still skipped, and the state is the same bytes as for a non-`Nullable` argument, the empty sketch when there were no values. The `If` combinator builds its own null adapter without asking the nested function, so `uniqApacheHLLStateIf` over a `Nullable` argument still writes the flag byte, as the other functions with this property do. Change the test to expect bare sketches, to compare `Nullable` and plain states, and to import an external sketch into a `Nullable` state type without a prefix byte. Not built or run. Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 --- .../AggregateFunctionUniqApacheHLL.cpp | 2 +- .../AggregateFunctionUniqApacheHLL.h | 12 ++++++++++++ ...138_uniq_apache_hll_null_and_empty.reference | 12 +++++++----- .../05138_uniq_apache_hll_null_and_empty.sql | 17 +++++++++++++---- 4 files changed, 33 insertions(+), 10 deletions(-) diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp index 72e55c032748..5cae6911a255 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp @@ -108,7 +108,7 @@ For interoperability, integers of at most 64 bits are hashed as 8-byte integers, Unsupported types and multiple arguments are rejected. Use `uniq`, `uniqCombined` or `uniqHLL12` for those inputs. `NULL` values and empty strings are ignored, as in the Java, Python and C++ implementations and in Spark; unlike `uniq`, an empty string is not counted as a value. -For a `Nullable` argument the state is the sketch preceded by a single `0x01` byte, which is how the `Null` combinator marks its state, so an external consumer has to skip that byte. +A `Nullable` argument gives the same state as a non-`Nullable` one, a bare DataSketches sketch. Merging can switch from the HIP estimator to the less accurate composite estimator, so results can depend on partitioning across threads, parts and shards. Merging a lower-resolution sketch permanently lowers the result's resolution, regardless of the declared `lg_k`. diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index 4d49ea0e7a86..473115641ee3 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -4,6 +4,7 @@ #if USE_DATASKETCHES +#include #include #include #include @@ -184,6 +185,17 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper bool allocatesMemoryInArena() const override { return false; } + /// `NULL` rows are skipped as usual, but the state is left as a bare DataSketches sketch. The default adapter + /// would put a flag byte of the `Null` combinator before it, and an external implementation could not read that. + AggregateFunctionPtr getOwnNullAdapter( + const AggregateFunctionPtr & nested_function, + const DataTypes & arguments, + const Array & params, + const AggregateFunctionProperties & /*properties*/) const override + { + return std::make_shared>(nested_function, arguments, params); + } + void add(AggregateDataPtr __restrict place, const IColumn ** columns, size_t row_num, Arena *) const override { auto & data = this->data(place); diff --git a/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.reference b/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.reference index 1c5efbbc4df3..108f9422f1ee 100644 --- a/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.reference +++ b/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.reference @@ -9,10 +9,12 @@ NULL values 1 0 0 -the state of a Nullable argument is the sketch behind a 0x01 byte -011C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06 -01080201070C030C0000 -01080201070C030C0000 -01080201070C030C0000 +a Nullable argument gives the same state as a plain one +1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06 +1 +080201070C030C0000 +080201070C030C0000 +080201070C030C0000 +1 5 3 diff --git a/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.sql b/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.sql index da38d5687a13..94282a2c346b 100644 --- a/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.sql +++ b/tests/queries/0_stateless/05138_uniq_apache_hll_null_and_empty.sql @@ -24,14 +24,23 @@ SELECT uniqApacheHLL(x) FROM (SELECT arrayJoin(['', NULL, 'a']) AS x); SELECT uniqApacheHLL(CAST(NULL, 'Nullable(UInt64)')) FROM numbers(3); SELECT uniqApacheHLL(toNullable(number)) FROM numbers(0); -SELECT 'the state of a Nullable argument is the sketch behind a 0x01 byte'; --- The `Null` combinator always writes its flag, even when every row was `NULL`. +SELECT 'a Nullable argument gives the same state as a plain one'; +-- No flag byte of the `Null` combinator: the state is the bare DataSketches sketch, also when every row was `NULL`. SELECT hex(toString(uniqApacheHLLState(toNullable(number)))) FROM numbers(5) SETTINGS max_threads = 1; +SELECT + (SELECT hex(toString(uniqApacheHLLState(toNullable(number)))) FROM numbers(5)) + = (SELECT hex(toString(uniqApacheHLLState(number))) FROM numbers(5)) +SETTINGS max_threads = 1; SELECT hex(toString(uniqApacheHLLState(toNullable(number)))) FROM numbers(0); SELECT hex(toString(uniqApacheHLLState(CAST(NULL, 'Nullable(UInt64)')))) FROM numbers(3); SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(['', NULL]) AS x); --- Importing an external sketch into a Nullable state type needs the same byte. -SELECT finalizeAggregation(CAST(unhex('011C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, Nullable(UInt64))')); +-- NULL rows between values do not change the state. +SELECT + (SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin([NULL, 1, NULL, 2]) AS x)) + = (SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin([toNullable(1), 2]) AS x)) +SETTINGS max_threads = 1; +-- An external sketch can be imported into a Nullable state type as it is. +SELECT finalizeAggregation(CAST(unhex('1C0201070C03080500CBD7C2042BF2FB06862FF90D7581660781BC5D06'), 'AggregateFunction(uniqApacheHLL, Nullable(UInt64))')); -- Merging Nullable states, including one built from `NULL` values only. SELECT uniqApacheHLLMerge(s) FROM ( From a30cd5761e840b4ede7f0ff4b4db462470f4b42c Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Mon, 5 Oct 2026 19:13:40 +0900 Subject: [PATCH 17/22] Give the `uniqApacheHLL` state of a `Nullable` argument the type of the plain state The previous commit made the bytes of that state a bare DataSketches sketch, but its type still named the `Nullable` argument, so `AggregateFunction(uniqApacheHLL, Nullable(T))` and `AggregateFunction(uniqApacheHLL, T)` could not be mixed: no `UNION ALL`, no `INSERT ... SELECT` into the other column type, and `haveSameStateRepresentation` was false. Replace the generic `AggregateFunctionNullUnary` with a `Nullable` variant of the function itself, selected through `getOwnNullAdapter`. It skips `NULL` rows through the null map, overrides `getStateType` and `getNormalizedStateType` to return those of the plain function, whose state layout is identical, and ignores nullability of the arguments in `haveSameStateRepresentationImpl`, so a column declared with a `Nullable` argument stays interchangeable. `LowCardinality` needs no handling: the factory and the aggregator strip it before the function sees types or columns. The generic adapter is left alone; it is shared by `sumCount` and `intervalLengthSum`, whose state types would change. New test `05139` pins the state type, mixing with `UNION ALL`, `CAST` and `INSERT` in both directions, and `LowCardinality(Nullable(String))` arguments. Not built or run. Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 --- .../AggregateFunctionUniqApacheHLL.cpp | 4 +- .../AggregateFunctionUniqApacheHLL.h | 85 ++++++++++++++++--- ...q_apache_hll_nullable_state_type.reference | 17 ++++ ...39_uniq_apache_hll_nullable_state_type.sql | 46 ++++++++++ 4 files changed, 138 insertions(+), 14 deletions(-) create mode 100644 tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference create mode 100644 tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.sql diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp index 5cae6911a255..ee513ae78194 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp @@ -65,7 +65,7 @@ static AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( /// Exclude wide integers: DataSketches has no portable representation for them. if (!which.isInt128() && !which.isInt256() && !which.isUInt128() && !which.isUInt256()) { - AggregateFunctionPtr res(createWithNumericType( + AggregateFunctionPtr res(createWithNumericType( argument_type, lg_config_k, target_type, argument_types, params)); if (res) return res; @@ -108,7 +108,7 @@ For interoperability, integers of at most 64 bits are hashed as 8-byte integers, Unsupported types and multiple arguments are rejected. Use `uniq`, `uniqCombined` or `uniqHLL12` for those inputs. `NULL` values and empty strings are ignored, as in the Java, Python and C++ implementations and in Spark; unlike `uniq`, an empty string is not counted as a value. -A `Nullable` argument gives the same state as a non-`Nullable` one, a bare DataSketches sketch. +A `Nullable` argument gives the same state as a non-`Nullable` one, a bare DataSketches sketch, and the same state type, `AggregateFunction(uniqApacheHLL, T)`. Merging can switch from the HIP estimator to the less accurate composite estimator, so results can depend on partitioning across threads, parts and shards. Merging a lower-resolution sketch permanently lowers the result's resolution, regardless of the declared `lg_k`. diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index 473115641ee3..848e3eace249 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -4,10 +4,10 @@ #if USE_DATASKETCHES -#include #include #include #include +#include #include #include #include @@ -18,6 +18,7 @@ #include #include #include +#include #include #include #include @@ -27,6 +28,7 @@ #include +#include #include #include #include @@ -161,10 +163,15 @@ class HllSketchData }; -template -class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper> +/** `nullable` is the variant that takes a `Nullable(T)` argument, skips its `NULL` rows itself and keeps the state + * of the plain variant: a bare DataSketches sketch. The generic `Null` combinator would put its flag byte before + * the sketch and give the state a `Nullable` argument type, so a state of a `Nullable` column could be neither read + * by another implementation nor used where the state of a plain column is expected. + */ +template +class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper> { - using Base = IAggregateFunctionDataHelper>; + using Base = IAggregateFunctionDataHelper>; uint8_t lg_config_k; datasketches::target_hll_type target_type; @@ -185,24 +192,52 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper bool allocatesMemoryInArena() const override { return false; } - /// `NULL` rows are skipped as usual, but the state is left as a bare DataSketches sketch. The default adapter - /// would put a flag byte of the `Null` combinator before it, and an external implementation could not read that. AggregateFunctionPtr getOwnNullAdapter( - const AggregateFunctionPtr & nested_function, + const AggregateFunctionPtr & /*nested_function*/, const DataTypes & arguments, const Array & params, const AggregateFunctionProperties & /*properties*/) const override { - return std::make_shared>(nested_function, arguments, params); + if constexpr (nullable) + return nullptr; + else + return std::make_shared>(lg_config_k, target_type, arguments, params); + } + + /// The bytes of the state do not depend on the nullability of the argument, so neither does its type. + DataTypePtr getStateType() const override + { + if constexpr (nullable) + return getPlainFunction()->getStateType(); + else + return IAggregateFunction::getStateType(); + } + + DataTypePtr getNormalizedStateType() const override + { + if constexpr (nullable) + return getPlainFunction()->getNormalizedStateType(); + else + return IAggregateFunction::getNormalizedStateType(); } void add(AggregateDataPtr __restrict place, const IColumn ** columns, size_t row_num, Arena *) const override { + const IColumn * column = columns[0]; + + if constexpr (nullable) + { + const auto & nullable_column = assert_cast(*column); + if (nullable_column.isNullAt(row_num)) + return; + column = &nullable_column.getNestedColumn(); + } + auto & data = this->data(place); if constexpr (std::is_same_v) { - const auto value = columns[0]->getDataAt(row_num); + const auto value = column->getDataAt(row_num); /// Other DataSketches implementations ignore empty strings, and the sketch of a mixed /// input must be the same as theirs. @@ -213,7 +248,7 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper } else { - const auto & value = assert_cast &>(*columns[0]).getData()[row_num]; + const auto & value = assert_cast &>(*column).getData()[row_num]; if constexpr (std::is_same_v) { @@ -241,10 +276,19 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper } } - /// Serialized sketches carry their configuration, so parameters need not match. + /// Serialized sketches carry their configuration, so parameters need not match, and nullability of the argument does not matter. bool haveSameStateRepresentationImpl(const IAggregateFunction & rhs) const override { - return getName() == rhs.getName() && this->haveEqualArgumentTypes(rhs); + const auto & lhs_types = this->getArgumentTypes(); + const auto & rhs_types = rhs.getArgumentTypes(); + + return getName() == rhs.getName() + && std::equal( + lhs_types.begin(), + lhs_types.end(), + rhs_types.begin(), + rhs_types.end(), + [](const auto & lhs, const auto & rhs_type) { return removeNullable(lhs)->equals(*removeNullable(rhs_type)); }); } void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena *) const override @@ -266,8 +310,25 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper { assert_cast(to).getData().push_back(this->data(place).size(target_type)); } + +private: + /// The same function for the argument types without `Nullable`; its state layout is identical. + AggregateFunctionPtr getPlainFunction() const + { + DataTypes plain_argument_types; + plain_argument_types.reserve(this->getArgumentTypes().size()); + for (const auto & type : this->getArgumentTypes()) + plain_argument_types.push_back(removeNullable(type)); + + return std::make_shared>( + lg_config_k, target_type, plain_argument_types, this->getParameters()); + } }; +/// The factory needs a template with a single type parameter. +template +using AggregateFunctionUniqApacheHLLPlain = AggregateFunctionUniqApacheHLL; + } #endif diff --git a/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference b/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference new file mode 100644 index 000000000000..1a3b073311ce --- /dev/null +++ b/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference @@ -0,0 +1,17 @@ +the state type does not depend on the nullability of the argument +AggregateFunction(uniqApacheHLL, UInt64) +AggregateFunction(uniqApacheHLL, UInt64) +AggregateFunction(uniqApacheHLL(14, \'HLL_8\'), String) +LowCardinality is stripped from the arguments of aggregate functions +AggregateFunction(uniqApacheHLL, String) +2 +1 +states of Nullable and plain arguments mix +6 +a declared Nullable state type is interchangeable with the plain one +AggregateFunction(uniqApacheHLL, Nullable(UInt64)) +4 +4 +tables +6 +6 diff --git a/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.sql b/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.sql new file mode 100644 index 000000000000..05e75ee0f485 --- /dev/null +++ b/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.sql @@ -0,0 +1,46 @@ +-- Tags: no-fasttest +-- ^ DataSketches is not built in fast-test builds. + +-- The state of a `Nullable` argument is the same bare DataSketches sketch as the state of a plain one, +-- so it also has the same type, and the two can be mixed without a `CAST`. + +SELECT 'the state type does not depend on the nullability of the argument'; +SELECT toTypeName(uniqApacheHLLState(toNullable(number))) FROM numbers(1); +SELECT toTypeName(uniqApacheHLLState(number)) FROM numbers(1); +SELECT toTypeName(uniqApacheHLLState(14, 'HLL_8')(toNullable(toString(number)))) FROM numbers(1); + +SELECT 'LowCardinality is stripped from the arguments of aggregate functions'; +SELECT toTypeName(uniqApacheHLLState(toLowCardinality(toNullable(toString(number))))) FROM numbers(1); +SELECT uniqApacheHLL(x) FROM (SELECT arrayJoin(CAST(['a', NULL, 'b', 'a', ''], 'Array(LowCardinality(Nullable(String)))')) AS x); +SELECT + (SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['a', NULL, 'b'], 'Array(LowCardinality(Nullable(String)))')) AS x)) + = (SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(['a', 'b']) AS x)) +SETTINGS max_threads = 1; + +SELECT 'states of Nullable and plain arguments mix'; +SELECT uniqApacheHLLMerge(s) FROM +( + SELECT uniqApacheHLLState(toNullable(number)) AS s FROM numbers(3) + UNION ALL + SELECT uniqApacheHLLState(number + 3) AS s FROM numbers(3) +); + +SELECT 'a declared Nullable state type is interchangeable with the plain one'; +SELECT toTypeName(CAST(uniqApacheHLLState(number), 'AggregateFunction(uniqApacheHLL, Nullable(UInt64))')) FROM numbers(1); +SELECT finalizeAggregation(CAST(uniqApacheHLLState(number), 'AggregateFunction(uniqApacheHLL, Nullable(UInt64))')) FROM numbers(4); +SELECT finalizeAggregation(CAST(CAST(uniqApacheHLLState(number), 'AggregateFunction(uniqApacheHLL, Nullable(UInt64))'), 'AggregateFunction(uniqApacheHLL, UInt64)')) FROM numbers(4); + +SELECT 'tables'; +DROP TABLE IF EXISTS hll_plain_states; +CREATE TABLE hll_plain_states (k UInt8, s AggregateFunction(uniqApacheHLL, UInt64)) ENGINE = AggregatingMergeTree ORDER BY k; +INSERT INTO hll_plain_states SELECT 0 AS k, uniqApacheHLLState(toNullable(number)) FROM numbers(3); +INSERT INTO hll_plain_states SELECT 0 AS k, uniqApacheHLLState(number + 3) FROM numbers(3); +SELECT uniqApacheHLLMerge(s) FROM hll_plain_states; +DROP TABLE hll_plain_states; + +DROP TABLE IF EXISTS hll_nullable_states; +CREATE TABLE hll_nullable_states (k UInt8, s AggregateFunction(uniqApacheHLL, Nullable(UInt64))) ENGINE = AggregatingMergeTree ORDER BY k; +INSERT INTO hll_nullable_states SELECT 0 AS k, uniqApacheHLLState(toNullable(number)) FROM numbers(3); +INSERT INTO hll_nullable_states SELECT 0 AS k, uniqApacheHLLState(number + 3) FROM numbers(3); +SELECT uniqApacheHLLMerge(s) FROM hll_nullable_states; +DROP TABLE hll_nullable_states; From 2033e7bf0e12e9980bd2d066044f7e26e34fbce5 Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Mon, 5 Oct 2026 19:30:14 +0900 Subject: [PATCH 18/22] Let a function opt out of the flag and type of the `Null` and `If` wrappers The wrapper that the `Null` combinator puts around a function for `Nullable` arguments writes a flag byte before the state and gives it a type that names the `Nullable` argument. For `uniqApacheHLL` neither is needed, since the state of no values is the empty sketch, and both stop a state built from a `Nullable` column from being read by another DataSketches implementation or mixed with the state of a plain column. Add `IAggregateFunction::stateIsIndependentOfNullability`, `false` by default, next to `getOwnNullAdapter` and `getArgumentsThatCanBeOnlyNull`. When it returns `true` the `Null` combinator and the `If` combinator over `Nullable` arguments use the existing no-flag wrappers, `AggregateFunctionNullUnary` and the variadic and `If` equivalents, and such a wrapper reports the state type of the nested function. `AggregateFunctionIf` forwards the question to its nested function. The wrappers ask the nested function when the type is requested, so nothing is added to their constructors, and every function that does not override the method behaves as before. This replaces the `Nullable` variant of `uniqApacheHLL` from the previous commit, which covered only the `Null` combinator, with the generic mechanism, which also covers `uniqApacheHLLStateIf` and `uniqApacheHLLMergeIf` over `Nullable` arguments. `uniqApacheHLL` keeps `haveSameStateRepresentationImpl` ignoring nullability, so a column declared with a `Nullable` argument stays interchangeable. The test now also covers the state type and bytes of `StateIf` and `MergeIf` with a `Nullable` argument and a `Nullable` condition. Not built or run. Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 --- .../AggregateFunctionUniqApacheHLL.cpp | 2 +- .../AggregateFunctionUniqApacheHLL.h | 73 ++----------------- .../Combinators/AggregateFunctionIf.cpp | 27 +++++++ .../Combinators/AggregateFunctionIf.h | 5 ++ .../Combinators/AggregateFunctionNull.cpp | 14 ++++ .../Combinators/AggregateFunctionNull.h | 27 +++++++ src/AggregateFunctions/IAggregateFunction.h | 13 ++++ ...q_apache_hll_nullable_state_type.reference | 9 +++ ...39_uniq_apache_hll_nullable_state_type.sql | 17 +++++ 9 files changed, 120 insertions(+), 67 deletions(-) diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp index ee513ae78194..95144e70c22a 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.cpp @@ -65,7 +65,7 @@ static AggregateFunctionPtr createAggregateFunctionUniqApacheHLL( /// Exclude wide integers: DataSketches has no portable representation for them. if (!which.isInt128() && !which.isInt256() && !which.isUInt128() && !which.isUInt256()) { - AggregateFunctionPtr res(createWithNumericType( + AggregateFunctionPtr res(createWithNumericType( argument_type, lg_config_k, target_type, argument_types, params)); if (res) return res; diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index 848e3eace249..dbb2f6a8d09f 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -7,7 +7,6 @@ #include #include #include -#include #include #include #include @@ -163,15 +162,10 @@ class HllSketchData }; -/** `nullable` is the variant that takes a `Nullable(T)` argument, skips its `NULL` rows itself and keeps the state - * of the plain variant: a bare DataSketches sketch. The generic `Null` combinator would put its flag byte before - * the sketch and give the state a `Nullable` argument type, so a state of a `Nullable` column could be neither read - * by another implementation nor used where the state of a plain column is expected. - */ -template -class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper> +template +class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper> { - using Base = IAggregateFunctionDataHelper>; + using Base = IAggregateFunctionDataHelper>; uint8_t lg_config_k; datasketches::target_hll_type target_type; @@ -192,52 +186,16 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper bool allocatesMemoryInArena() const override { return false; } - AggregateFunctionPtr getOwnNullAdapter( - const AggregateFunctionPtr & /*nested_function*/, - const DataTypes & arguments, - const Array & params, - const AggregateFunctionProperties & /*properties*/) const override - { - if constexpr (nullable) - return nullptr; - else - return std::make_shared>(lg_config_k, target_type, arguments, params); - } - - /// The bytes of the state do not depend on the nullability of the argument, so neither does its type. - DataTypePtr getStateType() const override - { - if constexpr (nullable) - return getPlainFunction()->getStateType(); - else - return IAggregateFunction::getStateType(); - } - - DataTypePtr getNormalizedStateType() const override - { - if constexpr (nullable) - return getPlainFunction()->getNormalizedStateType(); - else - return IAggregateFunction::getNormalizedStateType(); - } + /// A `Nullable` argument only has its `NULL` rows skipped: the state is the same bare sketch, with the same type. + bool stateIsIndependentOfNullability() const override { return true; } void add(AggregateDataPtr __restrict place, const IColumn ** columns, size_t row_num, Arena *) const override { - const IColumn * column = columns[0]; - - if constexpr (nullable) - { - const auto & nullable_column = assert_cast(*column); - if (nullable_column.isNullAt(row_num)) - return; - column = &nullable_column.getNestedColumn(); - } - auto & data = this->data(place); if constexpr (std::is_same_v) { - const auto value = column->getDataAt(row_num); + const auto value = columns[0]->getDataAt(row_num); /// Other DataSketches implementations ignore empty strings, and the sketch of a mixed /// input must be the same as theirs. @@ -248,7 +206,7 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper } else { - const auto & value = assert_cast &>(*column).getData()[row_num]; + const auto & value = assert_cast &>(*columns[0]).getData()[row_num]; if constexpr (std::is_same_v) { @@ -310,25 +268,8 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper { assert_cast(to).getData().push_back(this->data(place).size(target_type)); } - -private: - /// The same function for the argument types without `Nullable`; its state layout is identical. - AggregateFunctionPtr getPlainFunction() const - { - DataTypes plain_argument_types; - plain_argument_types.reserve(this->getArgumentTypes().size()); - for (const auto & type : this->getArgumentTypes()) - plain_argument_types.push_back(removeNullable(type)); - - return std::make_shared>( - lg_config_k, target_type, plain_argument_types, this->getParameters()); - } }; -/// The factory needs a template with a single type parameter. -template -using AggregateFunctionUniqApacheHLLPlain = AggregateFunctionUniqApacheHLL; - } #endif diff --git a/src/AggregateFunctions/Combinators/AggregateFunctionIf.cpp b/src/AggregateFunctions/Combinators/AggregateFunctionIf.cpp index 1818e9499699..2d630965076b 100644 --- a/src/AggregateFunctions/Combinators/AggregateFunctionIf.cpp +++ b/src/AggregateFunctions/Combinators/AggregateFunctionIf.cpp @@ -113,6 +113,19 @@ class AggregateFunctionIfNullUnary final filter_is_only_null = arguments[num_arguments - 1]->onlyNull(); } +protected: + /// The nested function is the one without `If`, so the state type of the `If` function without Nullable arguments is built here. + DataTypePtr getNestedStateType() const override + { + DataTypes plain_argument_types; + plain_argument_types.reserve(this->getArgumentTypes().size()); + for (const auto & type : this->getArgumentTypes()) + plain_argument_types.push_back(type->onlyNull() ? type : removeNullable(type)); + + return std::make_shared(this->nested_function, plain_argument_types, this->getParameters())->getStateType(); + } + +public: void add(AggregateDataPtr __restrict place, const IColumn ** columns, size_t row_num, Arena * arena) const override { if (filter_is_only_null) @@ -488,6 +501,20 @@ AggregateFunctionPtr AggregateFunctionIf::getOwnNullAdapter( bool need_to_serialize_flag = return_type_is_nullable || properties.returns_default_when_only_null; + /// The function only skips NULL rows: no flag, and the state of the function without Nullable arguments, with its type. + if (nested_func->stateIsIndependentOfNullability()) + { + if (!properties.returns_default_when_only_null) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "Aggregate function {} cannot have the state independent of nullability unless it returns a default value for NULL values only", + nested_function->getName()); + + if (arguments.size() <= 2 && arguments.front()->isNullable()) + return std::make_shared>(nested_function->getName(), nested_func, arguments, params); + + return std::make_shared>(nested_function, arguments, params); + } + if (arguments.size() <= 2 && arguments.front()->isNullable()) { if (return_type_is_nullable) diff --git a/src/AggregateFunctions/Combinators/AggregateFunctionIf.h b/src/AggregateFunctions/Combinators/AggregateFunctionIf.h index 4161b7937148..a325e10c355f 100644 --- a/src/AggregateFunctions/Combinators/AggregateFunctionIf.h +++ b/src/AggregateFunctions/Combinators/AggregateFunctionIf.h @@ -64,6 +64,11 @@ class AggregateFunctionIf final : public IAggregateFunctionHelpergetNormalizedStateType(); } + bool stateIsIndependentOfNullability() const override + { + return nested_func->stateIsIndependentOfNullability(); + } + bool canMergeStateFromDifferentVariant(const IAggregateFunction & rhs) const override { if (!this->haveSameDefinition(rhs)) diff --git a/src/AggregateFunctions/Combinators/AggregateFunctionNull.cpp b/src/AggregateFunctions/Combinators/AggregateFunctionNull.cpp index fabc256c90da..f9cc1570c2bd 100644 --- a/src/AggregateFunctions/Combinators/AggregateFunctionNull.cpp +++ b/src/AggregateFunctions/Combinators/AggregateFunctionNull.cpp @@ -13,6 +13,7 @@ namespace DB namespace ErrorCodes { extern const int ILLEGAL_TYPE_OF_ARGUMENT; + extern const int LOGICAL_ERROR; } namespace @@ -126,6 +127,19 @@ class AggregateFunctionCombinatorNull final : public IAggregateFunctionCombinato return new_function; } + /// The function only skips NULL rows: no flag, and the state of the nested function, with its type. + if (nested_function->stateIsIndependentOfNullability()) + { + if (!properties.returns_default_when_only_null) + throw Exception(ErrorCodes::LOGICAL_ERROR, + "Aggregate function {} cannot have the state independent of nullability unless it returns a default value for NULL values only", + nested_function->getName()); + + if (arguments.size() == 1) + return std::make_shared>(nested_function, arguments, params); + return std::make_shared>(nested_function, arguments, params); + } + bool return_type_is_nullable = !properties.returns_default_when_only_null && nested_function->getResultType()->canBeInsideNullable(); bool serialize_flag = return_type_is_nullable || properties.returns_default_when_only_null; diff --git a/src/AggregateFunctions/Combinators/AggregateFunctionNull.h b/src/AggregateFunctions/Combinators/AggregateFunctionNull.h index c18c68098a8e..f5ca11d7e93e 100644 --- a/src/AggregateFunctions/Combinators/AggregateFunctionNull.h +++ b/src/AggregateFunctions/Combinators/AggregateFunctionNull.h @@ -50,6 +50,19 @@ class AggregateFunctionNullBase : public IAggregateFunctionHelper const AggregateFunctionPtr nested_function; const size_t prefix_size; + /// The type of the state of the nested function, which this wrapper has when the nested function + /// says its state does not depend on nullability. See `IAggregateFunction::stateIsIndependentOfNullability`. + virtual DataTypePtr getNestedStateType() const { return nested_function->getStateType(); } + + /// Only a wrapper without the flag has the memory layout of the nested state, which the nested function can claim for its own type. + bool hasStateOfNestedFunction() const + { + if constexpr (!result_is_nullable && !serialize_flag) + return nested_function->stateIsIndependentOfNullability(); + else + return false; + } + /** In addition to data for nested aggregate function, we keep a flag * indicating - was there at least one non-NULL value accumulated. * In case of no not-NULL values, the function will return NULL. @@ -107,6 +120,20 @@ class AggregateFunctionNullBase : public IAggregateFunctionHelper return nested_function->getName(); } + DataTypePtr getStateType() const override + { + if (hasStateOfNestedFunction()) + return getNestedStateType(); + return IAggregateFunction::getStateType(); + } + + DataTypePtr getNormalizedStateType() const override + { + if (hasStateOfNestedFunction()) + return nested_function->getNormalizedStateType(); + return IAggregateFunction::getNormalizedStateType(); + } + bool canMergeStateFromDifferentVariant(const IAggregateFunction & rhs) const override { if (!this->haveSameDefinition(rhs)) diff --git a/src/AggregateFunctions/IAggregateFunction.h b/src/AggregateFunctions/IAggregateFunction.h index b666df47c664..6f0cbacd7158 100644 --- a/src/AggregateFunctions/IAggregateFunction.h +++ b/src/AggregateFunctions/IAggregateFunction.h @@ -417,6 +417,19 @@ class IAggregateFunction : public std::enable_shared_from_this getArgumentsThatCanBeOnlyNull() const { return {}; } + /** When the function is wrapped with the Null combinator (or the If combinator over Nullable arguments), the state of the + * wrapper has a flag byte and a type that names the Nullable argument. So the state of a Nullable column is not the state of a + * plain one: another implementation cannot read it, and the two cannot be mixed. + * + * A function that only skips NULL values, and whose state of no values is the empty state (example: uniqApacheHLL), + * does not need that. If it returns true here, the wrapper only skips NULL rows, and its state has the same bytes and the + * same type as the state of this function for a non-Nullable argument. + * + * It requires returns_default_when_only_null: the result of a function that must return NULL when only NULL values + * were seen needs the flag to tell it from the state of no rows. + */ + virtual bool stateIsIndependentOfNullability() const { return false; } + /** Return the nested function if this is an Aggregate Function Combinator. * Otherwise return nullptr. */ diff --git a/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference b/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference index 1a3b073311ce..01d72c226fcd 100644 --- a/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference +++ b/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference @@ -8,6 +8,15 @@ AggregateFunction(uniqApacheHLL, String) 1 states of Nullable and plain arguments mix 6 +the If combinator over Nullable arguments keeps the same state and type +AggregateFunction(uniqApacheHLLIf, UInt64, UInt8) +AggregateFunction(uniqApacheHLLIf, UInt64, UInt8) +AggregateFunction(uniqApacheHLLIf, UInt64, UInt8) +1 +1 +5 +5 +5 a declared Nullable state type is interchangeable with the plain one AggregateFunction(uniqApacheHLL, Nullable(UInt64)) 4 diff --git a/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.sql b/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.sql index 05e75ee0f485..c75d1cf0153f 100644 --- a/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.sql +++ b/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.sql @@ -25,6 +25,23 @@ SELECT uniqApacheHLLMerge(s) FROM SELECT uniqApacheHLLState(number + 3) AS s FROM numbers(3) ); +SELECT 'the If combinator over Nullable arguments keeps the same state and type'; +SELECT toTypeName(uniqApacheHLLStateIf(toNullable(number), number % 2 = 0)) FROM numbers(1); +SELECT toTypeName(uniqApacheHLLStateIf(number, number % 2 = 0)) FROM numbers(1); +SELECT toTypeName(uniqApacheHLLStateIf(number, toNullable(number % 2 = 0))) FROM numbers(1); +-- The state is the bare sketch of the values 0, 2 and 4, without a flag byte. +SELECT + (SELECT hex(toString(uniqApacheHLLStateIf(toNullable(number), number % 2 = 0))) FROM numbers(5)) + = (SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin([0, 2, 4]) AS x)) +SETTINGS max_threads = 1; +SELECT + (SELECT hex(toString(uniqApacheHLLStateIf(number, toNullable(number % 2 = 0)))) FROM numbers(5)) + = (SELECT hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin([0, 2, 4]) AS x)) +SETTINGS max_threads = 1; +SELECT uniqApacheHLLIf(toNullable(number), number % 2 = 0) FROM numbers(10); +SELECT uniqApacheHLLMergeIf(s, c) FROM (SELECT uniqApacheHLLState(number) AS s, number % 2 = 0 AS c FROM numbers(10) GROUP BY number); +SELECT uniqApacheHLLMergeIf(s, toNullable(c)) FROM (SELECT uniqApacheHLLState(number) AS s, number % 2 = 0 AS c FROM numbers(10) GROUP BY number); + SELECT 'a declared Nullable state type is interchangeable with the plain one'; SELECT toTypeName(CAST(uniqApacheHLLState(number), 'AggregateFunction(uniqApacheHLL, Nullable(UInt64))')) FROM numbers(1); SELECT finalizeAggregation(CAST(uniqApacheHLLState(number), 'AggregateFunction(uniqApacheHLL, Nullable(UInt64))')) FROM numbers(4); From 71ed5a7a7e5fc51987cefc810944ffb1a2f001c1 Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Mon, 5 Oct 2026 23:00:36 +0900 Subject: [PATCH 19/22] Fix the expected output of two new `uniqApacheHLL` tests after the first CI run The first CI run built and ran the tests for the first time. Only these two failed. `05139_uniq_apache_hll_nullable_state_type`: `uniqApacheHLLStateIf` has the type `AggregateFunction(uniqApacheHLL, UInt64)`, not `AggregateFunction(uniqApacheHLLIf, UInt64, UInt8)` as the reference guessed. It is the same for a `Nullable` argument, a plain one and a `Nullable` condition, which is what the test is meant to show. `05137_uniq_apache_hll_hash_borders`: the `Float64` and `Float32` states took the largest values and the denormals as text, and the server parses text with its fast path, which is not correctly rounded: `5e-324` was lost and `3.4028235e38` was not the largest `Float32`. Build the values from their exact bit patterns instead, with `reinterpretAsFloat64` and `reinterpretAsFloat32`. The expected states are unchanged, since they were computed from those bit patterns. CI report: https://altinity-build-artifacts.s3.amazonaws.com/json.html?PR=2398&sha=e99077e33925c82fab2ed4da8ae16ad2519b9882&name_0=PR Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 --- .../0_stateless/05137_uniq_apache_hll_hash_borders.sql | 4 ++-- .../05139_uniq_apache_hll_nullable_state_type.reference | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.sql b/tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.sql index 19bce1c67c0a..589ec1b16a7d 100644 --- a/tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.sql +++ b/tests/queries/0_stateless/05137_uniq_apache_hll_hash_borders.sql @@ -19,8 +19,8 @@ SELECT 'Int64', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST SELECT 'UInt64', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['0', '9223372036854775807', '9223372036854775808', '18446744073709551615'], 'Array(UInt64)')) AS x) SETTINGS max_threads = 1; SELECT 'floats'; -SELECT 'Float64', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['0', '-0', '1.5', 'nan', 'inf', '-inf', '5e-324', '1.7976931348623157e308'], 'Array(Float64)')) AS x) SETTINGS max_threads = 1; -SELECT 'Float32', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin(CAST(['0', '-0', '0.1', '1.5', 'nan', 'inf', '-inf', '3.4028235e38', '1e-45'], 'Array(Float32)')) AS x) SETTINGS max_threads = 1; +SELECT 'Float64', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin([reinterpretAsFloat64(unhex('0000000000000000')), reinterpretAsFloat64(unhex('0000000000000080')), reinterpretAsFloat64(unhex('000000000000F83F')), reinterpretAsFloat64(unhex('000000000000F87F')), reinterpretAsFloat64(unhex('000000000000F07F')), reinterpretAsFloat64(unhex('000000000000F0FF')), reinterpretAsFloat64(unhex('0100000000000000')), reinterpretAsFloat64(unhex('FFFFFFFFFFFFEF7F'))]) AS x) SETTINGS max_threads = 1; +SELECT 'Float32', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin([reinterpretAsFloat32(unhex('00000000')), reinterpretAsFloat32(unhex('00000080')), reinterpretAsFloat32(unhex('CDCCCC3D')), reinterpretAsFloat32(unhex('0000C03F')), reinterpretAsFloat32(unhex('0000C07F')), reinterpretAsFloat32(unhex('0000807F')), reinterpretAsFloat32(unhex('000080FF')), reinterpretAsFloat32(unhex('FFFF7F7F')), reinterpretAsFloat32(unhex('01000000'))]) AS x) SETTINGS max_threads = 1; SELECT 'BFloat16', hex(toString(uniqApacheHLLState(x))) FROM (SELECT arrayJoin([toBFloat16(1.5), toBFloat16(-2.0), toBFloat16(0.5), toBFloat16(256.0)]) AS x) SETTINGS max_threads = 1; SELECT 'strings'; diff --git a/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference b/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference index 01d72c226fcd..52504bd4b578 100644 --- a/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference +++ b/tests/queries/0_stateless/05139_uniq_apache_hll_nullable_state_type.reference @@ -9,9 +9,9 @@ AggregateFunction(uniqApacheHLL, String) states of Nullable and plain arguments mix 6 the If combinator over Nullable arguments keeps the same state and type -AggregateFunction(uniqApacheHLLIf, UInt64, UInt8) -AggregateFunction(uniqApacheHLLIf, UInt64, UInt8) -AggregateFunction(uniqApacheHLLIf, UInt64, UInt8) +AggregateFunction(uniqApacheHLL, UInt64) +AggregateFunction(uniqApacheHLL, UInt64) +AggregateFunction(uniqApacheHLL, UInt64) 1 1 5 From 1bfd0e169897f59f3b9df64661ed88484b3d34f3 Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Tue, 6 Oct 2026 00:11:35 +0900 Subject: [PATCH 20/22] DCO remediation commit for UnamedRus I, UnamedRus , hereby add my Signed-off-by to this commit: 71ed5a7a7e5fc51987cefc810944ffb1a2f001c1 I, UnamedRus , hereby add my Signed-off-by to this commit: 2033e7bf0e12e9980bd2d066044f7e26e34fbce5 I, UnamedRus , hereby add my Signed-off-by to this commit: a30cd5761e840b4ede7f0ff4b4db462470f4b42c I, UnamedRus , hereby add my Signed-off-by to this commit: 30ad260b002913e3ee0787aa7acecac035ca74b3 I, UnamedRus , hereby add my Signed-off-by to this commit: 8d494605d2f45d2c65ee59a84a4391b3715ed976 I, UnamedRus , hereby add my Signed-off-by to this commit: 31e9e7ae07b276e1465e1f6376a32bc4b6b05ade I, UnamedRus , hereby add my Signed-off-by to this commit: 254c630a861a2f7735a4c5ebc2dc8f9ac4493630 I, UnamedRus , hereby add my Signed-off-by to this commit: b403cd03a8a09071f698c23b4365508feac3bb39 I, UnamedRus , hereby add my Signed-off-by to this commit: 12c65dda8740251446e5839bd2c7d2edaeb66308 I, UnamedRus , hereby add my Signed-off-by to this commit: 2440dbeaa7f8fca632278df0b922f415f81b91ca Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: UnamedRus From 970be3112788e80740e3958bfab79fa556d5f81b Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Tue, 6 Oct 2026 11:50:26 +0900 Subject: [PATCH 21/22] Stop converting `uniqApacheHLL` unions to the declared type to read or merge them The union inside a state always holds an `HLL_8` sketch; the declared type is applied only when the state is serialized. `size` converted a copy to the declared type just to read its estimate, and `merge` converted the other union to the declared type before feeding it into this one, which accepts a sketch of any type. Read the estimate from the union directly, and merge with its `HLL_8` result. In a standalone micro-benchmark against the vendored DataSketches at `-O2` (not ClickHouse), for `lg_k = 12`: - estimate through a `HLL_4` copy: 72 us; directly on the union: 9 ns - merge through a `HLL_4` result: 88 us; through an `HLL_8` result: 16 us For 60 combinations of `lg_k`, declared type and cardinality, the estimates and the serialized bytes after merging were identical either way. `write` still converts to the declared type, since the serialized state must have it. With the declared type unused, drop that parameter from `size` and `merge`. Tests `04327`, `05137`, `05138` and `05139` pass on a local Debug build. Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: UnamedRus --- .../AggregateFunctionUniqApacheHLL.h | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index dbb2f6a8d09f..38501a098848 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -88,19 +88,19 @@ class HllSketchData getSkUpdate(lg_config_k, tgt_type)->update(static_cast(data), size); } - UInt64 size(datasketches::target_hll_type tgt_type) const + UInt64 size() const { foldUpdateIntoUnionIfNeeded(); /// Rounding preserves exact cardinalities despite floating-point error. if (sk_union) - return static_cast(std::llround(sk_union->get_result(tgt_type).get_estimate())); + return static_cast(std::llround(sk_union->get_estimate())); if (sk_update) return static_cast(std::llround(sk_update->get_estimate())); return 0; } - void merge(const HllSketchData & rhs, uint8_t lg_config_k, datasketches::target_hll_type tgt_type) + void merge(const HllSketchData & rhs, uint8_t lg_config_k) { if (!rhs.sk_update && !rhs.sk_union) return; @@ -110,8 +110,10 @@ class HllSketchData /// `rhs` may hold both sketches; take both without modifying it. if (rhs.sk_update) u->update(*rhs.sk_update); + + /// The union accepts a sketch of any type, and `HLL_8` is its own type, so there is no re-encoding. if (rhs.sk_union) - u->update(rhs.sk_union->get_result(tgt_type)); + u->update(rhs.sk_union->get_result(datasketches::HLL_8)); /// Fold our own pending sketch last: a lower-resolution `rhs` sketch is then taken first, /// instead of a union at the declared `lg_k` being built and downsampled afterwards. @@ -251,7 +253,7 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper void merge(AggregateDataPtr __restrict place, ConstAggregateDataPtr rhs, Arena *) const override { - this->data(place).merge(this->data(rhs), lg_config_k, target_type); + this->data(place).merge(this->data(rhs), lg_config_k); } void serialize(ConstAggregateDataPtr __restrict place, WriteBuffer & buf, std::optional /* version */) const override @@ -266,7 +268,7 @@ class AggregateFunctionUniqApacheHLL final : public IAggregateFunctionDataHelper void insertResultInto(AggregateDataPtr __restrict place, IColumn & to, Arena *) const override { - assert_cast(to).getData().push_back(this->data(place).size(target_type)); + assert_cast(to).getData().push_back(this->data(place).size()); } }; From 5f06020dfabc767a600969a126efcf4fdebe135d Mon Sep 17 00:00:00 2001 From: UnamedRus Date: Tue, 6 Oct 2026 11:52:39 +0900 Subject: [PATCH 22/22] Drop the speculative ordering comment in `uniqApacheHLL` merge The comment claimed that folding the pending sketch last avoids building a union at the declared `lg_k` and downsampling it. That was read from the library source, never measured, and nothing depends on it; the code is correct in either order. Keep the comment about `rhs` holding both sketches, which explains why there are two separate branches. Comment only. Related: https://github.com/Altinity/ClickHouse/pull/2398 Co-Authored-By: Claude Sonnet 5.5 Signed-off-by: UnamedRus --- src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h | 2 -- 1 file changed, 2 deletions(-) diff --git a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h index 38501a098848..76a0cd41de68 100644 --- a/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h +++ b/src/AggregateFunctions/AggregateFunctionUniqApacheHLL.h @@ -115,8 +115,6 @@ class HllSketchData if (rhs.sk_union) u->update(rhs.sk_union->get_result(datasketches::HLL_8)); - /// Fold our own pending sketch last: a lower-resolution `rhs` sketch is then taken first, - /// instead of a union at the declared `lg_k` being built and downsampled afterwards. foldUpdateIntoUnionIfNeeded(); }