Issues with permissions using DataLakeCatalog with type=glue

Scenario
I have two different tables in two different databases in two different buckets:

I have two databases: clickhouse-regression-database-1 and clickhouse-regression-database-2
I created table_a under clickhouse-regression-database-1, stored in

s3://glue-bucket-db-a/table_one

I created table_b under clickhouse-regression-database-2, stored in

s3://glue-bucket-db-b/second_table

Let's consider different permission configurations of IAM user (this will be Policy_1):

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "GlueAccessToSpecificTable",
			"Effect": "Allow",
			"Action": [
				"glue:GetTable",
				"glue:GetTables",
				"glue:GetDatabase",
				"glue:GetDatabases",
				"glue:GetTableVersion",
				"glue:GetTableVersions"
			],
			"Resource": "*"
		},
		{
			"Sid": "S3AccessToIcebergTableData",
			"Effect": "Allow",
			"Action": [
				"s3:ListBucket",
				"s3:GetObject",
				"s3:GetBucketLocation"
			],
			"Resource": [
				"arn:aws:s3:::glue-bucket-db-a",
				"arn:aws:s3:::glue-bucket-db-a/table_one/*"
			]
		}
	]
}

What this policy means:

The IAM user has full read access to Glue metadata for all databases and tables.
The user has S3 access only to objects inside s3://glue-bucket-db-a/table_one/.
The user does not have S3 access to glue-bucket-db-b (used by table_b).

From clickhouse (25.6.6.29):

SET allow_experimental_database_glue_catalog = 1;
CREATE DATABASE glue
ENGINE = DataLakeCatalog
SETTINGS catalog_type = 'glue', region = 'eu-central-1', aws_access_key_id = '***', aws_secret_access_key = 'Z***';

SHOW TABLES FROM glue;

Both tables are visible from the Glue catalog, even though the IAM user doesn't have S3 access to one of them.


   ┌─name─────────────────────────────────────┐
1. │ clickhouse-regression-database-1.table_a │
2. │ clickhouse-regression-database-2.table_b │
   └──────────────────────────────────────────┘

Let's try to select from those tables:

SELECT *
FROM glue.`clickhouse-regression-database-1.table_a`

   ┌─string─┬─int─┬─binary─┐
1. │ aa     │   1 │ ᴺᵁᴸᴸ   │
2. │ ᴺᵁᴸᴸ   │   1 │ ᴺᵁᴸᴸ   │
3. │ aaaaa  │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘

3 rows in set. Elapsed: 0.463 sec.

SELECT *
FROM glue.`clickhouse-regression-database-2.table_b`

Elapsed: 0.351 sec. 

Received exception from server (version 25.6.6):
Code: 499. DB::Exception: Received from localhost:9000. DB::Exception: User: arn:aws:iam::***:user/agiliazova is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::glue-bucket-db-b/second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json" because no identity-based policy allows the s3:GetObject action: while reading key: second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json, from bucket: glue-bucket-db-b. (S3_ERROR)

SHOW CREATE TABLE works for both tables:

SHOW CREATE TABLE glue.`clickhouse-regression-database-1.table_a`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-1.table_a`↴│
   │↳(                                                           ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `binary` Nullable(String)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-a/table_one')          │
   └──────────────────────────────────────────────────────────────┘

SHOW CREATE TABLE glue.`clickhouse-regression-database-2.table_b`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-2.table_b`↴│
   │↳(                                                           ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `float` Nullable(Float32)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-b/second_table')       │
   └──────────────────────────────────────────────────────────────┘

Now I update policy, since I want to be able to see only table_a from clickhouse (this will be Policy_2):

{
	"Version": "2012-10-17",
	"Statement": [
		{
			"Sid": "GlueAccessToSpecificTable",
			"Effect": "Allow",
			"Action": [
				"glue:GetTable",
				"glue:GetTables",
				"glue:GetDatabase",
				"glue:GetDatabases",
				"glue:GetTableVersion",
				"glue:GetTableVersions"
			],
			"Resource": [
				"arn:aws:glue:eu-central-1:{user id}:catalog",
				"arn:aws:glue:eu-central-1:{user id}:database/clickhouse-regression-database-1",
				"arn:aws:glue:eu-central-1:{user id}:table/clickhouse-regression-database-1/table_a"
			]
		},
		{
			"Sid": "S3AccessToIcebergTableData",
			"Effect": "Allow",
			"Action": [
				"s3:ListBucket",
				"s3:GetObject",
				"s3:GetBucketLocation"
			],
			"Resource": [
				"arn:aws:s3:::glue-bucket-db-a",
				"arn:aws:s3:::glue-bucket-db-a/table_one/*"
			]
		}
	]
}

In Policy_2, Glue access is restricted to specific resources:

"Resource": [
				"arn:aws:glue:eu-central-1:{user id}:catalog",
				"arn:aws:glue:eu-central-1:{user id}:database/clickhouse-regression-database-1",
				"arn:aws:glue:eu-central-1:{user id}:table/clickhouse-regression-database-1/table_a"
			]

IAM user can only:

Access the Glue catalog
See only the specific database clickhouse-regression-database-1
See only the specific table table_a

Again from clickhouse (25.6.6.29):

SHOW TABLES FROM glue

   ┌─name─────────────────────────────────────┐
1. │ clickhouse-regression-database-1.table_a │
   └──────────────────────────────────────────┘

1 row in set. Elapsed: 0.955 sec.

Now as expected I see only one table.

I can select from it:

SELECT *
FROM glue.`clickhouse-regression-database-1.table_a`

   ┌─string─┬─int─┬─binary─┐
1. │ aa     │   1 │ ᴺᵁᴸᴸ   │
2. │ ᴺᵁᴸᴸ   │   1 │ ᴺᵁᴸᴸ   │
3. │ aaaaa  │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘

3 rows in set. Elapsed: 0.460 sec.

I can not select from table_b (which is expected):

SELECT *
FROM glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.6.6):
Code: 736. DB::Exception: Received from localhost:9000. DB::Exception: Exception calling GetTable for table clickhouse-regression-database-2.table_b: User: arn:aws:iam::{user id}:user/agiliazova is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:{user id}:database/clickhouse-regression-database-2 because no identity-based policy allows the glue:GetTable action. (DATALAKE_DATABASE_ERROR)

SHOW CREATE TABLE works only for table_a.

Now let's go back to Policy_1 and run all the same queries on altinity/clickhouse-server:25.3.3.20186.altinityantalya:

SET allow_experimental_database_glue_catalog = 1;
CREATE DATABASE glue
ENGINE = DataLakeCatalog
SETTINGS catalog_type = 'glue', region = 'eu-central-1', aws_access_key_id = '***', aws_secret_access_key = '***';

SHOW TABLES FROM glue

Received exception from server (version 25.3.3):
Code: 499. DB::Exception: Received from localhost:9000. DB::Exception: User: arn:aws:iam::{user id}:user/agiliazova is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::glue-bucket-db-b/second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json" because no identity-based policy allows the s3:GetObject action: while reading key: second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json, from bucket: glue-bucket-db-b. (S3_ERROR)

This shows that in ClickHouse version 25.3.3.altinityantalya, SHOW TABLES attempts to read Iceberg metadata (*.metadata.json) from S3 even during table listing.

As a result, if the IAM user does not have s3:GetObject access to all buckets, the SHOW TABLES command itself fails, even if the user is not querying the problematic table.

But I am still able to query table_a:

SELECT *
FROM glue.`clickhouse-regression-database-1.table_a`

   ┌─string─┬─int─┬─binary─┐
1. │ aa     │   1 │ ᴺᵁᴸᴸ   │
2. │ ᴺᵁᴸᴸ   │   1 │ ᴺᵁᴸᴸ   │
3. │ aaaaa  │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘

3 rows in set. Elapsed: 0.297 sec.

I am not able to query table_b:

SELECT *
FROM glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.3):
Code: 499. DB::Exception: Received from localhost:9000. DB::Exception: User: arn:aws:iam::{user id}:user/agiliazova is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::glue-bucket-db-b/second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json" because no identity-based policy allows the s3:GetObject action: while reading key: second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json, from bucket: glue-bucket-db-b. (S3_ERROR)

But surprisingly SHOW CREATE TABLE works for both tables:

SHOW CREATE TABLE glue.`clickhouse-regression-database-1.table_a`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-1.table_a`↴│
   │↳(                                                           ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `binary` Nullable(String)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-a/table_one')          │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.106 sec.

SHOW CREATE TABLE glue.`clickhouse-regression-database-2.table_b`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-2.table_b`↴│
   │↳(                                                           ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `float` Nullable(Float32)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-b/second_table')       │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.172 sec.

Not let's switch to Policy_2 and run all above queries again:

SHOW TABLES FROM glue

   ┌─name─────────────────────────────────────┐
1. │ clickhouse-regression-database-1.table_a │
   └──────────────────────────────────────────┘

1 row in set. Elapsed: 0.880 sec.

SELECT *
FROM glue.`clickhouse-regression-database-1.table_a`

   ┌─string─┬─int─┬─binary─┐
1. │ aa     │   1 │ ᴺᵁᴸᴸ   │
2. │ aaaaa  │   1 │ ᴺᵁᴸᴸ   │
3. │ ᴺᵁᴸᴸ   │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘

3 rows in set. Elapsed: 0.444 sec.

SELECT *
FROM glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.3):
Code: 736. DB::Exception: Received from localhost:9000. DB::Exception: Exception calling GetTable for table clickhouse-regression-database-2.table_b: User: arn:aws:iam::{user id}:user/agiliazova is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:{user id}:database/clickhouse-regression-database-2 because no identity-based policy allows the glue:GetTable action. (DATALAKE_DATABASE_ERROR)

SHOW CREATE TABLE glue.`clickhouse-regression-database-1.table_a`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-1.table_a`↴│
   │↳(                                                           ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `binary` Nullable(String)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-a/table_one')          │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.140 sec.

SHOW CREATE TABLE glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.3):
Code: 736. DB::Exception: Received from localhost:9000. DB::Exception: Exception calling GetTable for table clickhouse-regression-database-2.table_b: User: arn:aws:iam::{user id}:user/agiliazova is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:{user id}:database/clickhouse-regression-database-2 because no identity-based policy allows the glue:GetTable action. (DATALAKE_DATABASE_ERROR)

Policy_1 with `clickhouse/clickhouse-server:25.3`:

SHOW TABLES FROM glue

   ┌─name─────────────────────────────────────┐
1. │ clickhouse-regression-database-1.table_a │
2. │ clickhouse-regression-database-2.table_b │
   └──────────────────────────────────────────┘

2 rows in set. Elapsed: 1.210 sec.

SELECT *
FROM glue.`clickhouse-regression-database-1.table_a`

   ┌─string─┬─int─┬─binary─┐
1. │ aa     │   1 │ ᴺᵁᴸᴸ   │
2. │ aaaaa  │   1 │ ᴺᵁᴸᴸ   │
3. │ ᴺᵁᴸᴸ   │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘

3 rows in set. Elapsed: 0.599 sec.

SELECT *
FROM glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.6):
Code: 499. DB::Exception: Received from localhost:9000. DB::Exception: User: arn:aws:iam::{user id}:user/agiliazova is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::glue-bucket-db-b/second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json" because no identity-based policy allows the s3:GetObject action: while reading key: second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json, from bucket: glue-bucket-db-b. (S3_ERROR)

SHOW CREATE TABLE glue.`clickhouse-regression-database-1.table_a`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-1.table_a`↴│
   │↳(                                                           ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `binary` Nullable(String)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-a/table_one')          │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.105 sec.

SHOW CREATE TABLE glue.`clickhouse-regression-database-2.table_b`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-2.table_b`↴│
   │↳(                                                           ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `float` Nullable(Float32)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-b/second_table')       │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.125 sec.

Policy_2 with `clickhouse/clickhouse-server:25.3`:

SHOW TABLES FROM glue

   ┌─name─────────────────────────────────────┐
1. │ clickhouse-regression-database-1.table_a │
   └──────────────────────────────────────────┘

1 row in set. Elapsed: 0.638 sec.

SELECT *
FROM glue.`clickhouse-regression-database-1.table_a`

Query id: 8c57936c-5e44-4ace-b934-b2d8d515e769

   ┌─string─┬─int─┬─binary─┐
1. │ aa     │   1 │ ᴺᵁᴸᴸ   │
2. │ ᴺᵁᴸᴸ   │   1 │ ᴺᵁᴸᴸ   │
3. │ aaaaa  │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘

3 rows in set. Elapsed: 0.690 sec.

SELECT *
FROM glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.6):
Code: 736. DB::Exception: Received from localhost:9000. DB::Exception: Exception calling GetTable for table clickhouse-regression-database-2.table_b: User: arn:aws:iam::{user id}:user/agiliazova is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:{user id}:database/clickhouse-regression-database-2 because no identity-based policy allows the glue:GetTable action. (DATALAKE_DATABASE_ERROR)

SHOW CREATE TABLE glue.`clickhouse-regression-database-1.table_a`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-1.table_a`↴│
   │↳(                                                           ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `binary` Nullable(String)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-a/table_one')          │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.115 sec.

SHOW CREATE TABLE glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.6):
Code: 736. DB::Exception: Received from localhost:9000. DB::Exception: Exception calling GetTable for table clickhouse-regression-database-2.table_b: User: arn:aws:iam::{user id}:user/agiliazova is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:{user id}:database/clickhouse-regression-database-2 because no identity-based policy allows the glue:GetTable action. (DATALAKE_DATABASE_ERROR)

Policy_1 with `altinity/clickhouse-server:25.3.3.20143.altinityantalya`:

SHOW TABLES FROM glue

Received exception from server (version 25.3.3):
Code: 499. DB::Exception: Received from localhost:9000. DB::Exception: User: arn:aws:iam::{user id}::user/agiliazova is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::glue-bucket-db-b/second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json" because no identity-based policy allows the s3:GetObject action: while reading key: second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json, from bucket: glue-bucket-db-b. (S3_ERROR)

SELECT *
FROM glue.`clickhouse-regression-database-1.table_a`

   ┌─string─┬─int─┬─binary─┐
1. │ aa     │   1 │ ᴺᵁᴸᴸ   │
2. │ aaaaa  │   1 │ ᴺᵁᴸᴸ   │
3. │ ᴺᵁᴸᴸ   │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘

3 rows in set. Elapsed: 0.454 sec.

SELECT *
FROM glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.3):
Code: 499. DB::Exception: Received from localhost:9000. DB::Exception: User: arn:aws:iam::{user id}::user/agiliazova is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::glue-bucket-db-b/second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json" because no identity-based policy allows the s3:GetObject action: while reading key: second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json, from bucket: glue-bucket-db-b. (S3_ERROR)

SHOW CREATE TABLE glue.`clickhouse-regression-database-1.table_a`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-1.table_a`↴│
   │↳(                                                           ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `binary` Nullable(String)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-a/table_one')          │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.113 sec.

SHOW CREATE TABLE glue.`clickhouse-regression-database-2.table_b`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-2.table_b`↴│
   │↳(                                                           ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `float` Nullable(Float32)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-b/second_table')       │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.113 sec.

Policy_2 with `altinity/clickhouse-server:25.3.3.20143.altinityantalya`:

SHOW TABLES FROM glue

   ┌─name─────────────────────────────────────┐
1. │ clickhouse-regression-database-1.table_a │
   └──────────────────────────────────────────┘

1 row in set. Elapsed: 0.920 sec.

SELECT *
FROM glue.`clickhouse-regression-database-1.table_a`

   ┌─string─┬─int─┬─binary─┐
1. │ aa     │   1 │ ᴺᵁᴸᴸ   │
2. │ aaaaa  │   1 │ ᴺᵁᴸᴸ   │
3. │ ᴺᵁᴸᴸ   │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘

3 rows in set. Elapsed: 0.368 sec.

SELECT *
FROM glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.3):
Code: 736. DB::Exception: Received from localhost:9000. DB::Exception: Exception calling GetTable for table clickhouse-regression-database-2.table_b: User: arn:aws:iam::{user id}::user/agiliazova is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:{user id}::database/clickhouse-regression-database-2 because no identity-based policy allows the glue:GetTable action. (DATALAKE_DATABASE_ERROR)

SHOW CREATE TABLE glue.`clickhouse-regression-database-1.table_a`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-1.table_a`↴│
   │↳(                                                           ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `binary` Nullable(String)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-a/table_one')          │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.086 sec.

SHOW CREATE TABLE glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.3):
Code: 736. DB::Exception: Received from localhost:9000. DB::Exception: Exception calling GetTable for table clickhouse-regression-database-2.table_b: User: arn:aws:iam::{user id}:user/agiliazova is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:{user id}::database/clickhouse-regression-database-2 because no identity-based policy allows the glue:GetTable action. (DATALAKE_DATABASE_ERROR)

Conclusions:

Both Antalya builds behave similarly.
Antalya versions fail SHOW TABLES if any table’s S3 access is missing (IMPORTANT, need to check !!!).
Non-Antalya versions (25.3.6 / 25.6.6) allow SHOW TABLES despite missing S3 access. May confuse users by showing tables they can’t query.
All versions allow SHOW CREATE TABLE under Policy_1 for table_b even without S3 access. Potential leakage of metadata from glue:GetTable permissions. While not strictly wrong, this exposes S3 locations.

Recommendation: Use Policy_2 for Fine-Grained Permissions

Why Policy_2 is better:

Restricts access only to the required Glue database/table and S3 path
Avoids runtime errors when querying tables without S3 access

Scenario with caches disabled

I don't see any changes in behavior with caches disabled. Still antalya is different from upstream.

Prepare (hopefully) clean state:

SYSTEM DROP CONNECTIONS CACHE;
SYSTEM DROP FILESYSTEM CACHE;
SYSTEM DROP PAGE CACHE;
SYSTEM DROP PRIMARY INDEX CACHE;
SYSTEM DROP S3 CLIENT CACHE;
SYSTEM DROP SCHEMA CACHE;
SYSTEM DROP MMAP CACHE;
SYSTEM DROP ICEBERG METADATA CACHE;

SET use_uncompressed_cache = 0;
SET use_query_cache = 0;
SET use_query_condition_cache = 0;
SET compile_expressions = 0;
SET use_iceberg_metadata_files_cache = 0;
SET use_iceberg_partition_pruning=0;

After that I recreated DataLakeCatalog database named glue:

Full glue access but NO access to s3 bucket of the table_b

Policy_1 with 25.3.3.20186.altinityantalya:

SHOW TABLES FROM glue;

Received exception from server (version 25.3.3):
Code: 499. DB::Exception: Received from localhost:9000. DB::Exception: User: arn:aws:iam::407099639081:user/agiliazova is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::glue-bucket-db-b/second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json" because no identity-based policy allows the s3:GetObject action: while reading key: second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json, from bucket: glue-bucket-db-b. (S3_ERROR)

SELECT *
FROM glue.`clickhouse-regression-database-1.table_a`

   ┌─string─┬─int─┬─binary─┐
1. │ aa     │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘
   ┌─string─┬─int─┬─binary─┐
2. │ ᴺᵁᴸᴸ   │   1 │ ᴺᵁᴸᴸ   │
3. │ aaaaa  │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘

3 rows in set. Elapsed: 2.298 sec.

SELECT *
FROM glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.3):
Code: 499. DB::Exception: Received from localhost:9000. DB::Exception: User: arn:aws:iam::407099639081:user/agiliazova is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::glue-bucket-db-b/second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json" because no identity-based policy allows the s3:GetObject action: while reading key: second_table/metadata/00001-2898634d-aa90-44f5-b5bc-7e067f43663f.metadata.json, from bucket: glue-bucket-db-b. (S3_ERROR)

SHOW CREATE TABLE glue.`clickhouse-regression-database-1.table_a`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-1.table_a`↴│
   │↳(                                                           ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `binary` Nullable(String)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-a/table_one')          │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.238 sec.

SHOW CREATE TABLE glue.`clickhouse-regression-database-2.table_b`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-2.table_b`↴│
   │↳(                                                           ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `float` Nullable(Float32)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-b/second_table')       │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.197 sec.

Restricted glue access to the table_a only and still NO access to s3 bucket of the table_b

Policy_2 with 25.3.3.20186.altinityantalya:
I recreated database.

SHOW TABLES FROM glue

   ┌─name─────────────────────────────────────┐
1. │ clickhouse-regression-database-1.table_a │
   └──────────────────────────────────────────┘

1 row in set. Elapsed: 5.244 sec.

SELECT *
FROM glue.`clickhouse-regression-database-1.table_a`

   ┌─string─┬─int─┬─binary─┐
1. │ aaaaa  │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘
   ┌─string─┬─int─┬─binary─┐
2. │ aa     │   1 │ ᴺᵁᴸᴸ   │
3. │ ᴺᵁᴸᴸ   │   1 │ ᴺᵁᴸᴸ   │
   └────────┴─────┴────────┘

3 rows in set. Elapsed: 2.639 sec.

SELECT *
FROM glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.3):
Code: 736. DB::Exception: Received from localhost:9000. DB::Exception: Exception calling GetTable for table clickhouse-regression-database-2.table_b: User: arn:aws:iam::407099639081:user/agiliazova is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:407099639081:database/clickhouse-regression-database-2 because no identity-based policy allows the glue:GetTable action. (DATALAKE_DATABASE_ERROR)

SHOW CREATE TABLE glue.`clickhouse-regression-database-1.table_a`

   ┌─statement────────────────────────────────────────────────────┐
1. │ CREATE TABLE glue.`clickhouse-regression-database-1.table_a`↴│
   │↳(                                                           ↴│
   │↳    `string` Nullable(String),                              ↴│
   │↳    `int` Nullable(Int32),                                  ↴│
   │↳    `binary` Nullable(String)                               ↴│
   │↳)                                                           ↴│
   │↳ENGINE = Iceberg('s3://glue-bucket-db-a/table_one')          │
   └──────────────────────────────────────────────────────────────┘

1 row in set. Elapsed: 0.762 sec.

SHOW CREATE TABLE glue.`clickhouse-regression-database-2.table_b`

Received exception from server (version 25.3.3):
Code: 736. DB::Exception: Received from localhost:9000. DB::Exception: Exception calling GetTable for table clickhouse-regression-database-2.table_b: User: arn:aws:iam::407099639081:user/agiliazova is not authorized to perform: glue:GetTable on resource: arn:aws:glue:eu-central-1:407099639081:database/clickhouse-regression-database-2 because no identity-based policy allows the glue:GetTable action. (DATALAKE_DATABASE_ERROR)

@alsugiliazova

@alsugiliazova Is it possible to get full exception stacktrace from log?

@ianton-ru

@ianton-ru
here is the trace for SHOW TABLES FROM

2025.08.28 18:02:57.874423 [ 86 ] {} <Error> TCPHandler: Code: 499. DB::Exception: User: arn:aws:iam::407099639081:user/glue_clickhouse_regression_test_user is not authorized to perform: s3:GetObject on resource: "arn:aws:s3:::glue-bucket-db-b/table_two/metadata/00001-0a65135f-facd-4115-a079-dcb81f9ac977.metadata.json" because no identity-based policy allows the s3:GetObject action: while reading key: table_two/metadata/00001-0a65135f-facd-4115-a079-dcb81f9ac977.metadata.json, from bucket: glue-bucket-db-b. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000f4d95db
1. DB::S3Exception::S3Exception(String const&, Aws::S3::S3Errors) @ 0x0000000012036bad
2. DB::ReadBufferFromS3::sendRequest(unsigned long, unsigned long, std::optional<unsigned long>) const @ 0x000000001205b96c
3. DB::ReadBufferFromS3::nextImpl() @ 0x00000000120583be
4. void DB::readJSONObjectOrArrayPossiblyInvalid<String, void, (char)123, (char)125>(String&, DB::ReadBuffer&) @ 0x000000000f5a9a36
5. DB::IcebergMetadata::create(std::shared_ptr<DB::IObjectStorage> const&, std::weak_ptr<DB::StorageObjectStorage::Configuration> const&, std::shared_ptr<DB::Context const> const&)::$_0::operator()() const @ 0x0000000011f303de
6. DB::IcebergMetadata::create(std::shared_ptr<DB::IObjectStorage> const&, std::weak_ptr<DB::StorageObjectStorage::Configuration> const&, std::shared_ptr<DB::Context const> const&) @ 0x0000000011f2f07d
7. DB::DataLakeConfiguration<DB::StorageS3Configuration, DB::IcebergMetadata>::updateMetadataObjectIfNeeded(std::shared_ptr<DB::IObjectStorage>, std::shared_ptr<DB::Context const>) @ 0x0000000011ef5ce1
8. DB::DataLakeConfiguration<DB::StorageS3Configuration, DB::IcebergMetadata>::update(std::shared_ptr<DB::IObjectStorage>, std::shared_ptr<DB::Context const>) @ 0x0000000011ef5aee
9. DB::StorageObjectStorage::StorageObjectStorage(std::shared_ptr<DB::StorageObjectStorage::Configuration>, std::shared_ptr<DB::IObjectStorage>, std::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::optional<DB::FormatSettings>, DB::LoadingStrictnessLevel, bool, std::shared_ptr<DB::IAST>, bool, bool, std::optional<String>)::$_0::operator()() const @ 0x0000000011e1244e
10. DB::StorageObjectStorage::StorageObjectStorage(std::shared_ptr<DB::StorageObjectStorage::Configuration>, std::shared_ptr<DB::IObjectStorage>, std::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::optional<DB::FormatSettings>, DB::LoadingStrictnessLevel, bool, std::shared_ptr<DB::IAST>, bool, bool, std::optional<String>) @ 0x0000000011e11112
11. DB::StorageObjectStorageCluster::StorageObjectStorageCluster(String const&, std::shared_ptr<DB::StorageObjectStorage::Configuration>, std::shared_ptr<DB::IObjectStorage>, std::shared_ptr<DB::Context const>, DB::StorageID const&, DB::ColumnsDescription const&, DB::ConstraintsDescription const&, String const&, std::optional<DB::FormatSettings>, DB::LoadingStrictnessLevel, std::shared_ptr<DB::IAST>) @ 0x0000000011e338bd
12. DB::DatabaseDataLake::tryGetTableImpl(String const&, std::shared_ptr<DB::Context const>, bool) const @ 0x0000000012b4c429
13. DB::DatabaseDataLake::getLightweightTablesIterator(std::shared_ptr<DB::Context const>, std::function<bool (String const&)> const&, bool) const @ 0x0000000012b4e5ea
14. DB::detail::getFilteredTables(DB::ActionsDAG::Node const*, COW<DB::IColumn>::immutable_ptr<DB::IColumn> const&, std::shared_ptr<DB::Context const>, bool) @ 0x00000000113ea085
15. DB::ReadFromSystemTables::applyFilters(DB::ActionDAGNodes) @ 0x00000000113f374e
16. DB::QueryPlanOptimizations::optimizePrimaryKeyConditionAndLimit(std::vector<DB::QueryPlanOptimizations::Frame, std::allocator<DB::QueryPlanOptimizations::Frame>> const&) @ 0x00000000152c4c56
17. DB::QueryPlanOptimizations::optimizeTreeSecondPass(DB::QueryPlanOptimizationSettings const&, DB::QueryPlan::Node&, std::list<DB::QueryPlan::Node, std::allocator<DB::QueryPlan::Node>>&) @ 0x00000000152c339f
18. DB::QueryPlan::buildQueryPipeline(DB::QueryPlanOptimizationSettings const&, DB::BuildQueryPipelineSettings const&) @ 0x00000000152149a1
19. DB::InterpreterSelectQueryAnalyzer::buildQueryPipeline() @ 0x000000001361eff1
20. DB::InterpreterSelectQueryAnalyzer::execute() @ 0x000000001361e91e
21. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr<DB::IAST>&) @ 0x00000000139c84cb
22. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x00000000139c2ee4
23. DB::InterpreterShowTablesQuery::execute() @ 0x0000000013a1eaff
24. DB::executeQueryImpl(char const*, char const*, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*, std::shared_ptr<DB::IAST>&) @ 0x00000000139c84cb
25. DB::executeQuery(String const&, std::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) @ 0x00000000139c2ee4
26. DB::TCPHandler::runImpl() @ 0x0000000014cfd44c
27. DB::TCPHandler::run() @ 0x0000000014d1b639
28. Poco::Net::TCPServerConnection::start() @ 0x000000001844e567
29. Poco::Net::TCPServerDispatcher::run() @ 0x000000001844e9b9
30. Poco::PooledThread::run() @ 0x000000001841acfb
31. Poco::ThreadImpl::runnableEntry(void*) @ 0x00000000184191dd

Thanks!
Must be fixed in 25.6 with this commit
cf3e774
from
#949

Navigation Menu

Repository navigation

Issues with permissions using DataLakeCatalog with type=glue #954

What this policy means:

Policy_1 with `clickhouse/clickhouse-server:25.3`:

Policy_2 with `clickhouse/clickhouse-server:25.3`:

Policy_1 with `altinity/clickhouse-server:25.3.3.20143.altinityantalya`:

Policy_2 with `altinity/clickhouse-server:25.3.3.20143.altinityantalya`:

Recommendation: Use Policy_2 for Fine-Grained Permissions

Scenario with caches disabled

Full glue access but NO access to s3 bucket of the table_b

Restricted glue access to the table_a only and still NO access to s3 bucket of the table_b

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issues with permissions using DataLakeCatalog with type=glue #954

Description

What this policy means:

Activity

alsugiliazova commented on Aug 6, 2025

Policy_1 with clickhouse/clickhouse-server:25.3:

Policy_2 with clickhouse/clickhouse-server:25.3:

alsugiliazova commented on Aug 6, 2025

Policy_1 with altinity/clickhouse-server:25.3.3.20143.altinityantalya:

Policy_2 with altinity/clickhouse-server:25.3.3.20143.altinityantalya:

alsugiliazova commented on Aug 6, 2025

alsugiliazova commented on Aug 6, 2025

Recommendation: Use Policy_2 for Fine-Grained Permissions

alsugiliazova commented on Aug 22, 2025

Scenario with caches disabled

Full glue access but NO access to s3 bucket of the table_b

Restricted glue access to the table_a only and still NO access to s3 bucket of the table_b

ianton-ru commented on Aug 28, 2025

alsugiliazova commented on Aug 28, 2025

ianton-ru commented on Aug 29, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Policy_1 with `clickhouse/clickhouse-server:25.3`:

Policy_2 with `clickhouse/clickhouse-server:25.3`:

Policy_1 with `altinity/clickhouse-server:25.3.3.20143.altinityantalya`:

Policy_2 with `altinity/clickhouse-server:25.3.3.20143.altinityantalya`: