Steps to reproduce the behavior (Required)
- Create a Kudu table through Impala/Hive so that it is registered in Hive Metastore (HMS) with
kudu.table_name / Kudu input format, on a Kudu cluster with multiple tablet servers.
- Create a unified catalog backed by that HMS and pointing at the real Kudu masters:
CREATE EXTERNAL CATALOG unified_catalog
PROPERTIES (
"type" = "unified",
"unified.metastore.type" = "hive",
"hive.metastore.uris" = "thrift://<hms-host>:9083",
"kudu.master" = "<kudu-master-1>:7051,<kudu-master-2>:7051,<kudu-master-3>:7051"
);
SELECT * FROM unified_catalog.<db>.<kudu_table> LIMIT 10; → works while every Kudu tablet server is healthy.
- Stop one Kudu tablet server (crash, or
systemctl stop kudu-tserver for maintenance).
- Run the same
SELECT again.
Expected behavior (Required)
The query should still succeed. The Kudu client should contact the configured Kudu masters (kudu.master of the catalog), learn the new tablet leader / replica locations, and read from the remaining tablet servers, exactly as a native Kudu catalog ("type" = "kudu") does in the same situation.
Real behavior (Required)
The query fails as soon as one tablet server is unavailable:
ERROR 1064 (HY000): Failed to call the nextChunkOffHeap method of off-heap table scanner ...
BE / JNI reader logs show the Kudu reader being opened with an empty master address, and the Kudu Java client then falls back to the default localhost:7051, which it cannot reach:
Open kudu reader with master: , token: ...
Root cause (FE):
- For a unified catalog,
UnifiedMetadata routes Kudu tables to KuduMetadata, which, when an HMS is configured, returns metastore.get().getTable(dbName, tblName) directly (KuduMetadata.getTable).
- That HMS path builds the table via
KuduTable.fromMetastoreTable(...), which hard-codes masterAddresses to StringUtils.EMPTY. The masterAddresses that KuduMetadata was constructed with (from the catalog's kudu.master property) is never applied to the returned KuduTable.
KuduScanNode.addScanRangeLocations then sends kuduTable.getMasterAddresses() (empty string) to the BE as THdfsScanRange.kudu_master, and KuduSplitScanner.open() builds new KuduClient.KuduClientBuilder(""), which effectively resolves to localhost:7051.
While all tablet servers are up, the pre-built scan token already contains the tablet server locations, so the missing master address is not noticed. As soon as a tablet server goes down (or a scan token points at a replica that is no longer available), the client needs the master to re-resolve tablet locations, cannot reach localhost:7051 on the BE host, and the scan fails.
This makes Kudu tables under a unified catalog unusable during any tablet server outage or rolling maintenance, whereas a plain Kudu catalog on the same cluster keeps working.
Fix: apply the catalog's masterAddresses to KuduTable objects returned from HMS in KuduMetadata.getTable. PR #67012 implements this.
StarRocks version (Required)
- Reproduced against the current
main code path. The same code is present in 3.3, 3.4, 3.5, 4.0 and 4.1, so those branches are affected as well.
Steps to reproduce the behavior (Required)
kudu.table_name/ Kudu input format, on a Kudu cluster with multiple tablet servers.CREATE EXTERNAL CATALOG unified_catalog PROPERTIES ( "type" = "unified", "unified.metastore.type" = "hive", "hive.metastore.uris" = "thrift://<hms-host>:9083", "kudu.master" = "<kudu-master-1>:7051,<kudu-master-2>:7051,<kudu-master-3>:7051" );SELECT * FROM unified_catalog.<db>.<kudu_table> LIMIT 10;→ works while every Kudu tablet server is healthy.systemctl stop kudu-tserverfor maintenance).SELECTagain.Expected behavior (Required)
The query should still succeed. The Kudu client should contact the configured Kudu masters (
kudu.masterof the catalog), learn the new tablet leader / replica locations, and read from the remaining tablet servers, exactly as a native Kudu catalog ("type" = "kudu") does in the same situation.Real behavior (Required)
The query fails as soon as one tablet server is unavailable:
BE / JNI reader logs show the Kudu reader being opened with an empty master address, and the Kudu Java client then falls back to the default
localhost:7051, which it cannot reach:Root cause (FE):
UnifiedMetadataroutes Kudu tables toKuduMetadata, which, when an HMS is configured, returnsmetastore.get().getTable(dbName, tblName)directly (KuduMetadata.getTable).KuduTable.fromMetastoreTable(...), which hard-codesmasterAddressestoStringUtils.EMPTY. ThemasterAddressesthatKuduMetadatawas constructed with (from the catalog'skudu.masterproperty) is never applied to the returnedKuduTable.KuduScanNode.addScanRangeLocationsthen sendskuduTable.getMasterAddresses()(empty string) to the BE asTHdfsScanRange.kudu_master, andKuduSplitScanner.open()buildsnew KuduClient.KuduClientBuilder(""), which effectively resolves tolocalhost:7051.While all tablet servers are up, the pre-built scan token already contains the tablet server locations, so the missing master address is not noticed. As soon as a tablet server goes down (or a scan token points at a replica that is no longer available), the client needs the master to re-resolve tablet locations, cannot reach
localhost:7051on the BE host, and the scan fails.This makes Kudu tables under a unified catalog unusable during any tablet server outage or rolling maintenance, whereas a plain Kudu catalog on the same cluster keeps working.
Fix: apply the catalog's
masterAddressestoKuduTableobjects returned from HMS inKuduMetadata.getTable. PR #67012 implements this.StarRocks version (Required)
maincode path. The same code is present in 3.3, 3.4, 3.5, 4.0 and 4.1, so those branches are affected as well.