Skip to content

[BugFix] Kudu tables in unified catalog (HMS) lose kudu.master: query fails when a tablet server is down #79431

Description

@jhcnet3

Steps to reproduce the behavior (Required)

  1. Create a Kudu table through Impala/Hive so that it is registered in Hive Metastore (HMS) with kudu.table_name / Kudu input format, on a Kudu cluster with multiple tablet servers.
  2. Create a unified catalog backed by that HMS and pointing at the real Kudu masters:
    CREATE EXTERNAL CATALOG unified_catalog
    PROPERTIES (
        "type" = "unified",
        "unified.metastore.type" = "hive",
        "hive.metastore.uris" = "thrift://<hms-host>:9083",
        "kudu.master" = "<kudu-master-1>:7051,<kudu-master-2>:7051,<kudu-master-3>:7051"
    );
  3. SELECT * FROM unified_catalog.<db>.<kudu_table> LIMIT 10; → works while every Kudu tablet server is healthy.
  4. Stop one Kudu tablet server (crash, or systemctl stop kudu-tserver for maintenance).
  5. Run the same SELECT again.

Expected behavior (Required)

The query should still succeed. The Kudu client should contact the configured Kudu masters (kudu.master of the catalog), learn the new tablet leader / replica locations, and read from the remaining tablet servers, exactly as a native Kudu catalog ("type" = "kudu") does in the same situation.

Real behavior (Required)

The query fails as soon as one tablet server is unavailable:

ERROR 1064 (HY000): Failed to call the nextChunkOffHeap method of off-heap table scanner ...

BE / JNI reader logs show the Kudu reader being opened with an empty master address, and the Kudu Java client then falls back to the default localhost:7051, which it cannot reach:

Open kudu reader with master: , token: ...

Root cause (FE):

  • For a unified catalog, UnifiedMetadata routes Kudu tables to KuduMetadata, which, when an HMS is configured, returns metastore.get().getTable(dbName, tblName) directly (KuduMetadata.getTable).
  • That HMS path builds the table via KuduTable.fromMetastoreTable(...), which hard-codes masterAddresses to StringUtils.EMPTY. The masterAddresses that KuduMetadata was constructed with (from the catalog's kudu.master property) is never applied to the returned KuduTable.
  • KuduScanNode.addScanRangeLocations then sends kuduTable.getMasterAddresses() (empty string) to the BE as THdfsScanRange.kudu_master, and KuduSplitScanner.open() builds new KuduClient.KuduClientBuilder(""), which effectively resolves to localhost:7051.

While all tablet servers are up, the pre-built scan token already contains the tablet server locations, so the missing master address is not noticed. As soon as a tablet server goes down (or a scan token points at a replica that is no longer available), the client needs the master to re-resolve tablet locations, cannot reach localhost:7051 on the BE host, and the scan fails.

This makes Kudu tables under a unified catalog unusable during any tablet server outage or rolling maintenance, whereas a plain Kudu catalog on the same cluster keeps working.

Fix: apply the catalog's masterAddresses to KuduTable objects returned from HMS in KuduMetadata.getTable. PR #67012 implements this.

StarRocks version (Required)

  • Reproduced against the current main code path. The same code is present in 3.3, 3.4, 3.5, 4.0 and 4.1, so those branches are affected as well.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions