[Feature] Support format table for Hive-style file directories

[SteNicholas]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Java Paimon supports format tables via org.apache.paimon.table.FormatTable: a table provided as a directory of files in a given file format, organized like a standard Hive table — partitions are discovered from the directory structure, and there is no snapshot or manifest metadata. It is identified by the table option type = format-table (Options::TYPE), and its file format is given by file.format (Options::FILE_FORMAT).

paimon-cpp has no equivalent today, so it cannot read or write these plain Hive-style directory tables. Users that keep data as directories of CSV/Parquet/ORC files (e.g. produced by external engines) cannot access them natively through paimon-cpp, which diverges from the Java table set.

Solution

Introduce native support for format tables, aligned with the Java FormatTable semantics. Unlike managed paimon tables, format table reads return plain columns without the _VALUE_KIND field, and only INSERT rows can be written.

Public API under include/paimon/table/format/:

• FormatTable — table handle: Create(...) (load from a path or from a schema), format parsing, and option accessors (Location, GetFormat, PartitionKeys, FileCompression, …).
• FormatTableScan / FormatDataSplit / FormatTableRead — read path: partition discovery from the directory layout, equality partition filtering, projection, and limit.
• FormatTableWrite / FormatCommitMessage / FormatTableCommit — write path: INSERT-only, two-phase commit (data written to hidden .{name}.tmp files, renamed to data-{uuid}-{n}.{ext} on commit).

Supported file formats: PARQUET, ORC, CSV, TEXT, JSON. This also adds a new CSV file format module (src/paimon/format/csv/) implementing the FileFormat / FileFormatFactory interfaces (reader, writer, options, and factory). Format tables can be created and loaded through FileSystemCatalog.

Intentional deviations from Java:

• FileSystemCatalog can create format tables. In Java this is only possible via HiveCatalog, which paimon-cpp does not have; disallowing creation would make the feature unusable.
• SchemaManager::CreateTable skips managed-table schema validation for format tables (bucket/manifest options do not apply) but still enforces generic structural invariants (partition keys exist, no duplicates, primitive key types, reserved names).
• No offset-based splitting of large files (whole-file splits only).
• Read path supports equality partition filtering + projection + limit; no row-level predicate pushdown.
• CSV: no BINARY type (Java uses Base64), csv.mode = FAILFAST only, compression none only.

Anything else?

Review findings & follow-ups (from the multi-angle review of the initial implementation)

Each finding below was independently verified against the actual code. Fixes for the P1 items and most P2/cleanup items are being prepared.

P1 — Correctness

1. CSV column pruning can silently read wrong columns. CsvFileBatchReader::SetReadSchema re-maps an arbitrary projection to positions 0..N-1 when the file has no header and the requested field names do not match the generated names f0..fn. Since format table data files are written without a header by default (csv.include-header=false), projecting a non-prefix / reordered subset reads the wrong file columns. Fix: the read path passes the full non-partition schema to positional formats (CSV/TEXT/JSON) and prunes/reorders after decode, keeping name-based pushdown for self-describing formats (Parquet/ORC).

2. CSV writer never escapes the escape character. CsvFormatWriter::EscapeField only doubles the quote character; a value containing the escape character (default \) is written raw, and CsvParser consumes a lone escape char and emits nothing. Writing a\b reads back as ab. Fix: escape the escape character on write (symmetric with the parser).

3. CSV line splitting is not quote-aware. The writer quotes fields containing the line delimiter, but SplitLines splits on the delimiter unconditionally, so a quoted field with an embedded newline is broken into two malformed rows on read. Fix: quote/escape-aware line scanning.

4. Format tables skip all generic schema validation. SchemaManager::CreateTable skips SchemaValidation::ValidateTableSchema entirely for format tables, which also drops generic structural invariants (partition keys exist, no duplicates, primitive key types, reserved names). Fix: split generic structural validation from managed-table-specific validation and run the generic part for format tables.

5. Format table write does not validate field nullability. FormatTableWrite::Impl::Write imports the Arrow batch without ArrowUtils::CheckNullabilityMatch, so a null in a nullable=false field can be silently serialized. Java FormatTableWrite performs this validation. Fix: apply the same check.

6. CSV format library missing from test/benchmark link lists. TEST_PLUGIN_LINK_LIBS and PAIMON_BENCHMARK_STATIC_LINK_LIBS omit paimon_csv_file_format, so the CSV FileFormatFactory is not registered in core_test/integration/benchmark binaries.

P2

7. CSV reader loads the whole file before the first batch; no range splits. LoadContent materializes the entire file into memory; FormatDataSplit has no offset/length, so a single multi-GB CSV cannot be split and may OOM. Follow-up: add offset/length to split metadata and a streaming line reader.

8. Reader metrics lost after each file completes. FormatTableBatchReader::GetReaderMetrics only returns the current file reader's metrics; finished readers' metrics are not aggregated and the method returns null after EOF. Fix: aggregate finished + current reader metrics.

Conventions / reuse / efficiency

• FormatTableUtils::ParseIntOption hand-rolls std::stoi in try/catch instead of reusing exception-free OptionsUtils::GetValueFromMap<int32_t>.
• CsvOptions getters use snake_case, violating the PascalCase method naming convention.
• Write hot path exports the full Arrow schema per batch; read path re-parses constant partition scalars per batch — both cacheable.
• Redundant Exists() round-trip per discovered partition during scans.
• Duplicated temp-file cleanup loops in FormatTableWrite::Abort / FormatTableCommit::Abort; dead members in the CSV reader; SplitPartitionLevels duplicates StringUtils::Split.
• The "batches must be released before the reader that produced them" lifetime contract belongs on the BatchReader base contract, not only on FormatTableRead.

Follow-ups (not in the first fix pass)

• Streaming CSV line reader + offset/length range splits (P2 item 7).
• CSV read path: parse fields directly into typed builders instead of building an intermediate StringArray then casting.
• A shared home for text-format option keys (csv.field-delimiter, …).
• Reusable data-file naming convention shared between the format table writer and DataFilePathFactory.

Are you willing to submit a PR?

• I'm willing to submit a PR!