[Question] How does migrations on a table ending up working?

[hamin]

First of, this project looks fascinating!

I have a general question, what do migrations end up looking like? I'm guessing/assuming that since everything is powered by the WAL that the final resulting parquet files get columns migrated properly. Also if that's the case, are all DDL migrations safe and end up resulting in correct transformations/changes in the parquet files?

[viggy28]

Thanks, glad you find it interesting! 🙏

You've got the right intuition. Streambed detects schema changes from the WAL itself — when Postgres re-emits a logical-replication Relation message for a table, we diff it against the cached one and detect column-level changes. We then flush any buffered rows under the old schema, evolve the Iceberg table schema (new metadata version, monotonically-increasing field IDs that are never reused), and resume writing. So newly-written Parquet files use the new schema while old files stay readable via Iceberg's field-ID mapping.

ADD/DROP/ALTER columns are implemented - https://github.com/viggy28/streambed/blob/main/test/integration/schema_test.go

Rename column is not.