Added support for add column and drop column by subkanthi · Pull Request #41 · Altinity/ice

subkanthi · 2025-06-30T18:49:34Z

closes: #39

subkanthi · 2025-07-01T19:53:27Z

+    Types.NestedField field;
+
+    switch (type.toLowerCase()) {
+      case "string":


couldn't find any mapper from string to iceberg Types

subkanthi · 2025-07-01T21:06:16Z

+    return new ColumnSpec(columnName, field.type(), comment);
+  }
+
+  private static Types.NestedField parseColumnType(String name, String type, String comment) {


maybe this needs to be moved to a common class for field mapping.

shyiko

I think we should go with @aadant's proposal #39 (comment). Separate add-column/delete-column commands instead of single alter-table means you can't combine multiple changes in a single transaction.

… as arguments.

subkanthi · 2025-07-10T21:03:43Z

alter-table flowers.iris --operations '[{"drop": "age"}]'
2025-07-10 15:41:35 [main] INFO c.a.i.c.i.cmd.AlterTable > Dropping column 'age' from table: flowers.iris
2025-07-10 15:41:36 [main] INFO c.a.i.c.i.cmd.AlterTable > Successfully applied 1 operations to table: flowers.iris

Invalid operation

alter-table flowers.iris --operations '[{"rename": "age"}]'
2025-07-10 15:54:17 [main] ERROR com.altinity.ice.cli.Main > Fatal
java.lang.IllegalArgumentException: Invalid operation. Supported operations: add, drop
        at com.altinity.ice.cli.internal.cmd.AlterTable.validateOperation(AlterTable.java:117)
        at com.altinity.ice.cli.internal.cmd.AlterTable.run(AlterTable.java:65)
        at com.altinity.ice.cli.Main.alterTable(Main.java:459)
        at com.altinity.ice.cli.Main.lambda$main$0(Main.java:551) [9 skipped]
        at com.altinity.ice.cli.Main.main(Main.java:558) [1 skipped]

shyiko · 2025-07-11T19:41:29Z

+              names = {"--operations"},
+              required = true,
+              description =
+                  "JSON array of operations: [{'add': 'age:int:User age'}, {'drop': 'col_name'}]")


It would be best to stick to yaml/json for this as column names may contain colons, future operations may not be so easy to express in colon-separated values, etc.

It would also be nice if operation/files names would follow https://iceberg.apache.org/docs/latest/spark-ddl/#alter-table-rename-column naming.

alter-table flowers.iris --operations '[{"operation": "add column", "column_name": "age", "type": "int", "comment": "user age"}]' 2025-07-16 16:12:24 [main] INFO c.a.i.c.i.cmd.AlterTable > Adding column 'age' to table: flowers.iris 2025-07-16 16:12:24 [main] INFO c.a.i.c.i.cmd.AlterTable > Successfully applied 1 operations to table: flowers.iris

ice alter-table flowers.iris --operations '[{"operation": "drop column", "column_name": "age"}]' 2025-07-16 16:12:48 [main] INFO c.a.i.c.i.cmd.AlterTable > Dropping column 'age' from table: flowers.iris 2025-07-16 16:12:49 [main] INFO c.a.i.c.i.cmd.AlterTable > Successfully applied 1 operations to table: flowers.iris

alter-table flowers.iris --operations '[{"operation": "drop column", "column_name": "age"}, {"operation": "add column", "column_name": "marital_status", "type": "string", "comment": "Marital Status"}]' 2025-07-16 16:35:47 [main] INFO c.a.i.c.i.cmd.AlterTable > Dropping column 'age' from table: flowers.iris 2025-07-16 16:35:47 [main] INFO c.a.i.c.i.cmd.AlterTable > Adding column 'marital_status' to table: flowers.iris 2025-07-16 16:35:47 [main] INFO c.a.i.c.i.cmd.AlterTable > Successfully applied 2 operations to table: flowers.iris

…tions name

shyiko · 2025-08-18T17:57:20Z

Can you please confirm that this scenario works:

insert parquet file #1 containing N columns
run ice alter-table to replace last column with a different one (i.e. delete X + add Y)
insert parquet file #2 that contains a new set of columns. confirm it gets inserted and data from both files can still be queried

…-to-an-existing-table

subkanthi · 2025-08-20T02:25:19Z

Can you please confirm that this scenario works:

insert parquet file #1 containing N columns

run ice alter-table to replace last column with a different one (i.e. delete X + add Y)

insert parquet file #2 that contains a new set of columns. confirm it gets inserted and data from both files can still be queried

CREATE TABLE employees (id INTEGER, name VARCHAR, department VARCHAR, yearsInService INTEGER);
INSERT INTO employees VALUES (1,'John Smith','Engineering',5),(2,'Sarah Johnson','Marketing',3),(3,'Mike Davis','Engineering',7),(4,'Emma Wilson','HR',2),(5,'James Brown','Sales',4),(6,'Lisa Anderson','Engineering',6),(7,'Robert Taylor','Marketing',1),(8,'Maria Garcia','HR',3),(9,'David Martinez','Sales',5),(10,'Jennifer Lee','Engineering',8),(11,'Chris Wilson','Engineering',4),(12,'Amanda White','Sales',2),(13,'Kevin Thompson','Marketing',6),(14,'Michelle Harris','HR',3),(15,'Brian Clark','Engineering',9),(16,'Nancy Lewis','Sales',1),(17,'Steven Walker','Marketing',5),(18,'Laura Hall','HR',7),(19,'Gary Allen','Engineering',3),(20,'Sandra Young','Sales',4);
COPY employees TO 'employees_with_years.parquet' (FORMAT PARQUET);
SELECT * FROM 'employees_with_years.parquet' LIMIT 5;
D CREATE TABLE employees (
      id INTEGER,
      name VARCHAR,
      department VARCHAR,
      salary DECIMAL(10,2)
  );

D copy employees to 'employees.parquet' (FORMAT PARQUET);

ice  insert flowers.employees -p   file://employees.parquet

ice alter-table flowers.employees --operations '[{"operation": "drop column", "column_name": "salary"}, {"operation": "add column", "column_name": "yearsInService", "type": "int", "comment": "Years in service"}]'


D DROP TABLE IF EXISTS employees;
D CREATE TABLE employees (id INTEGER, name VARCHAR, department VARCHAR, yearsInService INTEGER);
D insert into employees values(55, 'John Foo', 'Engineering', 44);
D copy employees to 'employees_with_years.parquet' (format parquet);


**Exception**

insert flowers.employees -p   file://employees_with_years.parquet
2025-08-19 21:22:31 [52179791-79] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/config
2025-08-19 21:22:31 [52179791-77] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous HEAD v1/namespaces/flowers/tables/employees
2025-08-19 21:22:31 [52179791-47] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/flowers/tables/employees
2025-08-19 21:22:31 [52179791-77] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous GET v1/namespaces/flowers/tables/employees
2025-08-19 21:22:32 [52179791-47] INFO c.a.i.r.c.i.r.RESTCatalogServlet > @token:anonymous POST v1/namespaces/flowers/tables/employees/metrics
2025-08-19 21:22:32 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://employees_with_years.parquet: processing
2025-08-19 21:22:32 [-6-thread-1] INFO c.a.i.c.internal.cmd.Insert > file://employees_with_years.parquet: jvm: heap: init=1008M,used=23M,committed=80M,free=15984M,max=16008M, gc pressure: 100.00%(27ms), off-heap: init=7M,used=51M,committed=54M, open fds: 10
2025-08-19 21:22:32 [main] ERROR com.altinity.ice.cli.Main > Fatal
java.io.IOException: Error processing file(s)
        at com.altinity.ice.cli.internal.cmd.Insert.run(Insert.java:230)
        at com.altinity.ice.cli.Main.insert(Main.java:364)
        at com.altinity.ice.cli.Main.lambda$main$0(Main.java:605) [9 skipped]
        at com.altinity.ice.cli.Main.main(Main.java:612) [1 skipped]
Caused by: java.io.IOException: Error processing file://employees_with_years.parquet
        at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$3(Insert.java:215)
 [4 skipped]
Caused by: org.apache.iceberg.exceptions.BadRequestException: file://employees_with_years.parquet's schema doesn't match table's schema
        at com.altinity.ice.cli.internal.cmd.Insert.processFile(Insert.java:399)
        at com.altinity.ice.cli.internal.cmd.Insert.lambda$run$3(Insert.java:194)
        ... 4 common frames omitted

    MessageType type = metadata.getFileMetaData().getSchema();
    Schema fileSchema = ParquetSchemaUtil.convert(type); // nameMapping applied (when present)
    if (!sameSchema(table, fileSchema)) {
      throw new BadRequestException(
          String.format("%s's schema doesn't match table's schema", file));
    }

shyiko · 2025-10-21T05:56:29Z

Included in 0.9.0.

Added support for add column and drop column

2d7a0d6

subkanthi linked an issue Jun 30, 2025 that may be closed by this pull request

Add a column to an existing table #39

Closed

subkanthi commented Jul 1, 2025

View reviewed changes

subkanthi marked this pull request as ready for review July 1, 2025 21:14

shyiko requested changes Jul 8, 2025

View reviewed changes

subkanthi added 2 commits July 10, 2025 13:59

Merged changes from main.

a4746bd

Changed add-column, drop-column to alter-table that takes a JSON list…

701a98e

… as arguments.

subkanthi requested a review from shyiko July 11, 2025 15:11

Removed AddColumn, DropColumn classes.

26e15c1

shyiko reviewed Jul 11, 2025

View reviewed changes

subkanthi added 5 commits July 14, 2025 09:11

Replaced semicolon logic to split value with JSON keys. changed opera…

9651e88

…tions name

Changed JSON for add/drop column to include operation JSON key.

c4d10b1

Replace hard coded strings in AlterTable

0d01d9e

Fixed unit test.

f76cbf3

Fixed CLI description.

530f16f

subkanthi requested a review from shyiko July 16, 2025 21:14

subkanthi removed a link to an issue Aug 15, 2025

Add a column to an existing table #39

Closed

Merge branch 'master' of github.com:Altinity/ice into 39-add-a-column…

6ade046

…-to-an-existing-table

shyiko closed this Oct 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added support for add column and drop column#41

Added support for add column and drop column#41
subkanthi wants to merge 10 commits into
masterfrom
39-add-a-column-to-an-existing-table

subkanthi commented Jun 30, 2025 •

edited

Loading

Uh oh!

subkanthi Jul 1, 2025

Uh oh!

subkanthi Jul 1, 2025

Uh oh!

shyiko left a comment •

edited

Loading

Uh oh!

subkanthi commented Jul 10, 2025

Uh oh!

shyiko Jul 11, 2025

Uh oh!

subkanthi Jul 16, 2025 •

edited

Loading

Uh oh!

shyiko commented Aug 18, 2025 •

edited

Loading

Uh oh!

subkanthi commented Aug 20, 2025 •

edited

Loading

Uh oh!

shyiko commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

subkanthi commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

subkanthi Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

subkanthi Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

shyiko left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

subkanthi commented Jul 10, 2025

Uh oh!

shyiko Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

subkanthi Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shyiko commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

subkanthi commented Aug 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shyiko commented Oct 21, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

subkanthi commented Jun 30, 2025 •

edited

Loading

shyiko left a comment •

edited

Loading

subkanthi Jul 16, 2025 •

edited

Loading

shyiko commented Aug 18, 2025 •

edited

Loading

subkanthi commented Aug 20, 2025 •

edited

Loading