Manifests with all-DELETED entries are not dropped during manifest list construction

[debugmiller]

Apache Iceberg version

main (development)

Please describe the bug 🐞

Manifests with only DELETEs are not pruned when creating new snapshots, causing large numbers of extra manifests to accumulate.

overwriteFiles.existingManifests iterates each parent snapshot manifest with discardDeleted=true, which skips entries with status=DELETED. For a manifest where every entry is DELETED, the iterator yields nothing (since foundDeletedCount=0 and len(notDeleted)=0) which the code interprets as "manifest was unaffected by this operation" and retains instead of dropping it at

iceberg-go/table/snapshot_producers.go

Lines 190 to 194 in a7c4912

	if foundDeletedCount == 0 {
	existingFiles = append(existingFiles, m)
	continue
	}

I think the fix is small, at L190, only retain the manifest if the iterator actually found live entries:

 if foundDeletedCount == 0 {
-    existingFiles = append(existingFiles, m)
+    if len(notDeleted) > 0 {
+        existingFiles = append(existingFiles, m)
+    }
+    // len(notDeleted)==0: all entries are DELETED and were skipped by
+    // discardDeleted=true. The manifest is empty — drop it.
     continue
 }

We observed this when using overwrite on small tables, which produces one DELETE manifest via deletedFilesManifests. Since these are never pruned, they accumulate: after N overwrites the current snapshot's manifest list contains N all-DELETED manifests alongside the live data manifests, causing a growing set of unnecessary snapshots.

[tanmayrauth]

@debugmiller Thanks for the detailed diagnosis, it's spot on for the commit you referenced (a7c4912, May 26), where foundDeletedCount == 0 was checked before len(notDeleted) == 0, so an all-DELETED manifest got retained instead of dropped.

This was fixed on main by #1154 ("Avoid carrying empty overwrite manifests", a286d19, Jun 8), which reorders the two checks so the len(notDeleted) == 0 case is caught first and the empty manifest is dropped — functionally the same as your proposed fix. Current main (table/snapshot_producers.go:191-198) already has this.

a7c4912 predates that fix, so a rebase onto current main should resolve the accumulation. Could you confirm you still see it on latest main?