Skip to content

Abort should remove incomplete Parquet data files #1247

Description

@fallintoplace

Follow-up from #1241.

That PR changes the failed batch-write path to call Abort() instead of Close(), which is the right control flow for avoiding Parquet footer finalization after a write error. One piece it intentionally leaves alone: ParquetFileWriter.Abort() currently only closes the underlying file handle.

For local files this is mostly just an orphan cleanup concern. For object/blob backends, though, it can leave a footer-less object in storage. Readers should not see it because the snapshot is never committed, but the object can still sit around until orphan cleanup catches it.

Expected behavior:

  • Abort() closes the open writer/file handle.
  • Abort() removes the incomplete file path from the FileIO.
  • If close and remove both fail, both errors are returned, for example with errors.Join.

This also applies to RollingDataWriter's deferred abort path, since it calls the same writer-level Abort() when a stream exits with an open file.

Suggested tests:

  • Abort() removes the created file.
  • remove errors are returned.
  • RollingDataWriter abort removes the incomplete current file, if practical.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions