Skip to content

fix: resolve crash when on_csv_write='update' encounters empty column (#1367) - #1419

Open
bhoomikakk wants to merge 2 commits into
mlco2:masterfrom
bhoomikakk:Bhoomika
Open

bhoomikakk wants to merge 2 commits into
mlco2:masterfrom
bhoomikakk:Bhoomika

Conversation

@bhoomikakk

Copy link
Copy Markdown

Fixes #1367

Problem

When on_csv_write="update" is used, pd.read_csv() infers columns with all missing/empty values as float64. On subsequent flushes, attempting df[col].dtype.type(val) fails with ValueError: could not convert string to float when val is a string (e.g. gpu_model="NVIDIA RTX...").

Solution

  • Removed df[col].dtype.type(val) and updated DataFrame columns directly.
  • Added graceful fallback to object dtype (df[col] = df[col].astype(object)) when assigning values to columns whose pandas dtype cannot hold the value (e.g. string values in an all-NaN float64 column).
  • Added unit test test_file_output_out_update_empty_column_second_write in tests/output_methods/test_file.py.

Verification

  • Ran unit tests in tests/output_methods/test_file.py (24 passed).

@bhoomikakk
bhoomikakk requested a review from a team as a code owner September 19, 2026 10:51

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

on_csv_write="update" crashes on second write when a column is empty in all existing rows

1 participant