Not Pandas 3.0 compatible?

[larsoner]

pyproject says < 3.0 is required, but it has been out for a couple of months now it would be good to make it compatible:

hed-python/pyproject.toml

Line 49 in 0929da8

"pandas>=2.2.3,<3.0.0",

What's needed for 3.0 compat? I can take a look if there are things that need to be tweaked, we had to make some minor adjustments for MNE-Python I think...

[VisLab]

There were some issues with Pandas 3.0.0, particular for windows, so we deferred. We'll work on this next, as hedtools will clearly have to support Pandas 3.0.0.

[VisLab]

Here is the assessment on the move to Pandas 3.0.0. I'm putting it here for reference. There will be quite a few changes, but they should not break the 1.0.0 interface, so we deferred for a minor version update:

Date: January 28, 2026 (Assessment by Claude Sonnet)
Pandas Version Tested: 3.0.0
Test Results: 1007 tests run, 8 failures, 6 errors

Executive Summary

Pandas 3.0.0 introduces breaking changes that affect the HED-Python codebase. The issues fall into three main categories:

1. ChainedAssignmentError - Pandas 3.0 enforces Copy-on-Write (CoW) more strictly
2. DataFrame API changes - drop() method no longer accepts both axis and columns parameters
3. Type handling changes - NaN/empty cell handling differs (returns float instead of empty string)

Test Results Summary

• Total Tests: 1007
• Passed: 993 (98.6%)
• Failed: 8 (0.8%)
• Errors: 6 (0.6%)
• Skipped: 14

Issues by Category

1. ChainedAssignmentError (6 failures + 1 test)

Affected File: hed/models/df_util.py:51

Problem:

df[column][mask] = df[column][mask].apply(partial(_shrink_defs, hed_schema=hed_schema))

This is classic chained assignment that Pandas 3.0 now raises an error for due to Copy-on-Write (CoW) enforcement.

Affected Tests:

• test_shrink_defs_mixed_tags
• test_shrink_defs_multiple_columns
• test_shrink_defs_multiple_defs_same_line
• test_shrink_defs_normal
• test_shrink_defs_placeholder
• test_get_assembled_strings_def_expand

Fix:

# Instead of: df[column][mask] = df[column][mask].apply(...)
# Use .loc:
df.loc[mask, column] = df[column][mask].apply(partial(_shrink_defs, hed_schema=hed_schema))

Impact: Medium - Affects definition shrinking functionality

2. DataFrame.drop() API Change (1 error)

Affected File: hed/tools/util/data_util.py:64

Problem:

df.drop(columns=delete_cols, axis=1, inplace=True)

Pandas 3.0 no longer allows specifying both columns and axis parameters together.

Affected Test:

• test_delete_columns

Fix:

# Remove axis parameter when using columns parameter
df.drop(columns=delete_cols, inplace=True)

Impact: Low - Simple API change, easy fix

3. NaN/Empty Cell Handling (5 errors + 1 failure)

Problem: Pandas 3.0 changed how it handles missing values in DataFrames. Empty cells now return float (NaN) instead of empty strings in some contexts, causing .strip(), .startswith() calls to fail.

Affected Files:

• Schema conversion utilities (multiple files)
• DataFrame merging utilities

Affected Tests:

• test_from_dataframes - AttributeError: 'float' object has no attribute 'strip'
• test_convert_df_to_omn - AttributeError: 'float' object has no attribute 'strip'
• test_assign_hed_ids_section - AttributeError: 'float' object has no attribute 'startswith'
• test_schema_adding_tag - AttributeError: 'float' object has no attribute 'strip'
• test_schema_conversion_and_update - AttributeError: 'float' object has no attribute 'strip'
• test_fill_missing_values - AssertionError: 0 != ''
• test_empty_and_single_item_series_df - Assertion failure

Fix Approach:
Need to add proper type checking or fillna() before string operations:

# Before string operations, ensure we have strings not floats:
df['column'] = df['column'].fillna('')
# OR
if isinstance(value, str):
    value.strip()

Impact: High - Affects schema conversion and DataFrame operations throughout

Pandas 3.0.0 Breaking Changes

Copy-on-Write (CoW) Enforcement

Pandas 3.0 makes Copy-on-Write the default and mandatory behavior:

• Chained assignment now raises ChainedAssignmentError
• Must use .loc[] for setting values with boolean masks
• Affects all in-place modifications through chaining

API Changes

1. DataFrame.drop(): Can't specify both axis and columns/index parameters
2. Missing value handling: More consistent NaN handling, but returns float instead of empty string
3. Type coercion: Stricter about maintaining dtypes

Recommended Fixes Priority

High Priority (Must Fix)

1. Fix ChainedAssignment in df_util.py (line 51)

   • Use .loc[mask, column] instead of chained indexing
   • Affects 6+ tests

2. Add NaN handling in schema utilities

   • Add .fillna('') before string operations
   • Add type checks before .strip(), .startswith() calls
   • Affects 6+ tests

Medium Priority

3. Fix DataFrame.drop() call in data_util.py (line 64)
   • Remove axis=1 parameter when using columns=
   • Affects 1 test

Backward Compatibility

All fixes should maintain backward compatibility with Pandas 2.x:

• .loc[mask, column] works in both versions
• .fillna('') works in both versions
• Removing axis when using columns works in Pandas 2.2+

Implementation Plan

1. Phase 1: Fix critical ChainedAssignment errors

   • Update hed/models/df_util.py line 51
   • Update any other chained assignments found

2. Phase 2: Fix NaN handling

   • Add .fillna('') in schema conversion code
   • Add type checking before string methods
   • Update DataFrame initialization to specify dtypes

3. Phase 3: Fix API changes

   • Update data_util.py drop() call
   • Check for other deprecated API usage

4. Phase 4: Testing

   • Run full test suite with Pandas 3.0.0
   • Run full test suite with Pandas 2.3.3
   • Verify backward compatibility

Files Requiring Changes

1. hed/models/df_util.py - ChainedAssignment fix (line 51)
2. hed/tools/util/data_util.py - drop() API fix (line 64)
3. hed/schema/ (multiple files) - NaN handling in schema conversion
4. tests/ - May need test updates for changed behavior

Timeline Estimate

• High Priority Fixes: 2-4 hours
• Medium Priority Fixes: 1 hour
• Testing & Validation: 2-3 hours
• Total: ~1 day of work

Notes

• The KeyMap fix we implemented is working correctly with Pandas 3.0.0
• Most tests (98.6%) pass without issues
• Core HED functionality is not affected
• Issues are mainly in utility functions and schema conversion

References

• Pandas 3.0.0 Release Notes: https://pandas.pydata.org/docs/whatsnew/v3.0.0.html
• Copy-on-Write Documentation: https://pandas.pydata.org/docs/user_guide/copy_on_write.html
• Migration Guide: https://pandas.pydata.org/docs/dev/whatsnew/v3.0.0.html#migration-guide

[VisLab]

Fixed in PR #1292. @larsoner will be in next minor release -- unless there is hurry for immediate release.