Skip to content

Use direct NumPy MAD in the windowed correlation loop#197

Open
sappelhoff wants to merge 2 commits into
mainfrom
perf/mad-direct
Open

Use direct NumPy MAD in the windowed correlation loop#197
sappelhoff wants to merge 2 commits into
mainfrom
perf/mad-direct

Conversation

@sappelhoff

Copy link
Copy Markdown
Owner

Summary

Replace the two scipy.stats.median_abs_deviation(..., axis=1) calls in the windowed loop of find_bad_by_correlation with the equivalent direct NumPy expression np.median(np.abs(x - np.median(x, axis=1, keepdims=True)), axis=1).

Why

median_abs_deviation carries a scipy axis_nan_policy wrapper (broadcast/concatenate bookkeeping) that adds noticeable overhead when called many times on small arrays — here roughly 2× per correlation window (~1100 windows). On an 18.5-min, 99-channel @ 500 Hz recording:

Correctness

The default median_abs_deviation (scale 1.0, center = median) is exactly median(|x − median(x)|):

  • np.array_equal, maxdiff 0.0 against median_abs_deviation in a direct micro-check.
  • Full test suite passes (47 passed), including tests/test_matprep_compare.py.
  • get_bads(as_dict=True) and all _extra_info arrays (including the per-window max_correlations, noise_levels, channel_amplitudes) unchanged on the real recording (reject "omit"/None) and the eegbci fixture (matlab_strict False/True).

Note: the same swap would also speed up find_bad_by_nan_flat and find_bad_by_hfnoise; kept out here to keep the diff minimal — happy to follow up if wanted.

@codecov

codecov Bot commented Jun 14, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.86%. Comparing base (54e2554) to head (5e27631).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #197   +/-   ##
=======================================
  Coverage   97.86%   97.86%           
=======================================
  Files           7        7           
  Lines         842      844    +2     
=======================================
+ Hits          824      826    +2     
  Misses         18       18           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant