Add ephys v2 pipeline tables + force-ingestion script

[esutlie]

Summary

• Merges the ephys v2 pipeline from dj_pipeline_ephys (v0.2.0 tag on SWC/aeon_mecha) into the Works pipeline
• Adds force_ingest_external_sorting.py — a 19-step script to ingest Dario's pre-sorted Kilosort 2.5 data into the pipeline tables

New files (from merge)

• ephys.py — probe, epoch, chunk, and block tables
• spike_sorting.py — electrode groups, sorting tasks, spike sorting, curation, SyncedSpikes, UnitMatching
• spike_sorting_curation.py — manual/official curation workflow
• utils/ephys_utils.py — helper functions
• schema/ephys.py — data readers for HarpSync, OnixClock, Bno055

New file (this PR)

• scripts/force_ingest_external_sorting.py — 3-phase ingestion script:
  • Phase A (steps 1-8): Foundation tables (experiment, probe, epoch, chunks, block)
  • Phase B (steps 9-15): Sorting setup + force-insert KS2.5 spike data
  • Phase C (steps 16-19): Auto-approve curation, SyncedSpikes, UnitMatching
  • Includes spike alignment verification (.bin sample count vs pipeline Clock samples)
  • Includes ground truth validation against Dario's pre-computed HARP timestamps

Modified files (from merge)

• __init__.py — adds DJ_SUPPORT_FILEPATH_MANAGEMENT + filepath_checksum_size_limit (required by ephys filepath@dj_store columns)
• pyproject.toml — datajoint pin >=0.14,<2 (was >=0.13.7); adds spike_sorting extras
• paths.py — adds get_sorting_root_dir()
• 7 schema files — import path swc.aeon.schema → swc.aeon.schema.streams (both work at current aeon_api pin; the direct path becomes required in newer versions)

Discussion items

See comments below — posting each as a separate thread for easier discussion.

Test plan

• Static import verification (all files parse, all module references resolve)
• Phase 2 testing (after S3 upload + DB deployment) — tracked separately

[esutlie]

Discussion 1: Data provenance transparency

The force-ingestion script leaves several markers that data was sorted externally:

• execution_duration=0 in PreProcessing, SpikeSorting, PostProcessing
• Empty File part tables (PreProcessing.File, SpikeSorting.File, PostProcessing.File)
• "External KS2.5 sorting — data sorted outside the pipeline" note in SortingParamSet params

If the goal is to publish this data publicly through the platform, should we:

1. Keep the markers for internal traceability (current approach)
2. Remove them so the data looks indistinguishable from pipeline-processed data
3. Something else?

[esutlie]

Discussion 2: File part tables for force-ingested data

PreProcessing.File, SpikeSorting.File, and PostProcessing.File are left empty because the pipeline expects SpikeInterface wrapper objects (si_recording.pkl, si_sorting.pkl, sorting_analyzer/) that don't exist for this externally-sorted data.

The raw KS2.5 output files (spike_times.npy, templates.npy, etc.) DO exist on ceph. We could copy them into the expected directory structure and register them in SpikeSorting.File, but they wouldn't be loadable as SI objects.

Options:

1. Leave empty (current approach) — simplest, but downstream code expecting files will get nothing
2. Register raw KS files — files exist but aren't SI-compatible
3. Something else?

[esutlie]

Discussion 3: S3 upload — which files are included?

The force-ingestion script needs the small KS output files (spike_times.npy, spike_clusters.npy, templates.npy, cluster_KSLabel.tsv) but not the large .bin or temp_wh.dat files (TB-scale each).

Which files does the existing upload script include/exclude? In particular, will Dario's spike_index_harp_clock_binary_2_147.npy files be uploaded? We're using those as ground truth to validate that our spike time conversion is correct — they contain HARP-synchronized timestamps that Dario pre-computed independently.