Feat/python port

.github/copilot-instructions.md

@@ -0,0 +1,52 @@
+# IMOS Toolbox - Copilot Instructions
+
+## Project Overview
+IMOS Toolbox converts oceanographic instrument files into quality-controlled IMOS-compliant NetCDF. It supports time-series (moorings) and profile (casts) workflows with batch and GUI usage. MATLAB is the primary language, with Java utilities and a Python build script for standalone compilation.
+
+## Key Paths
+- MATLAB source: `IMOS/`, `Preprocessing/`, `AutomaticQC/`, `Parser/`, `Util/`, `NetCDF/`
+- Java utilities: `Java/`
+- Build tooling: `build.py`
+- Configuration defaults: `toolboxProperties.txt`
+- Tests/scripts: `runalltests.sh`, `runalltests.bat`, `batchTesting.m`
+
+## Conventions and Constraints
+- Prefer MATLAB `.m` edits for core logic; keep functions vectorized where possible.
+- Do not modify or regenerate binaries (`*.exe`, `*.bin`, `imosToolbox_*`) unless explicitly requested.
+- Keep toolbox mode options aligned with existing conventions: `timeSeries` or `profile`.
+- Maintain QC pipeline ordering when editing QC chains; avoid reordering without justification.
+- Use ASCII-only text unless the file already contains Unicode.
+
+## Configuration Patterns
+- Use `toolboxProperties.txt` as the canonical defaults for:
+  - `toolbox.mode`
+  - template directory and date formats
+  - preprocessing and auto-QC chains
+  - visual QC settings
+- If adding a new default setting, document it in `toolboxProperties.txt` and keep names consistent with existing keys.
+
+## Build and Runtime
+- Standalone builds are managed via `build.py` (uses `docopt`, `GitPython`, and `ant` for Java deps).
+- MATLAB version baseline for standalone is R2018b; keep compatibility with that runtime.
+- For Java updates, ensure `Java/` builds via `ant install`.
+
+## Testing
+- Use `runalltests.sh` (Linux) or `runalltests.bat` (Windows) for full test runs.
+- For batch regression, use `batchTesting.m` when appropriate.
+
+## Documentation
+- Prefer linking to the wiki for usage; update `README.md` only for repo-level changes.
+
+## Python Port Roadmap
+- Maintain the Python port plan and progress checklist in `python/ROADMAP.md`.
+- When completing a Python port task, check it off in that roadmap.
+
+## Python Parser Delivery Cycle
+- For parser roadmap work, implement one parser at a time and keep changes scoped to that parser + shared utilities only.
+- After wiring parser code/CLI/docs, always run the local quality gate from `python/`:
+  - `uv run ruff check src tests`
+  - `uv run mypy src`
+  - `uv run pytest -v`
+  - `uv run imos-toolbox --help`
+- Only push when all checks pass locally.
+- After push, monitor GitHub Actions `Python Port CI` and address failures before moving to the next parser.

.github/workflows/python-port-ci.yml

@@ -0,0 +1,47 @@
+name: Python Port CI
+
+on:
+  push:
+    paths:
+      - "python/**"
+      - ".github/workflows/python-port-ci.yml"
+  pull_request:
+    paths:
+      - "python/**"
+      - ".github/workflows/python-port-ci.yml"
+
+jobs:
+  python-port:
+    runs-on: ubuntu-latest
+    defaults:
+      run:
+        working-directory: python
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python 3.14
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.14"
+
+      - name: Install uv
+        run: python -m pip install uv
+
+      - name: Create and sync environment
+        run: |
+          uv venv --python 3.14 .venv
+          uv sync --extra dev
+
+      - name: Verify lock file sync
+        run: uv lock --check
+
+      - name: Lint
+        run: uv run ruff check src tests
+
+      - name: Type check
+        run: uv run mypy src
+
+      - name: Test
+        run: uv run pytest -v

.gitignore

@@ -12,8 +12,22 @@ data/*
 dist/*
 Geomag/sample_coords.txt
 Geomag/sample_out_IGRF13.txt
-.python-version
+/.python-version
+!/python/.python-version
 .standalone_canonical_version
 .build.py@*
 *.nc
 tmp/test_*
+
+# Python port workspace
+python/.venv/
+python/.pytest_cache/
+python/.ruff_cache/
+python/.mypy_cache/
+python/.coverage
+python/.coverage.*
+python/htmlcov/
+python/build/
+python/dist/
+python/*.egg-info/
+python/**/__pycache__/

python/.gitignore

@@ -0,0 +1,22 @@
+# Virtual environment
+.venv/
+
+# Python caches
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+
+# Coverage artifacts
+.coverage
+.coverage.*
+htmlcov/
+
+# Packaging/build artifacts
+build/
+dist/
+*.egg-info/
+
+# Jupyter checkpoints
+.ipynb_checkpoints/

python/.python-version

@@ -0,0 +1 @@
+3.14

python/AW_QAQC_IMOS_QC_OVERLAP.md

@@ -0,0 +1,116 @@
+# AW_QAQC ↔ IMOS Toolbox QC Overlap Analysis
+
+## IMOS QC Flag Scheme (Set 1 — IMOS Standard Flags)
+
+| Flag | Value | Description | Classes |
+|------|-------|-------------|---------|
+| 0 | `raw` | No QC performed | raw |
+| 1 | `good` | Good data | good |
+| 2 | `probablyGood` | Probably good data | probablyGood |
+| 3 | `probablyBad` | Bad data, potentially correctable | suspect, spike, step, dup |
+| 4 | `bad` | Bad data | bad, bound, seq, test, unreal, discont, land |
+| 9 | `missing` | Missing value | missing |
+
+## AW_QAQC → IMOS QC Flag Mapping
+
+| AW Flag | AW Anomaly Type | IMOS Flag | IMOS Class | Overlap Notes |
+|---------|-----------------|-----------|------------|---------------|
+| **Bad** | Large sudden spike | **4 (Bad)** | spike → 3 or bad → 4 | IMOS `imosTimeSeriesSpikeQC` covers this. IMOS maps spikes to flag 3 (probablyBad) by default; AW treats large spikes as outright Bad (4). |
+| **Bad** | Persistent values (flatline) | **4 (Bad)** | — | **No direct IMOS auto-QC equivalent.** IMOS has no flatline/stuck-value detector. Manual QC would flag as `bad`. |
+| **Bad** | Impossible values: out of sensor range | **4 (Bad)** | bound → 4 | Direct overlap: `imosGlobalRangeQC` flags out-of-range values as flag 4 (bad). |
+| **Bad** | Impossible values: zero values | **4 (Bad)** | — | **No specific IMOS auto-QC.** Partially caught by `imosGlobalRangeQC` if 0 is outside the global range, but no dedicated zero-value check. |
+| **Bad** | Impossible values: negative values | **4 (Bad)** | — | **No specific IMOS auto-QC.** Same as above — caught only if the global/regional range excludes negatives. |
+| **Bad** | Date from last maintenance >5 months | **4 (Bad)** | — | **No IMOS equivalent.** IMOS has no maintenance-schedule-based flagging. |
+| **Suspect** | Date from last maintenance >3 months | **3 (probablyBad)** | — | **No IMOS equivalent.** |
+| **Suspect** | Constant offset | **3 (probablyBad)** | step → 3 | Partially covered. IMOS `imosRateOfChangeQC` may detect sudden shifts. Manual QC comment `sensor_drift` is related. No dedicated offset detector. |
+| **Suspect** | RAW values requiring local correction | **3 (probablyBad)** | — | **No IMOS equivalent.** IMOS does not track calibration model status. |
+| **Bad** | Maintenance window | **4 (Bad)** | — | Partially covered by `imosInOutWaterQC` (flags data outside deployment window). No general maintenance-window flag. |
+| **Suspect** | Impossible values: out of range for water type | **3 (probablyBad)** | bound → 4 | Covered by `imosRegionalRangeQC`. IMOS flags these as flag 4 (bad) rather than suspect. |
+| **Suspect** | Sudden shift | **3 (probablyBad)** | step → 3 | Partially via `imosRateOfChangeQC`. Manual QC comment `sensor_instability` applies. |
+| **Suspect** | Drift | **3 (probablyBad)** | — | **No IMOS auto-QC.** Manual QC comment `sensor_drift` is the only coverage. |
+| **Suspect** | High variability / oscillation | **3 (probablyBad)** | — | **No direct IMOS auto-QC.** `imosStationarityQC` is related but specific to ADCP. |
+| **Suspect** | Clusters of spikes | **3 (probablyBad)** | spike → 3 | Partially covered by `imosTimeSeriesSpikeQC` (point-by-point), but no cluster-aware logic. |
+| **Suspect** | Small sudden spike | **3 (probablyBad)** | spike → 3 | Covered by `imosTimeSeriesSpikeQC` (threshold-dependent). |
+| **Suspect** | Missing values | **9 (Missing)** | missing → 9 | IMOS uses flag 9 for missing data. The concept maps, but IMOS does not flag gaps as "suspect" — they are simply marked missing. |
+| **Suspect** | Inter-sensor disagreement | **3 (probablyBad)** | — | **No IMOS auto-QC.** No cross-sensor consistency check exists in the toolbox. |
+
+## Summary of Overlap
+
+### Strong overlap (direct IMOS auto-QC equivalents)
+
+- Out-of-sensor-range → `imosGlobalRangeQC` (flag 4)
+- Out-of-range for water type → `imosRegionalRangeQC` (flag 4)
+- Large/small spikes → `imosTimeSeriesSpikeQC` (flag 3)
+- Missing values → IMOS flag 9
+
+### Partial overlap (related but not exact)
+
+- Sudden shift / constant offset → `imosRateOfChangeQC` (flag 3)
+- Maintenance window → `imosInOutWaterQC` (deployment bounds only)
+- Manual QC comments cover: spike, sensor_drift, climatology_outlier, zero_measurements, sensor_instability, invalid_data
+
+### No IMOS equivalent (gaps)
+
+| AW_QAQC Check | Gap Description |
+|----------------|-----------------|
+| Flatline / persistent values | No stuck-value detector |
+| Zero values (where impossible) | No parameter-specific zero check |
+| Negative values (where impossible) | No parameter-specific sign check |
+| Maintenance schedule flags (>3mo / >5mo) | No metadata-driven maintenance age check |
+| RAW values needing local correction | No calibration status tracking |
+| Drift detection | No trend/slope detector (only manual comment) |
+| High variability / oscillation | No general variance anomaly detector |
+| Spike clusters | No cluster-aware spike logic |
+| Inter-sensor disagreement | No cross-sensor consistency check |
+
+## Key Flag-Level Differences
+
+| Aspect | AW_QAQC | IMOS |
+|--------|---------|------|
+| Flag levels | 2 (Bad, Suspect) | 10 (0–9), effectively 5 used |
+| Spike severity | Distinguishes large vs small | Single spike test, threshold-tunable |
+| "Suspect" mapping | Orange / use with caution | Flag 3 = probablyBad (correctable) |
+| "Bad" mapping | Red / do not use | Flag 4 = bad |
+| Metadata-driven flags | Maintenance age, calibration | Deployment dates only (`imosInOutWaterQC`) |
+
+## IMOS Automated QC Routines Reference
+
+| # | Routine | Description |
+|---|---------|-------------|
+| 0 | `userManualQC` | User manual QC (interactive) |
+| 1 | `imosCorrMagVelocitySetQC` | Correlation magnitude velocity check |
+| 2 | `imosDensityInversionSetQC` | Density inversion check |
+| 3 | `imosEchoIntensitySetQC` | Echo intensity check |
+| 4 | `imosEchoIntensityVelocitySetQC` | Echo intensity velocity check |
+| 5 | `imosEchoRangeSetQC` | Echo range check |
+| 6 | `imosErrorVelocitySetQC` | Error velocity check |
+| 7 | `imosGlobalRangeQC` | Global range check |
+| 8 | `imosHorizontalVelocitySetQC` | Horizontal velocity check |
+| 10 | `imosImpossibleDateQC` | Impossible date check |
+| 11 | `imosImpossibleDepthQC` | Impossible depth check |
+| 12 | `imosImpossibleLocationSetQC` | Impossible location check |
+| 13 | `imosInOutWaterQC` | In/out of water check |
+| 14 | `imosPercentGoodVelocitySetQC` | Percent good velocity check |
+| 15 | `imosRateOfChangeQC` | Rate of change check |
+| 16 | `imosRegionalRangeQC` | Regional range check |
+| 17 | `imosSalinityFromPTQC` | Salinity from P & T check |
+| 18 | `imosSideLobeVelocitySetQC` | Side lobe velocity check |
+| 19 | `imosStationarityQC` | Stationarity check (ADCP) |
+| 20 | `imosSurfaceDetectionByDepthSetQC` | Surface detection by depth check |
+| 21 | `imosTier2ProfileVelocitySetQC` | Tier 2 profile velocity check |
+| 22 | `imosTiltVelocitySetQC` (probably good) | Tilt velocity check (probably good) |
+| 23 | `imosTiltVelocitySetQC` (bad) | Tilt velocity check (bad) |
+| 24 | `imosTimeSeriesSpikeQC` | Time series spike check |
+| 25 | `imosVerticalSpikeQC` | Vertical spike check |
+| 26 | `imosVerticalVelocityQC` | Vertical velocity check |
+
+## IMOS Manual QC Predefined Comments
+
+| Internal Name | Comment |
+|---------------|---------|
+| spike | spike |
+| sensor_drift | Sensor data is drifting |
+| climatology_outlier | Sensor data is inconsistent with climatology |
+| zero_measurements | Sensor data measured is zero |
+| sensor_instability | Instrument instability |
+| invalid_data | unexpected sensor data |