Skip to content

Fix memory copies getz#120

Merged
steven-murray merged 12 commits into
mainfrom
fix-memory-copies-getz
May 7, 2026
Merged

Fix memory copies getz#120
steven-murray merged 12 commits into
mainfrom
fix-memory-copies-getz

Conversation

@steven-murray

Copy link
Copy Markdown
Contributor

Stacks on #119

Fixes an issue where memory copies were made when using multiple beams.

@codecov

codecov Bot commented May 1, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.88%. Comparing base (f6f93ee) to head (f5cfd85).
⚠️ Report is 9 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #120      +/-   ##
==========================================
+ Coverage   98.46%   98.88%   +0.42%     
==========================================
  Files          22       22              
  Lines         975      989      +14     
  Branches       98      103       +5     
==========================================
+ Hits          960      978      +18     
+ Misses          8        6       -2     
+ Partials        7        5       -2     
Flag Coverage Δ
unittests 78.46% <64.70%> (+0.20%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses excess memory usage when simulating with per-antenna beam selection by avoiding fancy-indexing copies in Z-matrix formation, and extends the chunk-sizing logic with a configurable free-memory safety margin.

Changes:

  • Avoid fancy-indexing beam selection in ZMatrixCalc to prevent large temporary allocations when beam_idx is used.
  • Add a memory_buffer parameter to CPU/GPU simulate() and propagate it into chunk estimation utilities.
  • Adjust chunk-estimation thresholds to account for memory_buffer, and tweak docstring composition in the GPU entrypoint.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 8 comments.

File Description
src/matvis/gpu/gpu.py Adds memory_buffer to GPU simulate and passes it into chunk sizing.
src/matvis/cpu/cpu.py Adds memory_buffer to CPU simulate and documents the new parameter.
src/matvis/core/getz.py Replaces beam fancy-indexing with a per-antenna loop to avoid memory copies.
src/matvis/_utils.py Extends chunk-sizing helpers with memory_buffer and uses it in the stopping criterion.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/matvis/cpu/cpu.py Outdated
Comment thread src/matvis/cpu/cpu.py Outdated
Comment thread src/matvis/cpu/cpu.py
Comment thread src/matvis/_utils.py Outdated
Comment thread src/matvis/_utils.py
Comment thread src/matvis/_utils.py
Comment thread src/matvis/_utils.py
Comment thread src/matvis/gpu/gpu.py Outdated
@steven-murray

Copy link
Copy Markdown
Contributor Author

@copilot please re-review given the rebase

…and GPU simulate

Agent-Logs-Url: https://github.com/HERA-Team/matvis/sessions/b7466356-8902-46eb-b67c-a9b2b60b6fdf

Co-authored-by: steven-murray <1272030+steven-murray@users.noreply.github.com>

Copilot AI commented May 6, 2026

Copy link
Copy Markdown
Contributor

@copilot please re-review given the rebase

Re-reviewed after the rebase. The remaining open items were:

  1. Missing input validation for source_buffer and memory_buffer in cpu/cpu.py
  2. Incomplete validation in gpu/gpu.py (only checked source_buffer > 1.0, missing lower-bound and memory_buffer validation)

Both are now addressed in commit 945ac01 — both functions now raise ValueError when source_buffer or memory_buffer is outside (0, 1], consistent with the validation already in _utils.get_required_chunks. The previously-resolved items (_utils.py validation, docstring fixes, and memory_buffer tests) were already in place from prior commits.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • data.astropy.org
    • Triggering command: /usr/bin/python3 python3 -m pytest tests/test_utils.py tests/test_matvis_cpu.py -x -v _TOKEN&#34;; }; f (dns block)
  • www.astropy.org
    • Triggering command: /usr/bin/python3 python3 -m pytest tests/test_utils.py tests/test_matvis_cpu.py -x -v _TOKEN&#34;; }; f (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

@tyler-a-cox tyler-a-cox left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is all looking good to me. Thanks @steven-murray!

@steven-murray steven-murray merged commit 6eafd90 into main May 7, 2026
15 checks passed
@steven-murray steven-murray deleted the fix-memory-copies-getz branch May 7, 2026 07:58
kartikmandar added a commit to kartikmandar/matvis that referenced this pull request Jun 11, 2026
Integrates ~40 upstream commits (through PR HERA-Team#123) onto the polarized-sky
branch. Conflicts resolved in 4 files:

- core/getz.py: kept the polarized m_matrix Z-path; adopted upstream's
  memory-efficient unpolarized branch (PR HERA-Team#120, beam_idx loop instead of
  fancy indexing) and the reshape-rebind idiom in both branches.
- gpu/gpu.py: kept both upstream's memory_buffer param and our
  stokes/raise_on_negative_flux params.
- wrapper.py: kept both upstream's coord_method/coord_method_params/
  matprod_method params and our stokes/raise_on_negative_flux params,
  in the signature, docstring, and the per-frequency backend call.
- setup.cfg: accepted upstream's deletion (config migrated to
  pyproject.toml). Ported our additions into pyproject.toml: the
  `benchmark` pytest marker, and the E741 lint ignores for coherency.py
  and tests (flake8 -> ruff migration; E741 is in ruff's selected "E"
  group and our Stokes I/Q/U/V variable names trigger it).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants