Skip to content

Add resume functionality and better logging#5

Merged
smcguire-cmu merged 22 commits into
mainfrom
sean/logging
May 28, 2026
Merged

Add resume functionality and better logging#5
smcguire-cmu merged 22 commits into
mainfrom
sean/logging

Conversation

@smcguire-cmu

Copy link
Copy Markdown
Collaborator

Adds a logging module which uses python's logging module to write all pipeline outputs to both stdout and a log file which is configurable by CLI arguments. Updates all logs in the pipeline to use python logging instead of print/typer

Adds resume functionality throughout the pipeline, with each stage touching a completed file and skipping each completed stage.

Also adds individual resuming behavior for post-process and nesting, checking for already completed work, and improves the resume functionality in import stage, and makes sure that all stages are resumable.

@codecov

codecov Bot commented Apr 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 24.75728% with 155 lines in your changes missing coverage. Please review.
✅ Project coverage is 36.47%. Comparing base (caf1464) to head (5e9d08d).
⚠️ Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
src/rubin_dash/stages/nesting.py 11.62% 38 Missing ⚠️
src/rubin_dash/pipeline.py 12.50% 28 Missing ⚠️
src/rubin_dash/stages/postprocess.py 13.33% 26 Missing ⚠️
src/rubin_dash/log.py 25.00% 15 Missing ⚠️
src/rubin_dash/config.py 55.55% 12 Missing ⚠️
src/rubin_dash/cli.py 26.66% 11 Missing ⚠️
src/rubin_dash/stages/collections.py 33.33% 6 Missing ⚠️
src/rubin_dash/stages/crossmatch.py 33.33% 6 Missing ⚠️
src/rubin_dash/stages/butler.py 28.57% 5 Missing ⚠️
src/rubin_dash/stages/raw_sizes.py 33.33% 4 Missing ⚠️
... and 2 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main       #5      +/-   ##
==========================================
- Coverage   37.11%   36.47%   -0.64%     
==========================================
  Files          14       15       +1     
  Lines         741      891     +150     
==========================================
+ Hits          275      325      +50     
- Misses        466      566     +100     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread src/rubin_dash/stages/postprocess.py
Comment thread src/rubin_dash/stages/postprocess.py
@smcguire-cmu smcguire-cmu requested a review from camposandro May 27, 2026 20:52
@smcguire-cmu smcguire-cmu merged commit 9ecaedf into main May 28, 2026
5 of 7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants