fix: stage case subdirectories into and out of the run directory#67
Merged
Conversation
Per-case staging copied only top-level files (`if item.is_file()`) in both directions — compiled inputs (result -> temp) and run outputs (temp -> result). Any code whose input is a directory tree (e.g. an OpenFOAM case with system/, constant/, 0/) therefore ran without its subdirectories, and outputs written into a subdirectory never reached the output parser (the case showed `done` but the output column was null). Copy subdirectories recursively (shutil.copytree, dirs_exist_ok=True) in both prepare_temp_directories() and the temp->result copy-back. Compilation already recursed (rglob), so only the run-staging was affected. Found while validating an OpenFOAM dam-break wrapper end to end. Add a regression test in test_multiple_input_files.py that runs a case with an input subdirectory and an output subdirectory; verified it fails without the fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR fixes per-case staging in fzr/fzd so that directory-tree inputs and outputs (e.g., OpenFOAM-style system/, constant/, 0/, and output subdirs like postProcessing/) are copied recursively into the temporary run directory and back into the results directory, ensuring calculators run with complete inputs and output parsers can see subdirectory outputs.
Changes:
- Update run staging (results → temp run dir) to recursively copy subdirectories via
shutil.copytree(..., dirs_exist_ok=True). - Update copy-back staging (temp run dir → results) to recursively copy output subdirectories.
- Add a regression test covering input subdirectory staging and output subdirectory copy-back, and document the fix in
NEWS.md.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
fz/helpers.py |
Recursively stages directories both into the temp run dir and back into results to preserve directory-tree cases. |
tests/test_multiple_input_files.py |
Adds regression coverage for subdirectory inputs and outputs during fzr runs. |
NEWS.md |
Documents the staging fix and the new regression test. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
1003
to
+1006
| file_names = [f.name for f in all_files if f.is_file()] | ||
| log_debug(f"🔍 [Thread {thread_id}] {case_name}: Files in tmp_dir before copying: {file_names}") | ||
|
|
||
| # Copy each file from the already-retrieved list | ||
| # Copy each item from the already-retrieved list |
Comment on lines
1659
to
+1663
| files_copied = 0 | ||
| for item in result_dir.iterdir(): | ||
| if item.is_file() and item.name != ".fz_hash": | ||
| if item.name == ".fz_hash": | ||
| continue | ||
| if item.is_dir(): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
fz's per-case staging copied only top-level files (
if item.is_file()) in both directions:prepare_temp_directories()So any code whose input is a directory tree — e.g. an OpenFOAM case with
system/,constant/,0/— ran in a temp dir missing its subdirectories (blockMesh: cannot find system/controlDict), and any output written into a subdirectory (postProcessing/…) never reached the output parser. The case would even reportstatus=donewhile the output column came backnull.Compilation already recursed (
rglob), so only the run-staging was affected — which is whyfzclooked fine butfzr/fzdfailed.Fix
Copy subdirectories recursively (
shutil.copytree(..., dirs_exist_ok=True)) in both staging directions.Test
Added
test_directory_input_with_subdirectoriesintests/test_multiple_input_files.py: a case with a parameterized file in an input subdirectory and a runner that writes into an output subdirectory. Asserts both cases complete and the subdir-produced values are parsed. Verified it fails without the fix and passes with it.How it was found
Validating an OpenFOAM dam-break wrapper end to end (random sampling of obstacle height). With this fix, the wrapper runs through
fzr/fzdand an 8-sample study completes in ~30 s. Example doc comes in a follow-up PR.🤖 Generated with Claude Code