[Question] Command chaining

[bnubald]

@JimCircadian Could we set up an approach of chaining requests transparently, then having a final output command that processes them?

While there are cons - it would mean you would lose the independent outputs of each step, but, it would simplify the workflow, config outputs and reduce the storage required.

i.e., instead of outputting a processed_data/, then a processed/, it would just output the final processed/ which would include the regridded dataset and masking as well.

[JimCircadian]

Thanks @bnubald

We can look to do this in the future by managing the removal etc of assets and configurations more intelligently along the way. I don't actually feel this is required at this stage and needs to be optional if ever we do it. The toolbox embodies the unix design philosophy, particularly the principle of single action per command. It's an intentional design so that things can be stitched together in a variety of ways should the user want, or steps easily removed. There's nothing to say that people wouldn't use the processed_data datasets directly, or otherwise. The ML-readiness conversion of processed_data to processed is not the only potential endgame.

We can discuss more, but I don't want to create a complex and rigid workflow, but offer tools to allow people to build them themselves. However, usability can be improved, I appreciate that. If this ever has uptake, hopefully we can build some of this idea into the way we interface with the individual tools / methods and intermediate outputs.