This issue tracks the changes I made to benchmark the co-execution runner. I have grouped related changes to reduce the number of pull requests while also keeping each relatively easy to review and understand.
I will fill in the actual PR numbers once I open them.
PR 1 — Co-execution deadlocks & output collection (critical)
The Pulsar co-execution runner uses two runnables: one that runs the tool itself, and a second sidecar that stages input data and collects results. Both were run as foreground runnables (the default) and GCP Batch runs foreground runnable in sequence waiting for the first to complete before starting the next.
This PR fixes the sidecar/tool runnable ordering so both containers run, collects outputs written to the working/ directory (including UUID-named datasets), re-raises genuine infrastructure errors instead of swallowing them, and passes docker_extra_volumes (CVMFS) through to the Batch containers.
PR1.1 — Re-raise infrastructure errors.
Split out from #466 and now just re-raises infrastructure errors (ImportError, MemoryError, SystemError, and infrastructure OSErrors; disk full, I/O errors) so they fail the job . FileNotFound and 403 errors still flow through.
PR 2 — Custom VM boot disk image & size
Adds custom_vm_image and boot_disk_size_gb to GcpJobParams so jobs can boot
from a custom image (e.g. one with CVMFS pre-installed) and size the boot disk to fit it. Pairs with a small companion change in Galaxy's PulsarGcpBatchJobRunner (PASSTHROUGH_PARAMS) so the image URI can be set once at the runner level.
PR 3 — Dynamic VM sizing & job lifecycle
Sizes the VM from TPV cores/mem, injects GALAXY_SLOTS/GALAXY_MEMORY_MB and sets the container compute resource so tools use the whole VM, validates local-SSD counts for N2/N2D machine types, generates collision-free job names with a configurable job_id_prefix, and makes job deletion on cancel optional (plus a kill() for the AMQP client) so jobs can be retained for debugging.
This issue tracks the changes I made to benchmark the co-execution runner. I have grouped related changes to reduce the number of pull requests while also keeping each relatively easy to review and understand.
I will fill in the actual PR numbers once I open them.
PR 1 — Co-execution deadlocks & output collection (critical)
The Pulsar co-execution runner uses two runnables: one that runs the tool itself, and a second sidecar that stages input data and collects results. Both were run as foreground runnables (the default) and GCP Batch runs foreground runnable in sequence waiting for the first to complete before starting the next.
This PR fixes the sidecar/tool runnable ordering so both containers run, collects outputs written to the
working/directory (including UUID-named datasets), re-raises genuine infrastructure errors instead of swallowing them, and passesdocker_extra_volumes(CVMFS) through to the Batch containers.PR1.1 — Re-raise infrastructure errors.
Split out from #466 and now just re-raises infrastructure errors (
ImportError,MemoryError,SystemError, and infrastructureOSErrors; disk full, I/O errors) so they fail the job .FileNotFoundand403errors still flow through.PR 2 — Custom VM boot disk image & size
Adds
custom_vm_imageandboot_disk_size_gbtoGcpJobParamsso jobs can bootfrom a custom image (e.g. one with CVMFS pre-installed) and size the boot disk to fit it. Pairs with a small companion change in Galaxy's
PulsarGcpBatchJobRunner(PASSTHROUGH_PARAMS) so the image URI can be set once at the runner level.PR 3 — Dynamic VM sizing & job lifecycle
Sizes the VM from TPV
cores/mem, injectsGALAXY_SLOTS/GALAXY_MEMORY_MBand sets the container compute resource so tools use the whole VM, validates local-SSD counts for N2/N2D machine types, generates collision-free job names with a configurablejob_id_prefix, and makes job deletion on cancel optional (plus akill()for the AMQP client) so jobs can be retained for debugging.