[CI] use multiple cache from

[junpuf]

Changes

Simplifying the cache source resolution process and potentially increase cache hit by always using 3 --cache-from in the order of priority:

• CACHE_FROM: primary cache source
• CACHE_FROM_BASE_BRANCH: secondary cache source
• CACHE_FROM_MAIN: fallback cache source

Also used compression=zstd option to save ~1min for pushing layers to ECR.

See Multiple caches, Cache compression for details.

Testing

• https://buildkite.com/vllm/ci/builds/41984

docker buildx build --file docker/Dockerfile
  --build-arg max_jobs=16
  --build-arg buildkite_commit=$BUILDKITE_COMMIT
  --build-arg USE_SCCACHE=1
  --build-arg TORCH_CUDA_ARCH_LIST="8.0 8.9 9.0 10.0"
  --build-arg FI_TORCH_CUDA_ARCH_LIST="8.0 8.9 9.0a 10.0a"
  --build-arg VLLM_USE_PRECOMPILED=0
  --cache-to type=registry,ref=936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-test-cache:pr-30074,mode=max
  --cache-from type=registry,ref=936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-test-cache:pr-30074,mode=max
  --cache-from type=registry,ref=936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-postmerge-cache:latest,mode=max
  --cache-from type=registry,ref=936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-postmerge-cache:latest,mode=max
  --tag public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:$BUILDKITE_COMMIT
  --push
  --target test
  --progress plain .
...
#62 exporting to image
#62 DONE 636.5s
...
#65 preparing build cache for export 298.9s done
...
#65 DONE 298.9s

• Total job time: ~33min

  • docker build: ~17min
  • exporting/pushing image/cache: ~16min

• https://buildkite.com/vllm/ci/builds/41997 with compression=zstd to compress the cache output

docker buildx build
  --file docker/Dockerfile 
  --build-arg max_jobs=16 
  --build-arg buildkite_commit=858dc70e304bc2567c3202b340dde54e4a18db2f 
  --build-arg USE_SCCACHE=1 
  --build-arg TORCH_CUDA_ARCH_LIST="8.0 8.9 9.0 10.0" 
  --build-arg FI_TORCH_CUDA_ARCH_LIST="8.0 8.9 9.0a 10.0a" 
  --build-arg VLLM_USE_PRECOMPILED=0 
  --cache-to type=registry,ref=936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-test-cache:pr-30074,mode=max,compression=zstd 
  --cache-from type=registry,ref=936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-test-cache:pr-30074,mode=max 
  --cache-from type=registry,ref=936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-postmerge-cache:latest,mode=max 
  --cache-from type=registry,ref=936637512419.dkr.ecr.us-east-1.amazonaws.com/vllm-ci-postmerge-cache:latest,mode=max 
  --tag public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:858dc70e304bc2567c3202b340dde54e4a18db2f 
  --push 
  --target test 
  --progress plain .
...
#62 exporting to image
#62 pushing manifest for public.ecr.aws/q9t5s3a7/vllm-ci-test-repo:858dc70e304bc2567c3202b340dde54e4a18db2f@sha256:2cb38012927c90d206e215ca3a9fa2bea6ef1635c251872f8eaddd07f99dc4ba 1.2s done
#62 DONE 646.7s
...
#65 preparing build cache for export 219.2s done
#65 writing cache image manifest sha256:267c85bf34c4317e612a9b7c3dae91746e82bb5c5800e60c04796e4a2bd2c669 0.6s done
#65 DONE 219.2s

• Total job time: ~32min
  • docker build: ~17min
  • exporting/pushing image/cache: ~15min

related: vllm-project/vllm#28642

[amrmahdi]

LGTM