Kustomize CI step sometimes fails in konflux CI pipeline

[kubasikus]

As noted by #1156 (comment) , the kustomize step sometimes fails due to intermittent download/install failure. We need to fix this to avoid dev frustration with unreliable CI. This is tracked by https://redhat.atlassian.net/browse/PVO11Y-5255

[kubasikus]

/fs-triage

[kubasikus]

/fs-triage

[fullsend-ai-triage]

Triage Summary

Severity: Medium | Category: Bug (CI reliability)

Problem

The kustomize Makefile target (line 20) downloads and runs the kustomize install script via curl -s ... | bash from the upstream repository at master HEAD. This has no retry logic and the -s flag suppresses curl errors, so transient download failures cause silent corruption leading to CI pipeline failures.

As noted by the reporter, the curl downloads an install script (not the binary directly), so adding retry logic around the install script invocation should be sufficient.

Root Cause Hypothesis

When the download from raw.githubusercontent.com fails or is incomplete (due to network issues, rate limiting, or CDN hiccups), the install script either doesn't execute properly or downloads a corrupted tarball. The lack of retry logic means a single transient failure breaks the entire CI step.

Prior Fix Attempt

PR #1174 attempted a comprehensive fix (version pinning, direct binary download, checksum verification, retry logic) but was closed without merging. Its approach and review feedback provide a good starting point for a new PR.

Recommended Fix

1. Add retry logic around the install script invocation (e.g., 3 attempts with a short delay)
2. Use curl --fail instead of curl -s to detect HTTP errors immediately
3. Consider pinning a specific kustomize version rather than pulling from master for reproducibility

Proposed Test Case

- Run `make kustomize` multiple times to confirm reliable installation
- Verify that transient curl failures trigger retries
- Confirm clear error messaging after retries are exhausted
- Verify `make kustomize-build` succeeds end-to-end

Previous runs

Triage Summary (run 2)

Severity: Medium | Category: Bug (CI reliability)

Problem

The kustomize Makefile target (line 20) downloads and runs the kustomize install script via curl -s ... | bash from the upstream repository at master HEAD. This has no retry logic and the -s flag suppresses curl errors, so transient download failures cause silent corruption leading to tar extraction errors in CI.

As noted in a follow-up comment, the curl downloads an install script (not the binary directly), so adding retry logic around the install script invocation should be sufficient.

Recommended Fix

1. Add retry logic around the install script invocation
2. Use curl --fail instead of curl -s to detect HTTP errors
3. Consider pinning a specific kustomize version

---

Triage Summary (run 1)

Problem

The kustomize Makefile target downloads kustomize at build time via curl -s ... | bash from the upstream kustomize repository. This download intermittently fails, causing CI pipeline failures unrelated to code changes.

Root Cause Hypothesis

No retry logic, no version pinning, and re-downloads on every CI run. Vulnerable to GitHub rate limiting, transient network errors, and upstream script changes.

---

Labels: The pr-open label should be removed since PR #1174 was closed without merging; bug label is already correctly applied.

Previous run

Triage Summary

Severity: Medium | Category: Bug (CI reliability)

Problem

The kustomize Makefile target (line 20) downloads and runs the kustomize install script via curl -s ... | bash from the upstream repository at master HEAD. This has no retry logic and the -s flag suppresses curl errors, so transient download failures cause silent corruption leading to tar extraction errors in CI.

Root Cause Hypothesis

When the download from raw.githubusercontent.com fails or is incomplete (due to network issues, rate limiting, or CDN hiccups), the install script either doesn't execute properly or downloads a corrupted tarball. The lack of retry logic means a single transient failure breaks the entire CI step.

As noted in a follow-up comment, the curl downloads an install script (not the binary directly), so adding retry logic around the install script invocation should be sufficient.

Recommended Fix

1. Add retry logic — wrap the curl | bash invocation in a retry loop (e.g., 3 attempts with a short delay)
2. Use curl --fail instead of curl -s to detect HTTP errors immediately
3. Consider pinning a specific kustomize version rather than pulling from master for reproducibility

Proposed Test Case

- Run `make kustomize` multiple times to confirm reliable installation
- Verify that transient curl failures trigger retries
- Confirm clear error messaging after retries are exhausted
- Verify `make kustomize-build` succeeds end-to-end

Previous run (2)

Triage Summary

Severity: Medium | Category: Bug (CI reliability)

Problem

The kustomize Makefile target downloads kustomize at build time via curl -s ... | bash from the upstream kustomize repository. This download intermittently fails, causing CI pipeline failures unrelated to code changes. The referenced PR comment shows the tar extraction failing because the download was incomplete or missing.

Root Cause Hypothesis

The current approach has no retry logic, no version pinning, and re-downloads on every CI run. This makes it vulnerable to GitHub rate limiting, transient network errors, and upstream script changes.

Recommended Fix

Replace the unreliable dynamic download with a more robust approach:

1. Pin a specific kustomize version instead of pulling from master
2. Add retry logic to the download step
3. Consider using a container image that already includes kustomize, or caching the binary

The simplest fix would be updating the kustomize Makefile target to pin a version and add retries.

Proposed Test Case

- Verify `make kustomize-build` succeeds reliably across multiple runs
- Simulate a download failure and confirm clear error messaging
- If retry logic is added, verify transient failures are retried successfully

---

Labels: This is a CI bug causing intermittent pipeline failures, matching the 'bug' label.

[kubasikus]

The curl downloads an install script, not the kustomize binary directly. I think we just need to retry the install script if it fails.

[kubasikus]

/fs-triage

[fullsend-ai-coder]

An open PR already addresses this issue — skipping automated implementation.

• fix(#1173): make kustomize download reliable with version pinning and retries #1174 by @app/fullsend-ai-coder

To override, comment /fs-code --force on this issue.

_{Posted by fullsend pre-code check}

[kubasikus]

/fs-triage