Skip to content

fix: only deploy newest build per app in scanExistingJobs #103

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tdussmann wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix: only deploy newest build per app in scanExistingJobs #103

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
74 changes: 54 additions & 20 deletions src/server/services/standalone-services/build-watch.service.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,49 +55,83 @@ class BuildWatchService {
console.log('[BuildWatch] Scanning existing build jobs...');
try {
const jobs = await k3s.batch.listNamespacedJob(BUILD_NAMESPACE);

// Group successful jobs by appId so we only deploy the newest per app.
// Without this, multiple successful jobs for the same app trigger
// sequential deployments in alphabetical order, causing the last-processed
// (often older) job to "win" and roll back the deployment.
const succeededByApp = new Map<string, V1Job[]>();

for (const job of jobs.body.items) {
const jobName = job.metadata?.name;
if (!jobName) continue;

const status = buildService.getJobStatusString(job.status);

if (status === 'FAILED') {
// Mark as processed so watch won't re-handle it
this.processedJobs.add(jobName);
continue;
}

if (status === 'SUCCEEDED') {
// Check if deployment already reflects this build via git commit comparison
const appId = job.metadata?.annotations?.[Constants.QS_ANNOTATION_APP_ID];
const projectId = job.metadata?.annotations?.[Constants.QS_ANNOTATION_PROJECT_ID];
const jobGitCommit = job.metadata?.annotations?.[Constants.QS_ANNOTATION_GIT_COMMIT];

if (!appId || !projectId) {
this.processedJobs.add(jobName);
continue;
}

try {
const deployment = await deploymentService.getDeployment(projectId, appId);
const deployedGitCommit = deployment?.spec?.template?.metadata?.annotations?.[Constants.QS_ANNOTATION_GIT_COMMIT];

if (jobGitCommit && deployedGitCommit && jobGitCommit === deployedGitCommit) {
// Already deployed with this commit
this.processedJobs.add(jobName);
console.log(`[BuildWatch] Job ${jobName} already deployed (commit=${jobGitCommit}), skipping.`);
} else {
// Not yet deployed — trigger deployment
this.processedJobs.add(jobName);
console.log(`[BuildWatch] Job ${jobName} not yet deployed, triggering deployment.`);
await this.handleSucceeded(job);
}
} catch (e) {
console.error(`[BuildWatch] Error checking deployment for app ${appId}:`, e);
this.processedJobs.add(jobName);
const existing = succeededByApp.get(appId);
if (existing) {
existing.push(job);
} else {
succeededByApp.set(appId, [job]);
}
}
}

// For each app, pick the newest successful job (by creationTimestamp)
// and only trigger deployment for that one.
succeededByApp.forEach(async (appJobs, appId) => {
// Sort descending by creationTime: newest first
appJobs.sort((a, b) => {
const timeA = a.metadata?.creationTimestamp
? new Date(a.metadata.creationTimestamp).getTime()
: 0;
const timeB = b.metadata?.creationTimestamp
? new Date(b.metadata.creationTimestamp).getTime()
: 0;
return timeB - timeA;
});

const newestJob = appJobs[0];
const jobName = newestJob.metadata?.name;
if (!jobName) return;

const projectId = newestJob.metadata?.annotations?.[Constants.QS_ANNOTATION_PROJECT_ID];
const jobGitCommit = newestJob.metadata?.annotations?.[Constants.QS_ANNOTATION_GIT_COMMIT];

try {
const deployment = await deploymentService.getDeployment(projectId!, appId);
const deployedGitCommit = deployment?.spec?.template?.metadata?.annotations?.[Constants.QS_ANNOTATION_GIT_COMMIT];

if (jobGitCommit && deployedGitCommit && jobGitCommit === deployedGitCommit) {
console.log(`[BuildWatch] Job ${jobName} already deployed (commit=${jobGitCommit}), skipping.`);
} else {
console.log(`[BuildWatch] Job ${jobName} not yet deployed (newest of ${appJobs.length} job(s) for this app), triggering deployment.`);
await this.handleSucceeded(newestJob);
}
} catch (e) {
console.error(`[BuildWatch] Error checking deployment for app ${appId}:`, e);
}

// Mark all jobs for this app as processed so the watch won't re-handle them
for (const job of appJobs) {
const name = job.metadata?.name;
if (name) this.processedJobs.add(name);
}
});
} catch (e) {
console.error('[BuildWatch] Error during startup scan:', e);
}
Expand Down
Loading