Repository navigation
Jenkins sometimes does not show "Resume build" option #4449
Description
Activity
When this happens, collaborators have to start a new CI.
Odd. Both seem to have been started manually by @jasnell and on the same PR so there's no obvious reason why one would show it and the other wouldn't. I'm not sure what we can do about this but I've never noticed the absence of the button before although James mentioned he has seen it before. I've "locked" both of the jobs referenced in that issue for now in case we can do some diagnosis on it:
- https://ci.nodejs.org/job/node-test-pull-request/76776/ (with resume button)
- https://ci.nodejs.org/job/node-test-pull-request/76787/ (no resume button)
I will also note that the way we are doing our PR test builds is with a deprecated/unmaintained jenkins plugin so there will be limited chance of getting it addressed if the failure is there.
Also spotted on two of @codebytere's recent jobs (both on separate PRs): https://ci.nodejs.org/job/node-test-pull-request/76791/ and https://ci.nodejs.org/job/node-test-pull-request/76792/
The ones without the resume button do not seem to have this section in the build's
build.xml:< <com.tikal.jenkins.plugins.multijob.MultiJobResumeBuild> < <run class="com.tikal.jenkins.plugins.multijob.MultiJobBuild" reference="../../.."/> < </com.tikal.jenkins.plugins.multijob.MultiJobResumeBuild>None of the PR test jobs between 76789 and 76795 had the resume link. All of those were started by @codebytere although the one before those at https://ci.nodejs.org/job/node-test-pull-request/76788/ was also by her and has
resume build... although unlike the ones that don't have it 76788 was started from aresume buildlink 🤔I think that in order for the Resume Build button to appear, the
node-test-pull-requestjob has to complete (the spawned child jobs can fail). I think this is related to #4453 -- if thenode-test-pull-requestjob gets disconnected, it doesn't add the Resume Build button.I think @richardlau's read is right from what i can see in the plugin:
MultiJobBuild$MultiJobRunnerImpl.run()only doesgetBuild().addAction(new MultiJobResumeBuild(...))aftersuper.run(listener)returns normally with a non-SUCCESS result. Ifsuper.run()throws, the action is never attached and there's no Resume link.Every no-resume build i checked (76787, 76791, 76792, 76955) has the same ending in its console: the child
node-test-commitfinishes, the parent printsStarting to gather test results!, and thenAbstractBuild$AbstractBuildExecution.run->Launcher$RemoteLauncher.kill->Channel.callthrowsChannelClosedExceptiononJNLP4-connect connection from 67.158.54.159, which istest-mnx-ubuntu2204-x64-1(jenkins-workspace). The parent held an executor there for the whole 2-10 h, the agent's remoting channel reconnected at some point in between (the traces show two differentChannel@…ids across the week), and the end-of-build process-tree kill on the stale channel is what throws. Builds that ended FAILURE or ABORTED without that trace (76776, 76788, 76967, 76993, 77016) all have/resume/; 76966, which i aborted by hand beforenode-test-commithad started, lacks it for the same reason via the abort's interrupt. Who started the run, or whether it came in throughncu-ci, looks like coincidence; what the no-resume builds share is that agent's connection not surviving the wait.The same exception also explains the original report in #4453 (parent red with all children green): the throw happens after the children are collected, so the parent is marked FAILURE regardless of their results.
One thing that made this harder to see from scripts:
MultiJobResumeBuildisn't@Exported, soapi/json?tree=actions[_class]never lists it even when the link is there; checking for/resume/in the build page HTML is the only way i found.Possible directions, for whoever knows the agent setup better than i do: find out why that MNX host's JNLP channel drops (it's also the box from #4442), or pin
node-test-pull-request/node-test-commitparents to an agent local to the controller since they only do the git/rebase work and then sit waiting.Reacted by Mike McCready
Refs: nodejs/node#65652 (comment)