feat(worker): defer SERVING until child processes are warm

[enochtangg]

Refs: STREAM-1244

In push mode, a new taskworker pod advertises gRPC health SERVING immediately after server.start(), with zero warm children. Each child must run import_app() + app.load_modules() (~30-40s) before it consumes from the queue. Because the k8s readiness probe and the GCP NEG both trust this gRPC health signal, the pod enters Envoy's routable set ~30s before it can actually serve. As a result, the child queue fills, the worker returns busy, and the broker's bounded push pool jams on the slow pushes.

This PR is responsible for holding the gRPC health service at NOT_SERVING until children are warm. Changes:

• the child_process takes an optional ready_counter (multiprocessing.Value) and increments it once, right before the consume loop. The increment happens after load_modules() and after the SIGTERM/SIGINT handlers are installed, so any child counted as warm can also drain on shutdown.
• Setsready_count in parent thread and passes it to each child.
• On startup, updated the push worker to _await_children_warm() between server.start() and the SERVING flip. Polling ready_count >= min_ready every 250ms tick. This is also bounded by warmup_timeout.

Testing:

• Tested changes locally
• Added unit tests

[linear-code]

STREAM-1244

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 8c73b6f. Configure here.}

[george-sentry]

On one hand, I can see why having this lever could be useful. On the other hand, it adds an extra lever that needs to be configured properly. Why not simply wait for all the children to become ready?

[enochtangg]

You're right, we should reduce the number of levers if possible. The lever allows us to tolerate some level of stuck child threads and enables other threads to start working. All that is speculative for v1 though. I'll remove this as we can always add back if we find it useful in practice in prod.

[george-sentry]

Note that we need to account for livenessProbe and readinessProbe values here. For example, if the livenessProbe is less than this timeout, then the pod may be restarted before the workers have warmed up since we don't set the status to SERVING until they are all ready.

Is this something we should worry about? If so, how should we deal with it?

[enochtangg]

Good call, k8s offers startupProbe so that liveness can't kill the pod mid-warmup. We would just need to make sure it comfortably exceeds warmup_timeout.

[george-sentry]

We can also use a different mechanism for the liveness probe (so that it succeeds right away) while keeping the existing health endpoint for the readiness probe (so that it's not considered ready until SERVING).