celld adopts a new deployment without restarting the process. This is great, but it also means that during deployment and version upgrades, it removes the signal a k8 orchestrator would rely on. Therefore, correct deployment and stale fleet look like the same.
In our testing, we had a three-node 0.4.1 fleet where celld deploy exited 0, the pointer advanced, and yet the three nodes were still serving the previous version for 45 minutes. The only way to see that that nodes were refusing to serve the new version was in the /reload response body:
HTTP 422 stateless Worker failed to load: instantiate: SyntaxError: The requested module 'cloudflare:workers' does not provide an export named 'DurableObject'
(To be clear, the underlying error above was fixed in 0.5.0, but the general deploy problem stays.)
An implementation idea would be to put the deployment version in the bucker under nodes/node_<id>.json, since it doesn't carry it yet. It would ensure that orchestrators wanting to check adoption just read the bucket, and don't have to probe the pods or the internal listener or stuff like that.
celldadopts a new deployment without restarting the process. This is great, but it also means that during deployment and version upgrades, it removes the signal a k8 orchestrator would rely on. Therefore, correct deployment and stale fleet look like the same.In our testing, we had a three-node 0.4.1 fleet where
celld deployexited 0, the pointer advanced, and yet the three nodes were still serving the previous version for 45 minutes. The only way to see that that nodes were refusing to serve the new version was in the/reloadresponse body:(To be clear, the underlying error above was fixed in 0.5.0, but the general deploy problem stays.)
An implementation idea would be to put the deployment version in the bucker under
nodes/node_<id>.json, since it doesn't carry it yet. It would ensure that orchestrators wanting to check adoption just read the bucket, and don't have to probe the pods or the internal listener or stuff like that.