Description:
When the Envoy Gateway controller runs in the same namespace as the Gateway resources it manages (i.e. the controller's --envoyGatewayNamespace equals the Gateway's namespace), the AI Gateway controller fails to reconcile the Gateway with:
ERROR Reconciler error {"controllerKind": "Gateway", "Gateway": {"name":"ai-gateway","namespace":"inference"}, ...,
"error": "failed to get objects for gateway ai-gateway: found gateway-labeled objects in multiple namespaces: [inference inference]"}
Note the namespace list is [inference inference] — the same namespace reported twice.
Because reconciliation aborts, the controller never regenerates the per-gateway filter-config Secret. After the controller image is upgraded, any freshly rolled extproc pod then crash-loops with:
error: failed to start config watcher: failed to load initial config:
config version mismatch: expected "v1.0.0", got "v0.6.0". Likely in the middle of rolling update
i.e. the data plane can never converge to the new version.
Root cause:
In internal/controller/gateway.go, getObjectsForGateway scans for gateway-labeled pods/deployments/daemonsets across both the Gateway's namespace and the configured Envoy Gateway namespace, without de-duplicating:
var distinctNamespaces []string
for _, ns := range []string{gw.Namespace, c.envoyGatewayNamespace} {
// ... list pods/deployments/daemonsets in ns ...
if len(ps.Items) > 0 || len(ds.Items) > 0 || len(dss.Items) > 0 {
distinctNamespaces = append(distinctNamespaces, ns)
}
}
if len(distinctNamespaces) > 1 {
err = fmt.Errorf("found gateway-labeled objects in multiple namespaces: %v", distinctNamespaces)
return
}When gw.Namespace == c.envoyGatewayNamespace, the loop lists the same namespace twice, finds the same objects on both passes, and appends the namespace twice — so len(distinctNamespaces) == 2 and the guard fires even though everything is in a single, valid namespace.
This is a regression introduced after v0.7.0 (v0.7.0's getObjectsForGateway did a single cluster-wide List and did not have this loop). It affects any deployment where Envoy Gateway is installed in the same namespace as the Gateways it manages — e.g. a Helm install that sets envoyGateway.namespace to the Gateways' namespace.
Repro steps:
- Install Envoy Gateway and the AI Gateway controller in namespace
inference, with --envoyGatewayNamespace=inference.
- Create a
Gateway (GatewayClass = eg) in inference and an AIGatewayRoute attached to it.
- Observe the controller log:
found gateway-labeled objects in multiple namespaces: [inference inference], and the filter-config Secret never updating to the controller's version.
Environment:
- Envoy AI Gateway: v1.0.0 (and current
main)
- Envoy Gateway: v1.7.3
Fix: de-duplicate the namespaces before scanning so a shared namespace is examined once. Genuinely distinct namespaces continue to trip the guard. PR attached.
Description:
When the Envoy Gateway controller runs in the same namespace as the
Gatewayresources it manages (i.e. the controller's--envoyGatewayNamespaceequals the Gateway's namespace), the AI Gateway controller fails to reconcile theGatewaywith:Note the namespace list is
[inference inference]— the same namespace reported twice.Because reconciliation aborts, the controller never regenerates the per-gateway filter-config
Secret. After the controller image is upgraded, any freshly rolled extproc pod then crash-loops with:i.e. the data plane can never converge to the new version.
Root cause:
In
internal/controller/gateway.go,getObjectsForGatewayscans for gateway-labeled pods/deployments/daemonsets across both the Gateway's namespace and the configured Envoy Gateway namespace, without de-duplicating:When
gw.Namespace == c.envoyGatewayNamespace, the loop lists the same namespace twice, finds the same objects on both passes, and appends the namespace twice — solen(distinctNamespaces) == 2and the guard fires even though everything is in a single, valid namespace.This is a regression introduced after v0.7.0 (v0.7.0's
getObjectsForGatewaydid a single cluster-wideListand did not have this loop). It affects any deployment where Envoy Gateway is installed in the same namespace as the Gateways it manages — e.g. a Helm install that setsenvoyGateway.namespaceto the Gateways' namespace.Repro steps:
inference, with--envoyGatewayNamespace=inference.Gateway(GatewayClass =eg) ininferenceand anAIGatewayRouteattached to it.found gateway-labeled objects in multiple namespaces: [inference inference], and the filter-configSecretnever updating to the controller's version.Environment:
main)Fix: de-duplicate the namespaces before scanning so a shared namespace is examined once. Genuinely distinct namespaces continue to trip the guard. PR attached.