metrics-apiserver zap logger not respected

[drornir]

The chart doesn't have conditionals on log arguments on metrics-apiserver, meaning the deprecated behavior is the only one that is respected.

The args are here

This is what we have now:

          args:
          - --logtostderr=true
          - --stderrthreshold={{ .Values.logging.metricServer.stderrthreshold }}
          - --v={{ .Values.logging.metricServer.level }}
          - --zap-log-level={{ .Values.logging.metricServer.zapLevel }}
          - --zap-encoder={{ .Values.logging.metricServer.zapEncoder }}
          - --zap-time-encoding={{ .Values.logging.metricServer.zapTimeEncoding }}

But there needs to be some logic to not pass --v and the other deprecated flags. Something like

          args:
          {{- if not .Values.logging.metricServer.useZapLogger }}
          - --logtostderr=true
          - --stderrthreshold={{ .Values.logging.metricServer.stderrthreshold }}
          - --v={{ .Values.logging.metricServer.level }}
          {{- end }}
          - --zap-log-level={{ .Values.logging.metricServer.zapLevel }}
          - --zap-encoder={{ .Values.logging.metricServer.zapEncoder }}
          - --zap-time-encoding={{ .Values.logging.metricServer.zapTimeEncoding }}

Expected Behavior

Logs are in json format when setting these values:

logging:
  metricServer:
    zapLevel: info
    zapEncoder: json

for example after manually editing the deployment to remove the depracated flags:

{"level":"info","ts":"2025-11-25T13:26:54Z","logger":"keda_metrics_adapter","msg":"KEDA Version: 2.18.1"}
{"level":"info","ts":"2025-11-25T13:26:54Z","logger":"keda_metrics_adapter","msg":"Git Commit: 3f590ba6027a67b2d6f79ece605061f6d201c54b"}
{"level":"info","ts":"2025-11-25T13:26:54Z","logger":"keda_metrics_adapter","msg":"Go Version: go1.24.7"}
{"level":"info","ts":"2025-11-25T13:26:54Z","logger":"keda_metrics_adapter","msg":"Go OS/Arch: linux/amd64"}
...
# There are still logs not in json format but I think this is an upstream problem, not a chart problem
I1125 13:26:54.874926       1 secure_serving.go:211] Serving securely on [::]:6443
...

Actual Behavior

Unstructured logs, for example:

2025-11-25T13:50:57.030Z    INFO    keda_metrics_adapter    KEDA Version: 2.18.1
2025-11-25T13:50:57.030Z    INFO    keda_metrics_adapter    Git Commit: 3f590ba6027a67b2d6f79ece605061f6d201c54b
2025-11-25T13:50:57.030Z    INFO    keda_metrics_adapter    Go Version: go1.24.7
2025-11-25T13:50:57.030Z    INFO    keda_metrics_adapter    Go OS/Arch: linux/amd64
...
I1125 13:50:57.330081       1 secure_serving.go:211] Serving securely on [::]:6443
...

Steps to Reproduce the Problem

deploy (or just render) the chart with helm values that include

logging:
  metricServer:
    zapLevel: info
    zapEncoder: json

Specifications

• KEDA Version: 1.28.1
• Platform & Version: irrelevant
• Kubernetes Version: irrelevant
• Scaler(s): irrelevant

[der-eismann]

We just stumbled upon the same issue, JSON logs are quite important for us.

[pierluigilenoci]

The root cause of this issue is a long-standing klog bug (kubernetes/klog#212) where -stderrthreshold is completely ignored when -logtostderr=true (the default). This was fixed in klog v2.140.0 via kubernetes/klog#432 with a new opt-in flag -legacy_stderr_threshold_behavior.

I've opened a draft PR on the main KEDA repo to adopt this fix: kedacore/keda#7568

The PR updates klog to v2.140.0 and sets -legacy_stderr_threshold_behavior=false in all three KEDA binaries (operator, webhooks, adapter), which makes -stderrthreshold work correctly. Users can then set -stderrthreshold=WARNING or -stderrthreshold=ERROR to reduce stderr noise.

[pierluigilenoci]

The underlying klog bug that causes -stderrthreshold to be ignored when -logtostderr=true is being fixed in kedacore/keda#7568. That PR bumps klog/v2 to v2.140.0 and opts into the corrected behavior.

Once that PR is merged, the --stderrthreshold=ERROR flag in the chart will actually work as expected, and the metrics-apiserver will only send ERROR+ messages to stderr.

[owen-c]

I just tried this on 2.20.1 and it does not seem to be working.

I cannot get json logs without setting --logtostderr=false

     containers:
      - args:
        - --port=8080
        - --secure-port=6443
        - --logtostderr=true
        - --stderrthreshold=ERROR
        - --disable-compression=true
        - --metrics-service-address=keda-operator.keda.svc.cluster.local:9666
        - --client-ca-file=/certs/ca.crt
        - --tls-cert-file=/certs/tls.crt
        - --tls-private-key-file=/certs/tls.key
        - --cert-dir=/certs
        - --v=0
        - --zap-log-level=info
        - --zap-encoder=json
        - --zap-time-encoding=rfc3339
        command:
        - /keda-adapter

produces

2026-06-10T21:29:38.594Z    INFO    keda_metrics_adapter    Starting metrics server
2026-06-10T21:29:38.595Z    INFO    keda_metrics_adapter    KEDA Version: 2.20.1
2026-06-10T21:29:38.595Z    INFO    keda_metrics_adapter    Git Commit: 875675ce5cd1d34772fe00a228a71b8bf05b3b43
2026-06-10T21:29:38.595Z    INFO    keda_metrics_adapter    Go Version: go1.26.2
2026-06-10T21:29:38.595Z    INFO    keda_metrics_adapter    Go OS/Arch: linux/amd64
2026-06-10T21:29:38.595Z    INFO    keda_metrics_adapter    Running on Kubernetes 1.33    {"version": "v1.33.11-eks-40737a8"}
20

      containers:
      - args:
        - --port=8080
        - --secure-port=6443
        - --logtostderr=false
        - --stderrthreshold=ERROR
        - --disable-compression=true
        - --metrics-service-address=keda-operator.keda.svc.cluster.local:9666
        - --client-ca-file=/certs/ca.crt
        - --tls-cert-file=/certs/tls.crt
        - --tls-private-key-file=/certs/tls.key
        - --cert-dir=/certs
        - --v=0
        - --zap-log-level=info
        - --zap-encoder=json
        - --zap-time-encoding=rfc3339
        command:
        - /keda-adapter

produces

{"level":"info","ts":"2026-06-10T21:34:21Z","logger":"keda_metrics_adapter","msg":"Starting metrics server"}
{"level":"info","ts":"2026-06-10T21:34:21Z","logger":"keda_metrics_adapter","msg":"KEDA Version: 2.20.1"}
{"level":"info","ts":"2026-06-10T21:34:21Z","logger":"keda_metrics_adapter","msg":"Git Commit: 875675ce5cd1d34772fe00a228a71b8bf05b3b43"}
{"level":"info","ts":"2026-06-10T21:34:21Z","logger":"keda_metrics_adapter","msg":"Go Version: go1.26.2"}
{"level":"info","ts":"2026-06-10T21:34:21Z","logger":"keda_metrics_adapter","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":"2026-06-10T21:34:21Z","logger":"keda_metrics_adapter","msg":"Running on Kubernetes 1.33","version":"v1.33.11-eks-40737a8"}