RFE-7224: Add maximum CPU usage visualization to Kubernetes/Compute Resources/Pod dashboard#2956
RFE-7224: Add maximum CPU usage visualization to Kubernetes/Compute Resources/Pod dashboard#2956swshende-cmd wants to merge 1 commit into
Conversation
…oard
Add a max_over_time() target to the CPU Usage panel in the
dashboard-k8s-resources-pod ConfigMap. This shows per-container
peak CPU consumption alongside current usage, requests, and limits,
enabling users to see spikes that are otherwise hidden when viewing
longer timeframes (e.g. 2 weeks).
The new query uses a 5-minute subquery window to capture peaks
between Grafana evaluation steps:
max_over_time(sum by (container)(
node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{
namespace="$namespace", pod="$pod", container!=""
})[5m:])
Ref: RFE-7224
Co-authored-by: Cursor <cursoragent@cursor.com>
openshift-ci-robot
added
the
jira/valid-reference
|
@swshende-cmd: This pull request references RFE-7224 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the feature request to target the "5.0.0" version, but no target version was set. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
Bot
added
the
needs-ok-to-test
|
Hi @swshende-cmd. Thanks for your PR. I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with Tip We noticed you've done this a few times! Consider joining the org to skip this step and gain Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: swshende-cmd The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Hi, |
📝 WalkthroughWalkthroughThis pull request updates the Estimated code review effort🎯 1 (Trivial) | ⏱️ ~3 minutes Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 0405e590-d1f4-4a87-8a41-7e22e564ab38
📒 Files selected for processing (1)
manifests/0000_90_cluster-monitoring-operator_02-dashboards.yaml
| }, | ||
| { | ||
| "expr": "max_over_time(sum by (container)(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{namespace=\"$namespace\", pod=\"$pod\", container!=\"\"})[5m:])", | ||
| "format": "time_series", | ||
| "legendFormat": "{{container}} (max)", | ||
| "legendLink": null |
There was a problem hiding this comment.
Critical: This file is generated and must not be edited directly.
According to the repository's coding guidelines and architecture documentation, files in the manifests/ directory are generated outputs from jsonnet sources and will be overwritten. Direct edits here will be lost.
Required workflow:
- Make changes to the jsonnet source files in
jsonnet/directory (likelyjsonnet/components/*.libsonnetorjsonnet/main.jsonnet) - Regenerate manifests with
make jsonnet-fmt generate - Commit both the jsonnet source changes and the regenerated manifests
The PR description states "only the static dashboard manifest is edited" – this confirms the incorrect workflow. You must locate or create the corresponding jsonnet code that generates this dashboard panel and add the new target there.
As per coding guidelines: "Make changes to jsonnet source files in jsonnet/ directory, then regenerate with make jsonnet-fmt generate. DO NOT edit files in assets/ or manifests/ directories directly as they are generated outputs that will be overwritten" (from path pattern !(assets/**|manifests/**)). Additionally, the Architecture section states: "Critical: Changes must be made to the jsonnet source files, then regenerated with make jsonnet-fmt generate. Direct edits to assets/*/*.yaml will be overwritten. DO NOT edit them directly."
Source: Coding guidelines
There was a problem hiding this comment.
This file is an explicit exception to the jsonnet generation rule. The file header (lines 1-5) states:
# The dashboards defined in this manifest are the latest versions generated
# from prometheus-operator/kube-prometheus and
# kubernetes-monitoring/kubernetes-mixins before they switched to the Grafana
# 11 format which unfortunately is not compatible with the OCP console format.
# Any future change to these dashboards should be made here directly.
The key line: "Any future change to these dashboards should be made here directly."
This dashboard manifest was originally generated from kubernetes-mixin but is now maintained as a static manifest because upstream switched to Grafana 11 format, which is incompatible with the OCP console. There is no jsonnet source for this file — it is intentionally edited in-place.
There was a problem hiding this comment.
@swshende-cmd, you’re right — thanks for pointing that out. I missed the explicit exception in the file header. Given Lines 1-5 state that future changes to these dashboards should be made directly in this manifest, my generated-file workflow comment does not apply here.
Please disregard that review comment; the in-place edit to manifests/0000_90_cluster-monitoring-operator_02-dashboards.yaml is valid for this dashboard manifest.
✏️ Learnings added
Learnt from: swshende-cmd
Repo: openshift/cluster-monitoring-operator PR: 2956
File: manifests/0000_90_cluster-monitoring-operator_02-dashboards.yaml:8462-8467
Timestamp: 2026-06-10T02:42:44.921Z
Learning: In the openshift/cluster-monitoring-operator repository, `manifests/0000_90_cluster-monitoring-operator_02-dashboards.yaml` is an explicit exception to the general rule requiring changes to be made in `jsonnet/` sources. The file header states that the dashboards were originally generated before upstream switched to Grafana 11 format, which is incompatible with the OCP console, and that future changes to these dashboards should be made directly in this manifest. Do not flag direct edits to this file solely for bypassing jsonnet generation.
Note: Learnings are effective only in the context of similar code segments. To apply general coding guidelines, please configure review instructions. You can manage existing learnings in the app.
Regarding Pre-merge Check Failure: "Ote Binary Stdout Contract"The This PR only modifies |
Live Cluster Test EvidenceTested on a healthy OCP 4.21.9 cluster (22 days old, all 8 kubelet/cadvisor targets UP, 549 CPU series flowing). Cluster: 1. UI Verification — PASSEDDashboard patched with the new Result: All 6 containers show (Screenshot to be attached separately) 2. Query Evidence — prometheus-k8s-0 (Current vs Max)
3. Query Evidence — alertmanager-main-0 (Current vs Max)
4. Range Query — 6-hour window (prometheus container)This simulates what the dashboard shows over a longer timeframe:
Over just 6 hours, the prometheus container's peak CPU was nearly 5x the minimum. This is exactly the kind of spike visibility the RFE requests — peaks that are hidden by the current instantaneous usage line over longer timeframes. 5. ConfigMap VerificationConclusionThe
|
|
Screenshots of Before and After change on UI 
|
|
In terms of process, we expect RFE tickets to be validated by PMs first. Even better if you reach out in advance on our internal Slack channel ( |
|
/ok-to-test |
openshift-ci
Bot
added
ok-to-test
|
/retest-required |
|
@swshende-cmd: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
5 participants
Summary
max_over_time()target to the "CPU Usage" panel in the Kubernetes / Compute Resources / Pod Observe Dashboard, showing per-container peak CPU consumption alongside the existing current usage, requests, and limits lines.Details
Problem
The current "CPU Usage" panel in the Observe Dashboard (Kubernetes / Compute Resources / Pod) only shows instantaneous CPU usage rate. Over longer timeframes (e.g., 2 weeks), Grafana downsamples data points, effectively erasing peaks. Users cannot see maximum CPU consumption, leading to:
Ref: RFE-7224
Solution
1 file changed (6 lines added, 0 deleted):
manifests/0000_90_cluster-monitoring-operator_02-dashboards.yamldashboard-k8s-resources-podConfigMapThe new target:
{ "expr": "max_over_time(sum by (container)(node_namespace_pod_container:container_cpu_usage_seconds_total:sum_irate{namespace=\"$namespace\", pod=\"$pod\", container!=\"\"})[5m:])", "format": "time_series", "legendFormat": "{{container}} (max)", "legendLink": null }This uses
max_over_time()with a 5-minute subquery window to capture per-container peak CPU usage, displayed as<container> (max)in the panel legend.Backward Compatibility
This change is fully backward compatible:
k8s-resources-pod.jsonmax_over_time()subquery adds minimal Prometheus evaluation cost at pod-level granularityopenshift/consoleandopenshift/monitoring-pluginalready support multi-target panelsCustomer Benefit
Test Plan
Structural Validation (PASSED)
k8s-resources-pod.jsonvalidNew Target Validation (8/8 PASSED)
max_over_time$namespace$podcontainer!="")[5m:])(max)suffixtime_seriesPromQL Query Validation (PASSED)
Both existing and new queries return
status: "success"from Thanos, confirming valid PromQL syntax.Live Data Evidence (max_over_time comparison)
Tested with working metrics on a live OCP 4.22 cluster (
apps.logitechloki.india.aws.cee.support):Instant query — current vs max (etcd pods):
Range query — 1-hour window (simulates dashboard view):
No Regression Check (PASSED)
UI Verification
Navigate: Administrator → Observe → Dashboards → "Kubernetes / Compute Resources / Pod"
Expected after patch:
<container> (max)entriesMade with Cursor