Identify and remediate the remaining Coroot latency incident

Problem and impact

A Coroot latency incident continues to recur after current cluster and deployment health checks recovered. The incident needs a verified cause and remediation; passing deployment checks does not establish that the latency problem is resolved.

Fresh read-only checks on 2026-10-01 show the affected application has healthy instances and no observed availability failure. The previous observed incident resolved automatically, with no open incidents at the 20:04 UTC read, but a new runtime latency incident started at 20:06 UTC and is still open at the subsequent read. Its latency compliance remains below the existing objective, confirming the earlier recovery was temporary. Profiling identifies the runtime process family, but the available aggregate profile does not identify the slow request or establish its cause.

Acceptance criteria

Establish the affected runtime identity and join slow requests to their actual dependencies using existing authorized read-only telemetry.
Reproduce or independently verify the root cause and implement the repair in the repository that owns it.
Verify normal latency against the existing objective and Coroot incident recovery after delivery.
Preserve alerting and access boundaries throughout; do not resolve the incident manually or relax its objective to manufacture clearance.

Remaining work: The runtime trace view returns no traces, so the slow-request/dependency join needed for a verified diagnosis remains unavailable. Profiling partially resolves the earlier process-family attribution gap, but aggregate CPU samples do not establish request latency. Obtain the missing evidence through an already authorized read-only path; any new access must be governed by the platform boundary. Full telemetry and receipts remain in private operator notes.

A fresh read-only Coroot check at 2026-10-01 18:14–18:16 UTC confirms a current high-latency recurrence. Instance availability is healthy, but the existing latency objective is still exceeded. The selected incident trace search returned no traces, and instance metadata still does not provide the process/request attribution needed to establish a repair.

This issue remains open on the existing attribution blocker. The cluster health check and recovered application workloads do not clear this runtime incident. No alert was resolved manually, no objective changed, and no access boundary expanded. Detailed telemetry remains in the private operator receipt.

A fresh read-only check around 2026-10-01 18:43–18:49 UTC shows another current high-latency recurrence. Profiling now identifies the runtime process family, which narrows the earlier attribution gap. The profile aggregates the selected time window and does not establish which request was slow or its cause. The runtime trace view still returned no traces.

The existing latency objective remains in effect. This issue stays open until the slow requests can be joined to their dependencies, the repair is delivered, and recovery is verified. No alert, objective, access boundary, or runtime configuration was changed. Detailed profiling and incident evidence remains in the private operator receipt.

Fresh read-only Coroot evidence at 2026-10-01 20:04 UTC shows the latest observed runtime incident resolved automatically, with no open incidents listed. No alert or objective was changed. The incident-window availability remains 100%, while historical latency compliance remains below its existing objective.

This closes the earlier unknown about that incident's resolution, but not the recurring latency issue: the request/dependency join and verified cause remain unavailable. This issue stays open for durable diagnosis and remediation. Full incident identifiers, times and telemetry remain in private operator notes.

The recurrent runtime latency incident remained open in the fresh Coroot read at 20:34 UTC. The incident's slow/error trace search explicitly returned no eBPF traces. Visible network connectivity and RTT checks were healthy, which does not establish the cause of the latency shortfall.

A bounded read-only sample from two current nodes found completed image pulls lasting 10.805 and 4.103 seconds, plus two sandbox operations lasting 659 and 554 milliseconds. These occurred around 20:16, roughly ten minutes after the incident began, with no parsed runtime operation near onset in that sample. The records lack an individual HTTP duration/destination and Coroot request or trace identity, so they do not causally join those operations to the affected requests. The cause remains UNKNOWN and the issue remains open. No SLO, alert threshold, access or runtime setting was changed.

The latest bounded Coroot read still shows the recurrent latency incident open, with 100% availability. All seven runtime instances report healthy availability and restart checks; network RTT and connectivity checks remain healthy.

The additional onset-window investigation found seven successful sandbox creations lasting 1.75–3.18 seconds and seven container starts lasting 620–795 milliseconds on one node. The isolated metrics error follows that same container's exit by about 79 milliseconds; the other warnings concern network cleanup. This is evidence of slow CRI operations around onset, but there is still no request/trace identity joining them to the Coroot measurements, and it does not establish the cause of the latency incident.

Current normal-reader metrics show no node pressure or failed runtime services. These current measurements cannot rule out historical contention, and one remaining node's retained log starts after onset. The incident stays open pending a causal request/operation join; no latency objective or alert has been relaxed.

Fresh read-only Coroot evidence on 2026-10-02 still shows the recurring runtime latency incident open, with 100% availability and latency compliance below the unchanged objective. The refreshed slow/error trace view explicitly returns no traces.

A broader retained onset-window sample found successful image pulls and slow sandbox creation grouped around scheduled Job startup. Several sandbox operations lasted multiple seconds. This gives a concrete concurrency/startup lead, but no request or trace identity joins those operations to the Coroot SLO measurements, so it does not establish causality.

Current normal-reader node checks show no pressure or failed runtime services. The request/dependency join remains the named blocker; healthy node or deployment checks cannot clear this incident. No alert, latency objective, access boundary or runtime setting was changed. Full telemetry remains private.

The startup-latency investigation now has a direct resource join: all 36 sampled slow sandbox requests match successful Cilium endpoint creation by exact Pod UID, node and timestamps. Every observed creation interval lies inside its corresponding runtime request, and all matched lifecycle records are informational.

Observed endpoint creation took 0.86–6.34 seconds. For 20 of 36 requests, more than half the delay occurred before the first endpoint-creation log. The longest 17.23-second request spent 15.28 seconds in that earlier, unobserved interval; its logged endpoint creation took 1.28 seconds. This does not prove the delay is outside Cilium: earlier CNI/runtime work or admission/queueing could precede the logged handler stage.

The incident remains open. The current network inspections are healthy, but no runtime-instance/request/span join to Coroot is available, and the earlier interval has no proven limiter, lock, storage or provider cause. No tuning, alert suppression, privileged sign-in or credential access was performed. The next useful evidence is a timing boundary inside that earlier interval and a direct join to the incident's requests.

Blocker: Exact Coroot request attribution and the cause of the interval before the first endpoint-creation log remain unverified.

A fresh Coroot warning led to a bounded read-only API-log check using the unchanged scoped reader. Across all three Ready API replicas, the 2026-10-01 22:40 UTC to 2026-10-02 02:10 UTC interval contains 19 long-running watch timeout messages and 16 post-timeout records for ordinary GET requests. The ordinary-request records span several resource families, so the warning cannot be dismissed solely as normal watch expiry. Each log read remained below its fixed 2 MiB cap.

Coroot currently reports the API availability and latency inspections as OK. Those inspections do not erase the observed request timeouts or establish their cause. This is additional incident evidence, not a demonstrated join to the runtime latency incident; the runtime slow-request attribution is still missing. No objective, alert rule, access scope or running resource was changed.

Blocker: Establish the missing runtime request/dependency attribution and independently identify the cause of the observed delays before choosing a repair. The full logs and object identities remain in private operator evidence.

The daily Coroot view was rechecked without an alert-specific filter at 2026-10-02 02:21 UTC. It shows another open runtime latency incident beginning at 02:18:40 UTC: availability is 100%, latency compliance is 91.96% against the unchanged 99% under-500-ms objective, and both burn-rate windows exceed the existing threshold. The new incident's loaded eBPF trace query again reports no traces. The earlier incident observation cannot be treated as lasting recovery.

As an additional independent check, finite service-log reads across all three datastore members returned 193 structured timed records in the interval after 22:40 UTC. Their longest recorded duration was 476 ms. These returned records do not identify a delay large enough to explain the observed API timeouts; they also do not provide complete request tracing or prove the datastore could never stall. No causal repair is established by this negative evidence.

The scheduled daily backup's two uploads associated with the fresh volume-staging warnings have independently reached Completed with all expected bytes transferred. Both temporary persistent volumes were subsequently confirmed absent. The overall backup remains in progress at that read, so this does not yet claim a completed daily backup.

Blocker: The runtime request/dependency join and reproducible delay cause remain unavailable through the currently authorized telemetry. Keep this issue open and the existing objectives intact; normal deployment health and a successfully verified backup fixture are separate evidence.

The scheduled daily backup has now independently reached Completed, at 2026-10-02 02:34:37 UTC, with 20,707 of 20,707 items backed up and no reported errors or warnings. Its two uploads associated with the earlier Coroot volume-staging warnings also completed with all expected bytes transferred; their temporary persistent volumes were confirmed absent through separate scoped reads.

This establishes recovery of that backup activity without manually resolving or suppressing the historical warnings. It does not establish the cause or recovery of the separate runtime latency incident, and it is not a database restore proof.

The latest runtime incident resolved automatically at 2026-10-02 02:26:39 UTC. A subsequent loaded daily census, with all four workload classes selected and no incident-specific filter, reports 0 Critical, 0 Warning, 50 Resolved incidents. No incident was manually resolved and no objective or alert rule was relaxed.

This is current recovery, not a demonstrated permanent repair: runtime latency recurred earlier in this round, and the missing request/dependency attribution remains unresolved. Keep #4329 open for that verified-cause work. Separately, the daily backup completed without reported errors, and the protected backup-fixture acceptance passed after #4346 merged.

The fully loaded Coroot read at 12:37Z still shows the same open critical incident. Availability is 100%; latency compliance is 90.64% against the existing 99%/500ms objective, and both the six-hour and fifteen-minute burn-rate windows remain above the configured threshold. This is not a cleared incident.

All seven existing runtime instances report healthy availability and restart inspections. The client table contains no attribution rows. The existing eBPF trace source, queried over six hours, contains no traces; the configured container-log view also contains no messages. Those empty inventories do not establish that requests are absent or that an observed dependency caused the latency. No profiling, collection, source mapping, threshold or alert setting was changed.

The remaining gate is unchanged: attribute the slow requests to a concrete operation and cause before proposing a durable repair. Aggregate readiness and healthy network checks cannot substitute for that join.

Fresh Coroot readback at 19:11 UTC shows one open critical latency incident, opened at 18:10 UTC, against the unchanged 99% below 500 ms objective. Availability remains healthy; the slow requests still lack the instance, protocol, timing and dependency join needed for a cause-specific repair.

At 19:21 UTC the alert inventory contains two warnings: a controller readiness event and the separately tracked kernel audit messages. Scoped Kubernetes readback finds the controller healthy with both replicas ready and no current warning events. These warnings are not asserted as causes of the latency incident.

An authorized read-only diagnostic captured bounded tails from 11 existing telemetry components, with all captures successful. The sample shows no collector error or dropped-metric report establishing an explanation. Ten tails reached their capture limit, so this is a bounded observation rather than proof of absence across the incident window. No incident, objective, credential, collection setting or production resource was changed. The attribution gate remains unresolved and this issue stays open.

The current runtime latency incident remains open. The latest fully loaded incident sample reports 100% request availability but 93.29% latency compliance against the unchanged 99%-under-500ms objective. The client table still provides no attributable request rows.

The operator evaluation in #4350 adds a finite private runtime-log sample: seven successful captures with unchanged node identities, 411 supported completed pairs, 137 returns within the requested interval, and 16 slow operations above five seconds (14 sandbox starts and two image pulls). One slow sandbox sample joined an exact live Pod UID through its Job and CronJob owners; the other 13 sampled sandbox pods were no longer live. That is workload correlation, not proof of which requests feed the Coroot incident.

The analyzer correctly returns GAPS with CorootAttribution=UNPROVEN: no input stream brackets the entire requested interval, and ordering/malformed/ambiguous evidence remains visible. Fixture controls passed; four independently calculated timestamp deltas matched the report, including a direct source-envelope check of the live workload sample. The observed absence of failed completed pairs is not a clearance claim.

The transient readiness warnings have cleared from the current alert view. The remaining kernel-audit warning is independently confirmed by fresh host logs on #4194. All seven nodes and eleven Flux Kustomizations are Ready, but those aggregate health checks do not clear either issue. No runtime settings, SLO objectives, credentials, or alert rules were changed. Keep #4329 open until request attribution supports a durable repair and post-repair observation.

Post-delivery readback, 2026-10-04 20:42 UTC: #4471 and #4478 are both merged through successful protected production deployments. All seven nodes are Ready and schedulable with no residual ownership-phase markers. All eleven Flux Kustomizations are Ready with force=false, all five Flux controllers are available at their current generations, and all thirty existing persistent-volume claims retain their UIDs and volume bindings. There are no Pending pods or non-Ready Running pods; all five retained Failed pods were already failed before deployment.

The latest fully loaded Coroot incident census still contains one critical runtime latency incident. Its alert view contains three warnings. Two new event warnings have an explicit current-owner readback: the reconciliation cancellation joins the exact live Kustomization UID, which is now Ready at its observed generation; the CronJob deletion warning concerns an already absent old job, while the latest job completed successfully and the exact CronJob UID has no active failure. These observations do not resolve or suppress alert objects.

The remaining kernel warning is current: the independent host sample on #4194 contains 140 fresh rate-limit messages on two nodes during the captured hour. The boot-image rollout hold remains. The runtime sample on #4350 still has incomplete coverage and unproven Coroot request attribution, and #4263 still tracks the storage inconsistency. No aggregate health result is being used to close these unresolved issues.

Fresh loaded Coroot readback at 2026-10-05 02:05 UTC still shows the runtime latency incident open: availability is 100%, but latency compliance is 91.32% against the unchanged 99%-under-500ms objective. This remains an unresolved performance incident; current deployment readiness is not a resolution.

The alert view shows nine warnings. The three new application readiness warnings are historical rollout events: each owning Deployment is now at its observed generation with 2/2 updated, ready and available replicas and NewReplicaSetAvailable. The other warnings include already-known cleanup-not-found events, Dex authentication cleanup errors and kernel audit rate limiting. They have not been dismissed or suppressed in this round.

Request attribution remains the repair gate on this issue. No SLO, runtime setting, credential, alert rule or paid telemetry feature was changed. Keep the incident open until a verified repair and post-delivery latency observation satisfy the existing acceptance criteria.

The live Coroot incident view was refreshed at approximately 02:54 UTC on 2026-10-05. The earlier latency/error incident is now marked resolved at 02:27:44 UTC, but a new high-latency incident opened at 02:48:44 UTC and remains open.

The new incident reports 100% availability and 91.63% latency compliance against the unchanged objective of 99% of requests under 500ms. Both latency burn windows show 8 against a threshold of 6. Its loaded eBPF trace view still returns “No traces found,” so there is no request-level evidence here to identify a durable latency repair.

This is a recurrence, not an incident-clearance receipt. The issue stays open; no objective, inspection, alert or incident was relaxed, suppressed or manually resolved.

Fresh Coroot readback at 03:42 UTC shows incident 5mtxouym resolved at 03:28:43 UTC after 39 minutes. The loaded incidents view reports zero open critical or warning incidents. Availability during that incident was 100%; latency compliance was 91.63% against the unchanged 99%-under-500ms objective.

This is observed recovery, not a root-cause fix. The recurring runtime latency still lacks a request-to-dependency join, so this issue remains open. No SLO, alert rule, or incident state was changed.

Coroot has recurred again: incident dkl6sdpz opened at 04:05:44 UTC and was still open on the 04:23 UTC readback. Availability is 100%; latency compliance is 92.61% against the unchanged 99%-under-500ms objective. The incident's loaded eBPF trace view reports no traces found.

The earlier natural recovery does not establish a durable repair. This issue remains open on the request-to-dependency evidence gap; no threshold, alert rule, or incident state was changed.

The final full-day Coroot check finds another active latency recurrence, starting at 2026-10-05T06:40:43Z. Availability is 100%; observed latency compliance is about 92.7% against the unchanged objective. Its loaded request-trace view again returns no traces, so a causal request-to-dependency join remains unavailable.

The protected Platform deliveries and scoped readiness checks succeeded, but they do not demonstrate clearance of this recurring latency condition. #4329 remains open. No objective, alert or incident was relaxed, suppressed or manually resolved.

The loaded full-day Coroot view at about 10:06 UTC still shows one critical latency incident, opened at 06:40:43 UTC. Availability is 100%; latency compliance is now 90.85% against the unchanged 99%-under-500ms objective. Its current-interval eBPF trace search again completes with no traces found.

The process and dependency panels narrow attribution but still do not join a slow request to its dependency or establish the cause. Aggregate profiling and healthy instances do not substitute for that evidence. This issue remains open; no objective, rule, alert or incident state was changed.

Additional read-only investigation on 2026-10-05: the runtime incident still reports 100% availability and 90.82% latency compliance against its unchanged objective. Its dedicated incident trace view explicitly uses eBPF data and still returns no traces after removal of an unrelated log-alert filter. The global trace page explicitly shows OpenTelemetry integrations only, so its empty result must not be treated as evidence of project-wide eBPF ingestion failure.

The live agent configuration includes a trace export endpoint and has no explicit L7-disable or trace-sampling override in the observed arguments or environment. A bounded two-hour sample, capped at 200 lines per agent pod, contains no matching export, instrumentation, or L7-loss diagnostic. Neither endpoint presence nor absence of errors in this bounded sample establishes successful ingestion or a request/dependency join. Coroot documents eBPF spans as capture of supported outbound requests: https://docs.coroot.com/tracing/ebpf-based-tracing/.

Also qualify the earlier network view: the supplementary ICMP pinger is disabled in both declared and live configuration. Its RTT OK indicator therefore does not prove measured RTT. The separate TCP/connectivity observations do not identify the slow request's cause.

The missing request evidence remains the blocker. No monitoring configuration, permission, sampling setting, or SLO objective was changed, and this investigation does not claim incident recovery.

Fresh incident readback at approximately 13:10 UTC still shows the runtime latency incident open: availability 100%, latency compliance 90.35% against the unchanged 99%-under-500ms objective, with 6h/15m burn rates 10/9 against threshold 6. Structural platform health passes separately; that does not clear this incident.

The existing eBPF CPU profile now identifies this application's process family as containerd and containerd-shim. It contains CRI container-status/statistics handlers, shim task statistics, and snapshot metadata/disk-usage scanning. The runtime's CPU inspection reports no node/container utilization alarm. This establishes runtime ownership and candidate work to correlate; an aggregate CPU profile does not establish which requests breached the latency objective or prove a root cause.

Per-request operation/timing correlation remains the acceptance gap. No profiling collection, permissions, SLO, alert suppression or incident resolution was changed.

Fresh loaded Coroot readback on 2026-10-06: one critical runtime latency incident remains open. Availability is 100%, while latency compliance is 91.43% against the unchanged 99%-under-500ms objective; both observed latency burn windows report 9 against threshold 6. The dedicated application tracing view finishes loading, explicitly identifies eBPF data, and returns no traces.

The application has healthy instance data and visible external dependencies, but those aggregate views still do not join a slow operation to a dependency. This confirms the existing attribution blocker remains current; it does not establish the cause or incident recovery. No objective, permissions, tracing configuration, alert or incident state was changed. Complete receipts stay private.

Fresh loaded readback at approximately 17:05 UTC confirms the runtime latency incident remains open against its existing objective. The incident-scoped slow/error trace query finishes loading and returns no traces from the eBPF tracer. That leaves the required slow-operation/dependency join unavailable; aggregate CPU and healthy-instance views still do not establish the cause or recovery.

The existing attribution blocker remains current. No objective, collection, access, alert or incident state was changed. Complete telemetry receipts remain private.

The current runtime latency investigation now has a verified metric-selection boundary. A read-only check of the deployed Coroot server identifies v1.22.2, and the live latency configuration shows inbound requests (built-in) with the existing 99%-under-500ms objective. The dialog was closed without saving. No custom query was selected.

In that exact server release, SLI construction prefers the application's inbound histogram and falls back to the built-in application histogram when no inbound histogram exists. The fallback recording rule attributes observed client request histograms to the destination application. Consequently, an older configuration comment describing this runtime SLO as outbound image pulls is not sufficient evidence for that attribution. The separate inbound metric constructor also aggregates server-side request histograms by the owning application.

Refreshed live readback at about 23:14 UTC on 6 October confirms a newer active runtime latency incident. Its objective remains unmet while availability is healthy. Existing CPU profiles provide diagnostic leads, and the completed incident query against the existing eBPF tracer reports no traces. CPU samples do not establish request latency or causation; a negative trace query and the empty client table do not establish an absence of slow or inbound requests.

The next required evidence remains the active histogram family and slow-operation/dependency join for the incident window, through the existing authorized read-only telemetry path. The earlier incident's terminal state has not been independently verified. No objective, alert, collection, access or incident state was changed; this is diagnostic progress, not remediation or recovery proof.

Navigation Menu

Repository navigation

Identify and remediate the remaining Coroot latency incident #4329

Problem and impact

Acceptance criteria

15 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Identify and remediate the remaining Coroot latency incident #4329

Description

Problem and impact

Acceptance criteria

Activity

devantler commented on Oct 1, 2026

devantler commented on Oct 1, 2026

devantler commented on Oct 1, 2026

devantler commented on Oct 1, 2026

devantler commented on Oct 1, 2026

devantler commented on Oct 2, 2026

devantler commented on Oct 2, 2026

devantler commented on Oct 2, 2026

devantler commented on Oct 2, 2026

devantler commented on Oct 2, 2026

devantler commented on Oct 2, 2026

15 remaining items

devantler commented on Oct 4, 2026

devantler commented on Oct 4, 2026

devantler commented on Oct 4, 2026

devantler commented on Oct 4, 2026

devantler commented on Oct 5, 2026

devantler commented on Oct 5, 2026

devantler commented on Oct 5, 2026

devantler commented on Oct 5, 2026

devantler commented on Oct 5, 2026

devantler commented on Oct 5, 2026

devantler commented on Oct 5, 2026

devantler commented on Oct 5, 2026

devantler commented on Oct 6, 2026

devantler commented on Oct 6, 2026

devantler commented on Oct 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions