Conversation
… cache config The store caches call this setting ttl and the query-frontend response cache calls it expiration. The response cache parses strictly, so a config that says ttl does not quietly fall back to a default, it stops the component from starting with "field ttl not found in type ...ResponseCacheConfig". Accept either name for the memcached and redis response caches, and reject a config that sets both to a non-zero value rather than silently picking one. A key that is present but zero does not count as set. The config is also built as a Go struct and marshalled in places, which emits both keys, and cfggen forbids omitempty on config fields. Signed-off-by: Jeremy Schoemaker <jeremy@shoemoney.com>
…esponse cache config Signed-off-by: Jeremy Schoemaker <jeremy@shoemoney.com>
…esponse cache config Signed-off-by: Jeremy Schoemaker <jeremy@shoemoney.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6996
Changes
The store caches spell the cache lifetime
ttl(pkg/store/cache/factory.go), the query-frontend response cache spells itexpiration, and the response cache parses withyaml.UnmarshalStrict. So a query-frontend config that saysttldoes not quietly fall back to a default, it stops the component from starting:This accepts
ttlas an alias forexpirationin the memcached and redis response cache configs, and errors when both are set instead of silently picking one. It is the shape @MichaHoffmann proposed on the issue: "I would accept both and then error if both were provided i think."Presence is detected by unmarshalling the backend config into a probe struct with pointer fields, because
DefaultRedisConfigalready setsExpiration: 24h, so a plain zero check cannot tell "user set expiration" from "default applied". A key that is present but zero does not count as set: the cache config is built as a Go struct and marshalled in places (test/e2e/query_frontend_test.go), which emits both keys, andscripts/cfggenrejectsomitemptyon config fields, so the conflict check has to key off the value rather than the key.Applied to both memcached and redis on purpose. Fixing only redis, which is what the issue reports, would leave memcached rejecting
ttland create a fresh inconsistency.What this does not do: it does not unify the query-frontend and store cache configs. @yeya24 noted on the issue that the QFE cache config comes from Cortex code and should be unified "at some point". That is a larger refactor and this is the small backward compatible step in the meantime, not a substitute for it.
Verification
Go 1.27.1, darwin/arm64, against
050d688f(main, 2026-09-18). Dated 2026-09-19.New table test
TestNewCacheConfig_TTLAliasinpkg/queryfrontend/config_test.go, red before the change and green after.Red, test added and
config.goreverted. This reproduces the exact error from the issue report:The
expirationcase passing while the rest fail is the control: it shows the test is not failing for an unrelated reason.Green, with the change:
Package suite, measured as a delta rather than a claim of green, because
pkg/queryfrontendis not clean on main in my environment. On050d688fwith no changes,TestLabelsCodec_DecodeResponsepanics (slice bounds out of range [:96] with length 1) andTestLabelsCodec_MergeResponsefails. Both reproduce identically with and without this change. Excluding the panicking test so the run completes, the counts are 8 pass / 1 fail on clean main and 9 pass / 1 fail with this change, the +1 being the new test.go vet ./pkg/queryfrontend/...is clean.gofmtis clean.The first push of this branch got this wrong and CI caught it: treating a present key as set made
TestQueryFrontendMemcachedCachefail withdocker container query-frontend-1 failed to start, because that test builds the config as a Go struct and marshals it, so both keys were emitted with zero values and the component refused to start. That path is now pinned byTestNewCacheConfig_MarshalledStructRoundTrip, which marshals the same struct for both backends and assertsNewCacheConfigaccepts it. I do not have Docker locally, so that unit test is my reproduction of the e2e failure rather than a local e2e run, and CI is the confirmation.golangci-lintv2.4.0, the version pinned in.bingo/golangci-lint.mod, produces byte identical output with and without this change (11 pre-existing typecheck findings, none in the touched files). Those findings are an artifact of my local Go 1.27 toolchain rather than the code: the first one iscould not load export data ... export data version 4 is greater than maximum supported version 2onresponse.pb.go, which cascades into the rest of the package. I could not get a clean lint baseline locally for that reason, so CI is the authority here.Docs:
docs/components/query-frontend.mdembeds generated config viamdox-exec="go run scripts/cfggen/main.go", so both affected blocks were regenerated with that command and the new field documented in prose alongside the existingexpirationsentence.CI
All checks are green on
d6ce8c9.Documentation checkfailed once on this commit with:That was mdox hitting the GitHub pulls API, not a docs diff. The same check passed on the previous push of this branch with the same generated blocks, and it has since been re-run on this same commit and passed, so no change was needed.
All 8
Thanos end-to-end testsshards pass, including shard 2, which is the one that caught the mistake described above, along withThanos unit tests, both linter jobs, CodeQL and codecov.This PR was written in part with the assistance of generative AI.
Supersedes #9039, which GitHub closed automatically after a bad push replaced the branch head with an unrelated commit. The branch has been restored to the reviewed head, unchanged from what CI verified there.