feat(fluentbit): fix anagine CloudWatch routing and add url=/report/ filter#58
Merged
Conversation
…filter - Replace single rewrite_tag (Keep true) with two-stage explicit routing to avoid re-emission loop ambiguity: - Stage 1: kube.* (anagine namespace) -> anagine.all (Keep false) - Stage 2: anagine.all -> anagine.filtered (Keep true) - Add dedicated CloudWatch output matching anagine.all so ALL anagine records are captured in CloudWatch (previously only kube.* records were matched, which could be lost when Keep true re-emitted them) - Update the three existing grep filters to match anagine.filtered instead of the old anagine.raw tag - Add 4th grep filter: url=/report/ on anagine.filtered (AND condition) - Update OpenSearch output to match anagine.filtered Pipeline flow: kube.* -> CloudWatch (all non-anagine logs) anagine.all -> CloudWatch (all anagine logs, unfiltered) anagine.filtered -> OpenSearch (INFO + auth user + POST + url=/report/) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Refactors the FluentBit pipeline in ardac1prd to make CloudWatch routing for the ardac-anagine namespace deterministic and to tighten the OpenSearch filter set. Replaces the single Keep true rewrite_tag (which re-injected kube.* records and likely caused inconsistent CloudWatch delivery) with a two-stage tag rewrite, adds a dedicated CloudWatch output for the unfiltered anagine stream, and narrows the OpenSearch filter to url=/report/.
Changes:
- Split anagine routing into two
rewrite_tagstages:kube.*→anagine.all(Keep false) andanagine.all→anagine.filtered(Keep true). - Add a second
cloudwatch_logsoutput matchinganagine.alland re-target existing grep filters and the OpenSearch output fromanagine.rawtoanagine.filtered. - Add a 4th grep filter on
anagine.filteredrequiringurl=/report/.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
esentuna
approved these changes
Jun 1, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two updates to the FluentBit config in
ardac1prd:url=/report/as a 4th AND filter for OpenSearch-bound recordsProblem
The previous
rewrite_tagusedKeep true, which re-emits the originalkube.*record back to the beginning of the filter chain. This can cause unpredictable re-processing behaviour and was the likely reason anagine records were not consistently appearing in CloudWatch.Changes
Two-stage explicit routing (filters)
Replaced the single
rewrite_tag(Keep true) with two stages:anagine_all_reemit)kube.*→anagine.allkube.*anagine_filtered_reemit)anagine.all→anagine.filteredanagine.allfor CloudWatchNew CloudWatch output (outputs)
Added a second
cloudwatch_logsoutput block matchinganagine.allso all anagine records are captured in CloudWatch regardless of whether they pass the OpenSearch filters.Updated grep filters (filters)
Updated the three existing grep filters from
Match anagine.raw→Match anagine.filtered.New grep filter (filters)
Added a 4th AND condition on
anagine.filtered:Resulting pipeline
kube.*anagine.allanagine.filteredmethod=POST+url=/report/