[Flaky Test] filebeat.tests.system.test_modules.Test test_fileset_file_052_auditd

[elastic-vault-github-plugin-prod]

Flaky Test

• Test Name: test_fileset_file_052_auditd
• Scope: filebeat.tests.system.test_modules.Test
• File: ../../../../.asdf/installs/python/3.9.13/lib/python3.9/unittest/case.py
• Location: ../../../../.asdf/installs/python/3.9.13/lib/python3.9/unittest/case.py:116
• Buildkite Link: https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/4ad58752-c5e9-8149-8d10-068670e04162
• Flaky Instances: 1
• Latest Occurrence: 2026-07-22T04:17:21.779Z

Details

{
  "id": "4ad58752-c5e9-8149-8d10-068670e04162",
  "web_url": "https://buildkite.com/organizations/elastic/analytics/suites/beats/tests/4ad58752-c5e9-8149-8d10-068670e04162",
  "scope": "filebeat.tests.system.test_modules.Test",
  "name": "test_fileset_file_052_auditd",
  "location": "../../../../.asdf/installs/python/3.9.13/lib/python3.9/unittest/case.py:116",
  "file_name": "../../../../.asdf/installs/python/3.9.13/lib/python3.9/unittest/case.py",
  "instances": 1,
  "latest_occurrence_at": "2026-07-22T04:17:21.779Z",
  "most_recent_instance_at": "2026-07-22T04:17:21.779Z",
  "last_resolved_at": null,
  "ownership_team_ids": [],
  "failure_examples_count": 1
}

Failure Examples

Example 1:
Run: https://api.buildkite.com/v2/analytics/organizations/elastic/suites/beats/runs/d8068140-0858-860c-9c8b-4eef22a552cf
Time: 2026-07-22T04:07:36.939Z
Stacktrace:

a = (<test_modules.Test testMethod=test_fileset_file_052_auditd>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../build/ve/linux/lib/python3.9/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
tests/system/test_modules.py:177: in test_fileset_file
    self.run_on_file(
tests/system/test_modules.py:296: in run_on_file
    self.wait_until(lambda: self.es.indices.exists(index=self.index_name),
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_modules.Test testMethod=test_fileset_file_052_auditd>, cond = <function Test.run_on_file.<locals>.<lambda> at 0x75e69cef28b0>, max_timeout = 20
poll_interval = 0.1
name = 'indices present for /opt/buildkite-agent/builds/bk-agent-prod-gcp-1784693080895481667/elastic/filebeat/filebeat/module/auditd/log/test/execve.log'
err_msg = ''

    def wait_until(self, cond, max_timeout=20, poll_interval=0.1, name="cond", err_msg=""):
        """
        TODO: this can probably be a "wait_until_output_count", among other things, since that could actually use `self`, and this can become an internal function
        Waits until the cond function returns true,
        or until the max_timeout is reached. Calls the cond
        function every poll_interval seconds.
    
        If the max_timeout is reached before cond() returns
        true, an exception is raised.
        """
        start = datetime.now()
        while not cond():
            if datetime.now() - start > timedelta(seconds=max_timeout):
                print("Test has failed, here are the Beat logs")
                if self.output_lines() == 0:
                    print("\n\nBeat had no output file")
                else:
                    print("\n\nHere is the beat's output file:")
                    for entry in self.read_output():
                        print(entry)
>               raise WaitTimeoutError(
                    f"Timeout waiting for condition '{name}'. Waited {max_timeout} seconds: {err_msg}")
E               beat.beat.WaitTimeoutError: Timeout waiting for condition 'indices present for /opt/buildkite-agent/builds/bk-agent-prod-gcp-1784693080895481667/elastic/filebeat/filebeat/module/auditd/log/test/execve.log'. Waited 20 seconds:

../libbeat/tests/system/beat/beat.py:453: WaitTimeoutError

[botelastic]

This issue doesn't have a Team:<team> label.

[github-actions]

TL;DR: The likely flake is in the filestream shutdown condition, not auditd parsing: the harness stops Filebeat when the total indexed count reaches the golden count, which can happen before the first expected auditd offsets have been flushed. The numeric test suffix is also not a stable fixture identifier, so the failing input should be recorded explicitly.

Recommendation

Harden test_fileset_file so the filestream path waits for the expected auditd event set (or at least the expected offset range) before stopping Filebeat, rather than only checking total document count. Keep the 60-second timeout as the failure boundary and prefer graceful termination after the condition is met. Also include the fixture basename in the test output/diagnostic so test_fileset_file_052_auditd identifies the actual input.

Findings

• filebeat/tests/system/test_modules.py:156 uses parameterized.expand(load_fileset_test_cases), while load_fileset_test_cases() enumerates with unsorted os.listdir() and glob.glob() at :96-126. Therefore the 052 suffix is an ordinal, not a stable auditd fixture name; in this checkout the auditd cases are ordinals 017-026.
• The filestream path removes --once at filebeat/tests/system/test_modules.py:212-217 and calls _wait_for_events_then_stop() at :281-285. That helper breaks when _indexed_event_count() >= expected at :328-342, then force-stops Filebeat. A total count of 100 does not prove that the first 100 input offsets—which are the documents selected by the sorted search at :299-303—are all indexed.
• filebeat/module/auditd/log/test/audit-rhel7.log contains 2,447 lines, but its golden file contains 100 events. The golden ends at log.offset: 21157 (filebeat/module/auditd/log/test/audit-rhel7.log:3324 is represented in the expected JSON), making this large fixture particularly exposed to early shutdown/output reordering.
• Auditd records are intentionally emitted as individual events by filebeat/module/auditd/log/ingest/pipeline.yml:16-22; the failure should not be addressed by merging audit records without evidence.

Verification

• python3 -m py_compile filebeat/tests/system/test_modules.py — passed.
• Fixture inspection found 10 auditd log fixtures. Their expected event counts are 10, 12, 100, 1, 6, 3, 91, 4, 15, and 8; audit-rhel7.log is the only 2,447-line fixture with a 100-event golden.
• The integration test was not run because it requires the Filebeat test binary and Elasticsearch services.

Detailed Action Plan

1. Run the auditd module case repeatedly with RUN_AS_FILESTREAM=true, logging module/fileset/test_file before launch, to identify whether the reported failure is audit-rhel7.log or another fixture.
2. Add a helper in filebeat/tests/system/test_modules.py that loads the golden offsets and make _wait_for_events_then_stop() wait until those offsets are present (or until the expected offset range is fully indexed), not merely until total count reaches the golden length.
3. Replace the unconditional post-condition force-stop with graceful process termination after the expected documents are flushed, retaining a force-stop fallback only after a bounded wait.
4. Make fixture enumeration deterministic (sorted(os.listdir(...)) and sorted(glob.glob(...))) and print the fixture basename in the parameterized test diagnostics.
5. Re-run the targeted auditd cases in both normal and filestream modes, including repeated runs of the large fixture.

Related Items

Type	Location	Relevance
Harness	filebeat/tests/system/test_modules.py:328-342	Count-only wait followed by process termination
Harness	filebeat/tests/system/test_modules.py:96-126	Unsorted fixture enumeration and unstable ordinal names
Fixture	filebeat/module/auditd/log/test/audit-rhel7.log	2,447 input lines
Golden	filebeat/module/auditd/log/test/audit-rhel7.log-expected.json	Only 100 expected events; final expected offset 21157
Pipeline	filebeat/module/auditd/log/ingest/pipeline.yml:16-22	One audit record per event behavior

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.