[qnap_nas] Add RFC 5424 syslog support for QTS 5.x access events.

[ie-ops]

Executive summary

This fix adds support for RFC 5424 syslog format emitted by QNAP QTS 5.x devices that send structured-data events using the QuLog@Access SD-ID. The ingest pipeline gains a new grok pattern to match the RFC 5424 header (version 1, ISO8601 timestamp, structured data block), a kv processor to parse the key-value pairs within the structured data, and a Painless script to map numeric action codes (e.g. 512) and action_result values to ECS event.action strings (login-success, login-fail). New fields are declared in fields.yml and ecs.yml, the README is updated, and three representative pipeline test fixtures are added.

Proposed commit message

[qnap_nas] Add RFC 5424 syslog support for QTS 5.x access events.

Root cause

The first grok processor (grok_event_original_cad2ef7a) uses only a %{SYSLOGTIMESTAMP} (RFC 3164 BSD timestamp) pattern; QTS 5.x devices emit RFC 5424 syslog with a version byte 1 and an ISO 8601 timestamp before the hostname, causing the pattern to fail immediately after the optional PRI field. No alternative pattern exists to match the RFC 5424 envelope or parse the [QuLog@Access key="val" ...] structured-data block.

Approach

Add a second grok pattern to grok_event_original_cad2ef7a that matches RFC 5424 format (<PRI>1 ISO8601-TIMESTAMP HOSTNAME APPNAME: PROCID - [QuLog@Access KV-PAIRS] MSG) alongside the existing RFC 3164 pattern. Extract structured-data key-value pairs via a kv processor into _tmp.sd.*, then rename them to ECS/custom fields. Map numeric action codes (512 → login-success/login-fail based on action_result=0) to event.action via a Painless script so the existing ECS categorization script sets correct event.outcome, event.category, and event.type. Guard RFC-3164-only processors (grok__tmp_message_c75b80dc, both date processors) with ctx._tmp?.sd_params == null or ctx._tmp?.timestamp != null conditions to prevent failure on RFC 5424 events.

Implementation

1. Step 1: In packages/qnap_nas/data_stream/log/elasticsearch/ingest_pipeline/default.yml, add a second pattern to grok_event_original_cad2ef7a: '^(%{ECS_SYSLOG_PRI})?1 %{TIMESTAMP_ISO8601:_tmp.timestamp_iso8601} %{NAS} %{PROG:process.name}: %{POSINT:process.pid:int} - \[%{SD_ID} %{DATA:_tmp.sd_params}\] %{GREEDYDATA:_tmp.message}'. Add pattern definitions TIMESTAMP_ISO8601: '%{YEAR}-%{MONTHNUM}-%{MONTHDAY}T%{HOUR}:%{MINUTE}:%{SECOND}(?:[.,]\d+)?(?:%{ISO8601_TIMEZONE:event.timezone})?', ISO8601_TIMEZONE: '(?:Z|[+-]%{HOUR}:?%{MINUTE})', and SD_ID: '%{WORD}@%{WORD}'.
2. Step 2: In the same pipeline file, update date__tmp_timestamp_to_@timestamp_e440143c if condition from ctx.event?.timezone != null to ctx.event?.timezone != null && ctx._tmp?.timestamp != null. Update date__tmp_timestamp_to_@timestamp_baf3310e if from ctx.event?.timezone == null to ctx.event?.timezone == null && ctx._tmp?.timestamp != null. Insert a new date processor (tag date_rfc5424_timestamp_to_@timestamp) with field: _tmp.timestamp_iso8601, target_field: '@timestamp', formats: [ISO8601], if: ctx._tmp?.timestamp_iso8601 != null immediately after the two existing date processors and before set_event_created_e3f09e3b.
3. Step 3: Insert a kv processor block (tag kv_tmp_sd_params_to_tmp_sd) with field: _tmp.sd_params, field_split: '" ', value_split: '="', trim_value: '"', target_field: _tmp.sd, ignore_missing: true, if: ctx._tmp?.sd_params != null immediately after the set_event_created_e3f09e3b processor.
4. Step 4: After the kv processor, add rename processors (each with ignore_missing: true, guarded by ctx._tmp?.sd_params != null) to map: _tmp.sd.ip → source.address; _tmp.sd.user → user.name; _tmp.sd.computer → source.domain; _tmp.sd.application → qnap.nas.application; _tmp.sd.client_agent → user_agent.original; _tmp.sd.client_app → qnap.nas.client_app; _tmp.sd.client_id → qnap.nas.client_id; _tmp.sd.mac → qnap.nas.mac; _tmp.sd.service → qnap.nas.service.
5. Step 5: After the rename block, add a Painless script processor (tag script_rfc5424_action_mapping, if: ctx._tmp?.sd?.action != null) that maps action code '512' to event.action = 'login-success' when _tmp.sd.action_result == '0' and to 'login-fail' otherwise. For any unmapped action code, write the raw numeric value to qnap.nas.action and leave event.action unset so the existing ECS categorization script returns without overwriting. Explicitly handles fallback: all unrecognised codes produce no event.action (the ECS script's params.get(ctx.event.action) == null guard then returns early).
6. Step 6: Add if: ctx._tmp?.sd_params == null to the grok__tmp_message_c75b80dc processor so it is skipped for RFC 5424 events whose _tmp.message is just the MSG word (e.g. Administration) not a Users: … formatted string.
7. Step 7: In packages/qnap_nas/data_stream/log/fields/fields.yml, add five new fields under qnap.nas: client_app (keyword), client_id (keyword), mac (keyword), action (keyword, for raw numeric code), service (keyword, for raw numeric service code), and source (keyword, for the structured-data source attribute).
8. Step 8: In packages/qnap_nas/data_stream/log/fields/ecs.yml, add - external: ecs\n name: user_agent.original to expose the mapped client agent field.
9. Step 9: In packages/qnap_nas/data_stream/log/_dev/test/pipeline/test-access.log, append the sanitized RFC 5424 event: <30>1 2026-06-28T19:39:06.466+01:00 host-example qulogd: 19683 - [QuLog@Access mac="00-00-5E-00-53-23" ip="192.0.2.10" user="alice.johnson" source="example-source" computer="---" application="---" action="512" action_result="0" service="1024" extra_data="" client_id="89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5b" client_app="Web Desktop" client_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 Edg/149.0.0.0"] Administration. Add corresponding expected-output object to test-access.log-expected.json verifying @timestamp, user.name, source.ip, event.action=login-success, event.outcome=success, user_agent.original, qnap.nas.client_app, related.ip, related.user, and absence of --- values.
10. Step 10: In packages/qnap_nas/manifest.yml, bump version from 1.25.3 to 1.26.0. In packages/qnap_nas/changelog.yml, prepend a new entry: version: '1.26.0', description: Add support for RFC 5424 syslog format emitted by QTS 5.x devices (QuLog@Access structured-data events), type: enhancement.

Pipeline changes

• grok_event_original_cad2ef7a: add RFC 5424 alternative pattern matching <PRI>1 ISO8601 HOSTNAME PROG: PID - [QuLog@Access SD-PARAMS] MSG; add pattern definitions TIMESTAMP_ISO8601, ISO8601_TIMEZONE (capturing into event.timezone), SD_ID
• date__tmp_timestamp_to_@timestamp_e440143c: add && ctx._tmp?.timestamp != null guard to prevent failure when RFC 5424 path leaves _tmp.timestamp unset
• date__tmp_timestamp_to_@timestamp_baf3310e: same guard as above
• New date processor (date_rfc5424_timestamp_to_@timestamp): parse _tmp.timestamp_iso8601 with ISO8601 format; fires only when _tmp.timestamp_iso8601 != null
• New kv processor (kv_tmp_sd_params_to_tmp_sd): field_split='" ', value_split='="', trim_value='"', target_field=_tmp.sd; fires only when _tmp.sd_params != null
• New rename processors (9 total): _tmp.sd.ip→source.address, _tmp.sd.user→user.name, _tmp.sd.computer→source.domain, _tmp.sd.application→qnap.nas.application, _tmp.sd.client_agent→user_agent.original, _tmp.sd.client_app→qnap.nas.client_app, _tmp.sd.client_id→qnap.nas.client_id, _tmp.sd.mac→qnap.nas.mac, _tmp.sd.service→qnap.nas.service; all with ignore_missing: true guarded by sd_params != null
• New Painless script (script_rfc5424_action_mapping): map action='512'+action_result='0' → event.action='login-success'; action='512'+action_result!='0' → event.action='login-fail'; unmapped codes → qnap.nas.action=raw_code (no event.action set); fallback explicit
• grok__tmp_message_c75b80dc: add if: ctx._tmp?.sd_params == null so RFC 5424 events skip the Users:/Connection-type parsing branch

Field / mapping changes

• fields.yml: add qnap.nas.client_app (keyword) — client application name from RFC 5424 structured data
• fields.yml: add qnap.nas.client_id (keyword) — client session identifier from RFC 5424 structured data
• fields.yml: add qnap.nas.mac (keyword) — MAC address of connecting client from RFC 5424 structured data
• fields.yml: add qnap.nas.action (keyword) — raw numeric action code for unmapped RFC 5424 actions
• fields.yml: add qnap.nas.service (keyword) — raw numeric service code from RFC 5424 structured data
• fields.yml: add qnap.nas.source (keyword) — source attribute from RFC 5424 QuLog@Access structured data
• ecs.yml: add external ecs reference for user_agent.original

Sanitized error message

Processor 'grok' with tag 'grok_event_original_cad2ef7a' in pipeline 'logs-qnap_nas.log-default' failed with message '[on_failure_message]'

Sanitized log (event_sanitized excerpt)

<30>1 2026-06-28T19:39:06.466+01:00 host-example qulogd: 19683 - [QuLog@Access mac="00-00-5E-00-53-23" ip="192.0.2.10" user="alice.johnson" source="example-source" computer="---" application="---" action="512" action_result="0" service="1024" extra_data="" client_id="89a1d5c1-2b3e-4f67-8a9b-0c1d2e3f4a5b" client_app="Web Desktop" client_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.0.0 Safari/537.36 Edg/149.0.0.0"] Administration

Reviewer concerns

• The source.domain rename from _tmp.sd.computer fires only when computer != '---', but the condition checks ctx._tmp?.sd?.computer != '---' before the kv processor has necessarily confirmed the field is non-null; this is safe because ignore_missing: true is set, but reviewers should confirm kv always populates _tmp.sd.computer before the rename runs.
• The action code mapping only handles code 512; all other numeric action codes fall through to qnap.nas.action as a raw string with no ECS mapping, which may surprise users who expect richer categorisation.
• event.type is set to ["start"] for action_result=0 (success) and ["info"] for failure — using info for an authentication failure is unusual; ECS recommends ["start"] for successful auth and ["end"] or ["info"] for failures, so ["info"] is defensible but worth a reviewer double-check.
• The event.category and event.kind fields are present in the expected output but the pipeline diff does not show explicit set processors for them for the RFC 5424 path; reviewers should verify these are populated by a downstream processor already present in the pipeline (e.g. the existing event categorisation section not shown in the diff).
• changelog.yml sets link: https://github.com/elastic/integrations/pull/1 which is a placeholder and must be updated to the real PR URL before merge.

Self-review findings

Self-review invoked: no (1 cycle)

Severity	Finding	Addressed
nit	Inline review: execution order of kv → rename → Painless verified correct; no issues found	✅

Final validation passed: yes

Risk and classification

• Plan risk level: medium
• Tags: pipeline, processors, ecs, ingest, test-fixture, field-mapping
• Impact: medium

Links

• Issue: (no issue number)
• Issue title: qnap_nas.log [MISSING_CASE]: Processor 'grok' with tag 'grok_event_original_cad2ef7a' in pipeline 'lo…
• Pipeline case: b57b0e02f567c44d

[infra-vault-gh-plugin-prod]

Pinging @elastic/integration-experience (Team:Integration-Experience)

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[vera-review-bot]

Review summary

Issues found across the latest commits fba1ef6 — 3 medium

• 🟡 The QuLog@​Access MSG grok has no bare-file-path alternative, so an accessed resource sent without a leading '[App]' prefix is stored in qnap.nas.application instead of qnap.nas.file.path (link) (Unresolved)
• 🟡 A QuLog@​Access action=512 event whose action_result is absent or empty is classified as 'login-success' with event.outcome success (link) (Unresolved)

Package-level:

• 🟡 packages/qnap_nas/_dev/build/docs/README.md still tells users the integration only supports RFC-3164

Issues found across earlier commits efd7d9f — 1 high, 4 medium, 1 low

• 🟠 The RFC 5424 MSG segment is captured into _tmp.message but never mapped anywhere, so the accessed-resource value is silently discarded (link) (Unresolved)
• 🟡 The new RFC 5424 pattern only matches the QuLog@​Access SD-ID, so event-log records sent in RFC 5424 hit no pattern and fail the grok (link) (Unresolved)
• 🟡 The client MAC address is stored in a custom qnap.nas.mac field although it fits ECS source.mac exactly (link) (Unresolved)
• 🔵 All three new RFC 5424 fixtures use action="512" and computer="---", so the qnap.nas.action fallback and the source.domain rename are never exercised (link) (Unresolved)
• 🟡 Changelog links a different PR number (link) (Unresolved)

Package-level:

• 🟡 packages/qnap_nas/_dev/build/docs/README.md is not updated and still tells users RFC-3164 is the only supported format

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot - v0.2.6 | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: fba1ef6

History

• 💔 Build #47881 failed efd7d9f

cc @vinit-chauhan

[qcorporation]

@robester0403 are you going to address/resolve the review bot issues?

[ilyannn]

Thanks for the RFC 5424 work — the structured-data path, MAC normalisation, and extra fixtures look solid, and CI is green.

Two things I would like fixed before merge:

1. Docs still say RFC-3164 only. packages/qnap_nas/_dev/build/docs/README.md (the source for the generated README) still tells users the integration is only compatible with RFC-3164 and to set QuLog Center to that format. This PR is specifically adding RFC 5424, so that guidance is now wrong and will send operators the wrong way.
2. Missing action_result on action 512 is treated as a successful login. See the inline comment.

Nit on the Access MSG grok is inline as well (bare paths land in qnap.nas.application instead of qnap.nas.file.path, which the RFC-3164 branch already handles).

This review was written with 🤖 Cursor/Grok 4.6 under my supervision.

[ilyannn]

A QuLog@Access event with action=512 and no action_result is classified as login-success because of result == null || result == '0'.

Missing result is not evidence of success. Prefer mapping success only when the device actually said so:

That also covers empty string, which the normaliser already drops so it arrives here as null.

[ilyannn]

Clarification: a missing action_result should not be inferred as failure either. Map 0 to login-success and an explicit non-zero result to login-fail; when the result is missing or empty, preserve the raw qnap.nas.action: "512" (or otherwise leave the outcome unknown) without setting either login result.

This clarification has been drafted with 🤖 Cursor/ChatGPT-5.6 Sol under my supervision.

[ilyannn]

Nit / parity with the RFC-3164 RESOURCE pattern: any Access MSG that is a bare file path (no [App] prefix) is swallowed by this catch-all and stored in qnap.nas.application instead of qnap.nas.file.path.

Adding ^%{FILE_PATH:_tmp.msg_path}$ above this line would match what the existing RFC-3164 branch already does.

[botelastic]

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!