EC DIGIT CSIRC Add-on for Sysdiagnose Analysis Framework (SAF)

Overview

EC DIGIT CSIRC Add-on for Sysdiagnose Analysis Framework (SAF) is a Splunk Technical Addon designed to facilitate the collection and processing of the artefacts generated by the Sysdiagnose Analysis Framework (SAF). This addon provides a set of configurations and scripts to streamline data ingestion and ensure proper handling within the Splunk environment.

Sysdiagnose Analysis Framework (SAF) produces the following type of artefacts:

• Parsed data from a case in JSON or JSONL format. Particularly, this Addon will look into JSONL files only.
• Logs in JSONL format with troubleshooting information from the parsing/analysing process.
• Case information related to a sysdiagnose archive file from the case management.

Features

This Addon maps all inputs to the index digit_sysdiagnose and all events (from all inputs) will obtain the time from the timestamp field.

It maps the SAF artefacts to the following splunk fields:

SAF artefact	SAF folder/file	host	source	sourcetype	input type
Parsed data	parsed_data	host_segment	filename	digit:saf:parseddata:json	Folder monitoring
Logs	logs	host_segment	filename	digit:saf:logs:json	Folder monitoring
Devices/Cases information	cases.json	serial_number	path/to/cases.json	digit:saf:cases:json	Scripted

It extracts the case_id from the path for the source types: digit:saf:parseddata:json and digit:saf:logs:json. So that you can correlate the information with digit:saf:cases:json by using the case_id field. Be aware that, due to digit:saf:cases:json source type comes from a scripted input the host field will also contain the hostname of the computer (Forwarder) where the script executed.

The scripted input attempts to be smart so that it keeps track of the already processed case_id by keeping a cache as json dict, where the key belongs to the serial number of the device and the value is the list of processed case id for that serial number. Example:

{
    "SERIAL1" : ["SERIAL1_20251125_091003", "SERIAL1_20251123_091003"],
    "SERIAL2" : ["SERIAL2_20251125_091003"]
}

If you encounter problems while ingesting cases, you may want to (1) take a look to the dedicated log file ec_digit_saf_ta_read_cases.log and backups ec_digit_saf_ta_read_cases.log.X (where X = 1, ...) and, (2) (selectively) delete the cache (entries) that you will find in the local folder.

Installation Instructions

1. Create a new index named digit_sysdiagnose.
2. Download the SAF Addon package.
3. Extract the contents to your Splunk app directory, typically located at $SPLUNK_HOME/etc/apps/.
4. Restart Splunk to load the new addon.

Instance type	Supported	Required	Description
Search head	Yes	Yes	This add-on provides search-time field extractions.
Indexer	Yes	Optional	Not required if you use heavy forwarders to collect data. Required if you use universal forwarders to collect data.
Heavy forwarder	Yes	See comments	This add-on supports heavy forwarder.
Universal forwarder	Yes	Conditional	This add-on needs the host to have Python installed and wrap the python scripted input in an OS script and declare that new (OS script based) scripted input.

Usage

The Addon comes with default configuration files. You will have to override the default configuration of inputs.conf and ec_digit_saf_ta_settings.conf.

• Copy those files to the local folder.
• Edit inputs.conf so that the monitor stanzas point to the proper folder path with SAF cases (default points to /opt/sysdiagnose/cases) and the host_segment field points to the correct segment according to the configured path (default to third segment, basically cases).
• Edit ec_digit_saf_ta_settings.conf so that the path within the section cases points to the path where to find the cases.json file. By default, at the root of the SAF cases folder (default value /opt/sysdiagnose/cases). Note: You can also tune the logging configuration.
• Make sure the environment variable $SPLUNK_HOME exists and that points to the Splunk installation directory. Otherwise the scripted input will not work. If you override the scripted input in the local folder, the Add-on will still attempt to create the log files ec_digit_saf_ta_read_cases.log.X under the folder /opt/splunk/var/log/splunk, which if it does not exist and the user under which the Splunk instance does not have write privileges will fail to create.

splunk cmd printenv | grep SPLUNK

You are ready to roll!

Support

Feel free to report issues, start new discussions, or create pull requests in Github.

A few tips and tricks to double check everything is ingested properly. Start with:

index=_internal sourcetype=splunkd digit:saf*
| stats count by log_level, component

You will see the warnings and errors, and from which component (DateParserVerbose, LineBreakingProcessor, etc.). That may help you to tune some properties.

It is worth mentioning LineBreakingProcessor because there are some data sources (logarchive is one of them) that contain very long lines. This SPL will help you to identify truncation problems.

index=_internal sourcetype=splunkd digit:saf*   component=LineBreakingProcessor
| rex field=event_message ">=\s*(?<data_length>\d+)"
| stats max(data_length) as data_length_max, min(data_length) as data_length_min, avg(data_length) as data_length_avg by data_source
| sort - data_length_max, data_source

OR
| top data_length_max, data_source

FAQ

Splunk complains about latency issues when ingesting data.

Warning message

Forwarder Ingestion Latency

Root Cause(s):

Indicator 'ingestion_latency_lag_sec' exceeded configured value. The observed value is XYZ. Message from GUID:IP:PORT

Answer

This is likely a false positive as explained in this article : The theoretical indexing latency can be calculated by subtracting the extracted time stamp (_time) from the time at which the event was indexed (_indextime).

This figure will of course be inaccurate when Splunk indexes historic logs. Negative latencies, if observed, usually indicate a system clock difference between the log writers and the indexer.

References

• Sysdiagnose Analysis Framework (SAF)
• Develop Splunk apps for Splunk Cloud Platform or Splunk Enterprise
• Splunk - Source types for add-ons

Licence

This project is released under the European Public Licence https://commission.europa.eu/content/european-union-public-licence_en