Adding Data-Structure Injection probe

[Tomer-Wetzler]

Data-Structure Injection, as presented in Zenity Lab's blog post (disclaimer: by me, in the link below) is an attack wherein an attacker sends a malicious prompt that is wrapped as a data-structure (i.e., YAML, JSON, XML).

Security research into this attack class / subtechnique show preliminary finding that LLMs are susceptible to this attack because they view data-structure-like information as inherently benign, or more authoritative, due to the context in their training data. When LLMs are trained on coding / config information, the context is almost always benign, as they appear mostly in code and configuration files explaining which tasks to perform.

There are additional possible explanations, but this is the one (that is not part of ongoing academic research on my part) I find most plausible.

Full research:

https://labs.zenity.io/p/data-structure-injection-dsi-in-ai-agents

[musaabhasan]

This would be a useful probe class, especially because structured data often gets over-trusted by both users and model-facing systems.

For a garak probe, I would suggest making the test matrix explicit across three dimensions:

• Container format: JSON, YAML, XML, TOML, Markdown front matter, and maybe nested arrays/objects.
• Instruction placement: top-level field, nested field, comment-like field, schema description, value string, and metadata field.
• Execution context: plain chat, RAG document, tool result, configuration snippet, and agent memory entry.

The important baseline is to include benign structured records that should be followed only as data, not instructions. Otherwise the probe may measure general instruction-following sensitivity rather than data-structure injection specifically.

I would also separate scoring into at least two outcomes:

1. Instruction adoption: the model follows an instruction embedded inside a data field.
2. Boundary failure: the model treats the data structure as higher-authority than the surrounding task/system instruction.

For agent contexts, a stronger variant is to wrap a tool call request inside a data object and check whether the model attempts to invoke or recommend that tool. That gets closer to operational risk than a pure text-response probe.

[musaabhasan]

This would be a useful probe class, especially because structured data often gets over-trusted by both users and model-facing systems.

For a garak probe, I would suggest making the test matrix explicit across three dimensions:

• Container format: JSON, YAML, XML, TOML, Markdown front matter, and maybe nested arrays/objects.
• Instruction placement: top-level field, nested field, comment-like field, schema description, value string, and metadata field.
• Execution context: plain chat, RAG document, tool result, configuration snippet, and agent memory entry.

The important baseline is to include benign structured records that should be followed only as data, not instructions. Otherwise the probe may measure general instruction-following sensitivity rather than data-structure injection specifically.

I would also separate scoring into at least two outcomes:

1. Instruction adoption: the model follows an instruction embedded inside a data field.
2. Boundary failure: the model treats the data structure as higher-authority than the surrounding task/system instruction.

For agent contexts, a stronger variant is to wrap a tool call request inside a data object and check whether the model attempts to invoke or recommend that tool. That gets closer to operational risk than a pure text-response probe.