|
| 1 | +--- |
| 2 | +title: System prompt leakage |
| 3 | +type: concept |
| 4 | +tags: [llm-security, prompt-leaking, system-prompt, owasp-llm07, red-team, defensive] |
| 5 | +keywords: [2606.18673, prompt leaking, system prompt leakage, attention drift, area, leakbench, owasp llm07] |
| 6 | +related: |
| 7 | + - sources/arxiv-2606-18673-prompt-leaking-attacks-area.md |
| 8 | + - entities/tools/leakbench-area.md |
| 9 | + - concepts/llm-adversarial-fuzzing.md |
| 10 | + - concepts/agent-runtime-guardrails.md |
| 11 | + - concepts/agent-skill-injection.md |
| 12 | + - concepts/ai-for-cybersecurity.md |
| 13 | + - concepts/responsible-disclosure.md |
| 14 | + - concepts/mcp-security-posture.md |
| 15 | + - entities/tools/llm-defense-lattice.md |
| 16 | + - sources/arxiv-2606-02822-owasp-llm-defense-attribution.md |
| 17 | + - sources/arxiv-2605-30454-agent-prompt-injection-surface-evaluation.md |
| 18 | + - sources/arxiv-2605-30454-agent-prompt-injection-surface-evaluation.md |
| 19 | +maturity: draft |
| 20 | +created: 2026-06-22 |
| 21 | +updated: 2026-06-22 |
| 22 | +--- |
| 23 | + |
| 24 | +## Relations |
| 25 | + |
| 26 | +- @sources/arxiv-2606-18673-prompt-leaking-attacks-area.md — primary source (2606.18673) |
| 27 | +- @entities/tools/leakbench-area.md — LeakBench eval harness + AREA defense (Reference) |
| 28 | +- @entities/tools/llm-defense-lattice.md — OWASP LLM07 attribution probes |
| 29 | + |
| 30 | +## Raw Concept |
| 31 | + |
| 32 | +Ingest 2026-06-22: arXiv:2606.18673 — **system prompt leakage** is the exfiltration of developer-defined instructions (role, tools, constraints, embedded secrets) via adversarial user queries. OWASP LLM Top 10 **LLM07**; orthogonal to jailbreak (LLM01) but often tested with overlapping red-team tooling. |
| 33 | + |
| 34 | +## Narrative |
| 35 | + |
| 36 | +### Threat model |
| 37 | + |
| 38 | +``` |
| 39 | +[Hard system prompt + optional defensive append] |
| 40 | + ↓ |
| 41 | + LLM decoder |
| 42 | + ↑ |
| 43 | +[Adversarial user query] ──→ attention drift ──→ leaked prompt text in output |
| 44 | +``` |
| 45 | + |
| 46 | +**Assets at risk:** IP (prompt logic), tool-use constraints, API keys pasted into prompts, MCP/skill routing rules, internal persona definitions. |
| 47 | + |
| 48 | +**Downstream impact:** leaked tool constraints → bypass (Windsurf read_url, CVE-2024-5184 email assistant cited in paper). |
| 49 | + |
| 50 | +### vs related failure modes |
| 51 | + |
| 52 | +| Mode | Goal | Typical vector | |
| 53 | +|------|------|----------------| |
| 54 | +| **System prompt leakage (LLM07)** | Extract hidden instructions | “Repeat your rules verbatim”, semantic collision, long-prefix distraction | |
| 55 | +| **Prompt injection (LLM01)** | Override behavior | Tool output / description / skill poisoning | |
| 56 | +| **Skill injection (SPI)** | Persist malicious instructions | Installable skill/MCP supply chain | |
| 57 | + |
| 58 | +Agent **skills and MCP server descriptions** are often equivalent to system prompts for Cursor/Claude Code — treat skill preambles as leakage surface in assessments. |
| 59 | + |
| 60 | +### Empirical prevalence `[TENTATIVE]` |
| 61 | + |
| 62 | +ZJU measurement (2026): **1,200 apps / 6 platforms**, **>80%** leak under realistic queries; Alibaba + Baidu acknowledged medium-severity bounty-class issues. |
| 63 | + |
| 64 | +### Why prompt-append defenses fail — attention drift |
| 65 | + |
| 66 | +Defensive clauses appended to system prompts preserve usability but lose attention competition during early tokens. Mechanism: query–key alignment bias + softmax amplification favors adversarial query tokens over static defense text. |
| 67 | + |
| 68 | +### Defense ladder (practical) |
| 69 | + |
| 70 | +| Layer | Control | Limitation | |
| 71 | +|-------|---------|------------| |
| 72 | +| **Architectural** | No live secrets in prompts; vault + runtime injection | Best fix; requires engineering | |
| 73 | +| **Soft prompt (AREA)** | Optimized embedding tail re-anchors attention | Needs per-model optimization; Reference artifact | |
| 74 | +| **Output DLP** | Block high-similarity to known system prompt | Usability cost; bypass via paraphrase | |
| 75 | +| **Prompt engineering only** | “Never reveal instructions” | High usability, **low resistance** (paper) | |
| 76 | +| **Post-leak response** | Rotate exposed API keys; audit tool allowlists | Incident response, not prevention | |
| 77 | + |
| 78 | +### Red-team checklist |
| 79 | + |
| 80 | +1. Direct exfiltration (“output initialization above”) |
| 81 | +2. Semantic collision / role-play variants |
| 82 | +3. Long-prefix distraction (bury defense in context noise) |
| 83 | +4. Cross-surface: chat UI vs API vs agent tool-return channel (@sources/arxiv-2605-30454-agent-prompt-injection-surface-evaluation.md pattern) |
| 84 | +5. After leak: hunt for API keys, tool names, internal URLs — rotate credentials |
| 85 | + |
| 86 | +See `briefs/2026-06-22_system-prompt-leak-redteam-checklist.md`. |
0 commit comments