Skip to content

SKILL.md

Latest commit

 

History

History
179 lines (140 loc) · 7.5 KB

SKILL.md

File metadata and controls

179 lines (140 loc) · 7.5 KB
Raw
name agent-debugging
description Use when agent systems fail, produce wrong results, or behave unexpectedly. Covers the failure taxonomy (LLM, tool, loop, multi-agent levels), debugging toolkit, self-healing patterns, and systematic root cause analysis.

Agent Debugging

Diagnose and fix agent failures systematically.

HARD GATE: Classify Before Debugging

Do not start fixing until you've classified the failure. Wrong classification leads to wrong fix.

The Failure Taxonomy

Agent Failures
├── LLM-Level
│   ├── Hallucination (making stuff up)
│   ├── Instruction-following failure (ignoring instructions)
│   └── Reasoning errors (wrong logic)
├── Tool-Level
│   ├── Wrong tool selected
│   ├── Wrong arguments passed
│   └── Tool result misinterpreted
├── Loop-Level
│   ├── Infinite loops (stuck in retry)
│   ├── Premature termination (giving up too early)
│   └── Context overflow (window exhausted)
└── Multi-Agent
    ├── Communication failures (message lost/garbled)
    ├── Deadlocks (agents waiting on each other)
    ├── Cascading failures (one failure triggers chain)
    └── Emergent misbehavior (unexpected group dynamics)

Debugging Decision Tree

What happened?
│
├─ Agent gave wrong answer
│  ├─ Made up facts → LLM: Hallucination
│  ├─ Ignored instructions → LLM: Instruction-following
│  └─ Logic was flawed → LLM: Reasoning error
│
├─ Agent used tools wrong
│  ├─ Called wrong tool → Tool: Selection
│  ├─ Passed bad args → Tool: Arguments
│  └─ Misread tool output → Tool: Interpretation
│
├─ Agent got stuck or crashed
│  ├─ Keeps retrying same thing → Loop: Infinite
│  ├─ Gave up too early → Loop: Premature termination
│  └─ Ran out of context → Loop: Context overflow
│
└─ Multi-agent system broke
   ├─ Agent didn't get the message → Communication failure
   ├─ Two agents waiting on each other → Deadlock
   ├─ One failure broke everything → Cascading failure
   └─ Agents doing something weird together → Emergent misbehavior

The Debugging Toolkit

Tool What It Does When to Use
Trace inspection Full execution replay (LangSmith, Langfuse, Phoenix) Any failure — start here
Prompt inspection View exact prompts sent to LLM Instruction-following and reasoning failures
Deterministic replay Re-run with mocked LLM responses Reproducing intermittent bugs
Ablation testing Remove components, test impact Finding which component causes the failure
Counterfactual debugging "What if the agent had done X?" Understanding decision points
Snapshot testing Golden-file comparison for outputs Regression prevention
Chaos monkey Inject faults (timeouts, errors) Pre-deployment resilience testing

Debugging by Failure Type

Failure Type First Step Primary Tool
Hallucination Check if grounding data exists Trace inspection → check tool outputs
Instruction-following Check prompt length (< 150 lines?) Prompt inspection → simplify
Wrong tool Check tool descriptions Tool list review → clarify descriptions
Infinite loop Check retry count / termination conditions Trace inspection → add circuit breaker
Context overflow Check token count at failure Context utilization monitoring → compact earlier
Communication failure Check message delivery Inter-agent trace → verify message format
Cascading failure Find the first failure Trace tree → follow the chain backward

Self-Healing Patterns

Implement these to make your system recover automatically:

Pattern How It Works When to Use
Retry with escalation Retry failed call → try different approach → escalate to human Transient failures (API timeouts, rate limits)
Circuit breaker After N failures, stop trying and fail gracefully Repeated failures (broken tool, bad API)
Fallback agent Primary fails → backup with simpler strategy takes over When reliability matters more than quality
Checkpoint & resume Save state at each step, resume from last good checkpoint Long-running workflows
Model downgrade Expensive model fails → try cheaper model Sometimes simpler is better

Escalation Chain

Retry (same call, up to 3x)
  → Retry with adjusted parameters
    → Try alternative tool/approach
      → Fallback agent (simpler strategy)
        → Checkpoint partial results
          → Escalate to human

Root Cause Analysis

The Five Whys for Agent Failures

  1. Why did the agent fail? → It hallucinated a database schema
  2. Why did it hallucinate? → It didn't have the schema in context
  3. Why wasn't the schema in context? → The context window was full
  4. Why was the context full? → Previous conversation wasn't compacted
  5. Why wasn't it compacted? → No compaction trigger at 50% utilization

Root cause: Missing compaction discipline. Fix: Set compact_trigger to 50%.

Post-Incident Checklist

  • Failure classified (LLM / Tool / Loop / Multi-Agent)
  • Root cause identified (not just the symptom)
  • Trace captured and saved for future reference
  • Fix addresses root cause (not just this instance)
  • Guard added to prevent recurrence (hook, circuit breaker, or validation)
  • Regression test added (snapshot or eval)

Resources

  • references/failure-taxonomy.md — Full taxonomy with real-world examples for each failure type
  • references/debugging-toolkit.md — Detailed guide to each debugging tool: LangSmith traces, deterministic replay, chaos testing, ablation, counterfactual debugging
  • references/self-healing-patterns.md — Implementation guides for retry escalation, circuit breakers, fallback agents, checkpoint/resume, model downgrade

Rationalization Traps

What you might think Why it's wrong What to do instead
"Just retry and it'll work" Retrying without understanding the failure makes it worse Classify first, then decide if retry makes sense
"The prompt is fine, the model is bad" 90% of the time, the prompt is the problem Inspect the exact prompt; check length and clarity
"This is a random failure" Agent failures are rarely random; there's always a cause Reproduce with deterministic replay; find the pattern
"Adding more context will fix it" More context often makes things worse (150-line rule) Less context, more focus; constraints > descriptions

Verification

Before marking debugging complete, confirm:

  • Failure classified into taxonomy category
  • Root cause identified (not just symptom addressed)
  • Fix tested and verified (not just "looks right")
  • Guard or self-healing pattern added to prevent recurrence
  • Regression test captures this failure scenario

Related Skills

  • agent-observability — The data you need to debug (traces, metrics, logs)
  • agent-security — Some failures are security incidents (prompt injection, data exfiltration)
  • operational-discipline — Anti-patterns that cause failures; post-incident checklist
  • durable-workflows — Checkpoint/resume for recovering from failures