Skip to content

agentic-plan-execute-verify.md

Latest commit

 

History

History
234 lines (183 loc) · 10.6 KB

agentic-plan-execute-verify.md

File metadata and controls

234 lines (183 loc) · 10.6 KB
Raw

Agentic Plan -> Execute -> Verify Integration

How to compose baloney-detection-kit with agentic runtimes that already use a Plan -> Execute -> Verify architecture, such as Robert Eichenseer's AgenticAI.PlanExecuteValidate.

This is an integration pattern, not an official adapter. Keep this repository a playbook. Put orchestration code, framework bindings, automated evaluators, and provider-specific details in the downstream system that uses the playbook.

Why the pattern fits

BDK describes a conversation behavior: before validating a weak, inflated, high-stakes, or novel-sounding claim, add epistemic friction. Plan -> Execute -> Verify systems provide a natural runtime shape for that behavior:

  1. Plan: decide whether the playbook should fire and which mode applies.
  2. Execute: run the chosen epistemic checks with available tools, agents, or human review.
  3. Verify: review whether the answer preserved epistemic integrity before it reaches the user.

AgenticAI.PlanExecuteValidate demonstrates this shape with a planner agent, an executor agent, a verifier agent, and activity artifacts that preserve plan, step results, final answer, and scores. BDK can be used as the behavioral policy inside that shape without importing the sample's sports-domain code.

Component mapping

BDK concept Plan -> Execute -> Verify component Integration rule
Trigger conditions Planner Detect material confidence-evidence-consequence mismatch. Treat novelty, suppression framing, dissent, validation requests, and multi-turn pressure as signals, not verdicts.
Light / full / stabilization mode Planner Select the lightest mode that preserves rigor. Do not force the full template on humble exploration.
Six-step protocol Plan steps Convert claim typing, scoped current knowledge, contribution, update conditions, evidence quality, credible alternatives, and calibration into explicit steps.
Evidence-backed practice Executor Use search, retrieval, citations, internal knowledge bases, or domain sources when available; record scope and evaluate provenance and independence.
External contrast Executor Assign distinct source-audit and hypothesis-test jobs when warranted. Send the object, not the prior answer; never count model agreement as evidence.
High-stakes handling Planner + Executor Separate epistemic uncertainty from action risk, require domain-appropriate sources, and route material decisions to qualified humans.
Manual review rubric Verifier Review protocol adherence, epistemic quality, utility, and adverse effects such as over-triggering, stubbornness, or false balance.
Review notes / transcript Activity artifact Preserve claim, mode, plan, tool results, reviewer outputs, uncertainties, and final answer for later audit.

Minimal integration flow

1. Intake: extract the object

Start by extracting the smallest reviewable object:

  • the user's atomic claim;
  • claim type and scope;
  • the domain and stakes;
  • the user's requested action;
  • any evidence already provided;
  • stated confidence and consequence if wrong;
  • whether the user is asking for validation, persuasion, or investigation.

Do not pass the whole conversation to downstream reviewers unless the conversation itself is the object being reviewed. For ordinary claims, summarize the claim and evidence neutrally.

2. Plan: create a BDK activity plan

The planner should output a plan that is specific enough to execute and narrow enough to avoid unnecessary friction.

Example conceptual shape:

{
  "claim": "A user claims X is a new discovery.",
  "claim_type": "empirical / interpretive",
  "mode": "full",
  "stakes": "scientific / low immediate harm",
  "steps": [
    "current_knowledge_and_scope",
    "prior_art_and_contribution",
    "update_conditions_and_evidence",
    "competing_explanations_and_discriminators",
    "calibration_and_next_step"
  ],
  "external_contrast": "not_required"
}

This is not a schema this repo promises to support. It is a readable sketch for teams that already have their own activity object.

3. Execute: run epistemic checks

The executor runs only the steps the planner selected:

  • Claim type: separate observation, explanation, significance, requested action, and confidence; choose evaluation rules that fit the claim.
  • Current knowledge: retrieve prior art or domain references and preserve the scope, date, and limits of the search.
  • Contribution: separate documented/rediscovered material, re-framing/application, new evidence/method/implementation, and "no close prior art found in this scoped search" from truth and importance.
  • Update conditions: identify what should strengthen or weaken the claim. Use falsification where empirical, values/tradeoffs where normative, and provenance/corroboration where interpretive.
  • Evidence quality: inspect relevance, method, independence, replication or corroboration, recency, provenance, incentives, and gaps.
  • Competing explanations: produce only credible alternatives and the evidence that would discriminate among them; do not force two sides.
  • External contrast: for high-stakes, uncertain, niche, or unusually inflated claims, assign distinct source-audit and hypothesis-test jobs. Model diversity may improve coverage but does not create independent evidence.
  • High-stakes escalation: when consequences are material, include qualified human expertise as the next step rather than treating the agent's answer as an action recommendation.

4. Verify: review the response before delivery

The verifier should not become an automated truth engine. It should check whether the response followed the playbook:

  • Did the trigger decision make sense?
  • Did it avoid treating novelty or dissent as a verdict?
  • Did it type the claim and check scoped prior art before validating?
  • Did it avoid fabricated sources and unsupported certainty?
  • Did it assess source quality and shared provenance?
  • Did it preserve useful contributions without flattering novelty or significance?
  • Did it avoid forced alternatives and reflexive contrarianism?
  • Did it re-open its own assessment when a premise or factual claim changed?
  • Did it handle high-stakes content with care?
  • Did it preserve reviewer disagreement and verify evidence instead of voting?
  • Did it leave a concrete, honest next step?

If verification fails, revise the response or mark the uncertainty plainly. Do not hide the failure behind a confidence score.

5. Return a user-facing answer

The final answer should still sound like BDK: kind, direct, specific, humble, and useful. The user does not need to see the internal activity object unless the product intentionally exposes audit traces.

Activity artifact

A downstream Plan -> Execute -> Verify runtime can preserve an activity artifact with fields like these:

Field Purpose
claim The atomic claim or decision being reviewed.
claim_type Empirical, causal/predictive, normative/policy, interpretive/historical, personal/experiential, or creative/hypothetical.
confidence_evidence_consequence Why friction was or was not proportionate.
trigger_reason Why BDK fired, or why it stayed in light mode.
mode light, full, or stabilization.
steps Planned BDK checks and their outputs.
sources Prior-art or evidence references used during execution.
reviewers External model reviewers and their distinct jobs, if used.
disagreements Preserved disagreements across reviewers or sources.
verifier_notes Protocol, epistemic-quality, utility, and adverse-effect review.
final_answer The answer delivered to the user.
next_step Source, expert, experiment, or narrowing action.

This artifact is useful for incident review, team calibration, and later measurement with tools such as robopsychology.

Guardrails

  • Do not turn agreement into evidence. Models can share sources and correlated errors. Verify the underlying claims.
  • Do not contaminate second opinions. Send the object to review, not the first model's reasoning or answer.
  • Do not use consensus as a verdict. Dissent can be well supported; consensus is one evidence signal.
  • Do not force falsifiability or alternatives. Match update rules to the claim type and avoid false balance.
  • Do not replace domain expertise. Medical, legal, financial, safety, and mental-health decisions need qualified humans when consequences are material.
  • Do not fabricate retrieval. If the runtime lacks browsing, search, or a trusted corpus, say so.
  • Do not collapse verification into scoring. A numeric score can help a product workflow, but BDK's verifier should remain rubric-based and auditable.
  • Do not copy external assets or code casually. Link to AgenticAI.PlanExecuteValidate as a reference pattern unless its license and your project policy explicitly allow reuse.
  • Do not move the framework into this repo. Keep provider-specific code, MCP wiring, notebooks, evaluators, and SDK adapters downstream.

Where to integrate

Use this pattern when you already have an agent runtime with one or more of:

  • a planner or task-decomposition stage;
  • tool execution, retrieval, MCP, or internal knowledge bases;
  • multiple reviewer models;
  • a verifier, judge, policy gate, or human review queue;
  • audit logs or activity artifacts.

If you only need personal use, copy ROOT_PROMPT.md. If your assistant supports skills, use skill/SKILL.md. Plan -> Execute -> Verify is for systems that already need orchestration.

Validation

Start with manual review:

For measurement, keep BDK as the prompt-side intervention and use an external instrument. The closed-loop protocol in validation/closed-loop/ shows how to compare a control prompt against BDK using robopsychology.

Reference

  • AgenticAI.PlanExecuteValidate by Robert Eichenseer: an end-to-end sample of a Plan -> Execute -> Verify orchestrator with planner, executor, verifier, activity objects, classic functions, agents, and MCP tools.