Merge pull request #916 from pipecat-ai/aleix/the-eval-loop

Rename Agent Self-Improvement to The Eval Loop

Author: markbackman

docs.json

@@ -102,7 +102,7 @@
               "pipecat/evals/scenarios",
               "pipecat/evals/suites",
               "pipecat/evals/library",
-              "pipecat/evals/agent-self-improvement",
+              "pipecat/evals/the-eval-loop",
               {
                 "group": "Third-party Platforms",
                 "pages": [

pipecat/evals/overview.mdx

@@ -29,7 +29,7 @@ Evals make agent behavior testable the way unit tests make code testable:
 - **Regression safety**: run your scenarios after every prompt, model, or pipeline change and catch breakage before users do.
 - **Fast iteration**: text-mode evals skip STT and TTS entirely, so a full conversation test runs in seconds with no audio service cost.
 - **Semantic assertions**: an LLM judge checks meaning ("the response says the capital is Berlin"), not exact strings, so tests don't break when wording changes.
-- **A feedback signal for AI coding assistants**: evals give a coding assistant a command it can run and a pass/fail result it can read, closing the loop between writing agent code and verifying it. See [Agent Self-Improvement](/pipecat/evals/agent-self-improvement).
+- **A feedback signal for AI coding assistants**: evals give a coding assistant a command it can run and a pass/fail result it can read, closing the loop between writing agent code and verifying it. See [The Eval Loop](/pipecat/evals/the-eval-loop).
 
 Pipecat itself relies on this framework: before every release, an eval suite drives 100+ example agents end to end.
 
@@ -126,39 +126,30 @@ Pipecat's other building blocks feed into any evaluation workflow: [Metrics](/pi
 
 <CardGroup cols={2}>
   <Card
-    title="Quickstart"
-    icon="rocket"
+    title="Writing Scenarios"
+    icon="file-pen"
     iconType="duotone"
-    href="/pipecat/evals/quickstart"
+    href="/pipecat/evals/scenarios"
   >
-    Run your first eval against an existing agent in a few minutes.
+    The full scenario format: turns, expectations, modalities, and the judge.
   </Card>
 
-<Card
-  title="Writing Scenarios"
-  icon="file-pen"
-  iconType="duotone"
-  href="/pipecat/evals/scenarios"
->
-  The full scenario format: turns, expectations, modalities, and the judge.
-</Card>
-
-<Card
-  title="Eval Suites"
-  icon="list-check"
-  iconType="duotone"
-  href="/pipecat/evals/suites"
->
-  Spawn multiple agents and run many scenarios concurrently from a manifest.
-</Card>
+  <Card
+    title="Eval Suites"
+    icon="list-check"
+    iconType="duotone"
+    href="/pipecat/evals/suites"
+  >
+    Spawn multiple agents and run many scenarios concurrently from a manifest.
+  </Card>
 
   <Card
-    title="Agent Self-Improvement"
+    title="The Eval Loop"
     icon="arrows-rotate"
     iconType="duotone"
-    href="/pipecat/evals/agent-self-improvement"
+    href="/pipecat/evals/the-eval-loop"
   >
-    Close the loop: let an AI coding assistant write, run, and fix against
-    evals.
+    Let a coding assistant write agent code, run evals, and iterate
+    automatically until the agent is better.
   </Card>
 </CardGroup>

pipecat/evals/quickstart.mdx

@@ -152,4 +152,4 @@ This guide takes an existing agent, starts it with the eval transport, and runs
 
 - Learn the full scenario format, including multi-turn conversations, function call assertions, interruptions, latency budgets, and text vs audio modes, in [Writing Scenarios](/pipecat/evals/scenarios).
 - Have many scenarios or agents? Let Pipecat spawn the agents for you with [Eval Suites](/pipecat/evals/suites).
-- Want your coding assistant to run these for you? See [Agent Self-Improvement](/pipecat/evals/agent-self-improvement).
+- Want your coding assistant to run these for you? See [The Eval Loop](/pipecat/evals/the-eval-loop).

pipecat/evals/scenarios.mdx

@@ -345,20 +345,12 @@ Bots often cancel their pipeline in `on_client_disconnected`, so a scenario with
 
 <CardGroup cols={2}>
   <Card
-    title="Eval Suites"
-    icon="list-check"
+    title="The Eval Loop"
+    icon="arrows-rotate"
     iconType="duotone"
-    href="/pipecat/evals/suites"
+    href="/pipecat/evals/the-eval-loop"
   >
-    Run many scenarios against many agents concurrently from a manifest.
-  </Card>
-
-  <Card
-    title="Using the Library"
-    icon="code"
-    iconType="duotone"
-    href="/pipecat/evals/library"
-  >
-    Load, build, and run scenarios from Python instead of YAML.
+    Let a coding assistant write agent code, run evals, and iterate
+    automatically until the agent is better.
   </Card>
 </CardGroup>

pipecat/evals/suites.mdx

@@ -96,25 +96,3 @@ The exit code makes suites CI-ready with no extra glue:
 ```
 
 For deterministic, key-free CI runs, prefer text-mode scenarios and an OpenAI-compatible judge endpoint you control. Audio-mode scenarios work in CI too, but need the harness's TTS and STT services available (local models by default, which also need more CPU).
-
-## Next steps
-
-<CardGroup cols={2}>
-  <Card
-    title="Using the Library"
-    icon="code"
-    iconType="duotone"
-    href="/pipecat/evals/library"
-  >
-    Orchestrate suites programmatically with `EvalManifest` and `EvalSuite`.
-  </Card>
-
-  <Card
-    title="Agent Self-Improvement"
-    icon="arrows-rotate"
-    iconType="duotone"
-    href="/pipecat/evals/agent-self-improvement"
-  >
-    Let an AI coding assistant run your suite and iterate until it's green.
-  </Card>
-</CardGroup>

pipecat/evals/agent-self-improvement.mdx pipecat/evals/the-eval-loop.mdxpipecat/evals/agent-self-improvement.mdx renamed to pipecat/evals/the-eval-loop.mdx

@@ -1,9 +1,11 @@
 ---
-title: "Agent Self-Improvement"
-description: "Close the loop: let an AI coding assistant write agent code, run evals, and iterate until they pass."
+title: "The Eval Loop"
+description: "Close the loop: evals give an AI coding assistant a pass/fail signal it can read, so it writes agent code, runs evals, and iterates automatically until the agent is better."
 ---
 
-Evals do more than catch regressions. They turn agent quality into a signal that an AI coding assistant can read, which changes how you build: instead of asking an assistant to "improve the prompt" and judging the result by hand, you describe the desired behavior as a scenario and let the assistant iterate until the eval passes.
+Evals turn agent quality into a signal an AI coding assistant can read. That closes the loop: instead of asking an assistant to "improve the prompt" and judging the result by hand, you describe the desired behavior as a scenario and let the assistant iterate until the eval passes, and the agent gets better with every pass.
+
+Think of it as a REPL for agent behavior: the assistant writes a change, evals it, reads a pass/fail result, and loops, except the eval step already contains the judgment, so the cycle can close without a human reading the output.
 
 ## The loop