🧠 AgentGenBench

A standardized benchmark and evaluation framework for LLM-based agent generation.
AgentGenBench provides structured specifications, ground-truth implementations, and evaluation metrics to assess how well AI frameworks can synthesize, test, and execute autonomous agents from natural language specifications.

---

🚀 Overview

The AgentGenBench benchmark is designed to systematically compare frameworks (such as YUMA, Gemini CLI, Copilot CLI, Spec-Kit and Kiro) in their ability to generate and evaluate LLM-driven agents from structured task specifications.
It provides:

• Four reference agent types (Simple, Tool, Memory, MCP)
• Formal specifications describing purpose, functionality, and edge cases
• Ground-truth implementations for reproducible comparison
• Evaluation scripts and metrics using LLMs as judges

All frameworks are tested under the same conditions, using standardized prompts and specifications.

---

🧩 Repository Structure

Directory	Description
bench/	Contains benchmark data — specifications, user stories, acceptance criteria, and reference (ground-truth) implementations for each agent.
eval/	Stores generated artifacts from each framework (code, prompts, logs, and reports).

---

🤖 Agent Types

AgentGenBench defines four standardized agents, each increasing in complexity and capability.

Agent	Description
Simple Agent	Processes user input and produces structured outputs without external tool usage.
Tool Agent	Integrates external APIs (e.g., weather service) through tool-calling mechanisms.
Memory Agent	Incorporates memory for context persistence and reasoning over multi-turn interactions.
MCP Agent	Interacts with external services using the Model Context Protocol (MCP) — e.g., managing GitHub issues through a remote MCP server.

Each agent has:

• A natural-language specification (purpose, roles, functionality)
• A set of user stories with Given–When–Then acceptance criteria
• Edge cases to test robustness and reasoning
• Ground-truth implementation that acts as the oracle for validation

---

📜 Specifications

Every specification in AgentGenBench follows a unified structure to ensure reproducibility and fairness:

1. Purpose & Scope — defines the agent’s high-level goal.
2. Intended Users — describes target usage scenarios.
3. Functional Requirements — outlines expected behaviors and API interactions.
4. User Stories — given–when–then format for objective evaluation.
5. Acceptance Criteria — measurable behavioral checkpoints.
6. Edge Cases — rare or error-prone situations to test adaptability.

This template ensures that every framework interprets the same requirements and can be compared directly on how accurately it converts specifications into working code.

---

⚙️ Evaluation Setup

The evaluation framework uses the Goal–Question–Metric (GQM) approach to assess each tool under four complementary dimensions:

1. Agent Code Generation

Measures the ability to generate syntactically valid, structurally correct agent code that matches the specification.

Metrics:

• Specification Compliance – adherence to the 6-point structural checklist:
  1. Defined workflow/graph
  2. LLM reasoning node
  3. External integration (MCP, Memory, or Tool)
  4. Core actions implemented
  5. Structured JSON output
  6. Edge case handling
• Layer Integration – real vs. mocked external connections
• Code Completeness – presence of all required components (graph, LLM, integration, CLI)

---

2. Test & Evaluation Artifact Generation

Assesses whether frameworks produce automated evaluation or testing scripts.

Metrics:

• Test Generation – creation of unit, behavioral, or structural tests.
• Agent Evaluation (LLM-as-a-Judge) – presence of automatic evaluation via LLM-based metrics such as G-Eval.

---

3. Executability & Resource Usage

Validates that the generated agents can actually run and measures the cost-efficiency of generation.

Metrics:

• Code Executability – does the code run without manual intervention?
• Execution Time – total duration of generation.
• Cost – API or token cost when applicable.
• Files Count – number of generated artifacts.

---

4. Behavioral Evaluation (LLM-as-a-Judge)

Executed using DeepEval and G-Eval, these evaluations test the behavior of each agent type against its ground truth.

Metrics:

• Clarity – responses are clear and structured.
• Relevance – responses address the request and edge cases.
• Completeness – responses contain all required information.
• Task Completion – successful execution of the specified task.
• Tool Correctness – correct external tool/API usage.
• Argument Correctness – correct parameter passing.
• Changed Lines – number of lines modified to make the code executable.

---

🧱 Ground Truth Implementations

Each agent has a reference implementation that serves as:

1. A functional oracle — defines the correct expected behavior.
2. A validation anchor — used to check outputs from generated agents.
3. A reproducibility baseline — ensures consistent evaluations across frameworks.

---

💬 Contextual Prompts

Every framework receives the same contextual prompt template, defining:

• The agent cycle: perception → decision → action
• The interaction model (e.g., reasoning via LLM, tool usage)
• The external endpoints (MCP, APIs)
• The expected JSON schema for responses

This ensures semantic alignment and fairness across generations.

---

🧮 Frameworks Evaluated

The following frameworks were evaluated in the AgentGenBench benchmark:

• YUMA – provides guided agent generation and built-in evals.
• Gemini CLI – uses Google’s generative models for code synthesis.
• Copilot CLI – GitHub Copilot-powered agent generation interface.
• Spec-Kit – open-source, code-oriented agent framework.
• Kiro - IDE that helps you go from prototype to production.

---

🧭 Methodology Summary

The assessment process follows four main phases:

1. Specification Delivery – all frameworks receive identical agent specs and prompts.
2. Code Generation – frameworks generate the agent code autonomously.
3. Execution & Testing – generated agents are run against the ground truth.
4. Evaluation & Scoring – results are measured via DeepEval and qualitative inspection.

---

🧠 Why AgentGenBench?

AgentGenBench establishes a standardized and reproducible foundation for evaluating the capabilities of LLM-based frameworks to generate, test, and run intelligent agents. It supports:

• Cross-framework comparison
• Fine-grained metrics
• Objective evaluation
• Reproducible experimentation

This enables researchers and developers to assess the maturity of agentic frameworks and guide future development of AI systems capable of autonomously translating specifications into functioning software agents.