LLM Eval Lab

LLM Eval Lab is a local-first evaluation workbench for comparing LLM prompts, models, and lightweight RAG behavior across quality, latency, cost estimates, and failure cases.

It is designed for teams that need a practical, inspectable way to answer:

• Which model performs better on our own questions?
• Which prompt is safer or more grounded?
• Which cases are failing, and why?
• Did a model, prompt, or retrieval change regress quality?

The project runs without API keys using a deterministic mock provider, and can also evaluate Ollama or OpenAI-compatible Chat Completions endpoints.

Features

• Dataset-based evaluation with included support QA and security-grounding benchmark packs.
• Prompt and model comparison across repeated runs.
• Matrix runs with --prompts and --models.
• Lightweight lexical retrieval over a Markdown knowledge base.
• Separate deterministic diagnostics for lexical answer correctness, answer relevancy, retrieved-context faithfulness, retrieval recall/hit rate, and conciseness.
• Per-case traces: question, retrieved context, rendered prompt, model answer, expected answer, scores, and evaluator reasoning.
• Strict regression gates reject incompatible baselines by default; an explicit intersection mode reports shared and missing cases for exploratory analysis.
• Experiment manifests include input fingerprints, exact case IDs, provider and sampling settings, evaluator configuration, runtime/dependency versions, and code revision.
• SQLite storage, Markdown reports, CSV exports, and a Streamlit dashboard.
• MIT license for broad personal and commercial use.

Quickstart

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e ".[dev]"

Run the local smoke test without API keys or external services:

llm-eval-lab --provider mock --model mock_good --case-limit 3 --min-score 0.80

Open the dashboard:

python -m streamlit run app/dashboard.py

Generated outputs:

• SQLite database: data/evals.db
• Markdown reports: reports/run_*.md
• CSV exports: reports/run_*.csv

Generated outputs are ignored by Git.

Compare Models And Prompts

Run a small comparison matrix:

llm-eval-lab `
  --dataset support_qa `
  --prompts baseline,strict_grounded `
  --provider mock `
  --models mock_good,mock_weak `
  --case-limit 5

Run a regression gate:

llm-eval-lab --dataset support_qa --prompt strict_grounded --provider mock --model mock_good --min-score 0.80

Compare a candidate against a previous run. The candidate's report includes overall and per-case deltas; the command exits non-zero if the allowed regression is exceeded.

llm-eval-lab --provider mock --model mock_good --case-limit 5
# Copy the emitted run id, then:
llm-eval-lab --provider mock --model mock_weak --case-limit 5 `
  --baseline-run run_YYYYMMDD_HHMMSS_abcdef --max-regression 0.02

Strict mode requires the same dataset and knowledge-base fingerprints, retrieval and evaluator configurations, and exact case set. To intentionally inspect only shared IDs, add --baseline-mode intersection; the report will mark the result as exploratory and list missing cases.

The dashboard reads historical runs from SQLite, so every completed run becomes available for comparison.

Providers

Mock

Use mock runs for local verification and CI:

llm-eval-lab --provider mock --model mock_good
llm-eval-lab --provider mock --model mock_weak

Ollama

$env:OLLAMA_BASE_URL="http://localhost:11434"
$env:OLLAMA_REQUEST_TIMEOUT_SECONDS="600"
$env:OLLAMA_NUM_PREDICT="160"
llm-eval-lab --provider ollama --model llama3.1:8b --case-limit 3

OpenAI Or Compatible Endpoints

$env:OPENAI_API_KEY="..."
$env:OPENAI_BASE_URL="https://api.openai.com/v1"
$env:OPENAI_INPUT_COST_PER_1K_USD="0"
$env:OPENAI_OUTPUT_COST_PER_1K_USD="0"
llm-eval-lab --provider openai --model gpt-4.1-mini --case-limit 3

Use .env.example as the reference for supported environment variables. See provider configuration for details.

Bring Your Own Data

Use datasets/support_qa.json as a template. security_grounding is a second built-in pack focused on authorization, secrets, retention, and incident-policy questions. Each case needs:

• id
• question
• expected_answer
• expected_context
• category
• difficulty

Use docs/support_docs.md as the default Markdown knowledge base, or pass a custom file with --kb-path.

Prompt templates live in prompts/ and can use:

• {question}
• {context}

See dataset format, evaluation methodology, and operating notes.

Metrics

The v2 evaluator keeps distinct experimental questions distinct:

• answer_correctness: normalized lexical and ordered-phrase agreement with the reference answer, with penalties for incompatible negation, numbers, and instruction-override artifacts.
• context_faithfulness: claim support from the context actually retrieved and delivered to the model. Annotated ideal context is never added to this score.
• retrieval_recall and retrieval_hit_rate: whether retrieved chunks cover the annotated expected evidence.
• answer_relevancy: mostly reference-answer coverage, with limited question alignment.
• conciseness: a transparent length diagnostic.

overall weights these signals at 35%, 25%, 20%, 10%, and 10% respectively. The metrics preserve frequency, include some word order, and detect common negation/numeric failures, but they remain lexical diagnostics—not semantic proof. See the methodology and adversarial limitations.

Architecture

datasets/*.json
        |
        v
prompts/*.md + docs/*.md
        |
        v
llm_eval_lab.run_eval
        |
        +--> lexical retriever
        +--> provider: mock, ollama, or openai
        +--> deterministic evaluator
        |
        v
data/evals.db
        |
        +--> reports/*.md and reports/*.csv
        +--> app/dashboard.py

Project Structure

app/
  dashboard.py
datasets/
  security_grounding.json
  support_qa.json
docs/
  dataset_format.md
  evaluation_methodology.md
  operating_notes.md
  provider_configuration.md
  security_grounding_docs.md
  support_docs.md
llm_eval_lab/
  datasets.py
  metrics.py
  prompts.py
  providers.py
  reports.py
  retrieval.py
  run_eval.py
  storage.py
prompts/
  baseline.md
  concise.md
  strict_grounded.md
tests/

Development

python -m pip install -e ".[dev]"
python -m pytest
python -m ruff check .
llm-eval-lab --provider mock --model mock_good --case-limit 3

Security And Data

Evaluation inputs, retrieved context, prompts, and model answers are stored in SQLite. Treat data/evals.db and generated reports as sensitive if your datasets contain private information.

Do not commit API keys, private datasets, generated databases, or reports.

Evaluation approach

The deterministic evaluator is dependency-free and designed for repeatable regression signals. Adversarial tests cover negation, wrong numbers, partial answers, common synonyms, retrieval misses, unsupported answers, injection artifacts, empty answers, and annotated abstention. High-stakes decisions still need reviewed benchmarks, human labels, or a semantic evaluator.

Every run stores a versioned experiment manifest and per-response provider metadata. Credentials are never included.

Roadmap

• Add optional semantic evaluators such as RAGAS, DeepEval, or LLM-as-judge.
• Add richer provider adapters and per-model pricing presets.
• Add export/import helpers for benchmark packs.
• Add dashboard screenshots and a short demo GIF.
• Add optional trace export to observability tools.

License

MIT. See LICENSE.