Know whether a model change is safe before it reaches production. LLM Preflight runs a small local preflight across providers and compares validated output, response speed, tokens, and estimated cost.
It is a local preflight tool—not a hosted evaluation platform, tracing system, RAG framework, or public leaderboard. Its results are evidence for your account, network, prompts, and validation rules.
Warning
Live benchmarks make paid API requests. Start with the no-key demo, preview the plan before a live run, and keep limits and repetitions small.
Create and run a deterministic local benchmark—no API key or network request:
llm-preflight --init
llm-preflight benchmark.json --no-saveFrom a source checkout:
python3 -m llm_preflight --init
python3 -m llm_preflight benchmark.json --no-save--init never overwrites an existing config. It creates a mock benchmark so
you can see the report and exit behavior before making a paid request.
- You are switching models or providers.
- A provider publishes a new model or changes a
latestalias. - You need to compare your own prompt's validity, latency, and cost.
- You want local result artifacts instead of a hosted dashboard.
It measures deterministic test validity, end-to-end latency (p50/p95), time to first token, throughput when the stream is incremental and usage is available, token totals, and estimated cost. Result files retain request metadata and per-request observations for reproducibility.
"Deterministic" describes the validator, not the model: every response is checked against explicit structural rules — a regular expression, a JSON shape, an exact routing label — so the same response always produces the same verdict. The tool does not score semantic quality; that is your task-specific evaluation, and it stays out of scope on purpose.
Real output from a cross-provider run (2026-07-16, one short support prompt, three repetitions per model, total spend under $0.05):
| Model | Success | Latency p50 | Latency p95 | TTFT p50 | Tokens/s p50 | Cost |
|---|---|---|---|---|---|---|
| gpt-5.6-luna | 100% | 1.379s | 1.579s | 0.726s | 139.0 | $0.001686 |
| gpt-5.4-mini | 100% | 1.779s | 3.628s | 1.061s | 126.6 | $0.001143 |
| claude-fable-5 | 100% | 6.497s | 7.573s | 2.908s | 56.5 | $0.033610 |
| claude-opus-4-8 | 100% | 3.859s | 4.267s | 1.350s | 54.3 | $0.010205 |
| gemini-3.5-flash | 100% | 2.924s | 2.955s | 2.895s | n/a | $0.001084 |
| minimax-m3 | 100% | 3.356s | 3.631s | 1.719s | 90.4 | n/a |
Tokens/s reads n/a when a provider delivers the response as a terminal
burst instead of an incremental stream — the observable window measures
transport, not generation, so no rate is reported. Cost reads n/a when
pricing for the model is unknown.
The report ends with a decision block:
- Fastest: gpt-5.6-luna — 1.423s mean latency.
- Cheapest: gemini-3.5-flash — $0.001084 total.
- Best value: gpt-5.6-luna — 88% composite score.
- Recommended: gpt-5.6-luna — passed every selected test and led the
qualified value ranking.
Numbers like these are evidence for one environment at one time, not a leaderboard. Latency depends on your network and region; run the preflight from the host that will serve production traffic.
The same comparison can be driven interactively — pick models and tests at the terminal, read the cost ceiling before anything is sent, watch each request report its own cost, and end on the decision. This capture is a real two-model paid run that cost half a cent (config, details):
Python 3.10+ is required. There are no third-party runtime dependencies:
pip install llm-preflight installs this package and nothing else, and the
CLI runs on the Python standard library alone. Development tools (pytest,
ruff, mypy) are optional extras that never reach a production install.
cp benchmark.example.json benchmark.json
cp .env.example .env.production
# Edit benchmark.json and add only the provider keys you use.
python3 -m llm_preflight benchmark.json --dry-run
python3 -m llm_preflight benchmark.jsonThe CLI reads .env.production beside the config without overriding environment
variables already set by your shell. Use --no-env-file or --env-file PATH
when needed. Runs print a terminal report and, unless --no-save is used,
write JSON and Markdown results under results/.
Install the command globally in a virtual environment if preferred:
python3 -m pip install llm-preflight
llm-preflight --initRun --doctor and --dry-run before the final command. They make no generation
requests; the final command is the paid work.
This is the core workflow. Put your approved model and candidate model in one config, then run the small response-and-contract preflight:
llm-preflight benchmark.json --migration-check --dry-run
llm-preflight benchmark.json --migration-checkIt sends three short representative cases to each selected model, once each. It answers: did the API work, did each response meet the basic contract, and how quickly did the provider start and finish responding? It is a cheap compatibility check, not a statistical performance conclusion.
When that passes, run the task-specific checks that match your application—for
example exact-routing-check or structured-output-check—before approving a
switch.
Use custom contract tests to express the outputs your
own feature must preserve.
I am new and want to see the tool safely. Start with the Getting started guide. It uses a no-key local mock before any provider request.
I know the current and candidate model IDs. Edit one config, run the migration check, then add a custom contract test for the output your feature must preserve. You do not need the catalogue.
I want to find and review provider releases. Use the local catalogue lifecycle below. It keeps broad provider metadata separate from the small set of models you approve for ongoing testing.
llm-preflight catalog init
llm-preflight catalog refresh benchmarks/watch.json
# If a model is shown as “Needs one probe”, review and confirm a minimal request:
llm-preflight catalog probe benchmarks/watch.json
llm-preflight catalog prepare benchmarks/watch.json \
--against benchmarks/approved.json --output benchmarks/candidates.json
llm-preflight benchmarks/candidates.json --interactive \
--approve-to benchmarks/approved.jsonRefresh reads metadata only. A probe sends one minimal request only for text candidates you select and confirm. The interactive benchmark then lets you approve passing models explicitly. Follow the complete catalogue tutorial for the decision points.
I am automating an established contract. Use
CI and JSON output, with a saved baseline and --ci where a
regression should fail the pipeline.
# Inspect configuration, credentials, and model selection without generation.
llm-preflight benchmark.json --doctor
llm-preflight benchmark.json --dry-run
llm-preflight benchmark.json --pricing-check
# Run a reduced live benchmark.
llm-preflight benchmark.json --smoke
# Run a single ad hoc prompt.
llm-preflight --quick "Return only valid JSON with a status field." \
--models openai:gpt-5.4-miniFor advanced discovery, interactive runs, CI, baselines, replay, and stop modes, see workflows. For models, environment files, custom prompts, and provider-specific options, see configuration.
- Keep prompts, system instructions, temperature, and output limits fixed.
- Validate outputs: a fast malformed response is a failed result.
- Run from the same host; network distance and provider load affect latency.
- Treat single-user latency and load testing as separate experiments.
- Prefer dated model IDs over moving aliases.
The CLI distinguishes API FAIL (transport, credentials, provider, or request
failure) from API OK / TEST FAIL (a response that fails your validator).
Recommendations only consider models that pass every selected test.
Several good tools live near this space. Use them when their job is your job:
- promptfoo, deepeval — full evaluation suites: scored quality metrics, red-teaming, large ongoing test matrices in CI. Use them to grade prompt and model quality over time.
- llmci — CI merge gates and prompt migration; it rewrites prompts for a new model. Use it when the prompt should adapt to the model.
- Braintrust, LangSmith — hosted platforms: tracing, dashboards, team collaboration, production observability.
llm(Simon Willison) — a general multi-provider CLI for running prompts, not a comparison harness.
LLM Preflight does one narrower job: the local go/no-go check in the moment before a model switch. Your prompt, candidate models, structural validation, latency, and cost — one command, one report, no hosted service, no telemetry, and no vendor between you and the verdict.
- Getting started — safe demo, first paid run, and choosing the right workflow.
- Workflows — discovery, smoke mode, CI, replay, matrix, baselines, and request safety.
- Configuration — providers, custom prompts, presets, aliases, and environment overlays.
- Configuration reference — every supported config key, default, and validation rule.
- Custom contract tests — copyable JSON extraction, exact-routing, and content-rule migration tests.
- CLI reference — every command-line option, default, and incompatibility.
- Interactive mode — selection, paid-run preview, and live progress.
- CI and JSON output — machine-readable output and exit-code gates.
- Result JSON schema — fields for integrations and saved benchmark evidence.
- Model watch and approval — discover, compare, and deliberately promote, re-test, and retire provider models.
- Troubleshooting — installation, credentials, catalogue, and benchmark-result recovery.
- Tests, pricing, and safety — built-in tests, validators, pricing confidence, retries, and sensitive data.
- Contributing — development setup and the TDD workflow.
- Security — reporting vulnerabilities.
Contributions are welcome; see CONTRIBUTING.md. Released under the MIT License.