Skip to content

Document latency limits for voice simulations - #1288

Merged
scott-lowe-vapi merged 2 commits into
mainfrom
scottlowe/sim-latency-expectations-docs
Oct 6, 2026
Merged

scott-lowe-vapi merged 2 commits into
mainfrom
scottlowe/sim-latency-expectations-docs

Conversation

@scott-lowe-vapi

Copy link
Copy Markdown
Contributor

Summary

Voice simulations can now fail when the assistant responds too slowly. The feature shipped in VapiAI/vapi#21959 (API and worker), VapiAI/vapi#21962 (run results) and VapiAI/vapi#21963 (editor), tracked in TEST-140. The API reference already shows the new latencyExpectations and latencyEvaluations fields through the nightly spec update; this PR covers the guides.

  • Simulations advanced: new Set latency limits section and card:
    • Dashboard and cURL setup;
    • the turn, model and voice metrics and the four aggregations, including that p95 equals max under 20 turns;
    • required versus optional limits;
    • when limits are skipped (chat mode, GPT-Live assistants under test) and that a voice run with no measured turns fails;
    • reading results in the run details and through results.latencyEvaluations;
    • two troubleshooting rows.
  • Simulations overview: the scenario definition, the "how it works" paragraph and the Evals comparison table now mention latency limits next to structured outputs.
  • Manage simulations: run results mention latency results and latencyEvaluations, "Design evaluations" points to latency limits, and the voice-or-chat table gains a "Gate on response latency" row.
  • GPT-Live testing: notes that latency limits are skipped when the assistant under test uses GPT-Live.

Every behavior described was checked against the merged code: field names, the default of a required median turn latency of 1,200 ms, the 1 to 60,000 ms range, the 20-limit maximum and the skip rules.

Not included

A "What's new" changelog entry. The weekly changelog files look curated by one author (e.g. #1280), so I left the entry to the next weekly post. Happy to add it here instead.

Testing

fern check with the pinned CLI 5.112.0 (node scripts/fern/run.cjs check): 0 errors. The 14 warnings are pre-existing discriminator warnings in the API spec. The preview link from CI is the place to check rendering.

🤖 Generated with Claude Code

Voice simulations can now fail when the assistant responds too slowly
(VapiAI/vapi#21959, #21962, #21963; TEST-140). Add a "Set latency limits"
section to Simulations advanced covering the Dashboard and API setup, the
turn, model and voice metrics, the aggregations, when limits are skipped
(chat mode, GPT-Live assistants) and how to read the results. Mention
latency limits where the overview, manage and GPT-Live testing pages
describe how a simulation is scored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@lightsage-app

lightsage-app Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Lightsage docs evals

Lightsage could not queue docs evals for this PR.

Docs URL: https://vapi-preview-01a11374-b04e-771e-a168-40438c9b76a6.docs.buildwithfern.com
Commit: d95a911
Reason: No custom evals are selected for GitHub PR evals. Open Lightsage > Custom Evals and enable the GitHub checkbox for at least one custom eval.

@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

@stephenvapiai stephenvapiai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two suggested wording changes for the latency limits section. Each suggestion can be accepted independently.

Comment thread fern/observability/simulations-advanced.mdx
Comment thread fern/observability/simulations-advanced.mdx Outdated
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

@scott-lowe-vapi
scott-lowe-vapi merged commit d09c2c6 into main Oct 6, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants