Skip to content

[wip] feat(assistant): hidden lk assistant voice assistant - #1025

Draft
royalfig wants to merge 12 commits into
mainfrom
ryan/lk-assistant
Draft

royalfig wants to merge 12 commits into
mainfrom
ryan/lk-assistant

Conversation

@royalfig

@royalfig royalfig commented Oct 9, 2026

Copy link
Copy Markdown

Adds a hidden lk assistant command: LK, a voice assistant for LiveKit in a macOS overlay.

  • Blurs the desktop and shows the LiveKit logo as an audio meter.
  • Talks with a LiveKit voice agent (console mode, local audio) that answers from the Docs MCP server.
  • Shows code and prompt cards with Copy buttons.
  • Runs lk commands (create a project from a template, install skills, list agents) after the user confirms by voice or on screen.
  • "Meet your agent": runs the user's new agent in the same audio pipeline and switches the conversation to it. Press B to come back.
  • X-ray mode shows the pipeline's models and metrics.
  • Saves each conversation as Markdown in ~/Library/Caches/livekit/assistant-sessions/.

Supporting changes, in separate commits:

  • lk agent init . and lk app create . create the project in the empty current directory.
  • pkg/console: playback audio metrics, and multiple agent connections with an active one.
  • pkg/overlay: native macOS overlay (NSWindow, blur, WKWebView). Other platforms get a stub.
  • simulate: entrypoint lookup stays inside the project directory.

Try it: lk assistant (needs uv and LiveKit Cloud credentials). --windowed runs it in a normal window, --demo plays a scripted session without audio.

Known gaps (WIP):

  • Starting a second guest agent leaks the first.
  • Tool results that finish while the guest agent is active are lost.
  • Overlay settings in localStorage don't persist.

🤖 Generated with Claude Code

royalfig and others added 11 commits October 15, 2026 10:52
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…init .`

`lk agent init .` and `lk app create .` fill the empty current directory and name the project after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The pipeline measures the agent's playback audio as well as the microphone, and can hold several agent connections and switch the active one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LK is a voice assistant in a macOS overlay. It answers from the LiveKit docs, shows code and prompts to copy, runs lk commands after the user confirms, and lets the user talk to the agent they created.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…guest status

Press / to type instead of speaking. Answer cards show short points with their docs source, link cards open a docs page, and backticked text in the transcript shows as code. The guest bar shows when the user's agent is starting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Scan the folder and coding agent setup at startup, and greet for what's there.
- Run only agents created this session; give the command for existing ones.
- Fail fast when a guest agent's job ends before it connects.
- Docs answers on screen with their source, copy by voice, and typed input.
- Confirm in one step when the user already agreed.
- Open in a window by default; --fullscreen covers the screen.
- Pronounce LK, lk, and LiveKit's correctly.
- Add onboarding evals, run with real Inference and stubbed lk tools.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Run LK in Agent Simulations: scenarios, mocked lk tools, and tool-call checks.
- Add a primer so basic questions get an expert answer without a lookup.
- Say a lookup line only when a lookup is slow.
- Short greetings that offer what fits the folder.
- Speak in Simplified Technical English; explain concretely.
- Close on a spoken exit even if the agent doesn't call end_call.
- Cover the screen while the user's agent starts.
- Open docs pages, not their Markdown, and darken the backdrop.
- Allow native full screen in windowed mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iable turns

- Create the user's agent in an AgentTask that code runs: one confirm card for
  the language, then create, then offer to talk to it. A narrow model call
  reads each answer, so the model can't skip a step or pick a language.
- Return to docs-first answers with show_code for code, and read pages with
  read_docs, which limits pages and text in code.
- Give the agent a trimmed docs table of contents to pick pages by title.
- Check each user turn in the background: close LK when the user asks to
  leave, and show a code card when they asked for code and got none.
- Announce the hand-off to the user's agent before starting it, with how to
  come back, matched to the Python or Node.js starter.
- Show link cards after the answer, and a read-more link when the agent read
  docs without one.
- Wait for the goodbye before closing; show when the microphone is off.
- Choice cards in the overlay, STT keyterms, and LK can use itself as an
  example.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Copy the latest card of the kind the user asked for (link, code, or
  prompt), matching titles loosely, and report exactly what was copied.
- Always confirm before restarting the user's agent.
- Don't repeat a code card that's already on screen, and show one link card
  per reply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Pass the context and the user's folder to resolveCredentials, which now
  reads .env files from that folder.
- Compare findEntrypoint's result with forward slashes, which it returns on
  every platform.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant