Skip to content

Reduce llama.cpp scoring latency - #19

Merged
NotXf1le merged 4 commits into
masterfrom
feature/llama-latency
Sep 24, 2026
Merged

NotXf1le merged 4 commits into
masterfrom
feature/llama-latency

Conversation

@NotXf1le

Copy link
Copy Markdown
Owner

What changed

  • Batch llama.cpp tokenization with a bounded fallback, and overlap vision capability checks with tokenization.
  • Remove a redundant image-prefix retokenization request.
  • Avoid extra logprob probes by default; retain exact scoring of missing choices with probeMissingLogprobs: true.

Why

These changes reduce HTTP round trips during choice scoring. The new default can assign zero probability to choices outside the returned top logprobs; callers that need their exact scores can opt in to additional probes.

Checks

  • npm run check — 166 tests passed
  • npm pack --dry-run
  • Live RTX llama.cpp checks of default and opt-in scoring

@NotXf1le
NotXf1le merged commit cd6d0cf into master Sep 24, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant