You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Introduce a unified, lightweight, zero-dependency telemetry layer to
replace the legacy openinference namespace. Group OTLP transport
options under a nested otlp object, and replace the format string
with a semantics list (allowing multiple active tracing conventions).
This implementation:
1. Allows users to specify semantics: [openinference, otel_genai].
2. Collects and compiles trace attributes for OpenInference and
OpenTelemetry GenAI in a single pass when both are active.
3. Integrates queue capacity, retry/backoff, and batch exporting.
4. Refactors code structure, C++ config unit tests, python integration
tests, CLI commands, and markdown documentation.
Internal endpoints are used for server control and configuration. By default, they are secured by `LEMONADE_ADMIN_API_KEY` (if set) to separate control privileges from standard inference operations.
Dynamically toggles telemetry tracing on the server and logs the change. This change is in-memory only and is not persisted to the `config.json` file (reverts on server restart).
1347
+
1348
+
#### Parameters
1349
+
1350
+
The request body must be a JSON object with the following fields:
1351
+
1352
+
| Parameter | Type | Required | Description |
1353
+
|-----------|------|----------|-------------|
1354
+
|`enabled`| boolean | Yes | Whether to enable (`true`) or disable (`false`) telemetry tracing. |
1355
+
1356
+
Example request:
1357
+
1358
+
```bash
1359
+
curl -X POST http://localhost:13305/internal/telemetry \
1360
+
-H "Content-Type: application/json" \
1361
+
-d '{
1362
+
"enabled": false
1363
+
}'
1364
+
```
1365
+
1366
+
#### Response Format
1367
+
1368
+
Returns a JSON object indicating success and the new state:
Forces the in-memory telemetry queue to flush all buffered trace spans immediately to the configured OTLP collector. This call blocks until all currently queued spans are serialized and sent.
1381
+
1382
+
#### Parameters
1383
+
1384
+
None.
1385
+
1386
+
Example request:
1387
+
1388
+
```bash
1389
+
curl -X POST http://localhost:13305/internal/telemetry/flush
1390
+
```
1391
+
1392
+
#### Response Format
1393
+
1394
+
Returns a JSON object indicating successful completion of the flush operation:
Copy file name to clipboardExpand all lines: docs/guide/cli.md
+25Lines changed: 25 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,6 +16,7 @@ The `lemonade` CLI is the primary tool for interacting with Lemonade Server from
16
16
-[Options for launch](#options-for-launch)
17
17
-[Options for bench](#options-for-bench)
18
18
-[Options for scan](#options-for-scan)
19
+
-[Options for telemetry](#options-for-telemetry)
19
20
20
21
## Commands
21
22
@@ -38,6 +39,7 @@ The `lemonade` CLI is the primary tool for interacting with Lemonade Server from
38
39
|`backends`| List supported recipes and backends or list all available recipes and backends with `--all`. Use `install` or `uninstall` to manage backends. |
39
40
|`cloud`| Manage cloud OpenAI-compatible providers. See command options [below](#options-for-cloud). |
40
41
|`scan`| Scan for network beacons on the local network. See command options [below](#options-for-scan). |
42
+
|`telemetry`| Dynamically enable or disable telemetry tracing. See command options [below](#options-for-telemetry). |
41
43
42
44
### Model Management
43
45
@@ -659,6 +661,29 @@ lemonade scan
659
661
lemonade scan --duration 5
660
662
```
661
663
664
+
## Options for telemetry
665
+
666
+
Dynamically toggle telemetry tracing on the server. This setting is applied immediately in-memory without requiring a server restart, but is not persisted to the server's `config.json` (meaning it will revert when the server restarts).
667
+
668
+
```bash
669
+
lemonade telemetry <on|off>
670
+
```
671
+
672
+
| Argument | Description |
673
+
|----------|-------------|
674
+
|`on`| Enable telemetry tracing. |
675
+
|`off`| Disable telemetry tracing. |
676
+
677
+
**Examples:**
678
+
679
+
```bash
680
+
# Enable telemetry tracing dynamically
681
+
lemonade telemetry on
682
+
683
+
# Disable telemetry tracing dynamically
684
+
lemonade telemetry off
685
+
```
686
+
662
687
## Options for bench
663
688
664
689
The `bench` command measures chat completion performance (TTFT and tokens-per-second) for one or more models across one or more installed backends, context sizes, and scenario workloads. It sends `POST /api/v1/chat/completions` requests and extracts timing data from the server response.
Copy file name to clipboardExpand all lines: docs/guide/configuration/README.md
+85-3Lines changed: 85 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -33,7 +33,7 @@ Values set in the user's `config.json` always take precedence over these seeded
33
33
34
34
```json
35
35
{
36
-
"config_version": 1,
36
+
"config_version": 2,
37
37
"port": 13305,
38
38
"host": "localhost",
39
39
"log_level": "info",
@@ -83,13 +83,30 @@ Values set in the user's `config.json` always take precedence over these seeded
83
83
"vulkan_bin": "builtin"
84
84
},
85
85
"flm": {
86
-
"args": "",
86
+
"args": ""
87
87
},
88
88
"ryzenai": {
89
89
"server_bin": "builtin"
90
90
},
91
91
"kokoro": {
92
92
"cpu_bin": "builtin"
93
+
},
94
+
"telemetry": {
95
+
"enabled": false,
96
+
"hide_inputs": false,
97
+
"hide_outputs": false,
98
+
"hide_thinking": false,
99
+
"max_queue_capacity": 1000,
100
+
"otlp": {
101
+
"endpoint": "http://localhost:4317",
102
+
"protocol": "http/protobuf",
103
+
"semantics": ["openinference", "otel_genai"],
104
+
"headers": {},
105
+
"max_retries": 0,
106
+
"retry_backoff_base_s": 5.0,
107
+
"send_batch_size": 100,
108
+
"batch_timeout_s": 1.0
109
+
}
93
110
}
94
111
}
95
112
```
@@ -171,6 +188,71 @@ Backend-specific settings are nested under their backend name:
171
188
172
189
API keys for these providers are **not** stored in `config.json` — they live in `LEMONADE_<PROVIDER>_API_KEY` env vars (persistent) or `lemond` process memory via `POST /v1/cloud/auth` (ephemeral). Manage providers with `lemonade cloud install/uninstall/auth/list` rather than editing this section by hand.
173
190
191
+
**telemetry** — Unified telemetry and tracing configurations:
|`semantics`| array of strings |["openinference", "otel_genai"]| Active trace semantics. Supported values: `"openinference"` and `"otel_genai"`. |
209
+
|`headers`| object | {} | Map of custom HTTP headers to pass to the OTLP receiver. |
210
+
|`max_retries`| int | 0 | Maximum number of retry attempts for failed exports. Set to `0` to disable retries and discard failed spans immediately. Must be `>= 0`. |
211
+
|`retry_backoff_base_s`| double | 5.0 | Base delay in seconds for exponential backoff retries. Must be `>= 0`. |
212
+
|`send_batch_size`| int | 100 | Target maximum number of spans to group in a single batched OTLP request. Must be `>= 1`. |
213
+
|`batch_timeout_s`| double | 1.0 | Maximum time to wait in seconds before exporting a partially filled batch of spans. Must be `> 0`. |
214
+
215
+
#### Environment Variables
216
+
217
+
The following environment variables can be used to override telemetry configuration at runtime:
218
+
219
+
-**`TELEMETRY_ENABLED`**: Overrides `telemetry.enabled` (e.g. `true` or `false`).
When both semantics are specified in `telemetry.otlp.semantics`, trace spans carry attributes for both conventions in a single network payload. This allows the collector to parse either convention without duplicate network requests.
241
+
-**Dynamic Attribute Prefixing**: Span attributes are dynamically prefixed based on the query type to simplify filtering:
242
+
-`llm.*` for standard chat and completion spans.
243
+
-`embedding.*` for text embedding generation spans.
244
+
-`reranker.*` for document reranking spans.
245
+
-**Token Tracking**: Captures and reports token usage metrics using semantic attributes depending on the enabled semantics:
246
+
- For **OpenInference**: Token count is prefixed with `llm.token_count` across all span kinds (`llm.token_count.prompt`, `llm.token_count.completion`, `llm.token_count.total`) alongside legacy keys like `llm.usage.prompt_tokens`.
247
+
- For **OpenTelemetry GenAI**: Token count uses standard fields like `gen_ai.usage.input_tokens` and `gen_ai.usage.output_tokens`.
248
+
-**Calculated Performance Metrics**: In streaming mode, the server automatically computes and records throughput (`llm.performance.tokens_per_second` / `gen_ai.usage.tokens_per_second` depending on semantic conventions) and prefill latency (`llm.performance.time_to_first_token` / `gen_ai.performance.time_to_first_token`) if not natively returned by the backend (e.g., for vLLM and Cloud models).
249
+
-**vLLM Engine Telemetry**: For the vLLM backend, the server queries the local `/metrics` endpoint on completion to attach scheduler queue metrics (`llm.vllm.num_requests_waiting`, `llm.vllm.num_requests_running`, `llm.vllm.num_requests_swapped`) and KV cache utilization (`llm.vllm.gpu_cache_usage_factor`, `llm.vllm.cpu_cache_usage_factor`) directly to the trace spans.
250
+
-**Reasoning Model Support**: For reasoning models (e.g., DeepSeek models), the server extracts and records `reasoning_content` from the assistant's generation. Any variant thought-termination tags (e.g., `</think|>`) are automatically standardized to the canonical `</think>` tag.
251
+
-**Exporter Retry Backoff**: When retries are enabled (i.e., `max_retries > 0`), the exporter uses an exponential backoff strategy combined with randomized jitter for failed posts. The base retry interval starts at `retry_backoff_base_s` seconds (defaulting to 5), doubling on each subsequent failure (e.g., 5s, 10s, 20s, 40s), up to a maximum cap of 60 seconds. A randomized jitter factor between `0.5` and `1.5` is applied to each calculated delay to prevent a "thundering herd" when the collector recovers. Permanent client errors (`4xx` HTTP status codes, excluding `429 Too Many Requests`) are classified as non-retryable and cause the batch to be dropped immediately to save resources.
252
+
-**OTLP Trace Batching**: Spans are aggregated in an in-memory queue buffer and exported in batches to minimize network overhead and maximize compression efficiency. Batching operates on a dual-trigger system: a batch is immediately serialized and dispatched if it reaches `send_batch_size` (default: `100`), or if `batch_timeout_s` (default: `1.0` second) has elapsed since the oldest span in the batch arrived. All remaining traces are flushed cleanly to the OTel collector upon server shutdown. Users can also trigger a manual flush at any time via the `POST /internal/telemetry/flush` endpoint.
253
+
-**Request Failure Tracing**: Captures request failures directly on the telemetry spans. If a model fails to load, a request is rejected by the router, or a streaming connection encounters an exception or a non-200 HTTP status code from the backend, the span is ended with `Error` status and the specific error message is attached.
254
+
-**Queue Blocking & Thundering Herd Prevention**: To prevent client requests from hanging and to avoid exhausting resources when the telemetry receiver endpoint is down, Lemonade employs a fail-fast mechanism. The exporter memory buffer is strictly bounded to a capacity of `max_queue_capacity` spans (default: `1000`). When full, a head-drop (FIFO) eviction policy is applied to drop the oldest telemetry spans to make room for newer ones, prioritizing current application state. If a telemetry transmission task fails all of its retries and is dropped, the endpoint is marked as **unreachable**. While in this unreachable state, subsequent spans in the transmission queue are attempted only once and immediately dropped without backoff delay if they fail, preventing the telemetry queue from blocking server operations. A single successful span delivery to the endpoint automatically resets the unreachable state and restores normal retry behavior.
255
+
174
256
### Backend binary selection
175
257
176
258
Every `*_bin` key (e.g. `llamacpp.vulkan_bin`, `whispercpp.cpu_bin`, `sdcpp.rocm_bin`) accepts the same set of values:
> **Important — `llamacpp.rocm_bin` version tags are channel-specific.** Each ROCm channel downloads from a different GitHub repository, so you must set the correct `rocm_channel` before pinning `rocm_bin` to a specific tag. See [Pinning to a Specific Version Tag](./llamacpp.md#pinning-to-a-specific-version-tag) for details.
270
+
> Note: `llamacpp.rocm_bin` version tags are channel-specific. Each ROCm channel downloads from a different GitHub repository, so you must set the correct `rocm_channel` before pinning `rocm_bin` to a specific tag. See [Pinning to a Specific Version Tag](./llamacpp.md#pinning-to-a-specific-version-tag) for details.
0 commit comments