Added registry setup for metrics, with accompanying CLI option (#863)

## Summary

Creates a registry-based input for metrics.

## Details

- The `kind` for the metrics input is `generative`, since it's designed to go with generative benchmarks.
- Includes options for `sample_size` and `prefer_response_metrics`.
- Example of the arg: `--metrics kind=generative,sample_size=2`


## Test Plan

Run a benchmark with a low sample size, and count them in the output. The `extract_conversation.py` script is a good one for this.


---

- [x] "I certify that all code in this PR is my own, except as noted below."

## Use of AI

- [x] Includes code generated or substantially modified by an AI agent
- [x] Includes tests generated or substantially modified by an AI agent

> NOTE: the `Generated-by` or `Assisted-by` trailers should be used in git commit messages when code or tests were generated or substantially modified by an AI agent, as described in the project's [`DEVELOPING.md`](https://github.com/vllm-project/guidellm/blob/main/DEVELOPING.md) file.





---

# git log

commit b4c4335
Author: Jared O'Connell <joconnel@redhat.com>
Date:   Thu Jun 25 17:36:35 2026 -0400

    Added registry setup for metrics, with accompanying CLI option
    
    Generated-by: Cursor AI Claude Opus 4.6
    Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit c50a786
Author: Jared O'Connell <joconnel@redhat.com>
Date:   Thu Jun 25 17:53:28 2026 -0400

    Added docs for the sample size setting
    
    Generated-by: Cursor AI Claude Opus 4.6
    Signed-off-by: Jared O'Connell <joconnel@redhat.com>

commit 2af6e9f
Author: Jared O'Connell <joconnel@redhat.com>
Date:   Thu Jun 25 18:26:30 2026 -0400

    Fix linter errors
    
    Signed-off-by: Jared O'Connell <joconnel@redhat.com>

---------

Generated-by: Cursor AI Claude Opus 4.6
Signed-off-by: Jared O'Connell <joconnel@redhat.com>

Author: jaredoconnell

docs/guides/outputs.md

@@ -94,6 +94,31 @@ guidellm run \
   --output kind=json,path=results/benchmark.json
 ```
 
+### Controlling Output File Size
+
+Long benchmark runs with thousands of requests can produce large JSON and YAML output files because, by default, every request's full data (prompt text, output text, tool calls) is retained. The `--metrics` option lets you limit how much request data is kept using reservoir sampling, while lightweight stats (latency, token counts, timing) are always preserved for every request.
+
+Use `sample_size` to set the maximum number of requests **per status group** (completed, errored, incomplete) that retain their full data:
+
+| Value             | Behavior                                                   |
+| ----------------- | ---------------------------------------------------------- |
+| Not set (default) | Keep full data for all requests                            |
+| `0`               | Strip all request data (stats only)                        |
+| `N` (e.g. `100`)  | Retain full data for N randomly sampled requests per group |
+
+```bash
+# Keep full data for only 100 sampled requests per group
+guidellm run \
+  --backend kind=openai_http,target=http://localhost:8000 \
+  --profile kind=sweep \
+  --constraint kind=max_requests,count=10000 \
+  --data kind=synthetic_text,prompt_tokens=256,output_tokens=128 \
+  --metrics kind=generative,sample_size=100 \
+  --output kind=json,path=results/benchmark.json
+```
+
+The `--metrics` option also accepts `prefer_response_metrics` (default `true`), which controls whether server-reported token counts are preferred over client-calculated counts when both are available. This rarely needs to be changed.
+
 ### Reloading Results
 
 JSON files can be reloaded into Python for further analysis using the `GenerativeBenchmarksReport` class. Below is a sample code snippet for reloading results:

src/guidellm/benchmark/benchmarker.py

@@ -64,7 +64,7 @@ async def run(
         environment: Environment,
         warmup: TransientPhaseConfig,
         cooldown: TransientPhaseConfig,
-        sample_requests: int | None = None,
+        sample_size: int | None = None,
         prefer_response_metrics: bool = True,
         progress: (
             BenchmarkerProgress[BenchmarkAccumulatorT, BenchmarkT] | None
@@ -81,8 +81,9 @@ async def run(
         :param environment: Environment for execution coordination
         :param warmup: Warmup phase configuration before benchmarking
         :param cooldown: Cooldown phase configuration after benchmarking
-        :param sample_requests: Number of requests to sample for estimation,
-            defaults to 20
+        :param sample_size: Maximum number of requests per status group
+            (completed, errored, incomplete) to retain full data for.
+            None keeps all, 0 strips all, N > 0 uses reservoir sampling.
         :param prefer_response_metrics: Whether to prefer response metrics over
             request metrics, defaults to True
         :param progress: Optional tracker for benchmark lifecycle events
@@ -117,7 +118,7 @@ async def run(
                         if constraints
                         else {}
                     ),
-                    sample_requests=sample_requests,
+                    sample_size=sample_size,
                     warmup=warmup,
                     cooldown=cooldown,
                     prefer_response_metrics=prefer_response_metrics,

src/guidellm/benchmark/entrypoints.py

@@ -30,6 +30,7 @@
     GenerativeBenchmark,
     GenerativeBenchmarkAccumulator,
     GenerativeBenchmarksReport,
+    GenerativeMetricsArgs,
     ProfileArgs,
 )
 from guidellm.data import (
@@ -433,6 +434,13 @@ async def benchmark_generative_text(
     """
     benchmark_args = resolve_to_single_benchmark(args.get_benchmarks())
 
+    metrics_args = benchmark_args.metrics
+    if not isinstance(metrics_args, GenerativeMetricsArgs):
+        raise TypeError(
+            f"Expected GenerativeMetricsArgs for generative text benchmark, "
+            f"got {type(metrics_args).__name__}"
+        )
+
     backend, model = await resolve_backend(
         backend_args=benchmark_args.backend,
         console=console,
@@ -481,10 +489,10 @@ async def benchmark_generative_text(
         profile=profile,
         environment=NonDistributedEnvironment(),
         progress=progress,
-        sample_requests=None,
+        sample_size=metrics_args.sample_size,
         warmup=warmup,
         cooldown=cooldown,
-        prefer_response_metrics=True,
+        prefer_response_metrics=metrics_args.prefer_response_metrics,
     ):
         if benchmark:
             report.benchmarks.append(benchmark)

src/guidellm/benchmark/schemas/__init__.py

@@ -30,7 +30,13 @@
     TransientPhaseConfig,
 )
 from .benchmark import GenerativeBenchmark
-from .entrypoints import BenchmarkArgs, BenchmarkMetadata, BenchmarkScenario
+from .entrypoints import (
+    BenchmarkArgs,
+    BenchmarkMetadata,
+    BenchmarkScenario,
+    GenerativeMetricsArgs,
+    MetricsArgs,
+)
 from .metrics import (
     GenerativeAudioMetricsSummary,
     GenerativeImageMetricsSummary,
@@ -64,10 +70,12 @@
     "GenerativeImageMetricsSummary",
     "GenerativeMetrics",
     "GenerativeMetricsAccumulator",
+    "GenerativeMetricsArgs",
     "GenerativeMetricsSummary",
     "GenerativeRequestsAccumulator",
     "GenerativeTextMetricsSummary",
     "GenerativeVideoMetricsSummary",
+    "MetricsArgs",
     "ProfileArgs",
     "RandomArgs",
     "RunningMetricStats",

src/guidellm/benchmark/schemas/accumulator.py

@@ -576,10 +576,12 @@ class GenerativeRequestsAccumulator(StandardBaseModel):
     retention (clearing request arguments and/or outputs for non-sampled requests).
     """
 
-    sample_requests: int | None = Field(
+    sample_size: int | None = Field(
         default=None,
         description=(
-            "Number of requests to sample and keep in the final benchmark for metrics"
+            "Maximum number of requests in each status group to retain full data "
+            "(prompt, output, tool calls) for. Lightweight stats are always kept. "
+            "None keeps all, 0 strips all, N > 0 uses reservoir sampling."
         ),
     )
     requests_stats: list[GenerativeRequestStats] = Field(
@@ -663,18 +665,18 @@ def update_estimate(
         current_index = len(self.requests_stats)
         self.requests_stats.append(stats)
 
-        if self.sample_requests is None:
+        if self.sample_size is None:
             # Keeping all requests, don't need to sample
             self.samples = None
-        elif self.sample_requests <= 0:
+        elif self.sample_size <= 0:
             # Not keeping any requests, clear out unnecessary memory usage for current
             self.clear_stats_data(stats)
-        elif self.sample_requests >= len(self.requests_stats):
+        elif self.sample_size >= len(self.requests_stats):
             # Add directly to samples, haven't filled yet
             if self.samples is None:
                 self.samples = []
             self.samples.append(current_index)
-        elif self.sample_requests / len(self.requests_stats) >= random.random():
+        elif self.sample_size / len(self.requests_stats) >= random.random():
             # Sampling logic: choose to replace with decreasing probability s / n
             # where s is sample size, n is current number of requests.
             # If chosen, choose random existing sample to replace.
@@ -802,16 +804,16 @@ def model_post_init(self, __context):
         """
         Initialize child accumulators with config values after model construction.
 
-        Propagates sample_requests from config to child request accumulators to ensure
+        Propagates sample_size from config to child request accumulators to ensure
         consistent sampling behavior across completed, errored, and incomplete request
-        collections. This ensures the --sample-requests option functions correctly.
+        collections. This ensures the --metrics-sample-size option functions correctly.
         """
         super().model_post_init(__context)
 
-        # Propagate sample_requests from config to child accumulators
-        self.completed.sample_requests = self.config.sample_requests
-        self.errored.sample_requests = self.config.sample_requests
-        self.incomplete.sample_requests = self.config.sample_requests
+        # Propagate sample_size from config to child accumulators
+        self.completed.sample_size = self.config.sample_size
+        self.errored.sample_size = self.config.sample_size
+        self.incomplete.sample_size = self.config.sample_size
 
     def update_estimate(
         self,

src/guidellm/benchmark/schemas/base.py

@@ -275,9 +275,12 @@ class BenchmarkConfig(StandardBaseDict):
     constraints: dict[str, dict[str, Any]] = Field(
         description="Constraint definitions applied to scheduler strategy execution",
     )
-    sample_requests: int | None = Field(
+    sample_size: int | None = Field(
         default=None,
-        description="Request count for statistical sampling in final metrics",
+        description=(
+            "Maximum number of requests per status group to retain full data for. "
+            "None keeps all, 0 strips all, N > 0 uses reservoir sampling."
+        ),
     )
     warmup: TransientPhaseConfig = Field(
         default_factory=TransientPhaseConfig,

src/guidellm/benchmark/schemas/entrypoints.py

@@ -11,8 +11,9 @@
 from __future__ import annotations
 
 import json
+from abc import ABC
 from pathlib import Path
-from typing import Any
+from typing import Any, ClassVar, Literal
 
 import yaml
 from pydantic import (
@@ -39,6 +40,7 @@
 )
 from guidellm.scheduler.constraints import ConstraintArgs
 from guidellm.schemas import (
+    PydanticClassRegistryMixin,
     ReloadableBaseModel,
     StandardBaseModel,
     standard_model_config,
@@ -49,6 +51,8 @@
     "BenchmarkArgs",
     "BenchmarkMetadata",
     "BenchmarkScenario",
+    "GenerativeMetricsArgs",
+    "MetricsArgs",
 ]
 
 
@@ -77,6 +81,61 @@ def default_kind_list(*kinds: str) -> list[dict[str, Any]]:
     return [default_kind(kind) for kind in kinds]
 
 
+class MetricsArgs(PydanticClassRegistryMixin["MetricsArgs"], ABC):
+    """Base class for metrics collection arguments.
+
+    :cvar schema_discriminator: Field name for polymorphic deserialization
+    """
+
+    model_config = standard_model_config()
+
+    schema_discriminator: ClassVar[str] = "kind"
+
+    @classmethod
+    def __pydantic_schema_base_type__(cls) -> type[MetricsArgs]:
+        """
+        Return base type for polymorphic validation hierarchy.
+
+        :return: Base MetricsArgs class for schema validation
+        """
+        if cls.__name__ == "MetricsArgs":
+            return cls
+
+        return MetricsArgs
+
+    kind: str = Field(
+        description="The kind of metrics configuration to use.",
+    )
+
+
+@MetricsArgs.register("generative")
+class GenerativeMetricsArgs(MetricsArgs):
+    """Metrics configuration for generative (autoregressive) benchmarks."""
+
+    kind: Literal["generative"] = Field(
+        default="generative",
+        description="The kind of metrics configuration to use.",
+    )
+    sample_size: int | None = Field(
+        default=None,
+        description=(
+            "Maximum number of requests per status group (completed, errored, "
+            "incomplete) to retain full data (prompt, output, tool calls) for in "
+            "the final benchmark. Lightweight stats (latency, token counts) are "
+            "always kept for every request. None keeps all request data, 0 strips "
+            "all request data, N > 0 uses reservoir sampling to retain N per group."
+        ),
+        examples=[None, 0, 100],
+    )
+    prefer_response_metrics: bool = Field(
+        default=True,
+        description=(
+            "Prioritize server-reported metrics over client-calculated metrics "
+            "when both are available."
+        ),
+    )
+
+
 class BenchmarkArgs(ReloadableBaseModel):
     """Common benchmark configuration arguments."""
 
@@ -164,6 +223,11 @@ class BenchmarkArgs(ReloadableBaseModel):
         ],
         json_schema_extra={"argument_alias": "output"},
     )
+    metrics: MetricsArgs = Field(  # type: ignore[assignment]
+        default_factory=lambda: default_kind("generative"),
+        description="Configuration for metrics collection and request sampling.",
+        json_schema_extra={"argument_alias": "metrics"},
+    )
 
 
 class BenchmarkMetadata(StandardBaseModel):