You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[GRV-27]: Resolve estimation API and catalog drift (#43)
## What & why
`llm-cost-estimation` re-exported attribution's implemented forecaster
but its catalog still described `ForecastIssueCost` as a planned stub,
and it exposed `forecastProjectCost` / `calibrate` as throwing `not
implemented` placeholders — even though their implemented analogs
already shipped in `llm-cost-attribution`. This PR makes the docs,
README, public exports, and tests agree on what is implemented vs.
unsupported, turning the catalog/API alignment check green for
estimation.
Resolves
[GRV-27](https://linear.app/riddimsoftware/issue/GRV-27/resolve-estimation-api-and-catalog-drift).
## Changes
- **`src/index.mjs`**
- `forecastProjectCost` now re-exports attribution's implemented project
forecaster — estimation-friendly signature `forecastProjectCost(issues,
usageSource?, options?)` with `{ size, model }` IssuePlans, consistent
with `forecastIssueCost`.
- Adds `calibrateCoverage`, re-exported from attribution's implemented
coverage backtester.
- `calibrate` kept as a deprecated compatibility shim; its error names
the supported replacement (`calibrateCoverage`) instead of `not
implemented`.
- **`docs/use-cases.md`** — rewrites `ForecastIssueCost` and
`ForecastProjectCost` to describe shipped behavior (no longer "Planned —
stub throws"), adds a `CalibrateCoverage` entry, and documents
`calibrate` as deprecated. `ForecastIssueCost` now reflects the real
`(cell, records)` shape rather than the never-built `CalibrationDataset`
model.
- **`README.md`** — drops `calibrate` from the ready-use import example;
documents the real `forecastProjectCost` / `calibrateCoverage`
signatures and the `calibrate` deprecation.
- **`test/smoke.test.mjs`** — no longer asserts ready APIs throw `not
implemented`: `forecastProjectCost` forecasts an empty project,
`calibrateCoverage` is asserted exported, and `calibrate`'s error is
asserted to name its replacement.
## Acceptance criteria
- [x] `docs/use-cases.md` accurately describes shipped behavior.
- [x] `ForecastIssueCost` no longer described as a planned stub.
- [x] README import examples do not advertise throwing APIs as
ready-to-use.
- [x] `forecastProjectCost` delegates to attribution's implemented
project forecaster with a documented estimation-friendly signature.
- [x] `calibrate` documented as deprecated and omitted from ready-use
examples; `calibrateCoverage` delegates to the implemented
calibration/coverage API.
- [x] Barrel smoke tests no longer assert advertised ready APIs throw
`not implemented`.
- [x] The one remaining throwing export (`calibrate`) names its
supported replacement.
- [x] Existing enrichment and CLI forecast behavior preserved (no
changes to `enrich.mjs`, `linear-estimate-source.mjs`, `bin/`).
- [x] Catalog/API alignment turns green for estimation.
- [x] Estimation package tests pass.
## Out of scope (unchanged)
No attribution forecast-math changes, no project-level quota forecasts,
no npm publishing / version bumps.
## Verification
```
node --test packages/llm-cost-estimation/test/ # 37 passing, 0 failing
node --check packages/llm-cost-estimation/src/*.mjs # OK
```
Re-verified after rebasing onto `origin/main` (which merged GRV-24's
attribution port extraction): the `forecastProjectCost` /
`calibrateCoverage` re-exports still resolve and all 37 tests pass.
Co-authored-by: riddim-developer-bot <developer-bot@riddimsoftware.com>
Copy file name to clipboardExpand all lines: packages/llm-cost-estimation/docs/use-cases.md
+47-24Lines changed: 47 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,6 +15,12 @@ The stats and forecaster core depends only on the `EstimateTaggedUsageSource` an
15
15
`LinearEstimateSource` ports. No Linear SDK, HTTP client, or filesystem import may
16
16
cross inward into the domain/application layer.
17
17
18
+
The forecast and calibration use cases below are re-exported from
19
+
`llm-cost-attribution`, which owns the empirical-quantile forecaster, the project
20
+
forecaster, and the coverage backtester. The estimation package surfaces them
21
+
through its public barrel so callers import one package; it does not re-implement
22
+
the math.
23
+
18
24
---
19
25
20
26
## EnrichUsageWithEstimate
@@ -43,17 +49,17 @@ absent.
43
49
44
50
| Field | Value |
45
51
|---|---|
46
-
|**Goal**| Forecast the expected LLM cost for a single issue before work begins, given its story-point estimate and a calibration dataset built from completed issues with known actual cost. |
|**Ports**|`EstimateTaggedUsageSource`— supplies completed-issue records with `estimate` already stamped, used at calibration time (see `calibrate()`). The `forecastIssueCost` call itself takes no port — it runs over an already-built `CalibrationDataset` value. |
51
-
|**Primary adapters**|`UsageJsonlEstimateTaggedSource` (planned) — reads enriched `usage.jsonl` files produced by `EnrichUsageWithEstimate`. |
52
+
|**Goal**| Forecast the expected LLM cost for a single issue before work begins, from the historical cost of completed issues that share its `{ size, model }` cell. |
53
+
|**Inputs**|`cell: { size, model }`; `records` — estimate-tagged usage records (the cell's history), the same source shape the attribution forecaster accepts. |
54
+
|**Outputs**|`IssueCostForecast` — `tokens`, `turns`, and `dollars` channels each with P50/P80 and `n`; an optional Codex `quota`-fraction P50/P80 (single-issue only, never aggregated to a project) with `quotaReason`; plus `lowConfidence`and `empty` markers. |
|**Ports**|`EstimateTaggedUsageSource` supplies the records; `PricingTable` (`priceFor(model, buckets)`) prices the dollars channel; `QuotaModel` (`quotaFractionFor(record)`) reads the Codex quota fraction. All three are injected — the forecaster imports none of their implementations. |
57
+
|**Primary adapters**|`usage.jsonl` reader (`EstimateTaggedUsageSource`), synthetic record source in tests, `pricing.mjs` (`PricingTable`), `quota.mjs` (`QuotaModel`) — all in `llm-cost-attribution`. |
|**Current implementation**|Re-exported from `llm-cost-attribution` (`packages/llm-cost-attribution/src/forecast.mjs`). |
54
60
55
61
**Boundary rule:** The empirical quantile forecaster is a pure function over the
56
-
calibration dataset. No Linear SDK, HTTP, or filesystem call occurs during
62
+
records and its injected ports. No Linear SDK, HTTP, or filesystem call occurs during
57
63
`forecastIssueCost`.
58
64
59
65
---
@@ -62,28 +68,45 @@ calibration dataset. No Linear SDK, HTTP, or filesystem call occurs during
62
68
63
69
| Field | Value |
64
70
|---|---|
65
-
|**Goal**| Forecast the aggregate LLM cost for an entire projectby Monte Carlo convolution over the per-issue cost distributions for each issue's estimate. |
66
-
|**Inputs**|`issues: { identifier: string, estimate: number }[]` (project issues with story-point estimates); `calibration: CalibrationDataset`. |
67
-
|**Outputs**|`ProjectCostForecast` — aggregate quantile distribution (p50, p80, p95) and dollar API-equivalent for the project total. |
|**Ports**|`EstimateTaggedUsageSource` — used at calibration time only (see `calibrate()`). The `forecastProjectCost`call itself takes no port. |
70
-
|**Primary adapters**|`UsageJsonlEstimateTaggedSource` (planned) — same adapter as `ForecastIssueCost`. |
71
+
|**Goal**| Forecast an entire project's aggregate cost by Monte Carlo convolution over its planned issues, so summed quantiles don't over-estimate the total (per-issue tail risks partly diversify). |
|**Outputs**|`ProjectCostForecast` — `tokens`, `turns`, and (when priced) `dollars` channels, each with P50/P80, `mean`, and `variance`; plus `iterations`, `seed`, `issues`, `lowConfidence`, and `empty`. |
74
+
|**Entities / values**|`IssuePlan` (`{ size, model }`); `ProjectCostForecast`; `ProjectChannelForecast`. |
75
+
|**Ports**|The per-issue forecaster's empirical cell sampler (`collectCellSamples`); the shared `PricingTable` (`priceFor(model, buckets)`); a seeded RNG. The call reads records through the sampler, not directly. |
76
+
|**Primary adapters**|Seeded RNG (mulberry32, in-module); `pricing.mjs` (`PricingTable`) — both in `llm-cost-attribution`. |
|**Current implementation**| Re-exported from `llm-cost-attribution` (`packages/llm-cost-attribution/src/project-forecast.mjs`). |
79
+
80
+
**Scope:** tokens, turns, and dollars only. Project-level quota and wall-clock are
81
+
excluded by construction — they are windowed / scheduling quantities that do not sum
82
+
across issues.
73
83
74
84
**Boundary rule:** The Monte Carlo convolution is pure. No Linear SDK, HTTP, or
75
85
filesystem call occurs during `forecastProjectCost`.
76
86
77
87
---
78
88
79
-
## calibrate (support function, not a use case)
89
+
## CalibrateCoverage
90
+
91
+
| Field | Value |
92
+
|---|---|
93
+
|**Goal**| Backtest the forecaster: hold out a deterministic fraction of each cell's issues, fit on the rest, and measure how often held-out actuals land at or below the predicted P80 — so a "P80" only keeps the name if it covers ~80% of held-out cost. |
|**Outputs**|`CalibrationReport` — per-cell and overall coverage, with cells whose coverage drifts from the target band beyond `deviationThreshold` flagged. |
0 commit comments