Skip to content

Commit 347b6dc

Browse files
riddim-developer-bot[bot]riddim-developer-bot
andauthored
[GRV-27]: Resolve estimation API and catalog drift (#43)
## What & why `llm-cost-estimation` re-exported attribution's implemented forecaster but its catalog still described `ForecastIssueCost` as a planned stub, and it exposed `forecastProjectCost` / `calibrate` as throwing `not implemented` placeholders — even though their implemented analogs already shipped in `llm-cost-attribution`. This PR makes the docs, README, public exports, and tests agree on what is implemented vs. unsupported, turning the catalog/API alignment check green for estimation. Resolves [GRV-27](https://linear.app/riddimsoftware/issue/GRV-27/resolve-estimation-api-and-catalog-drift). ## Changes - **`src/index.mjs`** - `forecastProjectCost` now re-exports attribution's implemented project forecaster — estimation-friendly signature `forecastProjectCost(issues, usageSource?, options?)` with `{ size, model }` IssuePlans, consistent with `forecastIssueCost`. - Adds `calibrateCoverage`, re-exported from attribution's implemented coverage backtester. - `calibrate` kept as a deprecated compatibility shim; its error names the supported replacement (`calibrateCoverage`) instead of `not implemented`. - **`docs/use-cases.md`** — rewrites `ForecastIssueCost` and `ForecastProjectCost` to describe shipped behavior (no longer "Planned — stub throws"), adds a `CalibrateCoverage` entry, and documents `calibrate` as deprecated. `ForecastIssueCost` now reflects the real `(cell, records)` shape rather than the never-built `CalibrationDataset` model. - **`README.md`** — drops `calibrate` from the ready-use import example; documents the real `forecastProjectCost` / `calibrateCoverage` signatures and the `calibrate` deprecation. - **`test/smoke.test.mjs`** — no longer asserts ready APIs throw `not implemented`: `forecastProjectCost` forecasts an empty project, `calibrateCoverage` is asserted exported, and `calibrate`'s error is asserted to name its replacement. ## Acceptance criteria - [x] `docs/use-cases.md` accurately describes shipped behavior. - [x] `ForecastIssueCost` no longer described as a planned stub. - [x] README import examples do not advertise throwing APIs as ready-to-use. - [x] `forecastProjectCost` delegates to attribution's implemented project forecaster with a documented estimation-friendly signature. - [x] `calibrate` documented as deprecated and omitted from ready-use examples; `calibrateCoverage` delegates to the implemented calibration/coverage API. - [x] Barrel smoke tests no longer assert advertised ready APIs throw `not implemented`. - [x] The one remaining throwing export (`calibrate`) names its supported replacement. - [x] Existing enrichment and CLI forecast behavior preserved (no changes to `enrich.mjs`, `linear-estimate-source.mjs`, `bin/`). - [x] Catalog/API alignment turns green for estimation. - [x] Estimation package tests pass. ## Out of scope (unchanged) No attribution forecast-math changes, no project-level quota forecasts, no npm publishing / version bumps. ## Verification ``` node --test packages/llm-cost-estimation/test/ # 37 passing, 0 failing node --check packages/llm-cost-estimation/src/*.mjs # OK ``` Re-verified after rebasing onto `origin/main` (which merged GRV-24's attribution port extraction): the `forecastProjectCost` / `calibrateCoverage` re-exports still resolve and all 37 tests pass. Co-authored-by: riddim-developer-bot <developer-bot@riddimsoftware.com>
1 parent 315cb62 commit 347b6dc

4 files changed

Lines changed: 137 additions & 57 deletions

File tree

packages/llm-cost-estimation/README.md

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -99,8 +99,8 @@ llm-cost-estimate --size 3 --model claude-sonnet-4-6 --from-usage ./usage.jsonl
9999
import {
100100
forecastIssueCost,
101101
forecastProjectCost,
102+
calibrateCoverage,
102103
enrichUsageWithEstimate,
103-
calibrate,
104104
createLinearEstimateSource,
105105
} from 'llm-cost-estimation';
106106
```
@@ -122,15 +122,38 @@ Core transform for adding estimates to usage telemetry.
122122
- Returns `{ records, unresolved, stats }`.
123123
- Issues with no estimate are left untouched and listed in `unresolved`.
124124

125-
### `forecastProjectCost(projectId, issues, options?)`
125+
### `forecastProjectCost(issues, usageSource?, options?)`
126126

127-
Public API placeholder for project rollups.
128-
Throws `Error('not implemented')` until the next sequencing issue lands.
127+
Forecast a whole project's cost by Monte Carlo convolution over its planned
128+
issues. Re-exported from [`llm-cost-attribution`](../llm-cost-attribution).
129129

130-
### `calibrate(completedIssues, options?)`
130+
- `issues` is an array of `{ size, model }` plans — one per planned issue, the
131+
same cell model `forecastIssueCost` reads.
132+
- `usageSource` is estimate-tagged usage records (the project's history).
133+
- `options` accepts `{ iterations, seed, minSampleSize, pricingTable }`.
134+
- Returns tokens / turns / (priced) dollars channels — each with P50, P80,
135+
`mean`, `variance` — plus `iterations`, `seed`, `issues`, `lowConfidence`, and
136+
`empty`.
131137

132-
Public API placeholder for empirical calibration from completed work.
133-
Throws `Error('not implemented')` until the next sequencing issue lands.
138+
Summing per-issue P80s over-estimates a project total because tail risks partly
139+
diversify; the convolution corrects for that. Project-level **quota** is not
140+
forecast — quota is a per-issue windowed quantity that doesn't sum.
141+
142+
### `calibrateCoverage(records, options?)`
143+
144+
Backtest the forecaster: on held-out issues, does the actual cost really land at
145+
or below the predicted P80 about 80% of the time? Re-exported from
146+
[`llm-cost-attribution`](../llm-cost-attribution).
147+
148+
- `records` are estimate-tagged usage records.
149+
- `options` accepts `{ seed, holdoutFraction, quantile, deviationThreshold, minHoldout, minTrain }`.
150+
- Returns a per-cell and overall coverage report; cells whose coverage drifts
151+
from the target band beyond `deviationThreshold` are flagged.
152+
153+
### `calibrate` — deprecated
154+
155+
`calibrate` was a never-implemented placeholder. It now throws an error naming
156+
its replacement, `calibrateCoverage` (above). Do not use it.
134157

135158
## What it doesn't do
136159

packages/llm-cost-estimation/docs/use-cases.md

Lines changed: 47 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,12 @@ The stats and forecaster core depends only on the `EstimateTaggedUsageSource` an
1515
`LinearEstimateSource` ports. No Linear SDK, HTTP client, or filesystem import may
1616
cross inward into the domain/application layer.
1717

18+
The forecast and calibration use cases below are re-exported from
19+
`llm-cost-attribution`, which owns the empirical-quantile forecaster, the project
20+
forecaster, and the coverage backtester. The estimation package surfaces them
21+
through its public barrel so callers import one package; it does not re-implement
22+
the math.
23+
1824
---
1925

2026
## EnrichUsageWithEstimate
@@ -43,17 +49,17 @@ absent.
4349

4450
| Field | Value |
4551
|---|---|
46-
| **Goal** | Forecast the expected LLM cost for a single issue before work begins, given its story-point estimate and a calibration dataset built from completed issues with known actual cost. |
47-
| **Inputs** | `estimate: number` (story-point estimate); `calibration: CalibrationDataset` (from `calibrate()`). |
48-
| **Outputs** | `IssueCostForecast`empirical quantile distribution of expected cost (p50, p80, p95) and dollar API-equivalent. |
49-
| **Entities / values** | `CalibrationDataset` — cost samples grouped by story-point estimate; `IssueCostForecast` — quantile estimates. |
50-
| **Ports** | `EstimateTaggedUsageSource` supplies completed-issue records with `estimate` already stamped, used at calibration time (see `calibrate()`). The `forecastIssueCost` call itself takes no port — it runs over an already-built `CalibrationDataset` value. |
51-
| **Primary adapters** | `UsageJsonlEstimateTaggedSource` (planned) — reads enriched `usage.jsonl` files produced by `EnrichUsageWithEstimate`. |
52+
| **Goal** | Forecast the expected LLM cost for a single issue before work begins, from the historical cost of completed issues that share its `{ size, model }` cell. |
53+
| **Inputs** | `cell: { size, model }`; `records` — estimate-tagged usage records (the cell's history), the same source shape the attribution forecaster accepts. |
54+
| **Outputs** | `IssueCostForecast``tokens`, `turns`, and `dollars` channels each with P50/P80 and `n`; an optional Codex `quota`-fraction P50/P80 (single-issue only, never aggregated to a project) with `quotaReason`; plus `lowConfidence` and `empty` markers. |
55+
| **Entities / values** | `Cell` / `FeatureRecord` (`{ size, model }`); `IssueCostForecast`. |
56+
| **Ports** | `EstimateTaggedUsageSource` supplies the records; `PricingTable` (`priceFor(model, buckets)`) prices the dollars channel; `QuotaModel` (`quotaFractionFor(record)`) reads the Codex quota fraction. All three are injected — the forecaster imports none of their implementations. |
57+
| **Primary adapters** | `usage.jsonl` reader (`EstimateTaggedUsageSource`), synthetic record source in tests, `pricing.mjs` (`PricingTable`), `quota.mjs` (`QuotaModel`) — all in `llm-cost-attribution`. |
5258
| **Linear issues** | GRV-3, GRV-5 |
53-
| **Status** | Planned — stub throws `Error('not implemented')`. |
59+
| **Current implementation** | Re-exported from `llm-cost-attribution` (`packages/llm-cost-attribution/src/forecast.mjs`). |
5460

5561
**Boundary rule:** The empirical quantile forecaster is a pure function over the
56-
calibration dataset. No Linear SDK, HTTP, or filesystem call occurs during
62+
records and its injected ports. No Linear SDK, HTTP, or filesystem call occurs during
5763
`forecastIssueCost`.
5864

5965
---
@@ -62,28 +68,45 @@ calibration dataset. No Linear SDK, HTTP, or filesystem call occurs during
6268

6369
| Field | Value |
6470
|---|---|
65-
| **Goal** | Forecast the aggregate LLM cost for an entire project by Monte Carlo convolution over the per-issue cost distributions for each issue's estimate. |
66-
| **Inputs** | `issues: { identifier: string, estimate: number }[]` (project issues with story-point estimates); `calibration: CalibrationDataset`. |
67-
| **Outputs** | `ProjectCostForecast`aggregate quantile distribution (p50, p80, p95) and dollar API-equivalent for the project total. |
68-
| **Entities / values** | `CalibrationDataset`; `ProjectCostForecast` — aggregate quantile estimates. |
69-
| **Ports** | `EstimateTaggedUsageSource` — used at calibration time only (see `calibrate()`). The `forecastProjectCost` call itself takes no port. |
70-
| **Primary adapters** | `UsageJsonlEstimateTaggedSource` (planned) — same adapter as `ForecastIssueCost`. |
71+
| **Goal** | Forecast an entire project's aggregate cost by Monte Carlo convolution over its planned issues, so summed quantiles don't over-estimate the total (per-issue tail risks partly diversify). |
72+
| **Inputs** | `issues: { size, model }[]` (one IssuePlan per planned issue); `usageSource` — estimate-tagged usage records; `options: { iterations, seed, minSampleSize, pricingTable }`. |
73+
| **Outputs** | `ProjectCostForecast``tokens`, `turns`, and (when priced) `dollars` channels, each with P50/P80, `mean`, and `variance`; plus `iterations`, `seed`, `issues`, `lowConfidence`, and `empty`. |
74+
| **Entities / values** | `IssuePlan` (`{ size, model }`); `ProjectCostForecast`; `ProjectChannelForecast`. |
75+
| **Ports** | The per-issue forecaster's empirical cell sampler (`collectCellSamples`); the shared `PricingTable` (`priceFor(model, buckets)`); a seeded RNG. The call reads records through the sampler, not directly. |
76+
| **Primary adapters** | Seeded RNG (mulberry32, in-module); `pricing.mjs` (`PricingTable`) — both in `llm-cost-attribution`. |
7177
| **Linear issue** | GRV-6 |
72-
| **Status** | Planned — stub throws `Error('not implemented')`. |
78+
| **Current implementation** | Re-exported from `llm-cost-attribution` (`packages/llm-cost-attribution/src/project-forecast.mjs`). |
79+
80+
**Scope:** tokens, turns, and dollars only. Project-level quota and wall-clock are
81+
excluded by construction — they are windowed / scheduling quantities that do not sum
82+
across issues.
7383

7484
**Boundary rule:** The Monte Carlo convolution is pure. No Linear SDK, HTTP, or
7585
filesystem call occurs during `forecastProjectCost`.
7686

7787
---
7888

79-
## calibrate (support function, not a use case)
89+
## CalibrateCoverage
90+
91+
| Field | Value |
92+
|---|---|
93+
| **Goal** | Backtest the forecaster: hold out a deterministic fraction of each cell's issues, fit on the rest, and measure how often held-out actuals land at or below the predicted P80 — so a "P80" only keeps the name if it covers ~80% of held-out cost. |
94+
| **Inputs** | `records` — estimate-tagged usage records; `options: { seed, holdoutFraction, quantile, deviationThreshold, minHoldout, minTrain }`. |
95+
| **Outputs** | `CalibrationReport` — per-cell and overall coverage, with cells whose coverage drifts from the target band beyond `deviationThreshold` flagged. |
96+
| **Entities / values** | `CalibrationReport`; `CalibrationCellReport`. |
97+
| **Ports** | None beyond the per-issue forecaster it fits with (`forecastIssueCost`). Pure and I/O-free; callers stream records in. |
98+
| **Primary adapters** | Seeded held-out split (mulberry32, in-module) — in `llm-cost-attribution`. |
99+
| **Linear issue** | GRV-3 |
100+
| **Current implementation** | Re-exported as `calibrateCoverage` from `llm-cost-attribution` (`packages/llm-cost-attribution/src/calibrate.mjs`). |
101+
102+
**Boundary rule:** The backtest is a pure function over the supplied records. No Linear
103+
SDK, HTTP, or filesystem call occurs during `calibrateCoverage`.
104+
105+
---
80106

81-
Builds a `CalibrationDataset` from completed issues whose actual cost is known.
82-
Used as a preparation step before calling the forecast functions.
107+
## calibrate (deprecated)
83108

84-
**Input:** `completedIssues: { identifier: string, estimate: number, actualCostUsd: number }[]`
85-
**Output:** `CalibrationDataset` — cost samples grouped by story-point estimate.
86-
**Ports used:** None (pure transform over caller-supplied data; I/O for fetching the
87-
source records is the caller's responsibility, typically via `EstimateTaggedUsageSource`).
88-
**Linear issue:** GRV-3
89-
**Status:** Planned — stub throws `Error('not implemented')`.
109+
The barrel exports a bare `calibrate` name only as a deprecated compatibility shim.
110+
It was never implemented; calling it throws an error naming its replacement,
111+
`calibrateCoverage` (above). It is omitted from the README's ready-to-use import
112+
example and carries no supported behavior.
Lines changed: 39 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
/**
22
* Public API for `llm-cost-estimation`.
33
*
4-
* Implemented exports are re-exported from their sub-modules; the remaining
5-
* stubs throw until their implementing issue lands. Import from this barrel —
6-
* do not import from sub-modules directly.
4+
* Every shipped export is implemented here or re-exported from
5+
* `llm-cost-attribution`, which owns the forecaster, project forecaster, and
6+
* coverage backtester. Import from this barrel — do not import from sub-modules
7+
* directly.
78
*/
89

910
/**
@@ -25,31 +26,49 @@ export { createLinearEstimateSource } from './linear-estimate-source.mjs';
2526
* cell from a set of estimate-tagged usage records. Re-exported from
2627
* `llm-cost-attribution`, which owns the empirical-quantile forecaster and
2728
* its `PricingTable` / `QuotaModel` adapters.
29+
*
30+
* Signature: `forecastIssueCost(cell, records, options?)` where `cell` is
31+
* `{ size, model }` and `records` are estimate-tagged usage records.
2832
*/
2933
export { forecastIssueCost } from 'llm-cost-attribution';
3034

3135
/**
32-
* Forecast the aggregate LLM cost for an entire Linear project, given per-issue
33-
* estimates and a calibration dataset.
36+
* Forecast an entire project's aggregate cost by Monte Carlo convolution over
37+
* its planned issues. Re-exported from `llm-cost-attribution`, which owns the
38+
* project forecaster.
3439
*
35-
* @param {string} projectId Linear project identifier.
36-
* @param {object[]} issues Array of `{ identifier, estimate }` objects.
37-
* @param {object} [options]
38-
* @returns {Promise<object>}
40+
* Estimation-friendly signature:
41+
* `forecastProjectCost(issues, usageSource?, options?)` where `issues` is an
42+
* array of `{ size, model }` IssuePlans (the same cell model
43+
* `forecastIssueCost` reads) and `usageSource` is estimate-tagged usage
44+
* records. Forecasts tokens / turns / dollars only — project-level quota is a
45+
* per-issue windowed quantity that does not sum, and is not forecast here.
3946
*/
40-
export async function forecastProjectCost(projectId, issues, options = {}) {
41-
throw new Error('not implemented');
42-
}
47+
export { forecastProjectCost } from 'llm-cost-attribution';
48+
49+
/**
50+
* Backtest the empirical forecaster: on held-out estimate-tagged usage records,
51+
* does the actual cost land at or below the predicted P80 about 80% of the
52+
* time? Re-exported from `llm-cost-attribution`, which owns the coverage
53+
* backtester. This is the supported calibration API and the replacement for the
54+
* never-shipped `calibrate` placeholder below.
55+
*
56+
* Signature: `calibrateCoverage(records, options?)`.
57+
*/
58+
export { calibrateCoverage } from 'llm-cost-attribution';
4359

4460
/**
45-
* Build or update a calibration dataset from a set of completed issues whose
46-
* actual cost is known. Returns calibration parameters used by the forecast
47-
* functions.
61+
* @deprecated Never implemented. The shipped calibration API is
62+
* `calibrateCoverage` (re-exported above) — a held-out P80 coverage backtest
63+
* over estimate-tagged usage records. This shim is retained only so old imports
64+
* fail loudly with a pointer to the replacement instead of silently resolving
65+
* to `undefined`.
4866
*
49-
* @param {object[]} completedIssues Array of `{ identifier, estimate, actualCostUsd }`.
50-
* @param {object} [options]
51-
* @returns {object}
67+
* @returns {never}
5268
*/
53-
export function calibrate(completedIssues, options = {}) {
54-
throw new Error('not implemented');
69+
export function calibrate() {
70+
throw new Error(
71+
'calibrate() was never implemented and is not part of the public API; ' +
72+
'use calibrateCoverage() instead.',
73+
);
5574
}

packages/llm-cost-estimation/test/smoke.test.mjs

Lines changed: 21 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,13 +4,16 @@ import {
44
enrichUsageWithEstimate,
55
forecastIssueCost,
66
forecastProjectCost,
7+
calibrateCoverage,
78
calibrate,
89
createLinearEstimateSource,
910
} from '../src/index.mjs';
1011

1112
// `enrichUsageWithEstimate` (GRV-2) and `forecastIssueCost` (GRV-3 + GRV-5 +
12-
// GRV-8 wiring) are implemented; `forecastProjectCost` (GRV-6) and `calibrate`
13-
// (planned) are still stubs.
13+
// GRV-8 wiring) ship from this barrel; `forecastProjectCost` and
14+
// `calibrateCoverage` are re-exported from the implemented attribution
15+
// forecasters. The bare `calibrate` name is a deprecated shim that names its
16+
// replacement.
1417
describe('llm-cost-estimation barrel', () => {
1518
it('forecastIssueCost returns the documented shape on an empty source', async () => {
1619
const forecast = await forecastIssueCost({ size: 'L', model: 'claude-sonnet-4-6' }, []);
@@ -23,6 +26,18 @@ describe('llm-cost-estimation barrel', () => {
2326
assert.equal(typeof forecast.quotaReason, 'string');
2427
});
2528

29+
it('forecastProjectCost forecasts an empty project over no history', async () => {
30+
const forecast = await forecastProjectCost(
31+
[{ size: 'L', model: 'claude-sonnet-4-6' }],
32+
[],
33+
);
34+
assert.equal(forecast.empty, true);
35+
assert.equal(forecast.lowConfidence, true);
36+
assert.equal(forecast.issues, 1);
37+
assert.equal(forecast.tokens.p50, null);
38+
assert.equal(forecast.dollars.priced, false);
39+
});
40+
2641
it('enrichUsageWithEstimate is exported from the barrel', () => {
2742
assert.equal(typeof enrichUsageWithEstimate, 'function');
2843
});
@@ -31,11 +46,11 @@ describe('llm-cost-estimation barrel', () => {
3146
assert.equal(typeof createLinearEstimateSource, 'function');
3247
});
3348

34-
it('forecastProjectCost throws "not implemented"', async () => {
35-
await assert.rejects(() => forecastProjectCost('proj-1', []), /not implemented/);
49+
it('calibrateCoverage is exported from the barrel', () => {
50+
assert.equal(typeof calibrateCoverage, 'function');
3651
});
3752

38-
it('calibrate throws "not implemented"', () => {
39-
assert.throws(() => calibrate([]), /not implemented/);
53+
it('calibrate is a deprecated shim whose error names calibrateCoverage', () => {
54+
assert.throws(() => calibrate([]), /calibrateCoverage/);
4055
});
4156
});

0 commit comments

Comments
 (0)