Skip to content

Commit 80f4385

Browse files
zozo123claude
zozo123
and
claude
authored
Build-cache data mart + MetricFlow semantic layer (10 measures + KPI) (#2)
* feat: build-cache data mart + MetricFlow semantic layer (10 measures + KPI) Adds the marts and semantic layer on top of staging: - dim_agents, fct_build_tasks (task grain, semantic), fct_builds (build grain) - enforced contract on the public fct_builds mart - MetricFlow semantic model with 10 governed measures - KPI cache_hit_rate (+ remote_execution_rate, cpu_hours_saved, cpu_efficiency, gb_served_from_cache) and a metricflow time spine - singular business-invariant test (cache_hits <= cacheable_tasks; rate in [0,1]) - CI: dbt build on DuckDB + KPI resolution via MetricFlow + agent-guardrail check Closes #1 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): seed before build on a persistent target so staging finds its sources dbt has no dependency edge from a source() to the seed that populates it, so a parallel `dbt build` on a fresh DB races the staging views ahead of the seed load. Load seeds first, then `dbt build --exclude resource_type:seed`, on a persistent file target (DBT_TARGET=ci) shared across steps incl. MetricFlow. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(ci): run on the default target; drop dead DBT_TARGET/ci config dbt has no built-in DBT_TARGET env var, so `env: DBT_TARGET: ci` was a no-op and CI actually ran on the dev target while the ci: output sat unused — misleading config that passed only by luck. MetricFlow can't select a target either (no --target flag); it uses the profile default. So consolidate on a single persistent dev target that seed/build/parse/mf all share, and document the warehouse target as a commented example. Found via adversarial review workflow. No behavior change to a passing build. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 36b8e66 commit 80f4385

13 files changed

Lines changed: 510 additions & 10 deletions

‎.github/workflows/dbt-ci.yml‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
name: dbt CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
8+
jobs:
9+
build:
10+
name: dbt build (DuckDB)
11+
runs-on: ubuntu-latest
12+
env:
13+
DBT_PROFILES_DIR: ${{ github.workspace }}
14+
steps:
15+
- uses: actions/checkout@v4
16+
17+
- uses: actions/setup-python@v5
18+
with:
19+
python-version: "3.12"
20+
21+
- name: Install dbt + MetricFlow
22+
run: |
23+
python -m pip install --upgrade pip
24+
pip install -r requirements.txt
25+
26+
- name: Generate seed data
27+
run: python scripts/generate_seeds.py
28+
29+
# Seeds stand in for raw sources here; load them first so the staging
30+
# models (which read via source()) find the tables. dbt has no dependency
31+
# edge from a source() to the seed that populates it, so we order it.
32+
- name: dbt seed
33+
run: dbt seed
34+
35+
- name: dbt build (run + test + contracts)
36+
run: dbt build --exclude resource_type:seed
37+
38+
- name: dbt parse (validate semantic layer)
39+
run: dbt parse
40+
41+
- name: Resolve KPI through the semantic layer
42+
run: mf query --metrics cache_hit_rate,cpu_hours_saved --group-by metric_time__year
43+
44+
agent-guardrail:
45+
name: agent-guardrail (execution-plane surfaces)
46+
runs-on: ubuntu-latest
47+
if: github.event_name == 'pull_request'
48+
steps:
49+
- uses: actions/checkout@v4
50+
with:
51+
fetch-depth: 0
52+
53+
- name: Flag changes that warrant a sandboxed run (ADR 0001)
54+
run: |
55+
base="origin/${{ github.base_ref }}"
56+
git fetch origin "${{ github.base_ref }}" --depth=1
57+
changed=$(git diff --name-only "$base"...HEAD)
58+
echo "Changed files:"; echo "$changed"
59+
risky=$(echo "$changed" | grep -E '(^|/)macros/|packages\.ya?ml|dependencies\.ya?ml|\.py$' || true)
60+
# Heuristic: hooks / run-operation introduced in the diff
61+
hooks=$(git diff "$base"...HEAD | grep -E '^\+.*(pre.?hook|post.?hook|on-run-(start|end)|run_operation|run-operation)' || true)
62+
if [ -n "$risky" ] || [ -n "$hooks" ]; then
63+
echo "::warning title=Execution-plane change::This PR touches macros/hooks/packages/Python models. Per docs/adr/0001-where-to-run-dbt.md, execute it only in a sandboxed runtime and review the code-execution surfaces, not just the SELECTs."
64+
echo "$risky"; echo "$hooks"
65+
else
66+
echo "No execution-plane surfaces touched — branch + DB role + CI confinement is sufficient."
67+
fi

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ seeds:
1010
. .venv/bin/activate && python scripts/generate_seeds.py
1111

1212
build: seeds
13-
. .venv/bin/activate && dbt build
13+
. .venv/bin/activate && dbt seed && dbt build --exclude resource_type:seed
1414

1515
test:
1616
. .venv/bin/activate && dbt test

‎README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -28,9 +28,10 @@ uv pip install -r requirements.txt
2828
# 2. generate the synthetic seed data (deterministic)
2929
python scripts/generate_seeds.py
3030

31-
# 3. build + test everything on DuckDB
31+
# 3. build + test everything on DuckDB (seed first: staging reads via source(),
32+
# which dbt cannot auto-order after the seed that fills it)
3233
export DBT_PROFILES_DIR=$PWD
33-
dbt build
34+
dbt seed && dbt build --exclude resource_type:seed
3435

3536
# 4. query the headline KPI through the semantic layer
3637
dbt parse

‎models/marts/core/_core.yml‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
version: 2
2+
3+
models:
4+
- name: dim_agents
5+
description: Agent (helper machine) dimension.
6+
columns:
7+
- name: agent_id
8+
description: Primary key.
9+
data_tests: [not_null, unique]
10+
- name: agent_class
11+
data_tests:
12+
- accepted_values:
13+
values: ['cloud', 'on_prem']
14+
15+
- name: fct_build_tasks
16+
description: >
17+
Task-grain fact and the semantic-layer consumption contract. One row per
18+
build task, denormalized with build + agent dimensions.
19+
columns:
20+
- name: task_id
21+
description: Primary key.
22+
data_tests: [not_null, unique]
23+
- name: build_id
24+
data_tests:
25+
- relationships:
26+
to: ref('fct_builds')
27+
field: build_id
28+
- name: cache_status
29+
data_tests:
30+
- accepted_values:
31+
values: ['hit', 'miss', 'skip']
32+
33+
- name: fct_builds
34+
description: >
35+
Build-grain fact: one row per build with cache + cpu rollups and a
36+
per-build cache hit rate. Public, contract-enforced — the schema is a
37+
promise to downstream consumers and breaks fail at build time.
38+
config:
39+
contract:
40+
enforced: true
41+
access: public
42+
columns:
43+
- name: build_id
44+
description: Primary key — one row per build.
45+
data_type: varchar
46+
constraints: [{ type: not_null }]
47+
data_tests: [unique]
48+
- name: tenant
49+
data_type: varchar
50+
- name: project
51+
data_type: varchar
52+
- name: build_type
53+
data_type: varchar
54+
- name: build_date
55+
data_type: date
56+
- name: started_at
57+
data_type: timestamp
58+
- name: ib_version
59+
data_type: varchar
60+
- name: duration_seconds
61+
data_type: double
62+
- name: total_tasks
63+
data_type: bigint
64+
- name: cache_hits
65+
data_type: bigint
66+
- name: cache_misses
67+
data_type: bigint
68+
- name: cacheable_tasks
69+
data_type: bigint
70+
- name: remote_tasks
71+
data_type: bigint
72+
- name: cpu_seconds
73+
data_type: double
74+
- name: cpu_seconds_saved
75+
data_type: double
76+
- name: bytes_served
77+
data_type: bigint
78+
- name: active_agents
79+
data_type: bigint
80+
- name: cache_hit_rate
81+
description: cache_hits / cacheable_tasks for this build (0–1).
82+
data_type: double

‎models/marts/core/dim_agents.sql‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
-- Agent dimension: one row per helper machine.
2+
with agents as (
3+
select * from {{ ref('stg_agents') }}
4+
)
5+
6+
select
7+
agent_id,
8+
region,
9+
cores,
10+
is_cloud,
11+
case when is_cloud then 'cloud' else 'on_prem' end as agent_class,
12+
created_at
13+
from agents

‎models/marts/core/fct_build_tasks.sql‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
-- Task-grain fact: the consumption contract for the semantic layer.
2+
-- One row per build task, denormalized with build + agent dimensions and
3+
-- pre-computed 0/1 helper columns so MetricFlow measures are simple SUMs.
4+
with tasks as (
5+
select * from {{ ref('stg_build_tasks') }}
6+
),
7+
8+
builds as (
9+
select * from {{ ref('stg_builds') }}
10+
),
11+
12+
agents as (
13+
select * from {{ ref('dim_agents') }}
14+
)
15+
16+
select
17+
t.task_id,
18+
t.build_id,
19+
t.agent_id,
20+
-- build dimensions
21+
b.tenant,
22+
b.project,
23+
b.build_type,
24+
b.build_date,
25+
b.started_at,
26+
-- agent dimensions
27+
a.region,
28+
a.agent_class,
29+
-- task attributes
30+
t.task_kind,
31+
t.cache_status,
32+
t.is_cacheable,
33+
t.executed_remotely,
34+
-- additive measure helpers (0/1) — keep semantic measures as plain SUMs
35+
1 as task_count,
36+
case when t.cache_status = 'hit' then 1 else 0 end as is_cache_hit,
37+
case when t.cache_status = 'miss' then 1 else 0 end as is_cache_miss,
38+
case when t.is_cacheable then 1 else 0 end as is_cacheable_int,
39+
case when t.executed_remotely then 1 else 0 end as is_remote,
40+
t.cpu_seconds,
41+
t.cpu_seconds_saved,
42+
t.bytes_served
43+
from tasks t
44+
inner join builds b on t.build_id = b.build_id
45+
left join agents a on t.agent_id = a.agent_id

‎models/marts/core/fct_builds.sql‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
-- Build-grain fact: one row per build, rolling up its tasks. The build-level
2+
-- data mart used for build-duration analysis and per-build cache efficiency.
3+
with tasks as (
4+
select * from {{ ref('fct_build_tasks') }}
5+
),
6+
7+
builds as (
8+
select * from {{ ref('stg_builds') }}
9+
),
10+
11+
task_rollup as (
12+
select
13+
build_id,
14+
cast(sum(task_count) as bigint) as total_tasks,
15+
cast(sum(is_cache_hit) as bigint) as cache_hits,
16+
cast(sum(is_cache_miss) as bigint) as cache_misses,
17+
cast(sum(is_cacheable_int) as bigint) as cacheable_tasks,
18+
cast(sum(is_remote) as bigint) as remote_tasks,
19+
cast(sum(cpu_seconds) as double) as cpu_seconds,
20+
cast(sum(cpu_seconds_saved) as double) as cpu_seconds_saved,
21+
cast(sum(bytes_served) as bigint) as bytes_served,
22+
cast(count(distinct agent_id) as bigint) as active_agents
23+
from tasks
24+
group by 1
25+
)
26+
27+
select
28+
b.build_id,
29+
b.tenant,
30+
b.project,
31+
b.build_type,
32+
b.build_date,
33+
b.started_at,
34+
b.ib_version,
35+
b.duration_seconds,
36+
r.total_tasks,
37+
r.cache_hits,
38+
r.cache_misses,
39+
r.cacheable_tasks,
40+
r.remote_tasks,
41+
r.cpu_seconds,
42+
r.cpu_seconds_saved,
43+
r.bytes_served,
44+
r.active_agents,
45+
-- per-build cache hit rate (guard divide-by-zero)
46+
cast(
47+
case
48+
when r.cacheable_tasks > 0
49+
then round(r.cache_hits * 1.0 / r.cacheable_tasks, 4)
50+
else null
51+
end as double
52+
) as cache_hit_rate
53+
from builds b
54+
inner join task_rollup r on b.build_id = r.build_id

‎models/marts/metrics/_time_spine.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
version: 2
2+
3+
models:
4+
- name: metricflow_time_spine
5+
description: Daily date spine for the MetricFlow semantic layer.
6+
time_spine:
7+
standard_granularity_column: date_day
8+
columns:
9+
- name: date_day
10+
description: One row per calendar day.
11+
granularity: day
12+
data_tests: [not_null, unique]

‎models/marts/metrics/metricflow_time_spine.sql‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
-- Daily time spine required by the MetricFlow semantic layer.
2+
-- Built with DuckDB's native generate_series — no package dependency, so the
3+
-- project never needs `dbt deps` (see docs/adr/0001-where-to-run-dbt.md).
4+
select cast(d as date) as date_day
5+
from unnest(
6+
generate_series(
7+
timestamp '2025-01-01',
8+
timestamp '2027-01-01',
9+
interval '1 day'
10+
)
11+
) as t(d)

0 commit comments

Comments
 (0)