Skip to content

Commit dd4ecdd

Browse files
committed
Record low-risk bbolt bucket coverage
Update Phase C docs with the next record-level storage slice: static objects, Worker scripts, plugin lifecycle records, and approvals now have tested bbolt APIs, while runtime storage and secret-bearing buckets remain explicitly pending. Constraint: Docs must keep JSON as the default store until the startup path changes Rejected: Call record-level storage complete | task/monitor/tunnel and secret-bearing buckets remain uncovered Confidence: high Scope-risk: narrow Directive: Do not document DDNS/OIDC/notify/users/tokens as record-level ready until encryption tests exist Tested: git diff --cached --check; cross-checked lattice-server main still uses store.OpenWithCipher Not-tested: Rendered docs site
1 parent 1bbae71 commit dd4ecdd

3 files changed

Lines changed: 98 additions & 9 deletions

File tree

docs/PRODUCT-VISION.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,12 @@ The bar is not "works." The bar is **惊艳 — a product that makes a self-host
1717
| **P1 · Trust** | Secure by default, fail-closed, auditable, nothing to misconfigure into insecurity | Strong: RBAC, rate-limit, signed plugins (fail-closed), tamper-evident audit WAL, 2FA, at-rest encryption |
1818
| **P2 · Identity** | Password + 2FA + SSO; the front door is frictionless and enterprise-ready | Password ✅ + TOTP 2FA ✅ + OIDC/SSO backend/UI ✅; **2FA policy + WebAuthn groundwork next** |
1919
| **P3 · Platform** | A real plugin system — install, verify, run, and extend safely; a marketplace of official + community plugins | Manifest + signing ✅, loader ✅, preflight verify ✅, host-API broker + server adapter ✅, lifecycle registry/API/UI ✅, runtime manager + runner contract ✅; **plugin artifacts don't execute yet** |
20-
| **P4 · Scale & durability** | Survives growth and crashes; backup/restore is a non-event | JSON state + at-rest encryption + audit WAL ✅; bbolt import/export + JSON migration/rollback CLI ✅; first record-level APIs ✅; **default store still whole-file JSON** |
20+
| **P4 · Scale & durability** | Survives growth and crashes; backup/restore is a non-event | JSON state + at-rest encryption + audit WAL ✅; bbolt import/export + JSON migration/rollback CLI ✅; record-level APIs for low-risk buckets ✅; **default store still whole-file JSON** |
2121
| **P5 · Experience (惊艳)** | A dashboard that is fast, legible, real-time, and beautiful; onboarding that takes minutes | Functional zero-dep dashboard with SSO + plugin lifecycle panels; **many endpoints still have no UI; design layer immature** |
2222

2323
## 3. Honest current state (2026-06-12)
2424

25-
Delivered and pushed: control plane + node agent + SDK + dashboard across 6 repos; security hardening; DDNS, monitoring, notifications, WireGuard mesh, CF Tunnel; plugin manifest/signing + loader + preflight verify + host-API broker/server adapter + lifecycle registry/API + dashboard panel + runtime manager + runner contract; TOTP 2FA; tamper-evident audit WAL; node-token rotation; bounded task execution; **AES-256-GCM at-rest encryption** (ADR-002); bbolt state import/export foundation with explicit JSON migration/rollback CLI and first record-level APIs for nodes, KV, and audit events. Minimal external Go deps are currently limited to the OIDC stack approved in ADR-001 (`oauth2`, `go-oidc`, transitive `go-jose`) plus bbolt for Phase C storage; still zero CGo. `go test -race` green; dashboard tests green.
25+
Delivered and pushed: control plane + node agent + SDK + dashboard across 6 repos; security hardening; DDNS, monitoring, notifications, WireGuard mesh, CF Tunnel; plugin manifest/signing + loader + preflight verify + host-API broker/server adapter + lifecycle registry/API + dashboard panel + runtime manager + runner contract; TOTP 2FA; tamper-evident audit WAL; node-token rotation; bounded task execution; **AES-256-GCM at-rest encryption** (ADR-002); bbolt state import/export foundation with explicit JSON migration/rollback CLI and record-level APIs for low-risk buckets (nodes, KV, audit, static, Workers, plugin lifecycle, approvals). Minimal external Go deps are currently limited to the OIDC stack approved in ADR-001 (`oauth2`, `go-oidc`, transitive `go-jose`) plus bbolt for Phase C storage; still zero CGo. `go test -race` green; dashboard tests green.
2626

2727
The three gaps that most separate us from 惊艳, in dependency order: **identity policy polish**, **the platform actually running plugins**, and **a storage engine that scales** — plus a **dashboard worthy of the backend**.
2828

@@ -43,7 +43,7 @@ Each phase has an exit bar. Phases ship as tested, reviewed, committed slices (t
4343
- **Exit:** a signed plugin installs, runs, is capability-confined, and its host calls are brokered + audited. Foundation for the user's own LatticeNet/* official plugins (sing-box/xray/sub-store).
4444

4545
### Phase C — Storage that scales *(item ④, next high-leverage backend slice)*
46-
- C1 · **Replace the whole-file JSON store with bbolt** (pure Go, preserves zero-CGo). Import/export + JSON migration/rollback CLI ✅; first record-level APIs for nodes/KV/audit ✅; broader per-bucket record writes, the at-rest encryption boundary (ADR-002) re-homed onto the default store, audit WAL head anchoring, and backup/restore still pending.
46+
- C1 · **Replace the whole-file JSON store with bbolt** (pure Go, preserves zero-CGo). Import/export + JSON migration/rollback CLI ✅; record-level APIs for low-risk buckets ✅; secret-bearing bucket writes, the at-rest encryption boundary (ADR-002) re-homed onto the default store, audit WAL head anchoring, and backup/restore still pending.
4747
- C2 · crash-safe migration from the JSON file, JSON export/import, retention policy for high-volume audit/monitor results.
4848
- **Exit:** writes are O(record) not O(state); a 10k-node/long-audit deployment stays responsive; crash-safe.
4949
- **Timing note:** done after A/B so it migrates a known-stable schema; the fresh at-rest boundary is re-homed deliberately, not rushed.
Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# Iteration 012 - bbolt Low-Risk Bucket Coverage
2+
3+
- **Date:** 2026-06-13
4+
- **Phase:** C1 storage foundation
5+
- **Repos:** `lattice-server`, `lattice`
6+
- **Status:** Verified low-risk bucket record APIs / default store not switched
7+
8+
## Goal
9+
10+
Extend the bbolt record-level foundation beyond nodes/KV/audit while avoiding
11+
secret-bearing buckets. This slice adds record-level APIs for static objects,
12+
Worker scripts, plugin lifecycle records, and approvals.
13+
14+
## Scope
15+
16+
- Add record-level static object writes and bucket reads:
17+
- `PutStatic`
18+
- `Static`
19+
- Add record-level Worker writes and reads:
20+
- `UpsertWorker`
21+
- `Workers`
22+
- Add record-level plugin lifecycle writes and reads:
23+
- `UpsertPluginInstallation`
24+
- `PluginInstallation`
25+
- `PluginInstallations`
26+
- `SetPluginStatus`
27+
- Add record-level approval writes and reads:
28+
- `UpsertApproval`
29+
- `Approval`
30+
- `Approvals`
31+
- Preserve current JSON-store behavior:
32+
- static objects sorted by path
33+
- Workers sorted by name
34+
- plugin lifecycle sorted by id
35+
- approvals sorted newest first
36+
- plugin status transition validation reused
37+
- Keep writes local to their bbolt buckets; record-level writes must not reset
38+
unrelated imported buckets.
39+
40+
## Non-Goals
41+
42+
- Do not switch `cmd/lattice-server` to bbolt.
43+
- Do not add `-data-engine=bolt` yet.
44+
- Do not add record-level writes for DDNS, notification channels, OIDC providers,
45+
TOTP challenges, users, tokens, or sessions yet.
46+
- Do not migrate or anchor the audit WAL into bbolt yet.
47+
48+
## Security Notes
49+
50+
- This slice still avoids reversible secret fields. Static content and Worker
51+
source can contain operator-provided sensitive text, but the current JSON store
52+
already persists those fields as ordinary state; this change does not widen
53+
their exposure or route runtime traffic to bbolt.
54+
- Secret-bearing per-record methods must be added with field-specific encryption
55+
tests before they are considered ready.
56+
- Plugin lifecycle reads clone capability slices so callers cannot mutate a
57+
returned slice and accidentally affect future in-memory state assumptions.
58+
59+
## Verification
60+
61+
Targeted tests added in `lattice-server/internal/store/bolt_state_test.go`:
62+
63+
- static, Worker, plugin lifecycle, and approval record-level writes work after
64+
a full import.
65+
- plugin lifecycle transition validation is preserved.
66+
- plugin capability slices are cloned on read.
67+
- records keep JSON-store ordering semantics.
68+
- record-level writes do not reset unrelated buckets.
69+
- records persist across close/reopen.
70+
71+
Commands run:
72+
73+
```sh
74+
GOCACHE=/tmp/lattice-review-go-build GOWORK=/Users/cdcd/roobli/RTFS_justTaste/Probe-Dashboards/Lattice/lattice/go.work go test ./internal/store -run TestBoltState -count=1
75+
GOCACHE=/tmp/lattice-review-go-build GOWORK=/Users/cdcd/roobli/RTFS_justTaste/Probe-Dashboards/Lattice/lattice/go.work go test ./internal/store -count=1
76+
GOCACHE=/tmp/lattice-review-go-build GOWORK=/Users/cdcd/roobli/RTFS_justTaste/Probe-Dashboards/Lattice/lattice/go.work go test ./... -count=1
77+
GOCACHE=/tmp/lattice-review-go-build GOWORK=/Users/cdcd/roobli/RTFS_justTaste/Probe-Dashboards/Lattice/lattice/go.work go vet ./...
78+
GOCACHE=/tmp/lattice-review-go-build GOWORK=/Users/cdcd/roobli/RTFS_justTaste/Probe-Dashboards/Lattice/lattice/go.work go test -race ./... -count=1
79+
```
80+
81+
## Residuals
82+
83+
- Remaining low/medium-risk buckets: tasks, task results, monitors, monitor
84+
results, tunnels.
85+
- Secret-bearing buckets need field-specific encryption tests before per-record
86+
methods: users, tokens, sessions, TOTP challenges, DDNS profiles, notification
87+
channels, OIDC providers/auth states.
88+
- Runtime store switching remains blocked until record-level coverage is broad
89+
enough and backup/restore drills are complete.

docs/roadmap.md

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -9,18 +9,18 @@
99
> encryption, tamper-evident audit WAL, signed plugin loader, lifecycle UI, host
1010
> API broker, and runtime runner contract are now landed. The durable storage
1111
> direction is **bbolt**, not SQLite, to preserve the pure-Go / zero-CGo rule;
12-
> bucketized bbolt import/export, JSON migration/rollback CLI, and first
13-
> record-level bbolt APIs have landed, but JSON is still the default server
14-
> store.
12+
> bucketized bbolt import/export, JSON migration/rollback CLI, and record-level
13+
> bbolt APIs for low-risk buckets have landed, but JSON is still the default
14+
> server store.
1515
1616

1717
## V1 Hardening
1818

1919
- Replace JSON storage with bbolt migrations, preserving AES-256-GCM secret
2020
encryption and moving hot/ephemeral records off whole-file rewrites.
21-
*(Bucketized import/export, JSON rollback CLI, and first record-level
22-
node/KV/audit APIs delivered; broader record-level writes and default store
23-
switch pending.)*
21+
*(Bucketized import/export, JSON rollback CLI, and record-level APIs for
22+
nodes/KV/audit/static/Workers/plugin lifecycle/approvals delivered;
23+
secret-bearing writes and default store switch pending.)*
2424
- Keep protobuf/ConnectRPC transport and generated TypeScript clients as a later
2525
API-boundary upgrade; current JSON APIs remain the bootstrap surface.
2626
- TOTP setup and recovery codes. *(Delivered 2026-06-12; enforce-2FA policy,

0 commit comments

Comments
 (0)