Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
067419c
racket: port core runtime and the diagnostics stack (phases 1-2 core)
Oct 1, 2026
6d2dc6a
racket: port the DoIP diagnostic path (phase 2 complete)
Oct 1, 2026
17ea77e
racket: port the resident daemon, CLI and client (phase 3)
Oct 1, 2026
c1df05b
racket: port the SCPI power driver (phase 4, first driver)
Oct 1, 2026
27a9469
racket: port the J-Link Commander flash driver (phase 4)
Oct 1, 2026
87f2ad5
fix: key derivers hash must use equal?-based keys
Oct 1, 2026
5916648
racket: key-deriver table must be equal-hash, not hasheq
turinglambdaai Oct 1, 2026
9c0ca2c
fix: key deriver lookup must use equal-based hash
Oct 1, 2026
cec07c3
merge: resolve key deriver conflict keeping equal-based hash
Oct 1, 2026
ecc3bde
racket: port real-hardware drivers (phase 4 complete)
turinglambdaai Oct 1, 2026
807dd2f
racket: port the SelfUpdater and wire the update command (phase 5a)
turinglambdaai Oct 1, 2026
174d100
racket: port the MCP adapter (phase 5b)
turinglambdaai Oct 1, 2026
0dc25a8
racket: port the E2E suite — 16 process-level tests green (phases 3-5…
turinglambdaai Oct 2, 2026
e869751
racket: the human-facing evidence layer — benchpilot report --format …
turinglambdaai Oct 2, 2026
298cec6
packaging: the Racket release pipeline + docs parity, version 0.6.0 (…
turinglambdaai Oct 2, 2026
fcfe246
retire the C# tree (ADR 0002 phase 5 gate)
turinglambdaai Oct 2, 2026
d6c8172
e2e/drivers: environment-agnostic assertions + best-effort DoIP sockets
turinglambdaai Oct 2, 2026
d0dc025
merge: bring in the PR #29 squash so the phase 3-5 line carries it
turinglambdaai Oct 2, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 10 additions & 31 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,36 +13,7 @@ concurrency:
cancel-in-progress: true

jobs:
build-test:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest]
steps:
- uses: actions/checkout@v7
- name: Setup .NET
uses: actions/setup-dotnet@v6
with:
dotnet-version: '10.0.x'
- name: Restore
run: dotnet restore
- name: Build
run: dotnet build --no-restore -c Release
- name: Test
run: dotnet test --no-build -c Release --logger "console;verbosity=normal"
- name: Resident runtime smoke
if: matrix.os == 'ubuntu-latest'
shell: bash
run: bash scripts/smoke-runtime.sh
- name: Graceful shutdown smoke
if: matrix.os == 'ubuntu-latest'
shell: bash
run: bash scripts/smoke-shutdown.sh

# Racket port gate (ADR 0002, Phase 0+): the ported tree must stay green
# on the same OS matrix as the C# implementation.
racket-port:
test:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
Expand All @@ -59,5 +30,13 @@ jobs:
version: '9.3'
- name: Install package
run: raco pkg install --auto --name benchpilot --link racket
- name: Test
- name: Test (unit + driver + MCP + E2E)
run: raco test racket/benchpilot
- name: Resident runtime smoke
if: matrix.os == 'ubuntu-latest'
shell: bash
run: bash scripts/smoke-runtime.sh
- name: Graceful shutdown smoke
if: matrix.os == 'ubuntu-latest'
shell: bash
run: bash scripts/smoke-shutdown.sh
88 changes: 39 additions & 49 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,13 +17,13 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- name: Verify tag matches Directory.Build.props version
- name: Verify tag matches the version constant
if: startsWith(github.ref, 'refs/tags/')
run: |
VERSION="$(sed -n 's:.*<VersionPrefix>\([^<]*\)</VersionPrefix>.*:\1:p' Directory.Build.props | head -n1 | tr -d '[:space:]')"
VERSION="$(sed -n 's/^(define benchpilot-version "\(.*\)")$/\1/p' racket/benchpilot/protocol/local-auth.rkt | head -n1)"
TAG_VERSION="${GITHUB_REF_NAME#v}"
if [ -z "$VERSION" ] || [ "$TAG_VERSION" != "$VERSION" ]; then
echo "tag $GITHUB_REF_NAME does not match Directory.Build.props VersionPrefix '$VERSION'" >&2
echo "tag $GITHUB_REF_NAME does not match benchpilot-version '$VERSION'" >&2
exit 1
fi
echo "release preflight: version $VERSION verified"
Expand All @@ -36,13 +36,17 @@ jobs:
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v7
- uses: actions/setup-dotnet@v6
- name: Setup Racket
uses: Bogdanp/setup-racket@v1.15
with:
dotnet-version: '10.0.x'
- name: Build
run: dotnet build -c Release
- name: Test (unit + e2e)
run: dotnet test --no-build -c Release
architecture: x64
distribution: full
variant: CS
version: '9.3'
- name: Install package
run: raco pkg install --auto --name benchpilot --link racket
- name: Test (unit + driver + MCP + E2E)
run: raco test racket/benchpilot
- name: Smoke runtime + graceful shutdown
if: runner.os == 'Linux'
run: |
Expand All @@ -51,39 +55,40 @@ jobs:

publish:
needs: [preflight, test]
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
rid: [win-x64, win-arm64, linux-x64, linux-arm64, osx-arm64, osx-x64]
include:
- rid: win-x64
os: windows-latest
- rid: linux-x64
os: ubuntu-latest
- rid: linux-arm64
os: ubuntu-24.04-arm
- rid: osx-arm64
os: macos-15
- rid: osx-x64
os: macos-13
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v7
- name: Setup .NET
uses: actions/setup-dotnet@v6
- name: Setup Racket
uses: Bogdanp/setup-racket@v1.15
with:
dotnet-version: '10.0.x'
architecture: '${{ matrix.rid == ''osx-arm64'' && ''arm64'' || matrix.rid == ''linux-arm64'' && ''arm64'' || ''x64'' }}'
distribution: full
variant: CS
version: '9.3'

- name: Determine version from tag
id: version
run: echo "version=${GITHUB_REF_NAME#v}" >> "$GITHUB_OUTPUT"

- name: Publish self-contained single-file shells
- name: Build distribution
shell: bash
run: |
VER="${{ steps.version.outputs.version }}"
RID="${{ matrix.rid }}"
STAGE="stage/$RID"
mkdir -p "$STAGE"
for project in Benchpilot.RuntimeHost Benchpilot.Cli Benchpilot.Mcp; do
dotnet publish "src/$project" \
-c Release -r "$RID" \
--self-contained true \
-p:PublishSingleFile=true \
-p:IncludeNativeLibrariesForSelfExtract=true \
-p:DebugType=None \
-p:DebugSymbols=false \
-p:Version="$VER" \
-o "$STAGE"
done
bash scripts/build-dist.sh "$VER" "${{ matrix.rid }}" "dist/${{ matrix.rid }}"

- name: Package
id: package
Expand All @@ -92,29 +97,12 @@ jobs:
VER="${{ steps.version.outputs.version }}"
RID="${{ matrix.rid }}"
NAME="benchpilot-$VER-$RID"
mkdir "$NAME"
# Ship exactly the three executables. Plain names on POSIX, .exe on
# Windows; publish intermediates (pdb, static web assets manifests)
# stay behind.
for exe in benchpilotd benchpilot benchpilot-mcp; do
if [[ -f "stage/$RID/$exe" ]]; then
mv "stage/$RID/$exe" "$NAME/"
elif [[ -f "stage/$RID/$exe.exe" ]]; then
mv "stage/$RID/$exe.exe" "$NAME/"
else
echo "::error::publish output missing: $exe"
ls -la "stage/$RID"
exit 1
fi
done
ls -la "$NAME"
mv "dist/$RID" "$NAME"
mkdir -p artifacts
if [[ "$RID" == win-* ]]; then
zip -q -r "artifacts/$NAME.zip" "$NAME"
echo "file=artifacts/$NAME.zip" >> "$GITHUB_OUTPUT"
else
tar -czf "artifacts/$NAME.tar.gz" "$NAME"
echo "file=artifacts/$NAME.tar.gz" >> "$GITHUB_OUTPUT"
fi

- name: Build Debian package
Expand All @@ -128,8 +116,10 @@ jobs:
run: |
sudo apt-get update -qq
sudo apt-get install -y ./artifacts/benchpilot-*-linux-x64.deb
test -x /opt/benchpilot/benchpilot
test -L /usr/bin/benchpilot
test -x /opt/benchpilot/bin/benchpilot
test -d /opt/benchpilot/lib
grep -q 'exec "/opt/benchpilot/bin/benchpilot"' /usr/bin/benchpilot
/usr/bin/benchpilot --version
sudo apt-get remove -y benchpilot
test ! -e /usr/bin/benchpilot

Expand Down
21 changes: 0 additions & 21 deletions Benchpilot.slnx

This file was deleted.

55 changes: 55 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,61 @@

All notable changes to BenchPilot are documented here.

## 0.6.0 - 2026-10-02

The Racket port completes ([ADR 0002](docs/adr/0002-racket-port.md)); the
C#/.NET tree of the 0.5.x line is removed and the release ships from the
Racket sources only, at full contract parity (same frozen JSON API, CLI
exit codes and E2E suite).

### Added

- Real-hardware drivers (phase 4): SocketCAN (Linux) and PCAN-Basic
(Windows) CAN transports behind the shared `can-iso-tp` profile driver,
and the `system-serial` serial console driver (POSIX termios + Windows
CommAPI backends) with bounded line buffering.
- The resident daemon now composes the same seven resource factories as
the C# RuntimeHost did (simulator, sim-diagnostics, can-iso-tp, doip,
system-serial, jlink, scpi-power).
- `benchpilot report --format html`: one self-contained static HTML
evidence report (bench status, readiness verdict, histories, newest
operation/observation evidence).
- MCP adapter (`benchpilot-mcp`) with the full 26-tool proxy surface over
the resident daemon.
- Debian package now ships the full Racket distribution tree under
`/opt/benchpilot` with thin `/usr/bin` wrappers; Homebrew formula and
the installers follow the same layout.

### Changed

- Release matrix drops `win-arm64`: Racket publishes no official Windows
ARM64 builds. Windows remains a single-file x64 build; scoop serves
64bit only.
- The Unix release archives now carry a `bin/` + `lib/` distribution tree;
installers wrap the real launchers so self-update keeps swapping in
place.
- The J-Link health check parses `ShowEmuList` output and applies the
deterministic USB probe selection policy (configured serial number wins;
multiple USB probes require one).
- SCPI health check performs a real identify query without changing the
output state.
- DoIP discovery returns an empty result on hosts without a broadcast
route instead of failing.

### Fixed

- Any request carrying a query string failed with an empty reply (the
HTTP server mis-indexed its query regex groups); the E2E suite was the
first harness to exercise query parameters.
- CLI exit codes for `busy` (5) and `deadline_exceeded` (6) were
unreachable — string error codes were compared against symbol datums.
- `benchpilot shutdown` stopped accepting work but never drained and
exited; it now follows the same graceful path as SIGTERM.
- Structured error fields (`deadlineMs`, `deadlineAtUtc`, `busyScope`,
`operationId`) survive into the CLI's printed error JSON.
- The J-Link driver consumed its subprocess pipes in the wrong order,
which would have hung real hardware runs.

## Unreleased

Racket port kickoff ([ADR 0002](docs/adr/0002-racket-port.md)):
Expand Down
13 changes: 0 additions & 13 deletions Directory.Build.props

This file was deleted.

40 changes: 25 additions & 15 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ No physical hardware is required for the simulator path.

### Requirements

- .NET SDK 10.0+
- nothing to *run* a release; Racket 9.3+ (CS) to build from source
- Git
- an MCP-capable client for Agent use (optional)
- for physical benches: the vendor/OS tools referenced by the selected profile, such as SEGGER J-Link Commander
Expand All @@ -110,8 +110,8 @@ No physical hardware is required for the simulator path.
```bash
git clone https://github.com/turinglambdaai/benchpilot.git
cd benchpilot
dotnet build
dotnet test
raco pkg install --auto --name benchpilot --link racket
raco test racket/benchpilot
```

## Quick start
Expand Down Expand Up @@ -149,7 +149,8 @@ From source instead:
```bash
git clone https://github.com/turinglambdaai/benchpilot.git
cd benchpilot
dotnet build -c Release
raco pkg install --auto --name benchpilot --link racket
racket packaging/launchers/benchpilot.rkt status --json
```

### 0.5. Stay updated
Expand Down Expand Up @@ -198,15 +199,15 @@ benchpilot power off --json
```

When installed from source, prefix the commands with
`dotnet run --project src/Benchpilot.Cli --` instead.
`racket packaging/launchers/benchpilot.rkt` instead.

Long mutations and serial observations can also carry a Runtime execution budget:

```bash
dotnet run --project src/Benchpilot.Cli -- \
racket packaging/launchers/benchpilot.rkt -- \
flash write build/app.elf --deadline-ms 30000 --json

dotnet run --project src/Benchpilot.Cli -- \
racket packaging/launchers/benchpilot.rkt -- \
serial wait Ready --timeout-ms 5000 --deadline-ms 7000 --json
```

Expand Down Expand Up @@ -290,6 +291,16 @@ After the target is ready, an Agent can execute a constrained bench loop such as

> Power on the ECU at 12 V, flash the selected firmware, wait for the console to print `Ready`, verify idle current is below the configured threshold, then power it off.

### 6. Attach a bench report

One static, self-contained HTML artifact covering the bench state, the
readiness verdict, operation/observation history and the newest evidence —
attach it to a bench session log or an ECU release note:

```bash
benchpilot report --out bench-report.html
```

## CLI exit codes

CLI exit codes are intentionally stable and machine-friendly:
Expand Down Expand Up @@ -458,14 +469,13 @@ See [ROADMAP.md](ROADMAP.md).

## Implementation language

The shipped Runtime is implemented in .NET/C#, and the project is porting it
to **Racket** ([ADR 0002](docs/adr/0002-racket-port.md), which supersedes
[ADR 0001](docs/adr/0001-runtime-language.md)). The port proceeds in staged
phases under a frozen API contract: until its final phase completes, the C#
tree remains the shipping implementation and the Racket sources under
`racket/` are the port under construction. GUI technology stays decoupled
from the Runtime; any future Studio GUI would speak the same versioned local
API rather than owning devices.
The Runtime is implemented in **Racket** (Racket CS) since v0.6.0
([ADR 0002](docs/adr/0002-racket-port.md), which supersedes
[ADR 0001](docs/adr/0001-runtime-language.md)). The C#/.NET tree of the
0.5.x line was retired in v0.6.0 after the port reached full contract
parity: same frozen JSON API, same CLI exit codes, same E2E suite. GUI
technology stays decoupled from the Runtime; any future Studio GUI would
speak the same versioned local API rather than owning devices.

## Long-term flashing direction

Expand Down
5 changes: 5 additions & 0 deletions ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@

BenchPilot is developed as an **agent-native ECU development runtime**, not as a feature-for-feature CANoe replacement.

> **2026-10-02 — v0.6.0:** the Racket port ([ADR 0002](docs/adr/0002-racket-port.md))
> completed at full contract parity; the C#/.NET tree is retired. The current
> gate is unchanged: physical bench validation and production-grade
> evidence/artifact handling.

The roadmap prioritizes one complete real-ECU loop over broad protocol coverage. The current gate is **physical bench validation and production-grade evidence/artifact handling**, not adding more protocols.

## Foundation — complete
Expand Down
Loading
Loading