Skip to content

Commit 1666e82

Browse files
docs: add orchestrator how-tos, config reference, and resume tutorial
Expand Diátaxis coverage for self-hosted operators: agent configuration, Postgres production, planning mode, Gateway routing, env reference, and DBOS crash recovery. Update docs hub and AGENTS.md pointers. Co-authored-by: Cursor <cursoragent@cursor.com>
1 parent 4a440a3 commit 1666e82

8 files changed

Lines changed: 713 additions & 87 deletions

File tree

agents.md

Lines changed: 40 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -1,112 +1,70 @@
1-
# Agent Context for Drover
1+
# Agent Context for Drover Orchestrator (`drover`)
22

33
Welcome, AI Agent. This document provides context, architecture, and instructions for working within the `drover` repository.
44

55
## Ecosystem Role
66

7-
> **Part of the Drover Ecosystem**: `drover` is **Drover Orchestrator**—parallel **task** dispatch in Git worktrees via DBOS. It invokes **Drover Code** (and other agents) locally per **worktree run**. Milestone A hosted path: **Drover Cloud** runs single **agent jobs** on workers; Orchestrator is self-hosted and out of scope until a future platform integration. Glossary: [`CONTEXT.md`](CONTEXT.md), [`../CONTEXT-MAP.md`](../CONTEXT-MAP.md).
7+
> **Part of the Drover Ecosystem**: `drover` is **Drover Orchestrator**—parallel **task** dispatch in Git worktrees via DBOS. It invokes **Drover Code** (and other agents) locally per **worktree run**. Milestone A hosted path: **Drover Cloud** runs single **agent jobs** on workers; Orchestrator is self-hosted and out of scope until a future platform integration.
8+
9+
**Glossary:** [`CONTEXT.md`](CONTEXT.md). **Org index:** [`../AGENTS.md`](../AGENTS.md).
810

911
## Project Overview
1012

11-
Drover is a durable workflow orchestrator designed to parallelize AI coding tasks. Instead of running a single agent (like Claude Code or Cursor) linearly, Drover breaks down a large epic into dependencies, respects blockers, and executes tasks using multiple agents concurrently inside isolated Git worktrees.
13+
Drover is a durable workflow orchestrator designed to parallelize AI coding tasks. Instead of running a single agent linearly, Drover breaks down a large epic into dependencies, respects blockers, and executes tasks using multiple agents concurrently inside isolated Git worktrees.
1214

1315
### Core Technologies
16+
1417
- **Language**: Go 1.26.3
15-
- **Workflow Engine**: DBOS v0.14.0 (Durable Operating System) — using `dbos.Go` for concurrent steps, `dbos.Sleep` for durable delays
16-
- **Database**: PostgreSQL (Production) / SQLite (Local Development)
17-
- **CLI Framework**: Cobra v1.10
18-
- **Observability**: OpenTelemetry v1.43.0
19-
- **Agent Interface**: Pluggable — supports Claude Code, Codex, Amp, OpenCode, Drover Code
18+
- **Workflow Engine**: DBOS `dbos.Go` for concurrent steps, `dbos.Sleep` for durable delays
19+
- **Database**: PostgreSQL (production) / SQLite (local)
20+
- **CLI**: Cobra
21+
- **Observability**: OpenTelemetry
22+
- **Agent interface**: Pluggable — Claude Code, Codex, Amp, OpenCode, Drover Code
2023

2124
## Architecture
2225

2326
### Package Map
2427

25-
| Package | Purpose | Coverage |
26-
|---------|---------|----------|
27-
| `internal/workflow` | Core orchestration (Orchestrator, DBOSOrchestrator, step functions) | 44.1% |
28-
| `internal/db` | SQLite/PostgreSQL persistence layer | 38.0% |
29-
| `internal/executor` | Pluggable agent interface (Claude, Codex, Amp, OpenCode, Drover Code) | 47.8% |
30-
| `internal/git` | Git worktree lifecycle (create, merge, prune, cleanup) | 69.5% |
31-
| `internal/config` | Configuration loading (env vars, flags, defaults) | 4.8% |
32-
| `internal/analytics` | Task execution analytics and reporting | 91.4% |
33-
| `internal/backpressure` | Adaptive concurrency control (AIMD algorithm) | 77.8% |
34-
| `internal/beads` | Hierarchical task ID parser (e.g., `task-123.1.2`) | 72.8% |
35-
| `internal/clock` | Deterministic time injection for testing ||
36-
| `internal/context` | Context window management for LLM prompts ||
37-
| `internal/taskcontext` | Recent-task context carrying between agents | 98.6% |
38-
| `internal/llmproxy` | LLM proxy server for rate limiting | 100% |
39-
| `internal/modes` | Execution mode selection (parallel, sequential, queue) | 90.8% |
40-
| `internal/search` | Task search and filtering | 65.7% |
41-
| `internal/template` | Prompt template engine | 100% |
42-
| `internal/webhooks` | Webhook notification system | 66.1% |
43-
| `internal/dashboard` | Real-time WebSocket dashboard (requires DB) | 39.2% |
44-
| `internal/memory` | Process memory tracking (platform-specific: Linux `/proc`, macOS `ps`/`sysctl`) | 80.6% |
45-
| `internal/callbacks` | Task lifecycle callbacks | 28.4% |
46-
| `internal/flags` | CLI flag parsing | 63.3% |
47-
| `pkg/telemetry` | OpenTelemetry traces, metrics, and spans | 66.5% |
48-
| `pkg/types` | Shared types (Task, TaskStatus, Epic) ||
28+
| Package | Purpose |
29+
|---------|---------|
30+
| `internal/workflow` | Core orchestration (Orchestrator, DBOSOrchestrator, step functions) |
31+
| `internal/db` | SQLite/PostgreSQL persistence |
32+
| `internal/executor` | Pluggable agent interface |
33+
| `internal/git` | Git worktree lifecycle |
34+
| `internal/config` | Configuration loading |
35+
| `internal/analytics` | Task execution analytics |
36+
| `internal/backpressure` | Adaptive concurrency (AIMD) |
37+
| `internal/beads` | Hierarchical task ID parser |
38+
| `internal/modes` | Execution modes (parallel, sequential, queue) |
39+
| `pkg/telemetry` | OpenTelemetry |
4940

5041
### Key Interfaces
5142

52-
- **`workflow.GitManager`**Defined in `internal/workflow/git_interface.go`. Decouples orchestration from `*git.WorktreeManager`. All worktree operations go through this interface.
53-
- **`executor.Agent`**Defined in `internal/executor/agent.go`. Pluggable agent execution with `ExecuteWithContext`, `SetProjectGuidelines`, `SetContextManager`, `SetTaskContext`.
54-
- **`clock.Clock`**Defined in `internal/clock/clock.go`. Injectable time source for deterministic testing.
43+
- **`workflow.GitManager`**worktree operations; do not use `*git.WorktreeManager` directly in orchestrator
44+
- **`executor.Agent`**pluggable agent execution
45+
- **`clock.Clock`**injectable time for tests
5546

5647
## Key Concepts
57-
- **Durable Workflows**: Every action is checkpointed by DBOS. If Drover crashes or gets interrupted, it can resume from exactly where it left off.
58-
- **Git Worktrees**: To allow multiple agents to modify the same repository simultaneously without conflict, each worker operates in an isolated Git worktree.
59-
- **Hierarchical Tasks**: Tasks can have sub-tasks (using the `task-123.1` format inspired by Beads). Sub-tasks run first, the parent runs last.
60-
- **Deterministic Clock**: The `clock.Clock` interface replaces raw `time.Now()` calls, enabling fully deterministic time-based unit tests via `clock.MockClock`.
61-
- **Workflow ID Generation**: Uses `sync/atomic.Int64` counter for thread-safe, unique workflow IDs.
62-
- **Concurrent Side-Effects**: Dashboard broadcasts, webhooks, analytics, and event recording use `dbos.Go` for fire-and-forget concurrent execution, keeping them off the critical task-execution path.
63-
- **Durable Delays**: Queue polling uses `dbos.Sleep` instead of `time.Sleep`, ensuring delays survive process crashes and DBOS recovery.
64-
- **Durable Timeouts**: Agent execution uses `dbos.Select` to race the task step against a configurable timeout step. If Drover crashes and recovers, the timeout race replays from the DBOS checkpoint.
65-
- **Dependency Cascade**: `OnTaskComplete` resolves blocker dependencies via `allBlockersComplete`, only enqueuing dependent tasks once ALL their blockers are marked complete. Uses `dbos.RunWorkflow` with `WithDelay(500ms)` for durable, staggered cascade scheduling.
66-
67-
## Development Workflow
68-
69-
When making modifications to `drover`:
70-
71-
1. Ensure changes to core loops or workflows respect DBOS checkpointing rules (e.g., only interact with the database via the DBOS transaction methods).
72-
2. Never introduce non-deterministic operations inside a DBOS workflow function without wrapping it in a DBOS step/activity.
73-
3. Use `dbos.Go` for fire-and-forget side effects (dashboard, webhooks, analytics). Use `dbos.RunAsStep` for operations whose results are needed.
74-
4. Use `dbos.Sleep` instead of `time.Sleep` inside workflows for durable delays that survive recovery.
75-
5. Use `dbos.Select` with `dbos.Go` channels for durable timeout races on long-running steps (e.g., agent execution).
76-
6. Use the `clock.Clock` interface for any time-dependent logic; never use `time.Now()` directly in testable code.
77-
7. When modifying the `GitManager` interface, update `mock_git_manager_test.go` to match.
78-
8. When modifying the `executor.Agent` interface, update `MockAgent` in `step_functions_test.go` to match.
79-
80-
### Build and Test
48+
49+
- **Durable workflows**: checkpointed by DBOS; resume after crash
50+
- **Git worktrees**: isolated parallel agent workspaces
51+
- **Hierarchical tasks**: `task-123.1` sub-tasks (Beads-style)
52+
- Use `dbos.Go` for fire-and-forget side effects; `dbos.RunAsStep` when results are needed
53+
- Use `dbos.Sleep` not `time.Sleep` inside workflows
54+
- Use `clock.Clock` not raw `time.Now()` in testable workflow paths
55+
56+
## Build and Test
8157

8258
```bash
83-
# Build the CLI
8459
go build -o drover ./cmd/drover
85-
86-
# Run all tests
8760
go test ./...
88-
89-
# Run with race detector
9061
go test -race ./...
91-
92-
# Run specific package tests with coverage
93-
go test ./internal/workflow/... -coverprofile=coverage.out
94-
95-
# View coverage report
96-
go tool cover -html=coverage.out
9762
```
9863

99-
### Environment-Dependent Tests
100-
101-
All 24 packages pass in the current environment. The following tests require specific infrastructure:
102-
103-
- `internal/workflow` — DBOS integration tests (`TestDBOSOrchestrator_*`) skip without `DBOS_SYSTEM_DATABASE_URL`
104-
- `internal/memory` — Platform-specific: uses build tags (`tracker_linux.go`, `tracker_darwin.go`)
64+
DBOS integration tests require `DBOS_SYSTEM_DATABASE_URL`.
10565

10666
## Known Guidelines
107-
- Never introduce non-deterministic operations inside a DBOS workflow function without wrapping it in a DBOS step/activity.
108-
- Use `dbos.Go` for fire-and-forget side effects (dashboard, webhooks, analytics). Use `dbos.RunAsStep` for operations whose results are needed by the workflow.
109-
- Use `dbos.Sleep` instead of `time.Sleep` inside workflows for durable, crash-safe delays.
110-
- Respect the OpenTelemetry spans; when adding new operations, ensure they are properly traced.
111-
- Use `clock.Clock` for time, `sync/atomic` for counters — no raw `time.Now()` or non-atomic state in workflow paths.
112-
- The `GitManager` interface is the standard for worktree operations — never reference `*git.WorktreeManager` directly in the orchestrator.
67+
68+
- No non-deterministic work inside workflow functions without a DBOS step
69+
- Respect OpenTelemetry spans on new operations
70+
- **Provisioning work** never runs through Orchestrator — that is **Drover Cloud** only

docs/README.md

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -22,25 +22,32 @@ Documentation follows [Diátaxis](https://diataxis.fr/): four types of content f
2222

2323
| Need | Type | Start here |
2424
|------|------|------------|
25-
| **Learn** by doing | Tutorial | [First parallel epic](tutorials/first-parallel-epic.md) |
26-
| **Accomplish** a task | How-to | [How-to guides](how-to/) *(coming soon)* |
27-
| **Look up** facts | Reference | [CLI reference](reference/cli.md) · [Feature specs](../spec/) |
25+
| **Learn** by doing | Tutorial | [First parallel epic](tutorials/first-parallel-epic.md) · [Resume after crash](tutorials/resume-after-crash.md) |
26+
| **Accomplish** a task | How-to | [How-to guides](how-to/) |
27+
| **Look up** facts | Reference | [CLI reference](reference/cli.md) · [Configuration](reference/configuration.md) · [Feature specs](../spec/) |
2828
| **Understand** concepts | Explanation | [Architecture overview](explanation/architecture.md) |
2929

3030
## Tutorials
3131

3232
Step-by-step lessons for newcomers.
3333

3434
- [First parallel epic](tutorials/first-parallel-epic.md)`drover init` → epic → blocked tasks → `drover run`
35+
- [Resume after crash](tutorials/resume-after-crash.md) — interrupt and recover with DBOS checkpoints
3536

3637
## How-to guides
3738

38-
Goal-oriented recipes. *(Quadrant in progress)*
39+
Goal-oriented recipes.
40+
41+
- [Configure agents](how-to/configure-agents.md) — Claude, Codex, Amp, OpenCode, worker subprocess isolation
42+
- [Planning vs building mode](how-to/planning-vs-building-mode.md) — plan review before implementation
43+
- [Set up PostgreSQL for production](how-to/set-up-postgres-production.md)`DBOS_SYSTEM_DATABASE_URL`
44+
- [Route LLM traffic through Gateway](how-to/route-llm-through-gateway.md) — agent CLI env + local Gateway
3945

4046
## Reference
4147

42-
- [Feature specifications](../spec/) — durable workflows, parallel execution, LLM proxy, webhooks, and more
4348
- [CLI reference](reference/cli.md)
49+
- [Configuration (environment variables)](reference/configuration.md)
50+
- [Feature specifications](../spec/)
4451

4552
## Explanation
4653

docs/how-to/configure-agents.md

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
---
2+
title: Configure orchestrator agents
3+
description: Select and verify Claude Code, Codex, Amp, OpenCode, or isolated worker subprocesses for drover run.
4+
product: drover-orchestrator
5+
audience: platform-operator
6+
doc_type: how-to
7+
topics:
8+
- agent-jobs
9+
surface: repo-docs
10+
---
11+
12+
# Configure orchestrator agents
13+
14+
**Drover Orchestrator** invokes a pluggable **orchestrator agent** inside each Git worktree. Configuration is per machine (environment variables) or per run (flags that override worker count and mode).
15+
16+
This path is **self-hosted** — not [Drover Cloud hosted agent jobs](../../../drover-cloud/docs/explanation/agent-jobs-vs-orchestrator.md).
17+
18+
## Supported agent types
19+
20+
| `DROVER_AGENT_TYPE` | Binary (`DROVER_AGENT_PATH`) | Notes |
21+
|---------------------|--------------------------------|-------|
22+
| `claude` (default) | `claude` | Anthropic Claude Code CLI |
23+
| `codex` | `codex` | OpenAI Codex agent |
24+
| `amp` | `amp` | Amp agent |
25+
| `opencode` | `opencode` | OpenCode CLI |
26+
| `worker` | inner agent via `DROVER_AGENT_PATH` | Runs the real agent inside a **`drover-worker`** subprocess for OOM isolation |
27+
28+
The factory in `internal/executor/agent.go` wires these types. A **`drover-code`** executor exists in the repo but is **not** selected by `DROVER_AGENT_TYPE` today — use Cloud [hosted agent jobs](../../../drover-cloud/docs/how-to/configure-gateway-hosted-agents.md) for headless Drover Code, or contribute wiring for local `drover-code`.
29+
30+
## Quick setup (Claude Code)
31+
32+
```bash
33+
# Verify the CLI is on PATH
34+
claude --version
35+
36+
export DROVER_AGENT_TYPE=claude
37+
export DROVER_AGENT_PATH=claude # optional if binary name is claude
38+
39+
cd my-repo
40+
drover init
41+
drover add "Smoke test task" --skip-validation
42+
drover run --workers 1 --verbose
43+
```
44+
45+
## Switch agent type
46+
47+
```bash
48+
# Codex
49+
export DROVER_AGENT_TYPE=codex
50+
export DROVER_AGENT_PATH=codex
51+
52+
# Amp with custom install location
53+
export DROVER_AGENT_TYPE=amp
54+
export DROVER_AGENT_PATH=/usr/local/bin/amp
55+
56+
# OpenCode
57+
export DROVER_AGENT_TYPE=opencode
58+
export DROVER_AGENT_PATH=opencode
59+
60+
drover run --workers 4
61+
```
62+
63+
`DROVER_CLAUDE_PATH` still works as an alias for `DROVER_AGENT_PATH` (deprecated).
64+
65+
## Process-isolated workers (OOM safety)
66+
67+
When many agents run in parallel, use the **`worker`** type so each task runs in a separate `drover-worker` process:
68+
69+
```bash
70+
export DROVER_AGENT_TYPE=worker
71+
export DROVER_AGENT_PATH=claude # agent the worker invokes
72+
export DROVER_USE_WORKER_SUBPROCESS=true # optional explicit toggle
73+
export DROVER_WORKER_BINARY=drover-worker
74+
export DROVER_WORKER_MEMORY_LIMIT=2G # optional per-worker RSS cap
75+
76+
drover run --workers 8
77+
```
78+
79+
Build `drover-worker` from this repo: `go build -o drover-worker ./cmd/drover-worker`.
80+
81+
## Project guidelines
82+
83+
Agents receive optional instructions from `.drover/task_template.yaml` (created by `drover init`) and per-task descriptions. Guidelines support template variables (`{{project}}`, `{{task_type}}`, `{{labels}}`) expanded in `internal/executor/guidelines.go`.
84+
85+
## Local testing without live LLM calls
86+
87+
For CI or scheduler testing, use mock-friendly settings documented in [internal testing checklists](../internal/testing/TESTING_CHECKLIST.md) (`DROVER_AGENT_TYPE=mock` where your branch supports it).
88+
89+
## Route LLM traffic through Gateway
90+
91+
To centralize provider keys, failover, and spend logging for self-hosted runs, see [Route LLM traffic through Drover Gateway](route-llm-through-gateway.md).
92+
93+
## Related
94+
95+
- [CLI reference](../reference/cli.md)`drover run` flags
96+
- [First parallel epic tutorial](../tutorials/first-parallel-epic.md)
97+
- [Planning vs building mode](planning-vs-building-mode.md)

0 commit comments

Comments
 (0)