behinddwalls · Sep 28, 2026 · Sep 28, 2026 · Sep 28, 2026 · Sep 28, 2026
@@ -38,42 +38,49 @@ request.Version = newVersion
 ```
 submitqueue/                        # repo root (Go module github.com/uber/submitqueue)
 ├── api/                            # Published wire contracts (cross-domain/external)
+│   ├── base/                       # Shared protos (change/, hook/, mergestrategy/, messagequeue/)
 │   ├── submitqueue/{gateway,orchestrator}/{proto,protopb}/   # RPC (proto)
 │   ├── stovepipe/{proto,protopb}/  # single-service RPC (proto) — no service segment yet
 │   ├── runway/{proto,protopb}/     # RPC (proto) — single-service domain, no service segment
 │   └── runway/messagequeue/        # external queue contracts (proto + protojson)
 ├── platform/                       # SHARED cross-domain packages — no domain deps
-│   ├── errs/, metrics/, consumer/, http/
+│   ├── errs/, metrics/, consumer/, http/, publish/
+│   ├── git/, hook/, lifecycle/, pipeline/
 │   ├── base/                       # SHARED entities (change/, messagequeue/, …)
 │   └── extension/                  # SHARED extension contracts + backends (counter/, messagequeue/, …)
 ├── submitqueue/                    # SubmitQueue domain
 │   ├── gateway/                    # Gateway service (port 8081) - entry point
-│   │   └── extension/              # Aggregates/backends only the gateway resolves (storage/)
+│   │   ├── core/request/           # Request-log materialization
+│   │   └── extension/storage/      # Aggregate, MySQL implementation, and schema
 │   ├── orchestrator/               # Orchestrator service (port 8082) - coordinates jobs
-│   │   └── extension/              # Aggregates/backends only the orchestrator resolves (storage/)
+│   │   ├── core/                   # request publish/terminate; batch helpers
+│   │   └── extension/storage/      # Aggregate, MySQL implementation, and schema
+│   ├── client/                     # Gateway client for CLIs and the demo
 │   ├── entity/                     # SubmitQueue-specific domain entities
 │   ├── extension/                  # SubmitQueue-specific extension contracts and implementations
-│   └── core/                       # SubmitQueue-internal shared infra (changeset, messagequeue, topickey)
+│   └── core/                       # Shared infra both services use (changeset, messagequeue, topickey)
 ├── stovepipe/                      # Stovepipe domain (single service)
 │   ├── controller/                 # RPC and queue-stage business logic
 │   ├── entity/                     # Stovepipe domain entities
 │   ├── extension/                  # Stovepipe-specific extension contracts and implementations
 │   └── core/                       # Stovepipe-internal queue contracts and shared infrastructure
 ├── runway/                         # Runway domain (single service — the domain *is* the service)
-│   └── controller/                 # Runway service controllers (consumes the merge queues; no gateway/orchestrator split)
+│   ├── controller/                 # Merge-queue controllers (no gateway/orchestrator split)
+│   └── extension/                  # Runway extensions (merger/)
 ├── tool/                           # Development and CI tooling
 ├── service/                        # Runnable server/client wiring (entry points + Docker Compose)
+│   ├── messagequeue/               # Shared queue MySQL pool and tenant configuration
 │   ├── submitqueue/                # Runnable SubmitQueue servers/clients + Docker Compose
 │   ├── stovepipe/                  # Runnable Stovepipe server/client + Docker Compose
 │   └── runway/                     # Runnable Runway server/client + Docker Compose
 ├── test/
-│   ├── e2e/submitqueue/            # End-to-end tests (full stack)
+│   ├── e2e/{submitqueue,stovepipe,runway}/  # End-to-end tests
 │   ├── integration/                # Integration tests (platform/, submitqueue/, stovepipe/, …)
 │   └── testutil/                   # Test utilities (ComposeStack, MySQL helpers)
 └── doc/                            # Documentation
 ```
 
-The `platform/` tree holds code reused across domains (infrastructure, shared entities, shared extension contracts). A multi-service **domain** (e.g. `submitqueue/`) keeps the same internal layout (`gateway/`, `orchestrator/`, `entity/`, `extension/`, `core/`); a domain's own `core/` (e.g. `submitqueue/core/`) holds infra shared only between that domain's services. A **single-service domain** collapses that split — the domain *is* the service, so its controllers live directly under the domain root (e.g. `runway/controller/`, `stovepipe/controller/`) with no `gateway/`/`orchestrator/` segment, and its wire contract is service-segment-free (`api/{domain}/`). `runway` is a consumer-only merge execution service with no gateway. `stovepipe` exposes ingestion RPC behavior and runs its own process, build, build-signal, record, hook, and DLQ queue stages.
+The `platform/` tree holds code reused across domains (infrastructure, shared entities, shared extension contracts). A multi-service **domain** (e.g. `submitqueue/`) keeps the same internal layout (`gateway/`, `orchestrator/`, `entity/`, `extension/`, `core/`); a domain's own `core/` (e.g. `submitqueue/core/`) holds infra shared by that domain's services — SubmitQueue's is `changeset`, `messagequeue`, and `topickey`. Helpers that serve one service live under that service: `submitqueue/gateway/core/request` materializes request logs, `submitqueue/orchestrator/core/request` publishes and terminates them, and `submitqueue/orchestrator/core/batch` moves batches and their queue membership records. A **single-service domain** collapses that split — the domain *is* the service, so its controllers live directly under the domain root (e.g. `runway/controller/`, `stovepipe/controller/`) with no `gateway/`/`orchestrator/` segment, and its wire contract is service-segment-free (`api/{domain}/`). `runway` is a consumer-only merge execution service with no gateway; it publishes merge results on the signal queues. `stovepipe` exposes ingestion RPC behavior and runs its own process, build, build-signal, record, hook, and DLQ queue stages.
 
 The `api/` tree holds **published** wire contracts — those depended on from outside the owning domain. RPC contracts live at `api/{domain}/{service}/` (`proto/` for `.proto` sources, `protopb/` for committed generated Go); for a single-service domain the service segment is dropped, so the contract lives directly at `api/{domain}/` (e.g. `api/runway/{proto,protopb}/`). A service package may hold multiple `.proto` files, all generating into the same `protopb/`. External message-queue contracts live at `api/{domain}/messagequeue/` (see Message Queue Contracts below). Internal queue contracts do **not** go here — they live under `{domain}/core/messagequeue/`.
 
@@ -175,7 +182,8 @@ Paths follow the directory layout: shared packages live under `platform/` at the
 - Domain extensions: `github.com/uber/submitqueue/{domain}/extension/{ext}[/{impl}]` (e.g. `.../submitqueue/extension/storage`)
 - Service-scoped extensions: `github.com/uber/submitqueue/{domain}/{service}/extension/{ext}[/{impl}]` (e.g. `.../submitqueue/orchestrator/extension/storage/mysql`)
 - Cross-domain consumer framework: `github.com/uber/submitqueue/platform/consumer`; internal topic keys live with the owning domain contract (for example `submitqueue/core/messagequeue` and `stovepipe/core/messagequeue`); external queue topic keys live with their published contract (for example `api/runway/messagequeue`)
-- Domain-internal infra: `github.com/uber/submitqueue/{domain}/core/{pkg}` (e.g. `.../submitqueue/core/request`)
+- Domain-internal infra: `github.com/uber/submitqueue/{domain}/core/{pkg}` (e.g. `.../submitqueue/core/changeset`, `.../submitqueue/core/messagequeue`)
+- Service-scoped infra: `github.com/uber/submitqueue/{domain}/{service}/core/{pkg}` (e.g. `.../submitqueue/gateway/core/request`, `.../submitqueue/orchestrator/core/request`, `.../submitqueue/orchestrator/core/batch`)
 - Shared entities: `github.com/uber/submitqueue/platform/base/{pkg}` (e.g. `.../platform/base/messagequeue`)
 - Shared extensions: `github.com/uber/submitqueue/platform/extension/{ext}[/{impl}]` (e.g. `.../platform/extension/messagequeue/mysql`)
 - Cross-domain infra: `github.com/uber/submitqueue/platform/{pkg}` (e.g. `.../platform/errs`, `.../platform/metrics`, `.../platform/http`)
@@ -246,7 +254,7 @@ make local-submitqueue-start        # Start full stack with Docker Compose
 make local-submitqueue-ps           # Show running containers and ports
 make local-submitqueue-logs         # View logs from all services
 make local-stop         # Stop all services
-make clean              # Clean Bazel cache
+make clean              # Clean Bazel cache and bin/
 ```
 
 ### Common Workflows

@@ -232,7 +232,7 @@ check-tidy: tidy ## Check that go.mod and MODULE.bazel are tidy
 	$(call assert_clean,make tidy)
 	@echo "Module files are up to date."
 
-clean: ## Clean generated files and binaries
+clean: ## Remove the Bazel cache and bin/ (generated proto: make clean-proto)
 	@echo "Cleaning with Bazel..."
 	@$(BAZEL) clean
 	@rm -rf bin/
@@ -485,7 +485,7 @@ local-submitqueue-ps: ## Show running containers and their ports
 	@echo "  mysql -h127.0.0.1 -P$$(docker port $(SUBMITQUEUE_LOCAL_PROJECT)-mysql-app-1 3306 2>/dev/null | cut -d: -f2 || echo 'PORT') -uroot -proot submitqueue"
 	@echo ""
 	@echo "  # Call Gateway gRPC"
-	@echo "  grpcurl -plaintext -d '{\"message\":\"test\"}' localhost:$$(docker port $(SUBMITQUEUE_LOCAL_PROJECT)-gateway-service-1 8080 2>/dev/null | cut -d: -f2 || echo 'PORT') submitqueue.SubmitQueueGateway/Ping"
+	@echo "  grpcurl -plaintext -d '{\"message\":\"test\"}' localhost:$$(docker port $(SUBMITQUEUE_LOCAL_PROJECT)-gateway-service-1 8080 2>/dev/null | cut -d: -f2 || echo 'PORT') uber.submitqueue.gateway.SubmitQueueGateway/Ping"
 	@echo ""
 	@echo "  # View logs"
 	@echo "  make local-submitqueue-logs"
@@ -541,12 +541,12 @@ local-submitqueue-stop: ## Stop the SubmitQueue stack (keeps PROVIDER=git's sand
 		echo "Sandbox repository left at $(SQ_GIT_SANDBOX_DIR); remove it with 'make local-submitqueue-clean'."; \
 	fi
 
-local-stop: ## Stop every local stack — SubmitQueue, Stovepipe, and Runway (keep data)
+local-stop: ## Stop every local stack — SubmitQueue, Stovepipe, and Runway
 	@echo "Stopping all services..."
 	@$(COMPOSE) -f $(COMPOSE_FILE) -p $(SUBMITQUEUE_LOCAL_PROJECT) down
 	@$(COMPOSE) -f $(STOVEPIPE_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) down
 	@$(COMPOSE) -f $(RUNWAY_COMPOSE_FILE) -p $(RUNWAY_LOCAL_PROJECT) down
-	@echo "Services stopped. Data volumes preserved."
+	@echo "Services stopped. Anonymous database volumes are not reused on the next start."
 
 local-stovepipe-debug-start: build-stovepipe-linux-debug ## Start Stovepipe under delve in Docker (attach IDE to :2345)
 	@echo "Starting Stovepipe service with compose (debug)..."
@@ -581,9 +581,34 @@ local-stovepipe-stop: ## Stop the Stovepipe service
 	@$(COMPOSE) -f $(STOVEPIPE_COMPOSE_FILE) -p $(STOVEPIPE_LOCAL_PROJECT) down
 	@echo "Stovepipe service stopped."
 
+# go generate does not walk the module; every tree with a //go:generate directive is listed here.
+GO_GENERATE_PACKAGES := \
+	./platform/consumer/... \
+	./platform/extension/consumergate/... \
+	./platform/extension/counter/... \
+	./platform/extension/hook/... \
+	./platform/extension/messagequeue/... \
+	./runway/extension/merger/... \
+	./stovepipe/core/requestlog/... \
+	./stovepipe/extension/buildrunner/... \
+	./stovepipe/extension/projectresult/... \
+	./stovepipe/extension/queueconfig/... \
+	./stovepipe/extension/sourcecontrol/... \
+	./stovepipe/extension/storage/... \
+	./submitqueue/core/changeset/... \
+	./submitqueue/extension/buildrunner/... \
+	./submitqueue/extension/changeprovider/... \
+	./submitqueue/extension/conflict/... \
+	./submitqueue/extension/queueconfig/... \
+	./submitqueue/extension/speculation/... \
+	./submitqueue/extension/storage/... \
+	./submitqueue/extension/validator/... \
+	./submitqueue/gateway/extension/storage/... \
+	./submitqueue/orchestrator/extension/storage/...
+
 mocks: ## Generate mock files using mockgen
 	@echo "Generating mocks..."
-	@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/gateway/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changeprovider/... ./platform/extension/counter/... ./platform/extension/consumergate/... ./platform/extension/hook/... ./platform/extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./runway/extension/merger/... ./submitqueue/extension/conflict/... ./submitqueue/extension/speculation/... ./submitqueue/extension/validator/... ./platform/consumer/... ./stovepipe/core/requestlog/... ./stovepipe/extension/storage/... ./stovepipe/extension/sourcecontrol/... ./stovepipe/extension/projectresult/...
+	@$(BAZEL) run @rules_go//go -- generate $(GO_GENERATE_PACKAGES)
 	@echo "Mocks generated successfully!"
 
 proto: ## Generate protobuf files from .proto definitions

@@ -90,20 +90,33 @@ brew install grpcurl
 
 ## Common Make Targets
 
+`make help` lists every target. The table below is the day-to-day set.
+
+CI runs `make lint`, `make check-tidy`, and `make check-gazelle`. `make lint` includes the format check, license headers, the tracked-binary check, the message-ID check, and the queue-shard check. `make fmt`, `make tidy`, and `make gazelle` apply the corresponding fixes. `make mocks` regenerates the checked-in mockgen files; `make check-mocks` fails when that output differs from the tree. `make clean` removes the Bazel cache and `bin/`. Generated protobuf Go files stay in the source tree until `make clean-proto`; `make proto` writes them again.
+
 | Target | Description |
 |--------|-------------|
 | `make build` | Build all services |
 | `make test` | Run unit tests |
 | `make integration-test` | Run all integration tests (Docker-based) |
 | `make e2e-test` | Run end-to-end tests |
+| `make fmt` | Format Go and YAML |
+| `make lint` | Run the linters CI runs |
+| `make tidy` | Tidy `go.mod` and `MODULE.bazel` |
+| `make check-tidy` | Fail if `go.mod` or `MODULE.bazel` is untidy |
+| `make check-gazelle` | Fail if `BUILD.bazel` files are stale |
+| `make gazelle` | Update `BUILD.bazel` files |
+| `make mocks` | Regenerate mockgen files |
+| `make check-mocks` | Fail if generated mocks are stale |
 | `make proto` | Regenerate protobuf files |
-| `make gazelle` | Update BUILD.bazel files |
+| `make clean` | Remove the Bazel cache and `bin/` |
+| `make clean-proto` | Remove generated protobuf Go files |
 | `make local-submitqueue-start` | Start full workflow stack (Gateway + Orchestrator + Runway + two MySQL databases) |
-| `make local-submitqueue-ps` | Show running containers and ports |
-| `make local-submitqueue-logs` | View logs from all services |
-| `make local-stop` | Stop all services |
-| `make clean` | Clean generated files and binaries |
-| `make help` | Show all available targets with descriptions |
+| `make local-submitqueue-ps` | Show running SubmitQueue containers and ports |
+| `make local-submitqueue-logs` | View logs from all SubmitQueue services |
+| `make local-submitqueue-stop` | Stop the SubmitQueue stack |
+| `make local-stop` | Stop SubmitQueue, Stovepipe, and Runway |
+| `make help` | List every target |
 
 ## Running Specific Tests
 

@@ -144,6 +144,8 @@ Shared (cross-domain) suites carry no domain segment — e.g. the shared queue e
 | SubmitQueue consumer (core) | `core-submitqueue-consumer` | `sq-test-core-submitqueue-consumer-…-mysql-1` |
 | SubmitQueue e2e (full stack) | `e2e-submitqueue` | `sq-test-e2e-submitqueue-def456-gateway-service-1` |
 
+The same shape covers the other suites, including `e2e-stovepipe`, `e2e-runway`, `e2e-submitqueue-git`, and `ext-messagequeue-vitess`.
+
 ### Parallel execution
 
 Each suite normally gets a distinct project name (`{context}-{shortid}`), where the short suffix is derived from the low 24 bits of the current nanosecond timestamp. It is useful for separating concurrent runs but is not a guaranteed unique identifier. Every compose service publishes **ephemeral host ports** (`- "3306"`, `- "8080"`), so suites can run **in parallel**. `make integration-test` runs suites concurrently via `--test_output=errors` (`--test_output=streamed` would force Bazel to serialize them). The domain-qualified context keeps container names understandable when many run at once.
@@ -158,11 +160,11 @@ For additional manual inspection:
 # See what tests are currently running
 docker ps --format "table {{.Names}}\t{{.Status}}" | grep sq-test
 
-# Find all containers from gateway test
-docker ps | grep sq-test-gateway
+# Find all containers from the SubmitQueue gateway integration test
+docker ps | grep sq-test-svc-submitqueue-gateway
 
 # Inspect a specific test's MySQL
-docker exec -it sq-test-ext-counter-2ce1d0-mysql-1 \
+docker exec -it sq-test-ext-counter-mysql-2ce1d0-mysql-1 \
   mysql -uroot -proot submitqueue -e "SHOW TABLES;"
 ```
 
@@ -277,7 +279,8 @@ grpcurl -plaintext -import-path . -proto api/submitqueue/gateway/proto/gateway.p
 | `make local-submitqueue-ps` | Show running containers and ports |
 | `make local-submitqueue-logs` | Follow logs from all services |
 | `make local-submitqueue-restart` | Rebuild and restart all services |
-| `make local-stop` | Stop all services (keep data) |
+| `make local-submitqueue-stop` | Stop the SubmitQueue stack (MySQL data does not survive; a `PROVIDER=git` sandbox is left in place) |
+| `make local-stop` | Stop SubmitQueue, Stovepipe, and Runway |
 | `make local-submitqueue-gateway-stop` | Stop Gateway service |
 | `make local-submitqueue-orchestrator-stop` | Stop Orchestrator service |
 | `make local-submitqueue-clean` | Stop and remove all services, volumes, and images |
@@ -352,6 +355,10 @@ docker network ls | grep sq-test | awk '{print $1}' | xargs docker network rm
 
 ## Writing New Tests
 
+Integration tests under `test/integration/` use the directory name as the package (`package gateway`, `package orchestrator`, `package stovepipe`, `package mysql`), not an external `*_test` package.
+
+End-to-end tests under `test/e2e/` are the exception: every suite there is `package e2e_test`.
+
 ### Adding Unit Tests
 
 1. Create `{file}_test.go` next to production code

@@ -2,9 +2,13 @@
 
 Fire-and-forget side effects for pipeline lifecycle events: one shared event contract, a durable hook topic per domain, pluggable hooks.
 
+## Status
+
+Implemented. The contract is `api/base/hook`, the dispatcher and DLQ reconciler are `platform/hook`, and the extension is `platform/extension/hook`. Stovepipe's `process` and `record` publish repository-scoped hook events, and `service/stovepipe/server` registers the stage. The SubmitQueue orchestrator pipeline registers a `submitqueue-hook` stage; no orchestrator controller publishes a hook event, and `service/submitqueue/orchestrator/server` resolves every event to noop. A deployment's real integrations are whatever its resolver returns.
+
 ## Problem
 
-The pipelines emit lifecycle transitions — a request lands or fails, a batch lands, a build finishes — but nothing can react outside pipeline state: no warehouse export, no PR comments or closes on land events, no notifications or audit trails. The log topic is not this seam: SubmitQueue request statuses only, consumed solely to build gateway read models.
+The pipelines emit lifecycle transitions — a request lands or fails, a batch lands, a build finishes — and a side effect needs a place outside pipeline state: warehouse export, PR comments or closes on land events, notifications, audit trails. The log topic is not this seam: SubmitQueue request statuses only, consumed solely to build gateway read models.
 
 Two requirements: side effects must never stall or fail the pipeline, and "fire and forget" must not mean lossy — a land-failure comment that silently never posts is a support ticket.