Email Black Box

A SendGrid-compatible flight recorder for transactional email.

Created and maintained by Oncel Cebeci.

Email Black Box is a self-hosted black box for critical transactional email. Drop it in front of SendGrid to record every send intent, provider attempt, webhook event, failure, bounce, and delivery trace in one operator-friendly timeline.

The service currently exposes a SendGrid-compatible /v3/mail/send endpoint. Existing SendGrid-shaped requests can be pointed at this proxy, where they are validated, tagged with an internal trace id, logged, optionally filtered against local preferences, and forwarded to SendGrid or a fallback provider.

Why This Exists

When email breaks, the hardest question is usually not "which API did we call?" It is:

• What did our app try to send?
• Which provider accepted or rejected it?
• Was the failure in our app, the relay, the provider, or the recipient side?
• Did a webhook arrive later and change the status?
• Can we prove what happened to a customer, support team, or internal stakeholder?
• Can we safely replay or reroute the message?

Provider dashboards help, but they only show the provider's slice of the truth. This project records the truth before the provider, then stitches provider responses and webhooks back into the same trace.

The product idea is not "another email sender." It is a flight recorder for transactional email.

Product Surface

The black-box surface should revolve around traces:

• Send intent - the exact payload your app attempted to send
• Trace id - an internal ns_... id injected into provider metadata
• Provider attempts - provider, status code, message id, duration, and error
• Event timeline - processed, delivered, bounced, dropped, opened, clicked, and related webhook events
• Operator evidence - request payload, recipients, subject/template, provider result, event history
• Safety controls - dry-run mode, preference filtering, provider health, and future replay decisions

Failover is useful, but it is not the main identity. The main identity is: know exactly what happened to important email.

Current Features

• SendGrid-compatible POST /v3/mail/send API
• Bearer-token API authentication
• Dry-run mode for testing email flows without sending real messages
• Internal _ns_send_id correlation between outbound sends and SendGrid webhooks
• Trace log with request payload, recipients, status, provider attempts, and errors
• Send detail timeline in the admin UI
• SendGrid Event Webhook ingestion at /webhooks/sendgrid
• Optional SendGrid Event Webhook signature verification
• Basic notification preference store for opt-outs
• Admin API for stats, traces, events, preferences, and provider health
• Queryable trace search by text, recipient, status, provider, template, sender, and date range
• React admin UI for operators
• Provider failover across SendGrid, AWS SES, and Postmark
• Provider cooldown after repeated failures
• In-memory rate limiting for send, admin, and webhook endpoints
• Append-only JSONL trace storage that rehydrates on startup
• Optional Postgres trace storage with indexed query tables
• Docker Compose setup for local/self-hosted deployment

What It Is Not Yet

This is not yet a production-grade black box. Important limitations:

• In JSONL mode, runtime reads are memory-backed and rehydrated from local JSONL files on startup.
• Postgres mode is available, but migrations are still boot-time table creation rather than a dedicated migration system.
• There is no replay workflow yet.
• SendGrid template-only sends require SendGrid and cannot fail over to SES/Postmark.
• SES and Postmark adapters support a practical subset of SendGrid payloads.
• SendGrid webhook signature verification is opt-in and must be configured before exposing the webhook endpoint publicly.
• Admin auth is a simple bearer token stored by the browser in localStorage.
• Rate limiting is process-local and not shared across multiple API instances.
• There is no queue, idempotency layer, scheduled retry worker, or tenant model.
• There is no deliverability management, domain setup workflow, or abuse handling.

Use it as a useful diagnostic tool, a self-hosted starter, or a base for a more serious transactional email trace system.

Architecture

flowchart LR
  App["Existing app"] -->|"SendGrid-shaped POST /v3/mail/send"| API["Black Box API"]
  API --> Auth["API key auth"]
  Auth --> Validate["Payload validation"]
  Validate --> Trace["Create trace id"]
  Trace --> Prefs["Preference filtering"]
  Prefs --> Store["Trace store"]
  Store --> Router["Provider manager"]
  Router --> SG["SendGrid"]
  Router --> SES["AWS SES"]
  Router --> PM["Postmark"]
  Router --> Attempts["Provider attempts"]
  Attempts --> Store
  SG -->|"Event Webhook"| Webhook["/webhooks/sendgrid"]
  Webhook --> Events["Event timeline"]
  Events --> Store
  Admin["Admin UI"] -->|"Bearer admin key"| AdminAPI["/api/*"]
  AdminAPI --> Store

Loading

Packages

• packages/api - Fastify API, provider routing, webhook handling, storage, tests
• packages/admin-ui - React/Vite admin console
• packages/shared - Shared types and constants

Quickstart

1. Configure environment

Copy the example file and fill in provider credentials:

cp .env.example .env

Minimum local dry-run config:

PORT=3100
API_KEYS=ns-key-local
ADMIN_API_KEY=admin-local
LOG_DIR=./logs
STORAGE_DRIVER=jsonl
DRY_RUN=true
ENABLE_PREFERENCE_CHECK=false
PROVIDER_ORDER=sendgrid

For real forwarding, set DRY_RUN=false and configure at least one provider credential.

To use Postgres-backed trace search, set:

STORAGE_DRIVER=postgres
DATABASE_URL=postgres://notification_service:notification_service@postgres:5432/notification_service

2. Run with Docker Compose

docker compose up --build

Services:

• API: http://localhost:3100
• Admin UI: http://localhost:3101

3. Send a test request

export EMAIL_BLACKBOX_API_KEY=ns-key-local

curl -i http://localhost:3100/v3/mail/send \
  -H "Authorization: Bearer ${EMAIL_BLACKBOX_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "personalizations": [
      { "to": [{ "email": "user@example.com" }] }
    ],
    "from": { "email": "sender@example.com" },
    "subject": "Hello from the Email Black Box",
    "content": [
      { "type": "text/plain", "value": "This is a test email." }
    ]
  }'

In dry-run mode this returns 202 and records the send without contacting a provider.

4. Open the admin UI

Visit http://localhost:3101 and enter ADMIN_API_KEY.

Local Development

Install dependencies:

npm install

Run API and admin UI together:

npm run dev

Run tests:

npm test

Run the real Postgres storage test against a container-backed database:

docker compose up -d postgres
cd packages/api
PG_TEST_DATABASE_URL=postgres://notification_service:notification_service@localhost:5432/notification_service npm run test:postgres

Build all packages:

npm run build

Configuration

Variable	Purpose	Default
PORT	API port	3100
SENDGRID_API_KEY	SendGrid API key	empty
API_KEYS	Comma-separated client API keys for /v3/mail/send	empty
ADMIN_API_KEY	Admin bearer token for /api/*	empty
LOG_DIR	JSONL log directory	./logs
MEMORY_STORE_MAX_ENTRIES	Max in-memory sends/events	10000
STORAGE_DRIVER	Storage backend: jsonl or postgres	jsonl
DATABASE_URL	Postgres connection string when STORAGE_DRIVER=postgres	empty
DRY_RUN	Log sends without forwarding	false
ENABLE_PREFERENCE_CHECK	Filter opted-out recipients	false
PROVIDER_ORDER	Comma-separated provider order	sendgrid
PROVIDER_MAX_FAILURES	Failures before cooldown	3
PROVIDER_COOLDOWN_MS	Provider cooldown duration	60000
RATE_LIMIT_ENABLED	Enable in-memory endpoint rate limits	true
RATE_LIMIT_WINDOW_MS	Rate-limit window size	60000
RATE_LIMIT_SEND_MAX	/v3/mail/send requests per token/IP per window	120
RATE_LIMIT_ADMIN_MAX	/api/* requests per token/IP per window	300
RATE_LIMIT_WEBHOOK_MAX	/webhooks/sendgrid requests per token/IP per window	600
SENDGRID_WEBHOOK_PUBLIC_KEY	SendGrid Event Webhook public verification key	empty
SENDGRID_WEBHOOK_REQUIRE_SIGNATURE	Reject SendGrid webhooks when no public key is configured	false
SENDGRID_WEBHOOK_MAX_AGE_SECONDS	Allowed timestamp skew for signed webhooks	600
AWS_ACCESS_KEY_ID	Optional SES access key	empty
AWS_SECRET_ACCESS_KEY	Optional SES secret	empty
AWS_SES_REGION	SES region	ap-southeast-2
POSTMARK_SERVER_TOKEN	Postmark server token	empty

Provider Compatibility

Capability	SendGrid	AWS SES	Postmark
Direct SendGrid payload forwarding	Yes	No	No
Plain text / HTML content	Yes	Yes	Yes
Attachments	Yes	Not yet	Yes
SendGrid dynamic templates	Yes	No	No
Webhook correlation	Yes	Send event only	Send event only
Provider message ID logging	Yes	Yes	Yes

SES and Postmark are fallback adapters. They are useful for content-based sends, but they are not full SendGrid emulators. A mature black box should show when a message is safe to fail over and when provider differences make failover unsafe.

Webhooks

Configure SendGrid Event Webhook to post to:

https://your-domain.example/webhooks/sendgrid

The service injects _ns_send_id into each outbound personalization's custom_args. SendGrid sends that value back in events, allowing the admin UI to show delivery/open/click/bounce/drop events on the original trace.

To verify signed SendGrid webhook requests, enable Signed Event Webhook in SendGrid, copy the public verification key, and set:

SENDGRID_WEBHOOK_PUBLIC_KEY=your-sendgrid-webhook-public-key
SENDGRID_WEBHOOK_REQUIRE_SIGNATURE=true

When SENDGRID_WEBHOOK_PUBLIC_KEY is set, the API verifies X-Twilio-Email-Event-Webhook-Signature and X-Twilio-Email-Event-Webhook-Timestamp against the raw request body before processing events.

PEM-formatted public keys are supported. If your environment provider requires a single-line value, keep the PEM markers and replace line breaks with \n.

Admin API

All admin endpoints require:

Authorization: Bearer <ADMIN_API_KEY>

Endpoints:

• GET /api/stats
• GET /api/providers/health
• GET /api/traces?page=1&pageSize=20&q=invoice&email=user@example.com&status=delivered&provider=sendgrid
• GET /api/traces/:id
• GET /api/events?page=1&pageSize=20
• GET /api/preferences?page=1&pageSize=20
• PUT /api/preferences

Legacy /api/logs routes are kept as compatibility aliases.

Token Guidance

Use separate, high-entropy values for API_KEYS and ADMIN_API_KEY. Client API keys can submit email sends; the admin key can read trace payloads and webhook history. Do not reuse the admin key as a send key, do not commit tokens to git, and rotate keys after any suspected exposure.

Security

See SECURITY.md for the current security model, known limitations, and vulnerability reporting guidance.

License

Apache License 2.0. See LICENSE and NOTICE.

Copyright notices use the legal owner name, Oncel Ozgebayram. Public project attribution uses Oncel Cebeci.

Trademarks

The Email Black Box and Black Box names, logo, and project branding are not licensed for use in ways that imply official endorsement. Forks are welcome, but should use a distinct name unless they are clearly presented as unofficial.