Skip to content

README.md

Latest commit

 

History

History
305 lines (240 loc) · 9.24 KB

README.md

File metadata and controls

305 lines (240 loc) · 9.24 KB

IPFS Website Gateway

License: MIT Go Version

A stateless edge IPFS gateway that serves DNSLink websites with strict access control via an internal API. Built with Go, Echo, and Boxo IPFS.

Features

  • DNSLink vhost support - Serves websites via virtual hosting using DNSLink records (_dnslink.{domain})
  • Access control - Validates website access against a central API via ipfs-sdk before serving
  • Status caching - In-memory LRU cache for website status queries (TTL-based, DoS protection)
  • Boxo IPFS integration - Lightweight Bitswap client-only node for content fetching from P2P network
  • Caddy On-Demand TLS - /allowed endpoint for certificate issuance validation with optional rate limiting and auth
  • Health monitoring - /healthz endpoint checking internal API and IPFS peer connectivity
  • Cache pre-warming - Automatically populates content cache when IPNS pubsub detects a changed content hash
  • Graceful shutdown - Proper cleanup on SIGINT/SIGTERM

Architecture

┌─────────────┐
│   Client    │
└──────┬──────┘
       │ HTTP Request
       ▼
┌─────────────────────────────────┐
│   Echo HTTP Server              │
│   - Recovery middleware         │
│   - Logger middleware           │
└──────┬──────────────────────────┘
       │ AccessControlMiddleware
       ▼
┌─────────────────────────────────┐
│   Request Pipeline              │
│   1. Extract domain from Host   │
│      or X-Forwarded-Host        │
│   2. Strip port (IPv6-safe)    │
│   3. Passthrough for IPs and   │
│      /ipfs/, /ipns/ paths      │
│   4. Check status cache (LRU)   │
│   5. Query internal API         │
│   6. Check result               │
│      (404/410/active)           │
│   7. Rewrite to /ipns/{domain} │
└──────┬──────────────────────────┘
       │
       ▼
┌─────────────────────────────────┐
│   Boxo Gateway Handler          │
│   - DNSLink resolution          │
│   - UnixFS content serving      │
│   - Headers (ETag, Cache-Control)│
│   - Range requests              │
└─────────────────────────────────┘

Installation

Prerequisites

  • Go 1.26 or later
  • Network access to internal API
  • DNS server with DNSLink records (for serving websites)

Build

go build -o gateway cmd/gateway/main.go

For other platforms:

GOOS=linux GOARCH=amd64 go build -o gateway-linux-amd64 cmd/gateway/main.go
GOOS=darwin GOARCH=amd64 go build -o gateway-darwin-amd64 cmd/gateway/main.go
GOOS=windows GOARCH=amd64 go build -o gateway-windows-amd64.exe cmd/gateway/main.go

Usage

Running the Gateway

# Using environment variables
export GATEWAY__API__URL=https://api.example.com
export GATEWAY__API__SECRET=my-secret-key
export GATEWAY__SERVER__PORT=8080
./gateway

# Using config file
./gateway --config /path/to/config/directory

# Using default config (creates ./gateway.yaml if not exists)
./gateway

CLI Flags

Flag Description
--config Path to config file directory

Configuration

Configuration is loaded from multiple sources in priority order:

  1. Environment variables (highest priority)
  2. Configuration file (YAML)
  3. Default values (lowest priority)

Config File Search

The gateway searches for gateway.yaml in these directories (in order):

  • /etc/lumeweb/gateway/
  • $HOME/.lumeweb/gateway/
  • ./ (current directory)

If no config file is found, gateway.yaml is created in the first writable directory.

Example Config File

server:
  port: 8080
  trusted_proxies: []
  allowed_secret: ""

api:
  url: https://api.example.com
  secret: my-secret-key
  timeout: 30s

ipfs:
  seed_peer: ipfs.pinner.xyz
  repo_path: ./data/ipfs

cache:
  status_cache_ttl: 5m
  status_cache_lru_size: 1000
  content_cache_path: /data/cache
  content_cache_max_bytes: 10737418240  # 10GB
  content_cache_lru_size: 100000

prewarm:
  enabled: true
  max_concurrency: 2
  retry_attempts: 2
  retry_delay: 1s

logging:
  level: info

rate_limit:
  enabled: false
  rate: 0.167
  burst: 10
  expires_in: 5m

Environment Variables

All configuration can be set via environment variables using the format GATEWAY__SECTION__KEY:

Required:

  • GATEWAY__API__URL - Internal API base URL
  • GATEWAY__API__SECRET - Gateway secret for API authentication

Optional:

Variable Description Default
GATEWAY__SERVER__PORT Server port 8080
GATEWAY__SERVER__TRUSTED_PROXIES Trusted proxy IPs []
GATEWAY__SERVER__ALLOWED_SECRET Auth secret for /allowed endpoint ""
GATEWAY__API__TIMEOUT API request timeout 30s
GATEWAY__IPFS__SEED_PEER IPFS seed peer ipfs.pinner.xyz
GATEWAY__IPFS__REPO_PATH IPFS repository path ./data/ipfs
GATEWAY__CACHE__STATUS_CACHE_TTL Status cache TTL 5m
GATEWAY__CACHE__STATUS_CACHE_LRU_SIZE Status cache size 1000
GATEWAY__CACHE__CONTENT_CACHE_PATH Content cache directory /data/cache
GATEWAY__CACHE__CONTENT_CACHE_MAX_BYTES Content cache max size 10737418240 (10GB)
GATEWAY__CACHE__CONTENT_CACHE_LRU_SIZE Content cache size 100000
GATEWAY__CACHE__IPNS_CACHE_PATH IPNS cache directory /data/ipns
GATEWAY__CACHE__IPNS_CACHE_LRU_SIZE IPNS cache size 140000
GATEWAY__LOGGING__LEVEL Log level info
GATEWAY__RATE_LIMIT__ENABLED Enable rate limiting on /allowed false
GATEWAY__RATE_LIMIT__RATE Rate limit (requests/second) 0.167
GATEWAY__RATE_LIMIT__BURST Rate limit burst 10
GATEWAY__RATE_LIMIT__EXPIRES_IN Rate limiter cleanup interval 5m
GATEWAY__PREWARM__ENABLED Enable cache pre-warming on IPNS updates true
GATEWAY__PREWARM__MAX_CONCURRENCY Max concurrent DAG walks 2
GATEWAY__PREWARM__RETRY_ATTEMPTS Block fetch retry attempts 2
GATEWAY__PREWARM__RETRY_DELAY Retry delay between attempts 1s

API Reference

Health Check

Endpoint: GET /healthz

Returns health status of gateway dependencies.

Response:

{
  "status": "pass",
  "checks": {
    "internal_api": {
      "status": "pass"
    },
    "ipfs_peer": {
      "status": "pass",
      "data": {
        "peer_count": 5
      }
    }
  }
}

Gateway Request

Endpoint: ANY /*path

Serves content from IPFS based on DNSLink resolution of the Host (or X-Forwarded-Host) header.

Headers:

  • Host - Domain name with DNSLink record
  • X-Forwarded-Host - Alternative domain header (takes precedence over Host)

Responses:

  • 200 OK - Content served successfully
  • 404 Not Found - Website not found or not active
  • 410 Gone - Website is broken or removed
  • 500 Internal Server Error - Server error

Example:

curl -H "Host: example.com" http://localhost:8080/

Caddy On-Demand TLS

Endpoint: GET /allowed?domain=...&secret=...

Validates whether a domain is allowed for TLS certificate issuance. Used by Caddy's On-Demand TLS feature.

Parameters:

  • domain - Domain name to validate
  • secret - Authentication secret (required if AllowedSecret is configured)

Responses:

  • 200 OK - Domain is allowed
  • 400 Bad Request - Domain not allowed or auth failed

Development

Project Structure

.
├── cmd/gateway/          # CLI entry point
├── docs/                 # Documentation
├── internal/
│   ├── api/             # Internal API client (ipfs-sdk)
│   ├── cache/           # Status and content cache
│   ├── config/          # Configuration management
│   ├── dns/             # DNSLink validation
│   ├── gateway/         # Boxo gateway handler + access control
│   ├── health/          # Health checks
│   ├── ipfs/            # IPFS node and fetcher
│   ├── prewarm/         # Cache pre-warming on IPNS updates
│   └── server/          # Echo server setup
├── pkg/types/           # Common types
├── vendor/              # Vendored dependencies
├── AGENTS.md            # Agent development guide
├── go.mod               # Go module definition
└── README.md            # This file

Running Tests

# Run all tests
go test ./...

# Run tests with coverage
go test -cover ./...

# Run tests with verbose output
go test -v ./...

# Run tests for specific package
go test ./internal/server/...

Generating Mocks

# Mockery is pre-installed at $HOME/go/bin/mockery
$HOME/go/bin/mockery

License

This project is licensed under the MIT License - see the LICENSE file for details.