|
| 1 | +# beyond/objects |
| 2 | + |
| 3 | +An S3-compatible object store backed by the local filesystem. Native REST API and full S3 wire protocol — no database, no buffering, authentication via HMAC-derived tokens. |
| 4 | + |
| 5 | +Objects are files on disk. Metadata lives in extended attributes. Writes are atomic: stream to a temp file, fsync, set xattrs, rename. Reads stream directly to the client without buffering the full body into memory. |
| 6 | + |
| 7 | +## Quick Start |
| 8 | + |
| 9 | +**Put and get an object:** |
| 10 | + |
| 11 | +```sh |
| 12 | +curl -T ./photo.jpg \ |
| 13 | + -H "Authorization: Bearer secret" \ |
| 14 | + -H "Content-Type: image/jpeg" \ |
| 15 | + http://localhost:9000/v1/default/photos/cat.jpg |
| 16 | + |
| 17 | +curl -H "Authorization: Bearer secret" \ |
| 18 | + http://localhost:9000/v1/default/photos/cat.jpg \ |
| 19 | + -o cat.jpg |
| 20 | +``` |
| 21 | + |
| 22 | +**Or use the TypeScript SDK:** |
| 23 | + |
| 24 | +```sh |
| 25 | +npm install @beyond.dev/objects |
| 26 | +``` |
| 27 | + |
| 28 | +```ts |
| 29 | +import { objects } from "@beyond.dev/objects"; |
| 30 | + |
| 31 | +await objects.put("photos/cat.jpg", file, { contentType: "image/jpeg" }); |
| 32 | +const stream = await objects.get("photos/cat.jpg"); |
| 33 | +await objects.delete("photos/cat.jpg"); |
| 34 | +await objects.close(); |
| 35 | +``` |
| 36 | + |
| 37 | +## Operations |
| 38 | + |
| 39 | +| Operation | HTTP | |
| 40 | +| ---------- | --------------------------------------------------- | |
| 41 | +| Upload | `PUT /v1/{bucket}/{key}` | |
| 42 | +| Download | `GET /v1/{bucket}/{key}` | |
| 43 | +| Metadata | `HEAD /v1/{bucket}/{key}` | |
| 44 | +| Delete | `DELETE /v1/{bucket}/{key}` | |
| 45 | +| Move | `PATCH /v1/{bucket}/{key}` + `{"key": "new/key"}` | |
| 46 | +| Copy | `POST /v1/{bucket}/{key}` + `{"source": "src/key"}` | |
| 47 | +| Set access | `PATCH /v1/{bucket}/{key}` + `{"access": "public"}` | |
| 48 | +| List | `GET /v1/{bucket}?prefix=photos/&limit=100&cursor=` | |
| 49 | + |
| 50 | +**Conditional writes** — `If-None-Match: *` rejects if the key exists; `If-Match: <etag>` rejects if the ETag doesn't match. Both are atomic. |
| 51 | + |
| 52 | +**Public objects** — upload with `X-Beyond-Access: public` (or `{ access: "public" }` in the SDK) to make a key readable without a token. |
| 53 | + |
| 54 | +**Byte ranges** — `GET` supports the `Range` header for partial downloads. |
| 55 | + |
| 56 | +## S3 Compatible |
| 57 | + |
| 58 | +Any S3 client works. Derive credentials from the root token — no separate credential store: |
| 59 | + |
| 60 | +```ts |
| 61 | +import { deriveS3Credentials } from "@beyond.dev/objects"; |
| 62 | + |
| 63 | +const { accessKeyId, secretAccessKey } = deriveS3Credentials({ |
| 64 | + rootToken: "secret", |
| 65 | + bucket: "uploads", |
| 66 | +}); |
| 67 | +``` |
| 68 | + |
| 69 | +```python |
| 70 | +import boto3 |
| 71 | +s3 = boto3.client( |
| 72 | + "s3", |
| 73 | + endpoint_url="http://localhost:9000", |
| 74 | + aws_access_key_id=access_key_id, |
| 75 | + aws_secret_access_key=secret_access_key, |
| 76 | + region_name="us-east-1", |
| 77 | +) |
| 78 | +s3.upload_file("photo.jpg", "uploads", "photos/cat.jpg") |
| 79 | +``` |
| 80 | + |
| 81 | +Supported: PutObject, GetObject, HeadObject, DeleteObject, CopyObject, ListObjectsV2, CreateMultipartUpload, UploadPart, CompleteMultipartUpload, AbortMultipartUpload, ListMultipartUploads, ListParts, CreateBucket, DeleteBucket, HeadBucket, ListBuckets. |
| 82 | + |
| 83 | +## Authentication |
| 84 | + |
| 85 | +The root token authenticates all buckets. Bucket-scoped tokens are derived with HMAC-SHA256 and require no database lookup — the server recomputes them on each request: |
| 86 | + |
| 87 | +```ts |
| 88 | +import { deriveToken } from "@beyond.dev/objects"; |
| 89 | + |
| 90 | +const uploadsBucketToken = deriveToken("secret", "uploads"); |
| 91 | +// Only valid for the "uploads" bucket — safe to share with clients |
| 92 | +``` |
| 93 | + |
| 94 | +## Upload Tokens |
| 95 | + |
| 96 | +Issue short-lived, key-scoped tokens for direct browser uploads without exposing the root or bucket token: |
| 97 | + |
| 98 | +```ts |
| 99 | +const { token, expiresAt } = await objects.createUploadToken("photos/cat.jpg", { |
| 100 | + ttlSecs: 300, |
| 101 | +}); |
| 102 | +``` |
| 103 | + |
| 104 | +Pass the token to the browser. The browser uploads directly to the server: |
| 105 | + |
| 106 | +```ts |
| 107 | +import { createObjectsClient } from "@beyond.dev/objects"; |
| 108 | + |
| 109 | +const client = createObjectsClient({ |
| 110 | + url: "http://localhost:9000", |
| 111 | + token: uploadToken, |
| 112 | + bucket: "uploads", |
| 113 | +}); |
| 114 | +await client.put("photos/cat.jpg", file, { contentType: "image/jpeg" }); |
| 115 | +``` |
| 116 | + |
| 117 | +**React:** |
| 118 | + |
| 119 | +```ts |
| 120 | +import { useUpload } from "@beyond.dev/objects/react"; |
| 121 | + |
| 122 | +const { upload, progress, error } = useUpload({ |
| 123 | + token: uploadToken, |
| 124 | + bucket: "uploads", |
| 125 | +}); |
| 126 | +await upload("photos/cat.jpg", file); |
| 127 | +``` |
| 128 | + |
| 129 | +## TypeScript SDK |
| 130 | + |
| 131 | +**Next.js** — reads `BEYOND_OBJECTS_URL`, `BEYOND_OBJECTS_ROOT_TOKEN`, and `BEYOND_OBJECTS_BUCKET` from the environment: |
| 132 | + |
| 133 | +```ts |
| 134 | +import { objects } from "@beyond.dev/objects"; |
| 135 | + |
| 136 | +export async function uploadAction(data: FormData) { |
| 137 | + "use server"; |
| 138 | + const file = data.get("file") as File; |
| 139 | + await objects.put(`uploads/${file.name}`, file, { contentType: file.type }); |
| 140 | +} |
| 141 | +``` |
| 142 | + |
| 143 | +**Listing with pagination:** |
| 144 | + |
| 145 | +```ts |
| 146 | +let cursor: string | undefined; |
| 147 | +do { |
| 148 | + const result = await objects.list({ prefix: "photos/", limit: 100, cursor }); |
| 149 | + for (const obj of result.objects) { /* ... */ } |
| 150 | + cursor = result.nextCursor; |
| 151 | +} while (cursor !== undefined); |
| 152 | +``` |
| 153 | + |
| 154 | +**mTLS** (Node/Bun/Deno): |
| 155 | + |
| 156 | +```ts |
| 157 | +const objects = createObjectsClient({ |
| 158 | + url: "https://objects.internal", |
| 159 | + token: "secret", |
| 160 | + tls: { ca: caPem, cert: certPem, key: keyPem }, |
| 161 | +}); |
| 162 | +``` |
| 163 | + |
| 164 | +## Buckets |
| 165 | + |
| 166 | +Buckets are directories on disk. Manage them with the root token: |
| 167 | + |
| 168 | +```sh |
| 169 | +curl -X POST http://localhost:9000/v1/buckets \ |
| 170 | + -H "Authorization: Bearer secret" \ |
| 171 | + -d '{"name": "uploads"}' |
| 172 | + |
| 173 | +curl -X PATCH http://localhost:9000/v1/buckets/uploads \ |
| 174 | + -H "Authorization: Bearer secret" \ |
| 175 | + -d '{"access": "public"}' |
| 176 | +``` |
| 177 | + |
| 178 | +Or via the SDK: |
| 179 | + |
| 180 | +```ts |
| 181 | +await objects.buckets.create("uploads", { access: "private" }); |
| 182 | +await objects.buckets.update("uploads", { access: "public" }); |
| 183 | +const list = await objects.buckets.list(); |
| 184 | +await objects.buckets.delete("uploads"); |
| 185 | +``` |
| 186 | + |
| 187 | +## Configuration |
| 188 | + |
| 189 | +| Env var | Default | Description | |
| 190 | +| ----------------------- | ----------------------- | --------------------------------------------------------------------------------- | |
| 191 | +| `OBJECTS_ROOT_TOKEN` | — | Root auth token; HMAC key for derived tokens (required) | |
| 192 | +| `OBJECTS_DATA_DIR` | `/data` | Root directory for buckets, temp files, and multipart state | |
| 193 | +| `OBJECTS_INDEX_DIR` | `/data/.index` | LSM-tree index directory for prefix listing | |
| 194 | +| `ADDRESS` | `0.0.0.0:9000` | Bind address | |
| 195 | +| `OBJECTS_URL` | — | Public base URL included in `url` fields on responses | |
| 196 | +| `SYNC_LINGER_MS` | `5` | fdatasync batching window — concurrent uploads within this window share one flush | |
| 197 | +| `DRAIN_TIMEOUT_SECS` | `30` | Grace period for in-flight requests during shutdown | |
| 198 | +| `GC_TEMP_TTL_SECS` | `3600` | Minimum age for orphan temp files eligible for garbage collection | |
| 199 | +| `GC_MULTIPART_TTL_SECS` | `86400` | Minimum age for abandoned multipart uploads eligible for garbage collection | |
| 200 | +| `BEYOND_TLS_CERT` | — | PEM-encoded TLS certificate | |
| 201 | +| `BEYOND_TLS_KEY` | — | PEM-encoded TLS private key | |
| 202 | +| `BEYOND_TLS_CA` | — | PEM-encoded CA cert; when set, mutual TLS is required on all connections | |
| 203 | +| `LOG_LEVEL` | `info` | Log verbosity | |
| 204 | +| `OTLP_ENABLED` | `false` | Export traces to an OTLP collector | |
| 205 | +| `OTLP_ENDPOINT` | `http://localhost:4317` | OTLP collector gRPC address | |
| 206 | +| `OTLP_SAMPLE_RATE` | `0.1` | Fraction of traces sampled (0.0–1.0) | |
| 207 | + |
| 208 | +Set `ENVIRONMENT=development` for human-readable logs. |
| 209 | + |
| 210 | +## Health |
| 211 | + |
| 212 | +| Path | Description | |
| 213 | +| ---------- | -------------------------------------------------------- | |
| 214 | +| `/livez` | Liveness — returns 200 when the process is up | |
| 215 | +| `/readyz` | Readiness — returns 200 when the index is open and ready | |
| 216 | +| `/metrics` | Prometheus metrics scrape endpoint | |
| 217 | + |
| 218 | +## Development |
| 219 | + |
| 220 | +```sh |
| 221 | +mise run format # format all source files |
| 222 | +mise run test # integration tests |
| 223 | +mise run bench # throughput benchmarks |
| 224 | +``` |
| 225 | + |
| 226 | +See [ARCHITECTURE.md](ARCHITECTURE.md) for on-disk layout, the atomic write path, HMAC derivation, S3 compatibility layer, and index design. |
| 227 | + |
| 228 | +## License |
| 229 | + |
| 230 | +AGPLv3. Self-host for any purpose, including commercial use. If you offer this software as a service, you must release your modifications under AGPLv3. |
0 commit comments