Runbook — Garage object store bootstrap (Coolify)¶

How to stand up the self-hosted Garage S3 store that backs the production S3FileStorageAdapter (STORAGE_BACKEND=s3), create the per-purpose buckets and application key, wire the staging/prod env, and rotate the key.

Context: see issue #178. The Node app authorizes at upload time and hands the client a pre-signed GET URL that resolves directly against Garage — the app stops touching the bytes after upload.

Topology¶

Two hostnames front the single Garage container (both terminated by Traefik, certs auto-issued by Coolify):

Hostname	Garage port	Purpose
`s3.beta.ikloze.com`	`3900`	S3 API — presigned URLs point here; the app's `S3_ENDPOINT` / `PUBLIC_STORAGE_URL`
`storage.beta.ikloze.com`	`3902`	Garage web endpoint (public object hosting) — not used by the adapter

SigV4 constraint (do not break this). Presigned URLs are signed over the host + exact path. The reverse proxy for s3.beta.ikloze.com must forward the path byte-for-byte to Garage — no prefix added, none stripped, no normalization. With path-style addressing the first path segment is the bucket (/ikloze/kyc/<key> → bucket ikloze, object kyc/<key>). Any path rewrite produces SignatureDoesNotMatch. This is why we use a dedicated S3 subdomain rather than proxying an /_storage sub-path on the API host.

1. Deploy the container (Coolify)¶

Image: pin a version, e.g. dxflrs/garage:v1.0.1 (never latest).
Persistent volumes:
/var/lib/garage/meta
/var/lib/garage/data
Config file /etc/garage.toml (mount or bake):

metadata_dir = "/var/lib/garage/meta"
data_dir = "/var/lib/garage/data"
db_engine = "lmdb"

replication_factor = 1

rpc_bind_addr = "[::]:3901"
rpc_public_addr = "127.0.0.1:3901"
rpc_secret = "<openssl rand -hex 32>"

[s3_api]
s3_region = "garage"            # MUST equal the app's S3_REGION
api_bind_addr = "[::]:3900"
root_domain = ".s3.beta.ikloze.com"

[s3_web]
bind_addr = "[::]:3902"
root_domain = ".storage.beta.ikloze.com"
index = "index.html"

Traefik labels: route s3.beta.ikloze.com → container :3900, storage.beta.ikloze.com → container :3902. Pass-through only; no path middleware on the S3 route.

exec into the container for the steps below: garage talks to the local node over RPC.

2. Cluster layout (single node)¶

# Get this node's id
garage node id -q          # → <NODE_ID>@127.0.0.1:3901

# Assign the whole capacity to the single node, in zone "dc1"
garage layout assign -z dc1 -c 10G <NODE_ID>
garage layout apply --version 1

garage status               # node should be UP and have a role

3. Create the bucket¶

We use a single bucket (ikloze) for all objects. The logical purpose the app passes (e.g. kyc) becomes a key prefix inside that bucket (kyc/<...>), so new purposes need no new bucket — just a new prefix, created implicitly on first upload. The adapter never auto-creates the bucket — provision it once here.

garage bucket create ikloze

4. Create the application key and grant access¶

garage key create ikloze-app
# → prints Key ID (GK...) and Secret key. Capture BOTH now; the secret is shown only once.

garage bucket allow --read --write --owner ikloze --key ikloze-app

5. Wire the Coolify env (staging/prod backend service)¶

Set on the ikloze-backend service:

STORAGE_BACKEND=s3
S3_ENDPOINT=http://garage:3900            # internal Docker-network endpoint (server-to-server)
S3_REGION=garage                          # MUST equal s3_region in garage.toml
S3_ACCESS_KEY_ID=<Key ID from step 4>
S3_SECRET_ACCESS_KEY=<Secret from step 4>
S3_FORCE_PATH_STYLE=true
PUBLIC_STORAGE_URL=https://s3.beta.ikloze.com   # host baked into presigned GET URLs
STORAGE_BUCKET=ikloze                     # single bucket holding all objects (default: ikloze)

S3_ENDPOINT is the fast internal path used by upload / delete / getStream. PUBLIC_STORAGE_URL is used only to pre-sign GET URLs handed to the browser. If unset it falls back to S3_ENDPOINT (fine for single-host dev, wrong for prod).

Redeploy the backend so the STORAGE_BACKEND=s3 factory picks up S3FileStorageAdapter.

6. Verify end-to-end¶

Opt-in contract test from a host that can reach S3_ENDPOINT and PUBLIC_STORAGE_URL:

STORAGE_LIVE_TESTS=1 \
S3_ENDPOINT=https://s3.beta.ikloze.com \
PUBLIC_STORAGE_URL=https://s3.beta.ikloze.com \
S3_REGION=garage S3_ACCESS_KEY_ID=... S3_SECRET_ACCESS_KEY=... \
STORAGE_BUCKET=ikloze \
pnpm jest tests/adapters/storage/s3.adapter.live.spec.ts

It uploads a small object, fetches it via the presigned URL (expects the bytes back), deletes it, and confirms a second fetch 404s.

Full KYC flow on staging: submit KYC (PUT via API → bytes land in Garage) → GET /kyc/me → the returned signed URLs resolve against https://s3.beta.ikloze.com and serve the file for 5 minutes.

7. Key rotation¶

Zero-downtime rotation (Garage allows multiple keys per bucket):

# 1. Create the replacement key
garage key create ikloze-app-2
garage bucket allow --read --write --owner ikloze --key ikloze-app-2

# 2. Update Coolify env S3_ACCESS_KEY_ID / S3_SECRET_ACCESS_KEY to the new key, redeploy backend.

# 3. Confirm traffic works (step 6), then revoke the old key:
garage bucket deny --read --write --owner ikloze --key ikloze-app
garage key delete ikloze-app

Presigned URLs already issued under the old key keep working only until their TTL (≤5 min) lapses, so revoke after a short drain.