A1 API Gateway — Installation and Operations

1. What this is, and what it commits you to
2. Requirements
3. Getting it running
4. Configuration and secrets
5. Domains and TLS
6. The first organisation and the first operator
7. Backups and restore
8. Upgrading
9. Security checklist
10. Troubleshooting

1. What this is, and what it commits you to

A1 API Gateway is a multi-tenant gateway that sits between the projects in your organisation — or your customers' organisations — and the AI vendors: Anthropic, OpenAI and xAI for chat, OpenAI for images and embeddings, ElevenLabs, AssemblyAI and OpenAI for speech and transcription, SendGrid for mail. Callers hold a gateway key that starts with sk-gw-. The gateway authenticates it, checks the caller's address, enforces rate limits and spending caps before the vendor is called, forwards the request with the real vendor key, meters the exact cost and writes it to that organisation's ledger.

You have bought the whole source and every future update, so you run it. That is two Node processes, one Redis and one reverse proxy.

Both processes are started by scripts/start.mjs, which reads APP: unset or all runs both, gateway or server runs one. If either child exits, the other is stopped and the container exits, so a partial failure is visible to the supervisor rather than silent. The background workers — rollups, credit sync, alerts, reconciliation, retention, snapshots, integrity — run inside the server process, and are switched off with RUN_WORKERS=0 (used when you start a second copy against a data directory for a release gate, §8.2).

1.1 Three things you are taking on

One box. A single node holds every organisation's file. Multi-node sharding is specified and not built (spec §19, Phase 3). The honest availability story is that of one machine and its backups, and it should be a stated one rather than an assumed one: if the box is down, every organisation is down. A corrupt or locked file for one organisation, on the other hand, affects only that organisation (§10).

Redis fails closed, by design. Budgets, the tenant ceiling, per-vendor-key caps, prepaid credit and the rate limits are all reserved atomically in Redis before a request reaches a vendor. If Redis is unreachable the gateway cannot know whether a request is within its caps, so it refuses with 503 gateway_unavailable rather than spending money it cannot account for. This is not a bug to work around. It means Redis is a hard dependency of serving traffic, and it needs the same care as the database.

2. Requirements

Vendor keys are not a requirement of the installation. They are added afterwards, in the console, either by you (platform keys, assigned to organisations, billed to you and paid for with prepaid credit) or by each organisation (its own keys, billed to it by the vendor). A freshly installed gateway with no vendor keys starts and serves its consoles; it simply answers 402 to requests until an organisation has a key or credit.

3. Getting it running

3.1 What is in deploy/

3.2 The steps

cd deploy
cp .env.example .env

# A master key you can never lose. Put a copy somewhere the backups are not.
openssl rand -base64 32                      # paste into GATEWAY_MASTER_KEY in .env

# Edit .env: BASE_DOMAIN, ACME_EMAIL, CLOUDFLARE_API_TOKEN (or your DNS provider's),
#            REGISTRATION_MODE, SENDGRID_API_KEY, EMAIL_FROM, BACKUP_TARGET_URL and its credentials.
chmod 600 .env

docker compose up -d --build                 # the build context is the repository root
docker compose logs -f gateway               # both processes announce the address they listen on

curl -fsS https://api.$BASE_DOMAIN/healthz   # {"ok":true,"mode":"full"}
curl -fsS https://api.$BASE_DOMAIN/readyz
# {"ready":true,"redis":"ok","registry":"ok","tenant_dir":"ok","platform_keys":"not_required"}

/healthz says the process is up. /readyz is the one that matters: it pings Redis, runs a statement against the registry and checks it is writable, checks the tenants directory is writable, and checks there is at least one healthy platform key or no prepaid organisations. It answers 503 when any of those fails, so it is what a load balancer or an uptime check should watch. It is served by the consumer API on 8080; the console server answers /healthz only, and the container's own health check uses that.

3.3 Running it without Docker

Nothing in the product needs a container. The two processes are apps/gateway/src/main.ts and apps/server/src/main.ts, run with tsx; scripts/start.mjs is a twelve-line supervisor that spawns them. A host installation is therefore: a Node 26 runtime, pnpm install --frozen-lockfile (not --prod, because tsx is a development dependency), pnpm --filter @aigw/admin build for the console bundle, two service units with the environment file loaded at 0600 and root-owned, a Redis instance, and your own reverse proxy in front. deploy/aiaw-ec2/ is a worked example of the proxy and certificate half of that.

4. Configuration and secrets

Configuration is read once at boot by loadConfig in packages/core/src/config.ts. That file is the authority: if a name or a default is not in it, it does not exist. Every value is validated at boot and a bad one throws a ConfigError naming the variable, so a misconfiguration is a failed start with a readable message, not a surprise at request time.

4.1 The three with no safe default

4.2 Everything else

Three more are read directly by the processes rather than through loadConfig: APP (all, gateway or server, in scripts/start.mjs), RUN_WORKERS (anything other than 0 runs the workers, in apps/server/src/main.ts), and ADMIN_DIST_DIR (where the built console bundle is, if it is not beside the server). Each vendor also takes an optional base URL — ANTHROPIC_BASE_URL, OPENAI_BASE_URL, XAI_BASE_URL, ELEVENLABS_BASE_URL, ASSEMBLYAI_BASE_URL, SENDGRID_BASE_URL, and ANTHROPIC_ADMIN_BASE_URL / OPENAI_ADMIN_BASE_URL for reconciliation — which exist so the test suites and the development stack can point at the in-repository mock vendor. An empty value means the vendor's own host. In production, leave them unset.

4.3 Where the secrets live

The compose profile keeps every secret in one file, deploy/.env, loaded into the container by env_file. Own that file: chmod 600, owned by the user that runs the stack, never committed, never copied into a build context. It holds the master key, the DNS token, the SendGrid key and the backup credentials. If you install on a host instead, the same applies to the systemd environment file, and its directory should be 0700.

Vendor key secrets themselves are never in the environment. They are stored as AES-256-GCM ciphertext — platform keys in the registry, an organisation's own keys inside that organisation's file — decrypted only for the upstream call, never returned by any API, never written to a log. All any response ever shows is a four-character hint. Several tests exist solely to keep that true, so do not add convenience that defeats it.

5. Domains and TLS

5.1 The four host shapes

The console listener picks the surface from the Host header: platform. requires the platform-admin flag, the apex is public, any other single label is looked up as an organisation slug. An unknown slug gets a generic 404 that does not reveal whether it ever existed. One login on the apex sets a session cookie on .{BASE_DOMAIN} and therefore works on every subdomain; membership of the organisation named by Host is re-checked on every request regardless, so the shared cookie does not widen anything.

5.2 The wildcard

You need one certificate covering the apex and *.{BASE_DOMAIN}, because a new organisation gets a subdomain the moment it registers and must not have to wait for certificate work. That means the ACME DNS-01 challenge, which means a Caddy binary built with your DNS provider's module. The compose file uses ghcr.io/caddybuilds/caddy-cloudflare:2 and passes CLOUDFLARE_API_TOKEN through to the tls block; for another provider, swap the image and the directive together. Scope the token to that one zone and nothing else.

The alternative is in the Caddyfile as comments: stock Caddy with tls { on_demand }, issuing one certificate per subdomain on first visit, gated by an ask endpoint that answers only for slugs that exist. It avoids the DNS module at the cost of a first-visit delay and one certificate per organisation. The wildcard is the recommendation.

5.3 Why TRUSTED_PROXIES is not a detail

X-Forwarded-For is written by whoever sends the request. The gateway's address allowlists, its per-address rate limits, the automatic suspension of a key seen from too many new addresses, and every line in the refusal log are all keyed on the address the gateway decides the caller has. Get this wrong in either direction and something breaks quietly:

• Trust too much — an empty list interpreted as "believe everyone", or a range wider than your proxy — and any caller can put any address in the header and walk straight through an allowlist that looks correct in the console.
• Trust too little, and every request appears to come from the proxy itself. Per-address rules collapse onto one address, the refusal log becomes useless, and auto-suspension either never fires or fires on everything.

The rule the code follows (spec §8.3): if the socket peer is not in TRUSTED_PROXIES, that peer is the client and the header is ignored entirely. Otherwise walk the forwarded chain from the right and take the first address that is not itself a trusted proxy. So TRUSTED_PROXIES should be exactly the range your proxy speaks from and nothing more. The compose file sets 172.16.0.0/12,10.0.0.0/8, which is the Docker bridge network Caddy sits on. If Caddy itself is behind Cloudflare or a cloud load balancer, add that range too or name the provider's own header in CLIENT_IP_HEADER — otherwise every caller looks like the load balancer.

6. The first organisation and the first operator

6.1 Registration modes

No mode grants anything. Registration gives no trial, no allowance and no credit, so an open gateway has no cost exposure from fake sign-ups: a new organisation can do nothing until it adds its own vendor key or you record prepaid credit for it. The question REGISTRATION_MODE answers is only who may create an organisation on your instance.

6.2 What a sign-up needs from you

The flow is: the form on the apex (organisation name, slug, name, email, password), then a single-use verification token by email that expires in 24 hours, then mandatory TOTP enrolment on first login with ten recovery codes shown once, then provisioning — the organisation's database file is created from the current schema and the owner membership written, in one transaction.

Every step but the first depends on email. Without SENDGRID_API_KEY the mailer only logs, and the boot line says so plainly: platform mail: not sent, only logged. The verification link is then in your logs, which is workable for your own first account and unworkable for anyone else. Set the key and a verified EMAIL_FROM before you invite a single person.

6.3 Making the first platform admin

Platform administration is a flag on a user in the registry, users.is_platform_admin. It is never granted through a role inside an organisation, on purpose: an organisation's owner must not be able to reach the platform console by promoting themselves.

Once one platform admin exists, the rest are made from the platform console or its API — POST /platform/v1/users with an email, which promotes an existing account or creates one and sends it a password-reset link, and DELETE /platform/v1/users/:id, which refuses to remove the last administrator. Both write to the platform audit log.

# With the sqlite3 command line, if you have it:
sqlite3 /data/registry.db "update users set is_platform_admin = 1 where email = 'you@example.com';"

# The runtime image does not carry sqlite3. Node 26 has SQLite built in, so this needs nothing extra:
node --input-type=module -e "
  import { DatabaseSync } from 'node:sqlite';
  const db = new DatabaseSync('/data/registry.db');
  console.log(db.prepare('update users set is_platform_admin = 1 where email = ?').run('you@example.com'));
"

6.4 DEV_SKIP_TOTP, and why it refuses to work

DEV_SKIP_TOTP turns off the mandatory second factor so a local stack can be signed into with a password alone. It checks BASE_DOMAIN against a list of local suffixes — localhost, test, local, localdomain, and the loopback addresses — and on anything else it throws at boot:

DEV_SKIP_TOTP is a local development switch and refuses to run on "gateway.example.com".
Two-factor authentication is mandatory (spec §6.2); unset DEV_SKIP_TOTP.

The reason it fails loudly rather than being ignored is that the quiet version of this mistake is a production console with no second factor and nothing to show for it. A variable copied from a development environment file into a production one should stop the deployment, not weaken it.

7. Backups and restore

7.1 What the system does on its own

• Litestream replicates /data/registry.db continuously to ${BACKUP_TARGET_URL}/registry, one-second sync, 720 hours of retention. The registry holds organisations, users, platform vendor keys, credit balances and the pricing catalogue.
• The snapshot worker runs nightly at 03:00 UTC. It flushes any open writer queue, takes a VACUUM INTO copy of the registry and of every organisation file into ${DATA_DIR}/snapshots/, and runs integrity_check on each copy. A snapshot that fails the check, or fails at all, fires storage.integrity to the operator alert channels.
• The integrity worker runs weekly, Sunday at 04:00 UTC, over every file.
• Retention and housekeeping run nightly in the same 03:00 pass: WAL checkpoints, chunked deletion on the retention windows, incremental vacuum.

Every one of these is a named job you can inspect and force. GET /platform/v1/workers lists each job with its schedule, whether it is running now, when it last started and finished, its run count and its last error; POST /platform/v1/workers/{name}/run runs one immediately and is refused if it is already running. The names are rollup, alerts, rebuild, reconciliation, housekeeping (retention, snapshots and deletion together), integrity and limits-summary. Use them before a planned restore rather than waiting for 03:00.

7.2 What it does not do, and you must

Also yours: the master key. It is not in the data directory and no backup of the data directory contains it. Keep it somewhere else, and make sure whoever would perform a restore at three in the morning knows where that is.

7.3 The restore drill

A backup nobody has restored from is not a backup, and neither path above has ever been exercised in the direction that counts (spec §18.4). Restoring is the only test that means anything, and it is a go-live requirement, not an improvement to schedule later.

1. Have at least two organisations on the running system, each with real ledger rows and its own vendor key.
2. Restore into a clean, empty directory — not over the live one.

   litestream restore -o /restore/registry.db ${BACKUP_TARGET_URL}/registry
   cp /data/snapshots/<tenant-id>.db /restore/tenants/<tenant-id>.db    # per organisation

3. Check that tenants.db_path in the restored registry points at where the files now are. Those paths are absolute, so a restore into a different directory needs them rewritten.
4. Start the stack against the restored directory with the same GATEWAY_MASTER_KEY.
5. Rebuild the Redis counters. Redis itself needs no backup — everything in it is derived from the ledgers and the registry — but a restored database and a stale or empty Redis disagree until the rebuild has run. It runs at startup and hourly on its own; to force it, POST /platform/v1/workers/rebuild/run. It recomputes each organisation's credit from top-ups minus the prepaid ledger, each key budget's spend for its current period, the monthly ceiling spend and each platform key's cap counter. In-flight holds are dropped, by design.
6. Prove three things, in this order. Both organisations' keys still authenticate. Their balances and ledgers are intact and agree with what you recorded before. And — the one that is usually skipped — their vendor keys still decrypt: press Test on a vendor key in each organisation's console, which makes one tiny real call with the decrypted secret. That is what proves the master key was backed up separately rather than merely assumed.
7. While you are there, confirm isolation: read one organisation's console and find nothing of the other's.

Restoring a single organisation, as opposed to everything, is copying one file back and restarting nothing.

8. Upgrading

An update arrives as source. You rebuild and restart, and the data is migrated. The whole risk of an upgrade is in that last clause, so this section is mostly about it.

8.1 How migrations run

• Versioned. Each migration has a number and its own transaction. The current version is stored in the file: platform_settings.schema_version for the registry, meta.schema_version for an organisation. Applying means running every migration newer than the stored number, in order.
• On open. The registry migrates when it is opened at boot. An organisation file migrates when it is opened, so a file belonging to an organisation nobody touches is migrated the first time somebody does. Opening a file with a version newer than the code expects is an error, not a downgrade.
• Forward-only. There are no down steps and there is no rollback of a migration. Rolling back to the previous release leaves the new schema in place, which is survivable only because each release is checked against the previous code before it ships (§8.2). Getting the data back means restoring a snapshot and losing everything written since.
• Replayable. SQLite can alter neither a CHECK constraint nor a unique key, so the migrations that change one rebuild the table, copy the rows and rename. Each of those first tests whether the change is already in place, so running it twice is harmless.
• Current versions are registry v8 and organisation-file v11.

8.2 The release procedure

This is the procedure the project uses for its own deployments, and it is the one to copy. It was last performed on 19 September 2026 for 0.1.0-alpha.6, the release that took the data to registry v8 and organisation-file v11, and the steps below are what was actually done. Its principle: nothing touches live data until the same migration has been run end to end on a copy of that data and checked, and until the previous release has been shown to survive the result.

1. Snapshot. Take a fresh copy of the data directory — VACUUM INTO per file, with integrity_check and foreign_key_check on each copy — into a timestamped folder. This is both the gate input and the rollback.
2. Make the gate copy. Copy that snapshot into a scratch directory, then point the copied registry's tenants.db_path values at the copied organisation files. They are absolute paths, so without this step the migration would run against production.
3. Dry run. Read-only, reports what would change per organisation, exits non-zero on the first error naming the file and the step.
4. Migrate the copy, then check it against the snapshot (§8.3).
5. Run the new release against the migrated copy on spare ports with RUN_WORKERS=0, and click through it. Check host-routed pages with curl and an explicit Host header; a 401 from the platform console without a session is correct, not a fault. If the gate shares Redis with production, give it a different database index so its refusals do not land in the live logs.
6. Prove the rollback both ways. Start the previous release against the migrated copy, and open the migrated files with the previous release's own database code. Forward-only migrations mean this is the only evidence you will get that a rollback is survivable.
7. Switch. Stop the services, take a second snapshot, migrate the live directory, run the check against that second snapshot, and only start the new release if it passes. Then verify health and traffic, with the rollback script already written.

8.3 The v11 release, as a worked example

The most recent data-model change gave every vendor key its own model list, environment and monthly cap, converted every key pinned to a single address into an ordinary allow rule for exactly that address, and added the refusal log. It shipped with two scripts, which are the model for the ones a future release will bring.

# 1. Dry run against the gate copy: reads only, writes nothing.
npx tsx scripts/migrate-v11.mts --data-dir /srv/gate/data --dry-run

# 2. Migrate the gate copy for real.
npx tsx scripts/migrate-v11.mts --data-dir /srv/gate/data

# 3. Compare the migrated copy with the snapshot it came from.
npx tsx scripts/migrate-check-v11.mts --before /srv/snapshots/20260919T2227Z --after /srv/gate/data

migrate-v11.mts migrates the registry and every organisation file in one pass, with the services stopped, so that nothing migrates in the middle of live traffic — opening a file still migrates it too, as a safety net. It prints a JSON report per organisation (vendor keys, virtual keys, the model-access selection, pinned keys converted, keys still waiting to pin) and a one-line summary on stderr. It stops at the first error and names the file and the step. Exit 0 means success.

migrate-check-v11.mts is the part that makes the gate worth running. It fails the release, exit 1, on any difference that is not a planned one:

• Row counts of every table in the registry and in every organisation file. Only key_ip_rules may grow, by exactly the number of converted pins, and audit_log by the single row the model-access copy writes. New tables are allowed only if they are the ones this migration adds.
• Every vendor key secret and every virtual key hash, compared byte for byte. A migration that re-encrypted or re-hashed anything would be caught here.
• PRAGMA integrity_check and PRAGMA foreign_key_check on every migrated file.
• For every virtual key: the same answer, allowed or not, for every model in the catalogue; and the same answer, allowed or refused, for every address in that key's address log. This is the check that matters, because it tests behaviour rather than schema. A migration that quietly widened or narrowed what a key can do fails here even when every table looks right.

A PASS line names the counts it compared. A FAIL lists every difference it found. Only tenants.db_path is permitted to differ, because the gate copy rewrites it.

9. Security checklist

10. Troubleshooting

For anything that leaves a trace, the places to look are: the structured request log (one JSON line per request, carrying the request id the caller was given, and never a body or a credential), the organisation's own Refusals page for anything refused before a vendor was called, the platform console's reconciliation page for a disagreement between the gateway's figures and a vendor's, and /metrics on the consumer API. Five series there answer most questions: