Deployment & server ops

How creddit runs in production, how code ships to it, and how to operate the box by hand. The app is a Next.js App Router site backed by a local PostgreSQL database; data is produced by off-chain cron refreshers (see Data Pipeline) and the app reads only.

Convention reminder: every trailing APY across every venue is annualizeRatio in src/lib/data/apy.ts (the realised ratio of an on-chain compounding index between two blocks, annualised by actual elapsed time). Never average per-snapshot annualised rates. This is what lets /carries compare venues like-for-like, and it is why a refresher is just an upsert job, not business logic the app re-derives.

1. Server topology

Single Hetzner box, shared with the DEX HQ stack. One host runs everything.

Item	Value
Host	dexhq.io (Hetzner)
SSH	ssh root@dexhq.io (key ~/.ssh/hetzner_ed25519)
Repo	/opt/onchain-credit — git checkout tracking origin/main
App process	pm2 onchain-credit on localhost:3001
Edge	nginx reverse-proxy → https://creddit.xyz, with Cloudflare in front
Database	local PostgreSQL; prod DB creddit, role onchain_credit, schema onchain_credit
Cron wrapper	scripts/run-cron.sh <script>
Daily backup	scripts/ops/backup-creddit.sh at 02:15 UTC
Log rotation	/etc/logrotate.d/onchain-credit (daily, 14 days) — see Log rotation

pm2 processes co-resident on the box:

pm2 process	Port	What it is
onchain-credit	3001	this app (prod)
onchain-credit-staging	3002	staging copy of this app (see §3)
rindexer	—	shared indexer
dexhq	—	DEX HQ app
creddit-indexer	—	shared indexer
creddit-event-ingester	—	always-on event-ledger ingester (portfolio 100k scale) — a manual add, see below

nginx terminates the public hostname and proxies to localhost:3001; Cloudflare sits in front of nginx for TLS and CDN. The app binds loopback only — it is never reachable except through nginx.

scripts/ops/ (backup, reseed, migrate, PII scrub, nginx template) is in the repo and ships via the normal deploy. The server crontab and the box-local secrets (.env.local, /root/.staging-basic-auth) live only on the box.

2. How code ships (promote-to-prod)

Two long-lived branches, each bound to one environment. You develop on staging and promote to prod:

feature/* ──PR──▶ staging ──auto-deploy──▶ staging.creddit.xyz   (develop + validate)
                     │
                     └──"release" PR──▶ main ──auto-deploy──▶ creddit.xyz   (promote)

• Feature work targets staging. Merge a feature PR into staging → deploy-staging.yml rebuilds /opt/onchain-credit-staging on :3002 and it appears on staging.creddit.xyz (basic-auth).
• Validate on staging.creddit.xyz.
• Promote with one staging → main release PR. Its diff is the release. The PR bumps version in package.json. Merge → deploy.yml ships prod, and release.yml tags v<version> + cuts a GitHub Release (see §2.3).
• main and staging are in sync after each release.

Design + rationale (same-box isolation, reseed model, tradeoffs): docs/plans/staging-environment-plan.md.

Branch protection is advisory, not enforced. This is a free-plan private repo, so GitHub cannot enforce PR-only main or required checks. ci.yml (tsc + tests on every PR into main/staging) and the scripts/ops/git-hooks/pre-push guard report but cannot block. Honor a red CI run; don't push to main directly. To make it enforced, upgrade to GitHub Pro (~$4/mo) and add a branch rule on main: require a PR, require the CI check green, and (optionally) restrict who can merge. Nothing else changes.

Shipping a change, step by step

The concrete loop, from an idea to it live on creddit.xyz:

1. Branch off staging (never main): git fetch && git switch -c feature/x origin/staging.
2. Do the work, then locally npx tsc --noEmit && npm test. (npm test now runs npm run typecheck:scripts first — tsconfig.json EXCLUDES scripts/, and the tests there run through tsx, which strips types without checking them, so every refresher and sync CLI was unchecked until tsconfig.scripts.json was added. That blind spot shipped a dead status guard that gone-marked the whole Morpho market universe; the gate would have failed it.) If you touched a page/metric/schema/refresher, update the matching docs/ page in the same change (the docs-in-the-PR rule).
3. Open a PR into staging. ci.yml runs tsc + tests. Merge it → deploy-staging.yml rebuilds staging and it appears on https://staging.creddit.xyz (basic-auth).
4. Validate on staging. Load the changed page; if the change needs data, run the refresher/backfill once on the box against creddit_staging (cd /opt/onchain-credit-staging && scripts/run-cron.sh <script>.ts).
5. Promote. Bump version in package.json on staging (minor for a feature, patch for a fix), then open a staging → main PR and merge it with a merge commit (not squash — that keeps staging an ancestor of main). This is the release; get explicit sign-off before merging, since it ships prod.
6. Merge → prod ships. deploy.yml deploys creddit.xyz; release.yml tags v<version> and cuts a GitHub Release. Then fast-forward staging back up to main so they match: git push origin origin/main:staging.
7. Post-deploy, if the change touched data. Apply prod migrations by hand (scripts/ops/migrate.sh against creddit), run the refresher/backfill, then re-run the deploy so the ISR pages re-prerender against the new data (see the ISR note).

Hotfix shortcut: for an urgent prod-only fix you can branch off main, PR into main, and back-merge to staging after — but the default is always through staging.

2.1 Production deploy pipeline (.github/workflows/deploy.yml)

Triggers on every push to main (in practice, the merge of a release PR). Code only. It does not run DB migrations, backfills, or data refreshers; those are manual server steps after deploy (see below).

The GitHub runner only opens an SSH session. The server fetches the new code from this repo and rebuilds in place. Step by step:

1. SSH in. Runner writes DEPLOY_SSH_KEY (repo secret) to a temp key and pins the server host key (StrictHostKeyChecking=yes, hard-coded known_hosts line), then ssh root@dexhq.io. The whole deploy script is passed as a single SSH argument so it runs from memory — a git reset rewriting files on disk cannot disturb the running script.
2. Record rollback target. PREV=$(git rev-parse HEAD) in /opt/onchain-credit.
3. Fetch + reset. git fetch https://x-access-token:$GH_TOKEN@github.com/<repo>.git main (ephemeral repo-scoped GITHUB_TOKEN) then git reset --hard FETCH_HEAD. .env.local and node_modules are gitignored, so the reset preserves the server's secrets and installed deps.
4. Install + build. npm ci --no-audit --no-fund && PG_POOL_MAX=4 npm run build. The PG_POOL_MAX=4 caps the pg pool for the build only: next build prerenders the ISR pages across several worker processes, each of which opens its own pool (see postgres.ts), so the default max of 12 per worker can trip an intermittent too many connections for role build failure that then rolls the deploy back. Four keeps the build under the role's connection cap; the runtime pm2 process leaves the var unset, so serving keeps the full pool of 12. If a build still fails this way it is transient — gh run rerun --failed clears it, no code change needed.
5. Restart (success path). pm2 restart onchain-credit --update-env, then log the deployed short SHA + timestamp.
6. Rollback (failure path). If npm ci/npm run build fails: git reset --hard $PREV, reinstall, rebuild from the previous commit, and exit 1. A failed deploy never leaves broken code or a half-built .next staged for the next process restart.

Other guarantees:

• One at a time. concurrency: group: deploy-production, cancel-in-progress: false — a second push waits its turn rather than racing.
• Least privilege. Workflow permissions: contents: read; the deploy key only authorises SSH to the box; the GitHub token is ephemeral and repo-scoped.
• Host-key pinning is verified out-of-band against /etc/ssh/ssh_host_ed25519_key.pub. Update the known_hosts line in the workflow if the server is ever rebuilt, or every deploy will fail host-key verification.

Manual post-deploy steps (when a change needs more than code)

The pipeline restarts the app with new code and a fresh build, nothing else. If your change touches the database or data, do these on the box after the deploy lands:

• Schema migration → apply the new scripts/sql/NNN-*.sql as the postgres owner (see §4, Migrations).
• Backfill → run the relevant scripts/backfill-*.ts via run-cron.sh (one-off history seeds for newly added wrappers/markets; they reuse the same annualisation math as the live refreshers).
• Refresher → run the affected scripts/refresh-*.ts via run-cron.sh so the new rows exist immediately instead of waiting for the next cron tick.
• Carry registry → after a new yield adapter or DEX-pool snapshot lands, re-run scripts/sync-carries.ts then --approve <key> the newly-PROPOSED carries (replaces the retired sync-fluid-vaults.ts; see processes A.0).

Portfolio taxonomy: first deploy (migrations 049 / 050 / 052)

The taxonomy release ships five new cron jobs, three migrations and two exhaustive universe ingests. The ordering below is not cosmetic — each step is what makes the next one correct.

1. Migrate first: 049, 050, 052. Additive and idempotent. refresh-portfolio.ts degrades gracefully if 049/052 are missing (empty wallet universe, static erc4626 set) but the WS5 backfill refuses to run against a degraded registry rather than mis-reading a wallet-venue-only holder as empty and pruning its history — so until these are applied, backfills abort loudly. That is intentional.

   cd /opt/onchain-credit && scripts/ops/migrate.sh creddit    # BARE DB NAME

2. Deep-ingest the two universes, once, by hand, before adding their crons. The first run walks the whole history and takes far longer than a daily tick:

   MORPHO_UNIVERSE_FROM_BLOCK=<morpho-blue-deploy>   scripts/run-cron.sh refresh-morpho-universe.ts
   METAMORPHO_FACTORY_FROM_BLOCK=<v1.0-factory-deploy> scripts/run-cron.sh refresh-metamorpho-factory.ts

   Steady-state runs scan only new blocks from the cursor, so the env overrides are for the FIRST run only.
3. One-time full reconcile over every seeded wallet. Required, and easy to miss. Migration 050 seeds a discovery watermark for every historically-active wallet — which authorises the loader to BOUND that wallet's reads. (Since the coverage hardening the authorising artifact is the CERTIFICATE on accounts, migration 061, and it must also be FRESH; the reasoning below is unchanged.) But the seed is derived from pre-T4 history, when the flow scans only admitted CURATED markets/vaults, so it cannot contain a wallet's dormant position in a previously-uncurated market, vault or fToken. Those positions now exist in the universe but not in the index. The weekly reconcile finds them eventually, but at PORTFOLIO_RECONCILE_MAX=50 LRU that is "within N weeks", not "within a week" — and each one fires a drift alert whose remediation text ("investigate discovery.ts") is misleading, because this is expected deploy-time state, not a discovery bug. So after step 2:

   PORTFOLIO_RECONCILE_MAX=100000 scripts/run-cron.sh refresh-portfolio-reconcile.ts
   
   > Since the coverage hardening this run **exits non-zero and pages** if it finds any aged divergence or dangling link. On the one-time flush that is expected: the whole point is to find and repair pre-existing drift. Read the first alert as the flush's REPORT, not as a cron failure.

   Expect drift rows on this run (that is the point) and none afterwards. The alternative is to drop every seeded watermark and let the /10min enrollment drain re-enroll against the widened universe; either is fine, doing neither is not.
4. Add the crontab entries (see the crontab table above).
5. Re-run the deploy so the prerendered pages build against the now-complete data (see the ISR note below).

Rolling back the app is NOT data-safe once venue='wallet' rows exist

Migrations 049/050/052 are rollback-safe, and the old code tolerates venue='wallet' rows existing. The data is the problem. Old pnl.ts classifies a bare wallet leg through its default branch and attributes it via the value path, but after a rollback the old cron stops emitting wallet rows and stops scanning wallet-token transfers. At the first interval where such a leg is present at t0 and absent at t1 it computes v1 = 0, netLegFlow = 0, and books yield = −v0 — the old code has a birth guard but no death guard. Every sUSDe/wstETH-class bare holding's USD/ETH book yield curve craters by the full position value.

So a code rollback requires deleting the wallet-venue rows first (or accepting corrupted curves until re-deploy):

-- Run BEFORE (or immediately after) rolling the app back past the taxonomy release.
DELETE FROM onchain_credit.portfolio_flow_events        WHERE venue = 'wallet';
DELETE FROM onchain_credit.portfolio_position_snapshots WHERE venue = 'wallet';

Re-deploying forward re-derives them (the 6h cron re-snapshots; a per-wallet backfill-portfolio-wallet.ts repair re-lays history).

ISR re-prerender note

Pages are App Router with per-page ISR (revalidate 1800 or 3600 seconds), prerendered at build time. The build snapshots whatever the DB held when npm run build ran. So:

If a refresher (or a manual backfill) writes new data after the deploy build has already run, the prerendered pages keep serving the stale snapshot until their revalidate window elapses (up to 30/60 min). To make fresh data appear immediately, re-run the deploy (push an empty commit or re-run the workflow) so the build re-prerenders against the now-current DB. This matters most for capacity/risk pages where a refresher and a deploy land close together.

The clean ordering is therefore: migrate → backfill/refresh the data → then deploy (or re-deploy), so the build prerenders against complete data.

2.3 Versioned releases

Every promotion to prod is a numbered release. .github/workflows/release.yml runs on push to main when package.json changed:

1. Reads version from package.json.
2. If a release for v<version> does not already exist, it tags v<version> at the merge commit and cuts a GitHub Release with auto-generated notes (the merged PRs/commits since the previous tag).
3. If the version was not bumped, or the release already exists, it is a no-op.

So: the release PR bumps version (semver: minor for features, patch for fixes), and merging it both ships prod (deploy.yml) and records the release (release.yml). The tag is the durable "what shipped, when" record; deploy.yml is still what actually deploys. git tag --list / the repo's Releases page is the prod history.

3. Staging

A second, independent copy of the app on the same box, fed by a scrubbed copy of prod data. This is where all development lands and is validated before promotion (see §2).

Item	Value
URL	https://staging.creddit.xyz (Let's Encrypt cert, HTTP→HTTPS)
Access	HTTP basic-auth (/etc/nginx/.htpasswd-staging; credential in /root/.staging-basic-auth) + noindex
Repo	/opt/onchain-credit-staging, tracking origin/staging
App process	pm2 onchain-credit-staging on :3002 (memory ceiling --max-memory-restart 700M)
Database	separate DB creddit_staging, role onchain_credit_staging (conn-limit 20, statement/idle timeouts)
Deploy key	separate DEPLOY_SSH_KEY_STAGING (a staging-deploy compromise ≠ prod-box access)

3.1 Deploy (.github/workflows/deploy-staging.yml)

Auto-deploys on push to staging (the develop flow). Same SSH + git reset + build + pm2 mechanism as prod, pointed at /opt/onchain-credit-staging and :3002, using the separate staging deploy key. It builds with the same PG_POOL_MAX=4 cap as prod; the creddit_staging role's connection limit is only 20, so the smaller build-time pool matters most here (this is where the too many connections for role build flake was first observed). After build it runs scripts/ops/migrate.sh against creddit_staging (additive migrations auto-apply on staging) and a :3002 health check, with the same build-failure rollback.

workflow_dispatch (with a ref input, default main) refreshes staging from any branch on demand — useful to reset staging to prod's exact state.

Post-deploy smoke

Once the deploy job succeeds, a second job in the same workflow (smoke) runs the Playwright end-to-end suite from tests/e2e against the build it just put on the box. It is a job in this workflow rather than a workflow of its own on purpose: a separate file triggered by workflow_run fires only from the copy that sits on the default branch, so merging one into staging would arm nothing.

It needs no secret that does not already exist. The runner opens an SSH tunnel with the deploy key this workflow already uses (DEPLOY_SSH_KEY_STAGING) and points the suite at the app directly instead of at the public hostname. The same two lines are the manual smoke, from a checkout of whatever is currently on staging:

ssh -f -N -L 3002:127.0.0.1:3002 root@dexhq.io   # staging's app port, firewalled from the internet
E2E_BASE_URL=http://localhost:3002 npm run e2e   # setting it also stops playwright.config.ts booting a local server

Going through https://staging.creddit.xyz instead would mean sending the NGINX basic-auth credentials, and NGINX holds only a one-way $apr1$ hash of that password. There is no BASIC_AUTH_* repo secret, none can be derived from the box's NGINX config, and none is needed.

Report only. The job is continue-on-error: true, so a red suite never paints the deploy run red: by then the deploy has succeeded and the new build is already live, and the build-failure rollback above stays the only automatic revert here. The signal is an error annotation, a block in the job summary, and, on failure, the Playwright HTML report plus traces uploaded as the run artifact staging-smoke-<run-id>-<attempt> (14-day retention). That artifact is what to read before opening the release PR.

Everything that ran passed, with at least 4 skipped, is the healthy result. Those four are always the authed /portfolio spec, once in each of the four viewport projects: it mints its own session cookie and skips when the server does not sign with the same secret, and staging's SESSION_SECRET is deliberately not handed to CI. A higher skip count is not lost coverage either. requireRows skips an interaction test when the deployment serves fewer rows than the test needs, so a nightly reseed that leaves a screener thin shows up here as an extra skip. Read it as a note about staging's data, not as a broken smoke.

The smoke runs inside the deploy-staging concurrency group, so a second merge queues behind it instead of restarting pm2 underneath a running suite. The group is held for the whole run, and the smoke step's timeout-minutes: 15 (with a timeout-minutes: 20 backstop on the job around it) is therefore also the longest a wedged staging can keep the next deploy waiting. When a recovery dispatch cannot wait, cancel the in-flight run from the Actions UI: its deploy job has already finished by the time the smoke is running, so cancelling costs only the report. Specs, fixture DB and the per-surface checklist: Processes → Verifying a UI change.

3.2 Data: reseed, not cron (by design)

Staging does not run refreshers. Its data is a nightly scrubbed copy of prod (03:00 UTC cron, after the 02:15 backup), plus on-demand, via scripts/ops/reseed-staging.sh:

# on the box
scripts/ops/backup-creddit.sh       # fresh prod dump -> /opt/backups (also the nightly backup)
scripts/ops/reseed-staging.sh       # drop+recreate creddit_staging from newest dump, scrub PII, re-grant

The reseed masks PII (scrub-staging-pii.sql: subscriber emails hashed to @staging.invalid, user agents dropped; the whole user-data graph — the four chat tables, accounts, account_wallets, the two portfolio history tables, portfolio_wallet_index and portfolio_backfill_state — TRUNCATEd in one statement, since a reseed restores the whole prod DB with no table-exclusion and a wallet's real positions must not ride into staging; one statement because every one of those tables FK-references accounts (as of migration 056 the chat tables and portfolio_wallet_index do too), and Postgres refuses to truncate an FK-referenced table unless every referrer is in the same statement) and fails closed if any unmasked email survives, or any account/portfolio table (accounts, account_wallets, both history tables, portfolio_wallet_index, portfolio_backfill_state) still holds rows after the scrub, or any per-wallet discovery watermark (chain_scan_cursors, scope portfolio:discover:%) survives it. That last one is not PII but a CORRECTNESS hazard: chain_scan_cursors has no FK to accounts, so before the coverage hardening a surviving scope row certified an index the TRUNCATE had just emptied — the nightly reseed manufactured the 2026-07-21 incident's poisoned state on staging for every wallet in the dump. The scrub now deletes those rows (prefix-scoped, so every other refresher cursor survives as it must). Because the four portfolio-workstream tables are emptied, re-run scripts/ops/seed-portfolio-fixtures.ts after each reseed to re-register the public test whales (or touch /root/.reseed-paused during a multi-day validation window so the reseed is skipped and the fixtures persist). It carries prod's current schema and its schema_migrations ledger, so after a reseed migrate.sh reports 0 pending. A reseed reverts any not-yet-promoted migration under test; pause it with touch /root/.reseed-paused. To validate a refresher change on staging, run it once by hand: cd /opt/onchain-credit-staging && scripts/run-cron.sh <refresher>.ts (its run-cron.sh sources the staging .env.local, so it writes creddit_staging).

3.3 Same-box safety guards

Staging shares the box and the Postgres cluster with prod, so the guards matter: the staging role's statement_timeout/idle_in_transaction_session_timeout/connection cap kill a runaway staging query, the PM2 --max-memory-restart restarts staging (not prod) on a leak, and hourly disk alerting bounds the backups + DB-copy growth. A dedicated VM is the upgrade path if OS/nginx/PG-version testing is ever needed.

4. Operating the server

SSH and pm2

ssh root@dexhq.io                       # key ~/.ssh/hetzner_ed25519

pm2 list                                # all processes + status
pm2 logs onchain-credit                 # tail prod app logs
pm2 logs onchain-credit --lines 200     # recent
pm2 restart onchain-credit --update-env # restart prod (re-reads .env.local)
pm2 restart onchain-credit-staging --update-env

Event ledger ingester (portfolio 100k scale)

The event-ledger ingester (scripts/ingester/ingest-events.ts) is the wallet-count-independent spine of the 100k-wallet plan. Unlike the refreshers it is not a cron: it is an always-on PM2 process with its own ~60s loop. Bringing it up is a manual server step, in order:

1. Apply migration 064 (additive + idempotent; staging auto-applies on deploy, prod is the usual gated migrate.sh). It creates raw_events (range-partitioned)

   • event_coverage and hands raw_events to the onchain_credit role so the ingester can auto-create partitions at runtime (see database.md).

   cd /opt/onchain-credit && scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)"

   Ownership caveat on staging. The parent ends up owned by onchain_credit (the prod role), and migration 064 also GRANTs that role CREATE on the schema so on prod the ingester and backfill auto-create partitions at runtime. Staging connects as onchain_credit_staging, which neither owns the parent nor is the grantee, so its runtime partition auto-create fails. The pre-created band covers only the live tip (the ingester's forward inserts), NOT the one-time backfill: the 2025-05-21 floor (~block 22.5M, partition p90) sits below the band's p98 floor, so step 3's backfill needs auto-create even for the validation window and will fail LOUD (ensurePartitions never does a silent bad insert). Before exercising the backfill on staging, re-own the parent: ALTER TABLE onchain_credit.raw_events OWNER TO onchain_credit_staging on that box.

2. Start the PM2 process. run-ingester.sh sources .env.local and execs the long-running loop, so PM2's SIGTERM reaches the ingester's graceful-shutdown handler (it finishes the current cycle, then exits 0):

   cd /opt/onchain-credit
   pm2 start scripts/ingester/run-ingester.sh --name creddit-event-ingester --time
   pm2 save                              # persist across reboots (and the --time flag)
   pm2 logs creddit-event-ingester

   It reuses ETHEREUM_RPC_URL (live head + live-window getLogs) and ETHEREUM_ARCHIVE_RPC_URL (catch-up history + block timestamps) from .env.local.

3. Run the one-time historical backfill once (attended; fills raw_events from the 2025-05-21 floor up to the ingester's live cursor; resumable, archive RPC):

   cd /opt/onchain-credit && ETHEREUM_ARCHIVE_RPC_URL=<archive> \
     DATABASE_URL="$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)" \
     npx tsx scripts/backfill-event-ledger.ts

A read-only live smoke (scripts/ingester/smoke-ledger.ts) checks the decode path against a live RPC without touching the DB. Phase A ships the ledger only — nothing reads it yet (the 6h cron keeps its own scans), so a missed step degrades nothing user-facing. Log-rotation note: the ingester writes PM2 logs like the app, but the /etc/logrotate.d/onchain-credit glob currently matches only the four creddit-app logs — add creddit-event-ingester-{out,error}.log to it (same copytruncate treatment) when the process is promoted to prod.

Portfolio 100k scale: Phase C envs + the snapshot repartition (migration 067)

New envs (all optional, safe defaults; set in .env.local on the box):

Env	Default	Effect
BACKFILL_WORKERS	4	concurrent wallet backfills per drain invocation (D3), each a separate child process
BACKFILL_GRID_CONCURRENCY	6	concurrent per-grid-point archive reads within one replay child (D2)
WALLET_TOPIC_CHUNK	500	wallets per getLogs owner-topic pass in the off/shadow legacy scans (D5)
JIT_LEDGER_LAG_BLOCKS	40	JIT ledger fast-path freshness gate (D4). The default keeps the JIT ledger paths OFF — the ingester's 64-block margin means the tip trails head by ≥ 64, which is not within 40. Raise it above ~64 (accepting up to that-many blocks of flow/position staleness on the JIT, self-healing on the next refresh/cron) only after PORTFOLIO_LEDGER_MODE=on has proven the ledger.

Effective archive concurrency = BACKFILL_WORKERS × BACKFILL_GRID_CONCURRENCY (default 4 × 6 = 24 concurrent grid reads, each firing several multicall/block requests → ~100–300 concurrent archive requests worst case). The two knobs MULTIPLY: the grid cap is per replay child, the worker pool runs that many children as separate processes, and there is no shared in-process RPC semaphore (finding 7). Size them against a provisioned high-throughput archive endpoint (Alchemy ETHEREUM_ARCHIVE_RPC_URL); on a free-tier archive default the burst rate-limits and grid reads fail → M9 abort → requeue → the same burst next tick. Raising BACKFILL_GRID_CONCURRENCY in isolation is a hidden ×BACKFILL_WORKERS.

The D1 backfill-discovery and D4 JIT fast paths are additionally gated by PORTFOLIO_LEDGER_MODE (they only engage when it is not off / is on) AND by event_coverage being live back to the read window — so they self-disable until the one-time backfill + ingester have caught the ledger up. No env flip is required to keep them off beyond leaving PORTFOLIO_LEDGER_MODE at its rollout value.

Snapshot repartition (migration 067, -- DESTRUCTIVE, MANUAL). 067 repartitions portfolio_position_snapshots into monthly range partitions. It is tagged -- DESTRUCTIVE, so the staging auto-apply and a plain migrate.sh SKIP it; it runs only under --allow-destructive. It renames the live table aside to portfolio_position_snapshots_preswap — including its index names, because a table rename leaves those behind and CREATE INDEX IF NOT EXISTS matches on name, so without that the new parent would silently end up with nothing but its PK — builds the partitioned replacement (identical columns / PK / indexes / FK / grants — the PK already carries snapshot_ts, so no conflict target changes), copies every row, and KEEPS _preswap for a manual DROP. Run it while the spine is near-empty (post-reset, the plan's clean-slate timing). The writers auto-create future month partitions at runtime (ensureMonthPartitions, a safe no-op before this migration). portfolio_flow_events is partitioned separately and differently, BY HASH (wallet), by migration 072 below. See the migration header and docs/database.md.

Runbook (staging first, then prod after a soak):

# 0. PAUSE THE NIGHTLY RESEED for the whole verification window (see the note below):
touch /root/.reseed-paused
# 1. Staging — apply the additive Phase C migrations (064/065/066 already; nothing new
#    additive in C), then apply 067 EXPLICITLY with --allow-destructive:
cd /opt/onchain-credit-staging && \
  scripts/ops/migrate.sh "postgresql://onchain_credit_staging@127.0.0.1:5432/creddit_staging" --allow-destructive
# 2. Verify: row counts match, the parent carries its FIVE indexes (4 + PK) and the month
#    bounds are exact UTC midnights, /portfolio spot-check, then drop the aside BY HAND:
#    psql> SELECT count(*) FROM onchain_credit.portfolio_position_snapshots;         -- new
#    psql> SELECT count(*) FROM onchain_credit.portfolio_position_snapshots_preswap; -- old (equal)
#    psql> SELECT c.relname FROM pg_class c JOIN pg_index i ON i.indexrelid = c.oid
#           WHERE i.indrelid = 'onchain_credit.portfolio_position_snapshots'::regclass;
#          -- expect: portfolio_pos_wallet_ts_idx, portfolio_pos_chain_ts_idx,
#          --         portfolio_pos_pendle_ptkey_idx, portfolio_pos_accounting_asset_idx,
#          --         portfolio_position_snapshots_pkey  (anything missing is missing FOREVER
#          --         once _preswap is dropped: migrate.sh has recorded 067)
#    psql> SELECT c.relname, pg_get_expr(c.relpartbound, c.oid) FROM pg_class c
#           JOIN pg_namespace n ON n.oid = c.relnamespace
#           WHERE n.nspname = 'onchain_credit' AND c.relname LIKE 'portfolio_position_snapshots_2%';
#          -- expect 00:00:00+00 boundaries; anything else means the month arithmetic and the
#          -- runtime helper's UTC literals disagree and will collide at the next rollover
#    psql> DROP TABLE onchain_credit.portfolio_position_snapshots_preswap;
# 3. Prod (gated on Fred's approval; run while the spine is near-empty):
cd /opt/onchain-credit && \
  scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)" --allow-destructive
#    Verify + DROP _preswap by hand exactly as on staging.
# 4. Only after the _preswap table is dropped on BOTH environments, resume the reseed:
rm -f /root/.reseed-paused

Because 067 renames + swaps inside migrate.sh's single transaction, a failure rolls the whole thing back and leaves the original table untouched — it is atomic. A re-run detects the table is already partitioned and no-ops.

Why step 0 exists. RENAME carries a table's constraints, so portfolio_position_snapshots_preswap keeps migration 056's FK to accounts plus a full copy of every wallet's history. Prod backs up at 02:15 UTC and reseed-staging.sh restores that dump at 03:00, so an aside table left standing overnight rides into staging — and the scrub's single TRUNCATE then fails (cannot truncate a table referenced in a foreign key constraint), aborting the reseed before its fail-closed leak checks and leaving staging serving the unscrubbed prod database, every night, until the aside table is dropped. portfolio_position_snapshots_preswap is listed in scripts/ops/scrub-staging-pii.sql and in the reseed's leak check (to_regclass-guarded, so it is a no-op when absent), which fixes it properly; the pause is the belt-and-braces half. scripts/ops/scrub-staging.test.ts keeps the two lists in lockstep, and portfolio_flow_events_preswap was pinned alongside it pre-emptively — migration 072 now produces exactly that table, so the fix was already in place when the flow repartition landed rather than being rediscovered by a failed reseed.

Partition maintenance after 067. Runtime creation is a backstop, not the primary path: the 6h cron pre-creates the CURRENT and NEXT month's partition for the spine, so the rollover fires from a quiet non-request process rather than from whichever writer happens to produce the first row of a new month. CREATE TABLE … PARTITION OF takes ACCESS EXCLUSIVE on the parent and Postgres's lock queue is FIFO, so the DDL runs under a 2s lock_timeout — queued behind one slow reader it gives up rather than parking every later /portfolio SELECT behind itself, and the next writer retries. Nothing to do operationally.

Applying 067 under running writers. The 6h tick, the minutely drain's children and the app all cache "is this table partitioned?" per process, and every one of them answered no before you ran the migration. That answer is NOT pinned: a negative expires after 60s, so a process that started before the migration begins creating month partitions within a minute, without a restart. A 23514 no partition of relation from a write also invalidates the cache immediately, which shortens that window for the writers whose caller survives the error (a drain child moves to the next wallet in-process); the 6h tick instead ends on the throw, so there the TTL is the whole mechanism and the cost is at most one missed snapshot window. Nothing to do operationally; noted because the failure it prevents is quiet (a snapshot missing beside a live balance reads as yield).

Flow-ledger repartition: hash by wallet (migration 072)

What it does. 072 rebuilds onchain_credit.portfolio_flow_events as PARTITION BY HASH (wallet) with MODULUS 16 — 16 partitions (portfolio_flow_events_h00 … h15), created ONCE by the migration because a hash modulus is fixed at CREATE time. ~6,250 wallets per partition at the 100k target. It renames the live table aside to portfolio_flow_events_preswap (including its index names — same trap as 067: a table rename leaves index names behind and CREATE INDEX IF NOT EXISTS matches on name, so without that step the new parent would end up carrying nothing but its PK), builds the partitioned replacement with identical columns / PK / CHECKs / the five indexes / the 056 FK / grants, copies every row, and KEEPS _preswap for a manual DROP. Tagged -- DESTRUCTIVE, so the staging auto-apply and a plain migrate.sh SKIP it; it runs only under --allow-destructive.

⚠ DEPLOY + ROLLBACK WARNING: the order is ONE-WAY. Ship the code release first, then apply 072. That release is a rollback floor for as long as this table is hash-partitioned. The PK genuinely does not change — wallet is already the second column of (chain_id, wallet, tx_hash, log_index, leg), so the partition key is inside the unique constraint Postgres requires it in and all three writers' 5-column ON CONFLICT targets match it exactly, before and after — but that buys the forward direction only:

• Forward (this release's code, table not yet partitioned) — FREE. The only code change in the release is the DELETION of the five ensureMonthPartitions calls naming this table — live.ts writeFlowRows, refreshers/portfolio.ts writeFlows, and three in backfill.ts (writeFinalFlowRowsTxn, upsertFlowRowsTxn, and writeGapSegment's segment pre-flight, whose sibling snapshot call stays) — all of them no-ops while the table was un-partitioned, plus the helper's partstrat guard. Nothing new is referenced, so the new code is correct against the old table for as long as it takes to get to the migration.
• Backward (the previous release's code, table already partitioned) — HARD-BROKEN. Every flow read and write is fine on its own; it names the parent, never a partition. What breaks is the vestigial DDL call in front of the write. The previous release's ensureMonthPartitions gates on relkind = 'p' alone, so 072 flipping the parent to 'p' stops those five call sites no-opping and starts them issuing CREATE TABLE … PARTITION OF <hash parent> FOR VALUES FROM (…) TO (…), which Postgres rejects: 42P16 invalid bound specification for a hash partition. It never self-heals (isMissingPartitionError requires 23514, so the self-heal path declines 42P16, and the positive relkind memo is permanent for the process lifetime; a restart re-probes and gets 'p' again). Verified end-to-end on PG 16 against the deployed helper.
• Blast radius, loud on one half and silent on the other. The 6h tick commits its snapshots before the first writeFlows, so it advances balances and then throws on the flow write — the cron alert fires, and a balance standing with no flow behind it reads as yield (the M3 violation). live.ts writeFlowRows sits inside the JIT try whose catch only logs mini flow scan failed, so the JIT flow persist and the provisional self-clean fail with no alert on every page load. All three backfill.ts call sites are pool-side, so they throw before their transaction ever opens.

Same class as 044 (hard-coupled, though 044 was coupled in both directions); unlike 067, whose forward direction needed the helper's memo semantics. This matters operationally because the bad order is reachable without an operator decision: deploy.yml reverts the tree with git reset --hard $PREV on an npm ci / npm run build failure (a documented recurring flake here), which lands the pre-072 code on a hash parent by itself. If the accompanying deploy fails its build, roll forward (re-run the deploy) rather than leaving the tree on $PREV. This is the same rule as the general one below: the deploy's code-rollback path does not roll back the DB.

Pick a quiet window as well: the copy is a single statement under migrate.sh's statement_timeout=300s, and the rename needs ACCESS EXCLUSIVE within lock_timeout=10s.

Runbook (staging first, then prod after a soak):

# 0. PRECONDITION — the DEPLOYED code must already be the release that ships 072. The
#    previous release's flow writers call ensureMonthPartitions behind a relkind-only probe,
#    so the moment the parent becomes hash-partitioned they emit month bounds at it and
#    EVERY flow write fails 42P16 (see the warning above). Confirm the box is running the
#    right build BEFORE touching migrate.sh — the partstrat guard is the marker:
#      cd /opt/onchain-credit-staging && git log -1 --format='%h %s'
#      grep -c partstrat src/lib/portfolio/partitions.ts      # must be > 0
#      grep -c 'portfolio_flow_events' <(grep -rh 'ensureMonthPartitions(' \
#        src/lib/portfolio scripts/refreshers)                # must be 0
#    If that fails, deploy first. If a deploy build FAILED and deploy.yml reverted the tree
#    to $PREV, re-run the deploy (roll forward) — do not migrate against $PREV.
# 0a. PAUSE THE NIGHTLY RESEED for the whole verification window — same reason as 067
#     (the aside table inherits the 056 FK to accounts and breaks the scrub's TRUNCATE):
touch /root/.reseed-paused
# 0b. PRE-FLIGHT: how big is the copy? migrate.sh runs every file under
#     statement_timeout=300s and lock_timeout=10s, and the copy is ONE statement.
#     psql> SELECT count(*), pg_size_pretty(pg_total_relation_size('onchain_credit.portfolio_flow_events'))
#            FROM onchain_credit.portfolio_flow_events;
#     A few million rows copy well inside 300s; if it would not, do NOT split the migration
#     (its atomicity is the safety property) — run that one file by hand with a raised
#     timeout instead, then record it in the ledger:
#       sudo -u postgres env PGOPTIONS='-cstatement_timeout=1800s -clock_timeout=10s' \
#         psql -v ON_ERROR_STOP=1 -X -q -1 -d <db> -f scripts/sql/072-repartition-portfolio-flows.sql
#       sudo -u postgres psql -q -d <db> -c "INSERT INTO onchain_credit.schema_migrations(filename) \
#         VALUES ('072-repartition-portfolio-flows.sql')"
# 1. Staging — 072 is DESTRUCTIVE, so it needs the explicit flag:
cd /opt/onchain-credit-staging && \
  scripts/ops/migrate.sh "postgresql://onchain_credit_staging@127.0.0.1:5432/creddit_staging" --allow-destructive
# 2. Verify — counts, the 16 partitions, the SIX indexes (5 + PK), and an EXPLAIN prune:
#    psql> SELECT count(*) FROM onchain_credit.portfolio_flow_events;         -- new
#    psql> SELECT count(*) FROM onchain_credit.portfolio_flow_events_preswap; -- old (equal)
#    psql> SELECT count(*) FROM pg_class c JOIN pg_inherits i ON i.inhrelid = c.oid
#           WHERE i.inhparent = 'onchain_credit.portfolio_flow_events'::regclass;   -- expect 16
#          -- 072 now ASSERTS this itself (16 partitions, 16 distinct remainders, before the
#          -- copy, so a shortfall aborts the atomic file). This query is the belt-and-braces
#          -- half: the case it guards is a relation already occupying one of the
#          -- portfolio_flow_events_hNN names, which `CREATE TABLE IF NOT EXISTS ...
#          -- PARTITION OF` matches on NAME and would otherwise adopt-around silently.
#    psql> SELECT c.relname FROM pg_class c JOIN pg_index i ON i.indexrelid = c.oid
#           WHERE i.indrelid = 'onchain_credit.portfolio_flow_events'::regclass;
#          -- expect: portfolio_flow_wallet_block_idx, portfolio_flow_wallet_pos_idx,
#          --         portfolio_flow_asset_idx, portfolio_flow_pendle_ptkey_idx,
#          --         portfolio_flow_provisional_idx, portfolio_flow_events_pkey
#          --         (anything missing is missing FOREVER once _preswap is dropped:
#          --          migrate.sh has recorded 072, so it never re-runs to reconcile)
#    psql> EXPLAIN (COSTS OFF) SELECT tx_hash, log_index FROM onchain_credit.portfolio_flow_events
#           WHERE chain_id = 1 AND wallet = '<a real lower-cased wallet>'
#           ORDER BY block_number DESC, log_index DESC;
#          -- expect ONE partition in the plan (e.g. ..._h06) and NO Append node. An Append
#          -- over all 16 means the wallet predicate is not pruning and the repartition has
#          -- bought nothing.
#    psql> ANALYZE onchain_credit.portfolio_flow_events;   -- fresh stats on the new relations
#    then a /portfolio + /portfolio events spot-check on a whale fixture wallet.
#    psql> DROP TABLE onchain_credit.portfolio_flow_events_preswap;
# 3. Prod (gated on Fred's approval; quiet window). RE-CHECK step 0's precondition here:
#    prod migrations are a separate manual step from the prod deploy, so this is the one
#    place the wrong order is easy to reach by hand.
cd /opt/onchain-credit && \
  scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)" --allow-destructive
#    Verify + ANALYZE + DROP _preswap by hand exactly as on staging.
# 4. Only after _preswap is dropped on BOTH environments, resume the reseed:
rm -f /root/.reseed-paused

Because 072 renames + creates + copies inside migrate.sh's single transaction, a failure anywhere rolls the whole thing back and leaves the original table untouched — it is atomic. A re-run detects the table is already partitioned and no-ops.

Why step 0a exists is exactly the 067 story: RENAME carries a table's constraints, so portfolio_flow_events_preswap keeps migration 056's FK to accounts plus a full copy of every wallet's flow history. Prod backs up at 02:15 UTC and reseed-staging.sh restores that dump at 03:00, so an aside table left standing overnight rides into staging and the scrub's single TRUNCATE then fails, aborting the reseed before its fail-closed leak checks. portfolio_flow_events_preswap was added to scripts/ops/scrub-staging-pii.sql and the reseed leak check pre-emptively (both to_regclass-guarded), so the mechanism is already in place and scripts/ops/scrub-staging.test.ts now checks both ends of the contract against 072 itself. The pause is the belt-and-braces half.

Partition maintenance after 072: there is none. All 16 partitions exist by construction, so unlike the snapshot spine there is no month rollover, no runtime DDL, no ACCESS EXCLUSIVE parent lock on a write path, and no reason to hand the parent to the app role — it stays owned by postgres with the app holding DML grants only (the 16 partitions pick up the schema's ALTER DEFAULT PRIVILEGES grant at CREATE time, but nothing depends on that: privileges are checked on the relation a query names, and tuple routing does not re-check them). The writers' partition calls for this table are deleted, and three layers keep them gone: ensureMonthPartitions' table parameter no longer ADMITS the flow ledger (MonthPartitionedTable excludes it, so the copy-paste does not compile), the helper refuses any non-RANGE parent at runtime (it probes pg_partitioned_table.partstrat), and a source tripwire in src/lib/portfolio/partitions.test.ts catches a literal call. Without those, month bounds against a hash parent are rejected outright (invalid bound specification for a hash partition) — which is the whole reason the deploy order above is one-way.

If the 16 partitions ever need to become 32 (they will not at the 100k target): 16 is a power of two, and Postgres allows mixed moduli as long as each divides the next, so it is an incremental DETACH of one MODULUS 16 partition plus ATTACH of two MODULUS 32 partitions with remainders r and r+16, one at a time — not another full rebuild.

Flow basis provisional (migration 070, additive). Widens the portfolio_flow_events.basis CHECK to admit provisional and adds the partial index the two provisional deletes use (see database.md). Staging auto-applies it on deploy; prod is the usual gated manual migrate.sh run. Apply it BEFORE the code restart — not "before or with", and never after: until the constraint is widened, the JIT's flow persist fails on any flow landing in the last ~64 blocks (the whole persist transaction rolls back and the JIT logs mini flow scan failed). Nothing is corrupted (a rejected insert writes nothing) and the flow is not lost, but two details make the window worse than it looks and neither is self-announcing:

• It does not heal on the next refresh. Every subsequent JIT refresh of that wallet fails on the same CHECK for as long as the flow sits in the reorg-exposed tail; only the next 6h cron tick — which writes live rows exclusively and so never touches the widened value — actually lands it. Up to 6h of "my deposit is not showing".
• Nothing pages you. The failure is caught by the JIT and written with console.error to the app's pm2 log, not to run-cron.sh's log, and the cron alert needs a non-zero exit and a matching line. JIT flow persistence can be dead for hours in silence, so grep pm2 logs for mini flow scan failed if you ever run the code ahead of 070.

The cron's settle sweep is cheap without the index (it is bounded by the tick's wallet list and block window, so it can fall back to portfolio_flow_wallet_block_idx), but it runs once per tick against the whole flow ledger — one more reason not to leave the window open. Rollback-safe in the other direction: the previous release only ever writes live/backfill, which the widened CHECK still admits.

Running a cron/refresher by hand

scripts/run-cron.sh is the wrapper for every scheduled TS job: it sources .env.local, resolves node_modules/.bin/tsx, runs scripts/<script> from the repo root, and appends output to /tmp/onchain-credit-cron/<script>.log. Extra args are forwarded to the script. It takes a non-blocking per-script flock keyed by the checkout ($LOG_DIR/$(basename "$DIR")-<script>.lock): a second run of the same script in the same checkout while the first is still going logs "skipping this tick" and exits 0 (so a slow 6h job never stacks a copy on itself), while the prod and staging checkouts get distinct lock files and never block each other. The script's real exit code is preserved for cron. NODE/TSX fall back to the box defaults (/usr/bin/node, the checkout's tsx) but are env-overridable for local testing.

Cron-failure alerting (WS8). On a non-zero exit, the wrapper best-effort POSTs a Telegram message naming the checkout, the script, and the exit code, so a failed job is not silent. It is gated on ALERT_TG_BOT_TOKEN + ALERT_TG_CHAT_ID (both must be in .env.local; if either is unset it skips silently, so an un-provisioned box never posts). The alert runs with errexit off and a time-bounded curl, so it can never change the job's real exit code or block the wrapper (a failed POST is logged as "ignored"). Because prod and staging share the box, the checkout label ($(basename "$DIR"), or the ALERT_ENV override) tells you which environment failed. This covers every cron job. Separately, the portfolio refresher posts its own unknown-asset alert to the same bot when an accounting asset it holds with value is unmapped in buckets.ts (see Data pipeline → refresh-portfolio.ts). To exercise the alert path locally, set ALERT_TG_API_BASE at a stub endpoint (defaults to https://api.telegram.org).

cd /opt/onchain-credit
scripts/run-cron.sh refresh-assets.ts
scripts/run-cron.sh refresh-vault-capacity.ts
scripts/run-cron.sh refresh-sofr.ts --since=2018-04-03   # args pass through
tail -f /tmp/onchain-credit-cron/refresh-vault-capacity.log

Server crontab (all UTC):

Schedule	Command	Cadence
0 */6 * * *	run-cron.sh refresh-assets.ts	every 6h
15 */6 * * *	run-cron.sh refresh-vault-capacity.ts	every 6h
30 */6 * * *	run-cron.sh refresh-collateral-exposure.ts	every 6h
45 */6 * * *	run-cron.sh refresh-lending-positions.ts	every 6h
50 */6 * * *	run-cron.sh refresh-portfolio.ts	every 6h (WS4; manual crontab add)
* * * * *	run-cron.sh drain-portfolio-backfills.ts	minutely (WS5 queue drain; manual crontab add — apply migration 045 FIRST; pre-045 the drain degrades to the attempts-free reclaim with a warning). Exit 2 (= Telegram alert) when a wallet is parked as error after exhausting its reclaim budget. The 6h portfolio tick warns if anything sits queued >30min (drain missing/broken).
*/10 * * * *	run-cron.sh refresh-portfolio-discovery.ts	every 10 min (portfolio taxonomy T4 §3.6; manual crontab add — apply 050 FIRST). Discovery enrollment drain: one FULL-universe current read per eligible wallet with no FRESH completeness certificate (accounts.discovery_scanned_*, migration 061), capped PORTFOLIO_ENROLL_MAX (default 10). Quiet no-op once every eligible wallet holds one. Selection is freshness-based, not presence-based, so it is also the REPAIR half of the staleness gate: a wallet demoted for a frozen certificate is re-certified within one run instead of paying the full-universe read tax indefinitely. Un-scanned wallets force the whole 6h batch to a full-universe read, so this is what makes the discovery bound ENGAGE for new signups.
20 3 * * *	run-cron.sh refresh-morpho-universe.ts	daily (T4 §3.5; manual crontab add — apply 041 FIRST). Exhaustive Morpho Blue market universe via a cursor-driven CreateMarket scan. First run: set MORPHO_UNIVERSE_FROM_BLOCK to the Morpho Blue deploy block so it walks the full history; steady-state scans only new blocks.
40 3 * * *	run-cron.sh refresh-metamorpho-factory.ts	daily (T5 §3.5; manual crontab add — apply 052 FIRST). Exhaustive MetaMorpho vault universe via a cursor-driven CreateMetaMorpho scan over BOTH factories. First run: set METAMORPHO_FACTORY_FROM_BLOCK to the v1.0 deploy block; steady-state scans only new blocks.
30 4 * * 0	run-cron.sh refresh-portfolio-reconcile.ts	weekly, Sun 04:30 (T5 §3.6; manual crontab add — apply 050 FIRST). Discovery reconciliation safety net: reads each SCANNED wallet against the COMPLETE universe and adds any membership the index missed, with a WS8 drift alert. Capped PORTFOLIO_RECONCILE_MAX (default 50), least-recently-reconciled first. Scheduled AFTER the two universe ingests so "the complete universe" really is complete. See the one-time post-ingest step below. Wallets rotate least-recently-RECONCILED first (accounts.discovery_reconciled_at, migration 061 — a column only this sweep writes, so the 6h tick's certificate advance cannot flatten the rotation). Exits non-zero on four conditions, each printing a [fail]-tagged line so the cron alert has both halves it needs: STARVED (eligible wallets present but an empty batch = the backstop checked nothing), an AGED DIVERGENCE (a membership the index lacked while the ledger already carried its flow — a membership the byproduct simply has not reached yet is repaired quietly, so a brand-new position never pages), a DANGLING LINK (counted as FOUND, not repaired, so a run whose repairs all failed pages more loudly rather than less), and an all-errors run. The dangling-link check runs BEFORE the batch selection, so an empty cohort — the situation a bulk account delete creates — cannot hide it.
30 4 * * 1	run-cron.sh sync-portfolio-tokens.ts	weekly, Mon 04:30 (T6; manual crontab add — apply 049 FIRST). Portfolio token-registry sync: propose + alert only, never writes on the cron. A human applies an entrant with --approve <SYMBOL>.
30 3 * * 1	run-cron.sh refresh-vault-risk.ts	weekly, Mon 03:30
0 13 * * 1-5	run-cron.sh refresh-sofr.ts	weekdays 13:00
10 4 * * 1	run-cron.sh chat-retention.ts	weekly, Mon 04:10 (chat retention, audit B6; manual crontab add). Prunes chat conversations idle >365d (messages cascade via the migration 038 FK) and chat_usage rows >400d, in one transaction. Portfolio history is untouched (retained in full by design). Prod only.
15 2 * * *	ops/backup-creddit.sh	daily DB backup, 02:15
0 3 * * *	ops/reseed-staging.sh	daily staging reseed from the 02:15 dump (pause: touch /root/.reseed-paused)
hourly	disk-usage alert	hourly

The staggered :00 / :15 / :30 / :45 on the 6h jobs spreads RPC/API load. Note ops/backup-creddit.sh (02:15) runs ahead of any data job, so the daily dump is a quiet-state snapshot.

psql

psql -U onchain_credit -d creddit                 # prod app role
psql -U onchain_credit_staging -d creddit_staging # staging

# inside psql
SET search_path TO onchain_credit;
\dt onchain_credit.*
SELECT max(snapshot_ts) FROM onchain_credit.assets;

Migrations 062 / 063 (coverage hardening, additive)

062 adds a (chain_id, snapshot_ts DESC) index on portfolio_position_snapshots (the 6h tick's flow-scan population asks "which wallets have any snapshot in the last 48h?", and every existing index leads with (chain_id, wallet, …)). 063 adds portfolio_backfill_state.covered_through_ts / covered_through_block, the durable coverage anchor.

Both are additive, idempotent and inert under a code rollback, so staging auto-applies them and prod is the usual gated manual step. Neither has a transient, unlike 061. But 063 is a HARD PREREQUISITE FOR THE DEPLOY, not a post-deploy step: the add-wallet path's requeue statement references covered_through_ts, and although it falls back to the pre-063 predicate on a 42703, the fallback costs a round trip on every add until the migration lands. Apply 063 BEFORE the release merge (the v0.9.0 lesson). with no 063 columns loadCoverageAnchor returns null and every backfill takes the fresh-replay path it takes today. Note the consequence: no wallet gets a gap patch until it has completed one run after 063, because an anchor is only written on a terminal done (or advanced by a tick). That is intentional — a wallet whose last completed replay predates the column has no trustworthy seam to patch from.

Migration 061 (discovery completeness certificates)

Additive and idempotent, so staging auto-applies it; prod stays the gated manual step and must run before the release merge. It adds accounts.discovery_scanned_block / discovery_scanned_at / discovery_reconciled_at and copies the surviving portfolio:discover:<wallet> scope rows onto them, taking chain_scan_cursors.updated_at and never now() (a now() stamp would launder every currently-poisoned wallet as freshly certified, which is the whole point of the migration).

Expect one transient after it runs, and it is by design. The copied stamps are as old as the migration-050 seed for most wallets, i.e. far past the 18h staleness horizon, so the whole population is DEMOTED on the first read afterwards. Because the discovery bound fails open for the whole batch, one un-fresh wallet puts every wallet on a full-universe read: expect the 6h tick to log discovery FULL (unscanned wallets) and take measurably longer for a tick or two. It resolves itself as the */10 enrollment drain certifies wallets (PORTFOLIO_ENROLL_MAX, default 10 per run), and the reads are correct throughout, just slower. To shorten it, run the drain by hand right after the migration:

cd /opt/onchain-credit && PORTFOLIO_ENROLL_MAX=50 scripts/run-cron.sh refresh-portfolio-discovery.ts

Deploy order is tolerant in both directions: before the migration the readers fall back to the legacy scope-row presence test (whole-batch, on a 42703), and the enrollment / reconcile / backfill writers dual-write the scope row so a code rollback still finds a current watermark. The 6h tick deliberately does NOT dual-write, so it cannot refresh those scope rows to now() and defeat the migration's copy policy.

Migrations

DDL lives in scripts/sql/NNN-*.sql, applied in numeric order as the postgres owner (the app role is read-only against the data and lacks DDL rights). Applied files are tracked in onchain_credit.schema_migrations, and the runner scripts/ops/migrate.sh is the canonical way to apply pending ones:

# on the box — apply any pending migrations to a database, idempotently
cd /opt/onchain-credit
scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)"        # prod
scripts/ops/migrate.sh "postgresql://onchain_credit_staging@127.0.0.1:5432/creddit_staging"  # staging

The runner lists scripts/sql/*.sql in order, skips any already recorded in schema_migrations, applies the rest inside a transaction, and records them. It is flock-serialized with a lock_timeout/statement_timeout so a blocked migration fails fast, and it refuses a file tagged -- DESTRUCTIVE unless passed --allow-destructive (expand/contract discipline; the deploy never auto-drops). A file needing to run outside a transaction (e.g. CREATE INDEX CONCURRENTLY) tags itself -- NO-TRANSACTION.

• Staging auto-applies additive migrations inside deploy-staging.yml.
• Prod migrations stay a gated manual step for now: after a release deploys, run migrate.sh against creddit by hand (it is a no-op if nothing is pending). Wiring migrate.sh into deploy.yml is the planned next step once the ledger has proven itself over a few releases.
• Migrations are forward-only and must be backward-compatible with the previous release (expand/contract): the deploy's code-rollback path does not roll back the DB, so a rolled-back build still runs against the migrated schema.

After applying, run the matching refresher/backfill so the new columns/tables fill, then (if data changed) re-deploy to re-prerender (see §2.1 ISR note).

Database backup

scripts/ops/backup-creddit.sh (in-repo) runs daily at 02:15 UTC from cron: pg_dump -Fc creddit → /opt/backups, integrity-checked (pg_restore --list), 14-day retention. The same custom-format dump is what reseed-staging.sh restores into staging, so the backup is continuously restore-tested. On-box only for now; an off-box encrypted copy is a documented follow-up (needs a storage target + key). RPO is up to 24h with no PITR, accepted because the data is re-derivable from chain (the refreshers rebuild it); the only non-derivable data is the newsletter/capacity email tables.

Log rotation

pm2 does not rotate its own logs, and pm2 restart does not truncate them. Left alone, /root/.pm2/logs/onchain-credit-error.log accumulates forever — it reached 62k lines / 4.1 MB spanning every deploy since the box was built before this was fixed.

Two independent pieces, both needed:

Piece	What it fixes	Where
/etc/logrotate.d/onchain-credit	dates and caps the files (daily, 14 days, maxsize 50M, compressed)	reference copy: scripts/ops/logrotate-onchain-credit.conf
pm2 --time on both creddit apps	timestamps the lines	persisted in the pm2 dump via pm2 save

Rotation alone is not enough: it dates the files but every line inside is still anonymous. --time is what makes a line answerable. It survives deploy.yml's pm2 restart because pm2 save writes it to /root/.pm2/dump.pm2.

Reading the error log after a deploy: don't trust the tail. Lines have no inherent ordering guarantee relative to now, and (pre---time) months-old lines sit directly above fresh ones. Three known-benign residents cost triage time repeatedly:

• Failed to find Server Action "<hash>" — a browser tab on the old build POSTs an action id the new build doesn't have. Expected on every deploy; self-heals on reload. Hundreds of distinct hashes = many deploys, not one incident.
• Failed to load external module pg-<hash> — alarming (pg is the Postgres driver) but historical; it does not recur and DB-backed pages serve 200.
• Single item size exceeds maxSize — Next data-cache notice, item too big to cache. Present on staging too. Not a page failure.

To tell live from historical, don't read — measure. Snapshot wc -l on the log, curl the DB-backed pages, then print only the newly appended lines:

L=/root/.pm2/logs/onchain-credit-error.log; B=$(wc -l < $L)
for p in / /portfolio /carries; do curl -s -o /dev/null -w "%{http_code} $p\n" "http://localhost:3001$p"; done
sleep 3; A=$(wc -l < $L); sed -n "$((B+1)),${A}p" $L

Anything not emitted by a fresh request is not the release's problem.

Scope is deliberate. The glob matches only the four creddit logs (onchain-credit-{out,error}.log, onchain-credit-staging-{out,error}.log). The co-resident DEX HQ logs (dexhq, rindexer, creddit-indexer) are not rotated by this config and are not ours to touch — rindexer-out.log alone is 286 MB. For the same reason the config uses copytruncate rather than a pm2 reloadLogs postrotate, which would reopen every pm2 process's handles box-wide.

Access gate (NEXT_PUBLIC_ACCESS_CODE, per environment)

src/components/AccessGate.tsx is an opaque full-screen overlay that covers the app until the visitor types an access code (then remembered per browser in localStorage). It is off unless the environment sets NEXT_PUBLIC_ACCESS_CODE, so each environment decides independently. It replaces the old hardcoded EarlyAccessGate (code letsgetonchain), which was removed in v0.3.0.

• Build-time, not runtime. Next inlines NEXT_PUBLIC_* into the client bundle and every page here is prerendered, so the value is baked by npm run build. Setting the var and only running pm2 restart does not arm the gate; a deploy (or a manual npm run build + restart) does.
• Soft gate, zero access control. NEXT_PUBLIC_ACCESS_CODE is inlined into the client bundle at build time and compared in client React state, so the code ships in the bundle, the localStorage flag is trivially set by hand, and page content stays server-rendered underneath (deliberately, so SEO/structured data still indexes). It is cosmetic friction and beta signalling, never a security control. Everything private is protected server-side by the session cookie on /api/portfolio/* and /api/chat* regardless of the gate.
• Rotating the code: bump STORAGE_KEY in the component (currently creddit_access_v2) to re-prompt browsers that already unlocked with the old one.
• Staging does not need it. Staging is protected at the edge by nginx basic auth (real auth, no bundle leak); the overlay would only add a second prompt.

# gate creddit.xyz
echo 'NEXT_PUBLIC_ACCESS_CODE=<code>' >> /opt/onchain-credit/.env.local
cd /opt/onchain-credit && npm run build && pm2 restart onchain-credit --update-env
# ungate: remove the line, rebuild, restart

Security response headers

Set centrally in next.config.ts (an async headers() block applied to /:path*, so every route including API routes and static assets gets them). There is no middleware.ts; keeping this in one place means the app ships secure headers with no per-route wiring. The same file sets poweredByHeader: false, which removes the default x-powered-by: Next.js response header so we do not advertise the framework/version.

Header	Value	Why
Strict-Transport-Security	max-age=63072000; includeSubDomains; preload	Force HTTPS for two years, including subdomains; preload-eligible. Only sent over HTTPS, so it is inert on local http://localhost.
X-Frame-Options	DENY	Block framing (clickjacking). Reinforced by the CSP frame-ancestors 'none'.
X-Content-Type-Options	nosniff	Stop MIME sniffing.
Referrer-Policy	strict-origin-when-cross-origin	Send only the origin (not the path/query) on cross-origin navigations.
Permissions-Policy	camera=(), microphone=(), geolocation=(), browsing-topics=()	Deny powerful features the app never uses, and opt out of the Topics API.
Content-Security-Policy-Report-Only	see below	Observe-and-report CSP. Does not block anything yet (see the rollout note).

CSP ships Report-Only first. The policy is emitted as Content-Security-Policy-Report-Only, not the enforcing Content-Security-Policy. In Report-Only mode the browser evaluates the policy and logs every violation but never blocks the resource, so we can watch real traffic and tune the policy before it can break a page. The current policy (assembled from the per-directive list in next.config.ts) is:

default-src 'self'; base-uri 'self'; object-src 'none'; frame-ancestors 'none'; form-action 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data: https:; font-src 'self' data:; connect-src 'self' https: wss:; manifest-src 'self'

The looser-than-'self' directives are deliberate and documented inline in next.config.ts: script-src 'unsafe-inline' (Next.js bootstrap scripts + inline JSON-LD from src/components/seo/JsonLd.tsx), style-src 'unsafe-inline' (Recharts injects an inline <style> in src/components/ui/chart.tsx + Next.js inline styles), img-src data: https: (inline icons + remote logos), font-src data: (self-hosted + data-encoded glyphs), and connect-src https: wss: (same-origin fetch/stream + the injected window.ethereum wallet, which opens its own RPC/WebSocket connections client-side). Third-party price/aggregator/RPC APIs (KyberSwap, Pendle, Ethereum RPC) are called from the server, so they are not connect-src entries.

Reading violation reports. With no report-uri/report-to endpoint configured, reports surface in the browser only: open DevTools and look for Content-Security-Policy-Report-Only warnings in the Console (each names the blocked directive and resource), or the Network panel. Exercise the real pages while watching (home, an asset profile with charts, /portfolio, /carries, /agent while a chat streams, and connect a wallet) so injected wallet/chart/JSON-LD behaviour is covered. If a first-party resource trips the policy, widen the specific directive; do not blanket-add hosts.

Flipping to enforcing. Once the console is clean across the surfaces above, change the header key in next.config.ts from Content-Security-Policy-Report-Only to Content-Security-Policy (value unchanged), deploy, and re-verify. Do this as its own change so a regression is easy to bisect and revert.

Verify after a deploy:

# prod edge: expect the six headers present and NO x-powered-by
curl -sI https://creddit.xyz/ | grep -iE 'strict-transport|x-frame|x-content-type|referrer-policy|permissions-policy|content-security|x-powered-by'

x-powered-by must be absent; content-security-policy-report-only (not content-security-policy) must be present until the flip. Note the edge (Cloudflare) may add or normalise some headers; the origin values above are what the app emits.

5. Verifying a deploy

After a push to main (or a manual deploy):

1. CI is green. Watch the run on GitHub Actions ("Deploy to production"); the final log line is Deployed <short-sha> at <iso-ts>. A red run that ends in Rolled back to <sha> means the build failed and the previous commit is live — fix and re-push.
2. Process is up and on the new code.

   pm2 list                                 # onchain-credit "online", restart count bumped
   ssh root@dexhq.io 'cd /opt/onchain-credit && git rev-parse --short HEAD'

   The short SHA must match the commit you deployed.
3. App responds locally + DB reachable.

   ssh root@dexhq.io 'curl -sI http://localhost:3001/ | head -1'          # expect HTTP 200
   ssh root@dexhq.io 'curl -s http://localhost:3001/api/health'           # expect {"ok":true,"db":"up",...}

   /api/health is a 200 + SELECT 1 probe; the deploy workflows curl it (staging today, prod once trusted) and fail the deploy if it is not healthy.
4. Edge serves it. Load https://creddit.xyz and the changed page. Remember ISR: if you also ran a refresher, the page may show the pre-refresh snapshot until the revalidate window elapses — re-deploy to force a re-prerender.
5. No errors in logs. pm2 logs onchain-credit --lines 100.
6. Data freshness (if relevant). Spot-check max(snapshot_ts) on the affected table via psql to confirm a manual refresher actually wrote.
7. End-to-end smoke. The Playwright suite already ran against staging on this exact code (§3.1). It reports, it does not gate: read its result before opening the release PR, but nothing blocks the merge on it. Nothing runs automatically against prod. To point the suite at https://creddit.xyz by hand (read-only page loads, but still prod traffic), get explicit permission first; the target is E2E_BASE_URL (set it and playwright.config.ts boots no local server). Prod is served on the public hostname with no basic auth in front of it, so no credentials are involved; to point the suite at a firewalled app port on either box, tunnel to it as the staging smoke does (§3.1). Full procedure: Processes → Verifying a UI change.

External dependencies

The deploy itself only needs GitHub + the box. The running app and its refreshers depend on (configured in .env.local, preserved across deploys):

Dependency	Used for	Config
Ethereum RPC (current state)	live on-chain reads	ETHEREUM_RPC_URL (fallback publicnode https://ethereum-rpc.publicnode.com)
Ethereum RPC (archive)	historical/backfill block reads	ETHEREUM_ARCHIVE_RPC_URL (fallback dRPC https://eth.drpc.org)
DefiLlama Coins API	USD prices; basis market price	src/lib/data/llama-prices.ts
NY Fed	SOFR rates	refresh-sofr.ts
Dune API	analytics (scarce credits — inspect before executing)	—
Morpho Blue GraphQL + Euler Goldsky subgraph	curator / market data	—
Telegram bot (cron alerting)	WS8 failure + unknown-asset alerts	ALERT_TG_BOT_TOKEN, ALERT_TG_CHAT_ID (both unset = alerting off); optional ALERT_TG_API_BASE (default https://api.telegram.org), ALERT_ENV (env label override)

RPC access goes through src/lib/data/rpc.ts. Token-basis redemption rate comes from the token_yield_apy table; market price for basis from DefiLlama.

Related

• Architecture — tech stack, project layout, full schema.
• Data Pipeline — the refreshers, cadences, vault-coverage sync, backfills.
• Database & schema — the canonical APY math and readers.

6. AI assistant (Creddit Agent)

The AI research assistant (Creddit Agent) is an in-app chat surface (/agent, with /chat redirecting to it) backed by /api/chat* routes that stream a tool-using Claude response over creddit's own data (carries, repo rates, capacity, oracles, basis, assets). It is code + schema + env: the code and migrations ship through the normal pipeline; the secrets and the nginx streaming block are box-local manual steps.

AI assistant env

Set in each environment's .env.local (preserved across deploys; next start re-reads it on pm2 restart). The feature stays dark (route returns 503) until CHAT_ENABLED is truthy and ANTHROPIC_API_KEY is set, so it can ship to staging before the key is wired in.

Var	Purpose
ANTHROPIC_API_KEY	Model access (secret). Never in the repo.
CHAT_ENABLED	Kill switch (1 to enable).
CHAT_MODEL	Model id (default claude-sonnet-5).
SESSION_SECRET	HMAC key for the account session cookie (creddit_session). Generate 32+ random bytes; required for sign-in. Falls back to CHAT_SESSION_SECRET if unset, so an existing server keeps working.
CHAT_SESSION_SECRET	Legacy name for the session-cookie HMAC key; still honored as the SESSION_SECRET fallback.
CHAT_DAILY_TOKEN_BUDGET	Per-user daily token cap (default 250000).
CHAT_GLOBAL_DAILY_TOKEN_BUDGET	Whole-app daily token cap (default 5000000).
SIWE_DOMAIN	Required in production. Pins the SIWE domain (creddit.xyz in prod, staging.creddit.xyz in staging) so the domain binding never derives from the client-controlled Host header, which closes a phishing-relay / Host-spoofing account-takeover path (a signature captured on an attacker origin cannot be replayed against this server). Falls back to CHAT_SIWE_DOMAIN. If unset, the request Host is trusted only when it is in a built-in allowlist of known creddit hosts (a safety net, not a substitute for pinning); any Host outside that allowlist fails verification. See Origin & SIWE hardening.
CHAT_SIWE_DOMAIN	Legacy name for the SIWE domain pin; still honored as the SIWE_DOMAIN fallback.

Migrations

038-chat-assistant.sql + 039-chat-profile.sql are additive and auto-apply on the staging deploy (migrate.sh); on prod run migrate.sh by hand after the release, per §4 Migrations.

044-portfolio-fluid.sql (FWS3). Read before promoting. It adds a leg column to portfolio_flow_events and swaps the PK to (chain_id, wallet, tx_hash, log_index, leg) (guarded DO $$ block, so a re-run is a no-op), and creates fluid_event_log + fluid_event_coverage. It is a coordinated change, NOT plain expand/contract: this release's three flow writers (writeFlows, writeFlowRows, writeBackfillFlows) reference the leg column and the 5-column ON CONFLICT, and 044 is what makes the schema accept them. Because deploy.yml is a single atomic ssh script (git reset --hard FETCH_HEAD; npm ci && npm run build; pm2 restart), the 044 SQL file arrives on the box with the new code, so a strict "migrate before the code deploys" order is not achievable in this pipeline. Both directions of the coordination therefore carry a hazard; the honest handling is to run the migration immediately after the deploy and to accept the rollback constraint, not to pretend an impossible ordering.

FORWARD (deploy, then migrate). Between the moment the new code restarts and the moment you run migrate.sh 044, EVERY flow insert fails for EVERY venue, because the new writers reference a schema that does not exist yet. The two failure signatures (both reproduced on a scratch DB): column "leg" ... does not exist against a fully un-migrated schema, and there is no unique or exclusion constraint matching the ON CONFLICT specification if leg exists but the PK swap has not run. This window is LOUD (the cron-failure Telegram alert fires), ATOMIC and non-corrupting (a rejected insert writes nothing; the snapshot/rate writes in the same tick are unaffected), and SELF-HEALING (the first 6h cron tick after 044 is applied writes normally). The JIT live path degrades to stored history (jit:false) while the window is open. So run migrate.sh 044 IMMEDIATELY after the deploy completes, before the next 6h refresh-portfolio.ts tick, to keep the window as short as possible.

ROLLBACK (code rollback after 044 is applied). The previous release's flow writers use the 4-column ON CONFLICT, which no longer matches the 5-column PK, so they are HARD-BROKEN and Postgres rejects every flow insert for ALL venues until the code rolls forward again. This is ACCEPTED (staging first; the two-release expand-then-contract sequencing was considered and rejected as not worth the delay for a staging-gated feature). A rollback that must restore flow writes has to revert 044 as well, or hotfix forward.

Manual server steps, in order (run as the deploy operator, right after the prod deploy restarts):

1. Apply migration 044 (idempotent; a no-op on a re-run):

   cd /opt/onchain-credit && scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)"

2. Run the one-time D7 deploy seed (fills the trailing-90d fluid_event_log cache; ~648 chunks, ~11 min, attended, free tier):

   cd /opt/onchain-credit && ETHEREUM_ARCHIVE_RPC_URL=<archive> \
     DATABASE_URL="$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)" \
     npx tsx scripts/seed-fluid-event-log.ts        # --days 90 default

The 6h refresh-portfolio.ts cron appends to the cache thereafter; registration backfills read it and never re-scan the chain. List both (the migrate.sh 044 run and the seed) in the release PR body.

048-account-wallets.sql (portfolio multi-wallet)

Additive and idempotent, so it auto-applies on the staging deploy and is the usual gated manual migrate.sh run on prod after the release. Unlike 044, it needs no special ordering: this release's code is written to run against the pre-048 schema.

That is deliberate, and it is what makes 048 boring. The deploy ships code before schema (staging builds and restarts before migrate.sh; prod migrations are a manual step after the release deploy), and every /api/portfolio/* route resolves its wallet through account_wallets. Left untreated, that window would have taken the whole portfolio surface down with a 500, not just the multi-wallet parts. So the three read paths tolerate a missing table (42P01) and degrade to exactly the pre-multi-wallet behaviour:

Path	Pre-048 behaviour
resolveWalletSelection (all five portfolio routes)	The account sees its own wallet. ?wallet=<other> is still a 403, never a 500.
GET /api/portfolio/wallets	Returns the account's own wallet, alone.
loadEligibleWallets (the 6h cron)	Falls back to the pre-048 predicate with a loud console.warn, so a tick landing in the window does not punch a hole in every wallet's history.
/api/auth/verify -> ensureSelfWalletRow	Inside the route's existing best-effort block: logged and swallowed, so sign-in is unaffected.

Only "relation does not exist" is swallowed. A permission error or a syntax error still throws, so a real fault cannot masquerade as "you have one wallet" (verified against a pre-048 database, and pinned by unit tests).

After migrate.sh runs, the next request and the next cron tick pick up the new behaviour with no restart. The scrub must go with it: scrub-staging-pii.sql now TRUNCATEs account_wallets in the same statement as accounts (Postgres refuses to truncate an FK-referenced table unless every referrer is in the same statement), so a staging box running the new SQL against an old checkout, or vice versa, is the one combination to avoid. Migration 056 widens this: it adds ON DELETE CASCADE FKs from the chat tables and portfolio_wallet_index to accounts, so once 056 is applied the reseed's TRUNCATE accounts requires the chat tables and portfolio_wallet_index in the same statement too — which the shipped scrub already does. Apply 056 with migrate.sh (it first deletes any orphaned uid/wallet rows so the constraints validate) and keep the scrub on the box in lockstep; an old scrub against a 056 database fails the reseed closed (safe, but it blocks the nightly reseed until the checkout is current).

nginx (streaming)

The chat response is a long-lived stream, so its location must disable proxy buffering. Add to each vhost (onchain-credit and onchain-credit-staging), above the catch-all location /:

location /api/chat {
    proxy_pass http://127.0.0.1:3001;   # 3002 on staging
    proxy_http_version 1.1;
    proxy_buffering off;
    proxy_cache off;
    proxy_read_timeout 300s;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
}

Then nginx -t && nginx -s reload. Verify tokens arrive incrementally (not one flush) through Cloudflare.

Access

Sending a message requires wallet sign-in (Sign-In with Ethereum; injected wallets only in stage 1). Signing costs nothing (a message signature). The chat page and history are viewable without signing in; the per-address + global daily token budgets are the real spend guard.