Release steps (one-time)

Every first-deploy note and every gated repair this project has written, verbatim, with the migration or release it belongs to and whether it has run.

This page is a log, not a runbook. Each step below belongs to ONE release and runs once per environment. The procedure that is true on any day — topology, shipping, the deploy pipeline, staging, versioning, the migrations procedure, the crontab, backup, verification and operations — is in Deployment & server ops, and one-time steps never go back there: scripts/runbook-split.test.ts fails the build if one does, and fails if an entry here has no executed: status.

Reading the `executed:` field

Value	Means
`YYYY-MM-DD`	the step ran on prod on that date, and the repo proves it
`pending`	the migration or release it belongs to has not shipped yet
`outstanding`	the release SHIPPED, and the repo shows the gated prod step has not been taken
`unverified`	it probably ran, and nothing in the repository says so

pending and outstanding are different questions: a gated step can be waiting on an operator long after its code is live on prod, and calling that pending would read as "not released yet" and calling it unverified would throw away evidence the repository actually holds.

unverified is the honest default, not a shrug: prod's onchain_credit.schema_migrations is the only authority on which migrations have been applied, and a repository cannot see it. To settle one, read the ledger on the box:

bash

sudo -u postgres psql -d creddit -c \
  "SELECT filename, applied_at FROM onchain_credit.schema_migrations ORDER BY filename"

and set the date here. A gated repair leaves no ledger row at all; its evidence is the report the run printed, which each step below names.

Per-migration first-deploy steps

Ordered by migration number, which is the order they were applied.

AI assistant migrations `038` / `039`, and the `044` coordinated migration

belongs to: migrations 038 / 039 / 044 (chat assistant, chat profile, portfolio Fluid) — v0.3.0
executed: unverified

This is the ONE heading on this page that was changed beyond a level shift. In the runbook it read simply Migrations, under the assistant section that gave it its subject; standing alone in a log it named nothing, and it collided with the runbook's own evergreen Migrations section, which would have made one of the two resolve as #migrations-1. The body below is verbatim.

038-chat-assistant.sql + 039-chat-profile.sql are additive and auto-apply on the staging deploy (migrate.sh); on prod run migrate.sh by hand after the release, per §4 Migrations.

044-portfolio-fluid.sql (FWS3) is spent, but it still RUNS. It widened portfolio_flow_events and created fluid_event_log + fluid_event_coverage; migration 095 drops all three, and Fluid positions are derived from the ingested event store like every other venue (the retirement runbook). The two files are tagged differently, so a new environment gets both halves and keeps the first: 044 carries no -- DESTRUCTIVE tag, so scripts/ops/migrate.sh and scripts/fixture/build.sh apply it like any other pending file, while 095 carries the tag on line 1, so the fixture build always skips it and migrate.sh skips it until a hand-run --allow-destructive. A fresh database therefore holds the three relations, unread and unwritten, until somebody runs the drop by hand.

`048-account-wallets.sql` (portfolio multi-wallet)

belongs to: migration 048 (portfolio multi-wallet) — v0.9.0
executed: unverified

Additive and idempotent, so it auto-applies on the staging deploy and is the usual gated manual migrate.sh run on prod after the release. Unlike 044, it needs no special ordering: this release's code is written to run against the pre-048 schema.

That is deliberate, and it is what makes 048 boring. The deploy ships code before schema (staging builds and restarts before migrate.sh; prod migrations are a manual step after the release deploy), and every /api/portfolio/* route resolves its wallet through account_wallets. Left untreated, that window would have taken the whole portfolio surface down with a 500, not just the multi-wallet parts. So the three read paths tolerate a missing table (42P01) and degrade to exactly the pre-multi-wallet behaviour:

Path	Pre-048 behaviour
`resolveWalletSelection` (all five portfolio routes)	The account sees its own wallet. `?wallet=<other>` is still a 403, never a 500.
`GET /api/portfolio/wallets`	Returns the account's own wallet, alone.
`loadEligibleWallets` (the 6h cron)	Falls back to the pre-048 predicate with a loud `console.warn`, so a tick landing in the window does not punch a hole in every wallet's history.
`/api/auth/verify` -> `ensureSelfWalletRow`	Inside the route's existing best-effort block: logged and swallowed, so sign-in is unaffected.

Only "relation does not exist" is swallowed. A permission error or a syntax error still throws, so a real fault cannot masquerade as "you have one wallet" (verified against a pre-048 database, and pinned by unit tests).

After migrate.sh runs, the next request and the next cron tick pick up the new behaviour with no restart. The scrub must go with it: scrub-staging-pii.sql now TRUNCATEs account_wallets in the same statement as accounts (Postgres refuses to truncate an FK-referenced table unless every referrer is in the same statement), so a staging box running the new SQL against an old checkout, or vice versa, is the one combination to avoid. Migration 056 widens this: it adds ON DELETE CASCADE FKs from the chat tables and portfolio_wallet_index to accounts, so once 056 is applied the reseed's TRUNCATE accounts requires the chat tables and portfolio_wallet_index in the same statement too — which the shipped scrub already does. Apply 056 with migrate.sh (it first deletes any orphaned uid/wallet rows so the constraints validate) and keep the scrub on the box in lockstep; an old scrub against a 056 database fails the reseed closed (safe, but it blocks the nightly reseed until the checkout is current).

Portfolio taxonomy: first deploy (migrations 049 / 050 / 052)

belongs to: migrations 049 / 050 / 052 (portfolio taxonomy) — v0.10.0
executed: unverified

The taxonomy release ships five new cron jobs, three migrations and two exhaustive universe ingests. The ordering below is not cosmetic — each step is what makes the next one correct.

Migrate first: 049, 050, 052. Additive and idempotent. refresh-portfolio.ts degrades gracefully if 049/052 are missing (empty wallet universe, static erc4626 set) but the WS5 backfill refuses to run against a degraded registry rather than mis-reading a wallet-venue-only holder as empty and pruning its history — so until these are applied, backfills abort loudly. That is intentional.
bash
```
cd /opt/onchain-credit && scripts/ops/migrate.sh creddit    # BARE DB NAME
```
Deep-ingest the two universes, once, by hand, before adding their crons. The first run walks the whole history and takes far longer than a daily tick:
bash
```
MORPHO_UNIVERSE_FROM_BLOCK=<morpho-blue-deploy>   scripts/run-cron.sh refresh-morpho-universe.ts
METAMORPHO_FACTORY_FROM_BLOCK=<v1.0-factory-deploy> scripts/run-cron.sh refresh-metamorpho-factory.ts
```
Steady-state runs scan only new blocks from the cursor, so the env overrides are for the FIRST run only.
One-time full reconcile over every seeded wallet. Required, and easy to miss. Migration 050 seeds a discovery watermark for every historically-active wallet — which authorises the loader to BOUND that wallet's reads. (Since the coverage hardening the authorising artifact is the CERTIFICATE on accounts, migration 061, and it must also be FRESH; the reasoning below is unchanged.) But the seed is derived from pre-T4 history, when the flow scans only admitted CURATED markets/vaults, so it cannot contain a wallet's dormant position in a previously-uncurated market, vault or fToken. Those positions now exist in the universe but not in the index. The weekly reconcile finds them eventually, but at PORTFOLIO_RECONCILE_MAX=50 LRU that is "within N weeks", not "within a week" — and each one fires a drift alert whose remediation text ("investigate discovery.ts") is misleading, because this is expected deploy-time state, not a discovery bug. So after step 2:
bash
```
PORTFOLIO_RECONCILE_MAX=100000 scripts/run-cron.sh refresh-portfolio-reconcile.ts

> Since the coverage hardening this run **exits non-zero and pages** if it finds any aged divergence or dangling link. On the one-time flush that is expected: the whole point is to find and repair pre-existing drift. Read the first alert as the flush's REPORT, not as a cron failure.
```
Expect drift rows on this run (that is the point) and none afterwards. The alternative is to drop every seeded watermark and let the /10min enrollment drain re-enroll against the widened universe; either is fine, doing neither is not.
Add the crontab entries (the table is in the runbook: Running a cron/refresher by hand).
Re-run the deploy so the prerendered pages build against the now-complete data (see the ISR note in the runbook).

Migration `061` (discovery completeness certificates)

belongs to: migration 061 (discovery completeness certificates) — v0.21.0
executed: unverified

Additive and idempotent, so staging auto-applies it; prod stays the gated manual step and must run before the release merge. It adds accounts.discovery_scanned_block / discovery_scanned_at / discovery_reconciled_at and copies the surviving portfolio:discover:<wallet> scope rows onto them, taking chain_scan_cursors.updated_at and never now() (a now() stamp would launder every currently-poisoned wallet as freshly certified, which is the whole point of the migration).

Expect one transient after it runs, and it is by design. The copied stamps are as old as the migration-050 seed for most wallets, i.e. far past the 18h staleness horizon, so the whole population is DEMOTED on the first read afterwards. Because the discovery bound fails open for the whole batch, one un-fresh wallet puts every wallet on a full-universe read: expect the 6h tick to log discovery FULL (unscanned wallets) and take measurably longer for a tick or two. It resolves itself as the */10 enrollment drain certifies wallets (PORTFOLIO_ENROLL_MAX, default 10 per run), and the reads are correct throughout, just slower. To shorten it, run the drain by hand right after the migration:

bash

cd /opt/onchain-credit && PORTFOLIO_ENROLL_MAX=50 scripts/run-cron.sh refresh-portfolio-discovery.ts

Deploy order was tolerant in both directions when this shipped: before the migration the readers fell back to the legacy scope-row presence test (whole-batch, on a 42703), and the enrollment / reconcile / backfill writers dual-wrote the scope row so a code rollback still found a current watermark. Both shims are gone and migration 106 deletes the scope rows.

Migrations `062` / `063` (coverage hardening, additive)

belongs to: migrations 062 / 063 (coverage hardening) — v0.21.0
executed: unverified

062 adds a (chain_id, snapshot_ts DESC) index on portfolio_position_snapshots (the 6h tick's derived population asks "which wallets have any snapshot in the last 48h?", and every existing index leads with (chain_id, wallet, …)). 063 adds portfolio_backfill_state.covered_through_ts / covered_through_block, the durable coverage anchor.

Both are additive, idempotent and inert under a code rollback, so staging auto-applies them and prod is the usual gated manual step. Neither has a transient, unlike 061. But 063 is a HARD PREREQUISITE FOR THE DEPLOY, not a post-deploy step: the add-wallet path's requeue statement references covered_through_ts, and although it falls back to the pre-063 predicate on a 42703, the fallback costs a round trip on every add until the migration lands. Apply 063 BEFORE the release merge (the v0.9.0 lesson). with no 063 columns loadCoverageAnchor returns null and every backfill takes the fresh-replay path it takes today. Note the consequence: no wallet gets a gap patch until it has completed one run after 063, because an anchor is only written on a terminal done (or advanced by a tick). That is intentional — a wallet whose last completed replay predates the column has no trustworthy seam to patch from.

Portfolio 100k scale: Phase C envs + the snapshot repartition (migration 067)

belongs to: migration 067 (snapshot repartition) — v0.25.0
executed: unverified

New envs (all optional, safe defaults; set in .env.local on the box):

Env	Default	Effect
`BACKFILL_WORKERS`	`4`	concurrent wallet backfills per drain invocation (D3), each a separate child process
`BACKFILL_GRID_CONCURRENCY`	`6`	concurrent per-grid-point archive reads within one replay child (D2)
`WALLET_TOPIC_CHUNK`	`500`	wallets per `getLogs` owner-topic pass in the registration probe's chain scans (D5)
`PORTFOLIO_WRITE_LOCK_WAIT_MS`	`600000`	how long a history writer may QUEUE for the shared advisory key before its acquire is cancelled. Raised for the acquire only; the role's `statement_timeout` is handed back before any write runs under it. Keep it below `CHILD_TIMEOUT_MS` (30 min) so a pathological queue reports a failure instead of being SIGKILLed. A typo falls back to the default, never to "wait forever".
`DUNE_RATE_LIMIT_COOLOFF_MS`	`60000`	how long a Dune 429 pauses this process's price-mirror executions (the rate-limit breaker). `0` disables it.
`JIT_LEDGER_LAG_BLOCKS`	`40`	the JIT POSITIONS fast path's freshness gate (D4b). The default keeps it OFF — the ingester's 64-block margin means the tip trails head by ≥ 64, which is not within 40, so a page load full-reads positions. Raise it above ~64 (accepting up to that-many blocks of position staleness, self-healing on the next refresh) to opt in. Production has since opted in (`200`); the current setting is documented in Environment variables. It gates no WRITE: the page load's ledger merge is unconditional and clamps its own top to how far ingestion has finished.

Effective archive concurrency = BACKFILL_WORKERS × BACKFILL_GRID_CONCURRENCY (default 4 × 6 = 24 concurrent grid reads, each firing several multicall/block requests → ~100–300 concurrent archive requests worst case). The two knobs MULTIPLY: the grid cap is per replay child, the worker pool runs that many children as separate processes, and there is no shared in-process RPC semaphore (finding 7). Size them against a provisioned high-throughput archive endpoint (Alchemy ETHEREUM_ARCHIVE_RPC_URL); on a free-tier archive default the burst rate-limits and grid reads fail → M9 abort → requeue → the same burst next tick. Raising BACKFILL_GRID_CONCURRENCY in isolation is a hidden ×BACKFILL_WORKERS.

The D1 registration-probe discovery and the D4b JIT positions fast path are gated by event_coverage being live back to the read window, so they self-disable until the one-time backfill + ingester have caught the ledger up. The probe falls back to a chain scan when coverage is short; the positions path falls back to a full read.

Snapshot repartition (migration 067, -- DESTRUCTIVE, MANUAL). 067 repartitions portfolio_position_snapshots into monthly range partitions. It is tagged -- DESTRUCTIVE, so the staging auto-apply and a plain migrate.sh SKIP it; it runs only under --allow-destructive. It renames the live table aside to portfolio_position_snapshots_preswap — including its index names, because a table rename leaves those behind and CREATE INDEX IF NOT EXISTS matches on name, so without that the new parent would silently end up with nothing but its PK — builds the partitioned replacement (identical columns / PK / indexes / FK / grants — the PK already carries snapshot_ts, so no conflict target changes), copies every row, and KEEPS _preswap for a manual DROP. Run it while the spine is near-empty (post-reset, the plan's clean-slate timing). The writers auto-create future month partitions at runtime (ensureMonthPartitions, a safe no-op before this migration). portfolio_flow_events is partitioned separately and differently, BY HASH (wallet), by migration 072 below. See the migration header and docs/database.md.

Runbook (staging first, then prod after a soak):

bash

# 0. PAUSE THE NIGHTLY RESEED for the whole verification window (see the note below):
touch /root/.reseed-paused
# 1. Staging — apply the additive Phase C migrations (064/065/066 already; nothing new
#    additive in C), then apply 067 EXPLICITLY with --allow-destructive:
cd /opt/onchain-credit-staging && \
  scripts/ops/migrate.sh "postgresql://onchain_credit_staging@127.0.0.1:5432/creddit_staging" --allow-destructive
# 2. Verify: row counts match, the parent carries its FIVE indexes (4 + PK) and the month
#    bounds are exact UTC midnights, /portfolio spot-check, then drop the aside BY HAND:
#    psql> SELECT count(*) FROM onchain_credit.portfolio_position_snapshots;         -- new
#    psql> SELECT count(*) FROM onchain_credit.portfolio_position_snapshots_preswap; -- old (equal)
#    psql> SELECT c.relname FROM pg_class c JOIN pg_index i ON i.indexrelid = c.oid
#           WHERE i.indrelid = 'onchain_credit.portfolio_position_snapshots'::regclass;
#          -- expect: portfolio_pos_wallet_ts_idx, portfolio_pos_chain_ts_idx,
#          --         portfolio_pos_pendle_ptkey_idx, portfolio_pos_accounting_asset_idx,
#          --         portfolio_position_snapshots_pkey  (anything missing is missing FOREVER
#          --         once _preswap is dropped: migrate.sh has recorded 067)
#    psql> SELECT c.relname, pg_get_expr(c.relpartbound, c.oid) FROM pg_class c
#           JOIN pg_namespace n ON n.oid = c.relnamespace
#           WHERE n.nspname = 'onchain_credit' AND c.relname LIKE 'portfolio_position_snapshots_2%';
#          -- expect 00:00:00+00 boundaries; anything else means the month arithmetic and the
#          -- runtime helper's UTC literals disagree and will collide at the next rollover
#    psql> DROP TABLE onchain_credit.portfolio_position_snapshots_preswap;
# 3. Prod (gated on Fred's approval; run while the spine is near-empty):
cd /opt/onchain-credit && \
  scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)" --allow-destructive
#    Verify + DROP _preswap by hand exactly as on staging.
# 4. Only after the _preswap table is dropped on BOTH environments, resume the reseed:
rm -f /root/.reseed-paused

Because 067 renames + swaps inside migrate.sh's single transaction, a failure rolls the whole thing back and leaves the original table untouched — it is atomic. A re-run detects the table is already partitioned and no-ops.

Why step 0 exists. RENAME carries a table's constraints, so portfolio_position_snapshots_preswap keeps migration 056's FK to accounts plus a full copy of every wallet's history. Prod backs up at 02:15 UTC and reseed-staging.sh restores that dump at 03:00, so an aside table left standing overnight rides into staging — and the scrub's single TRUNCATE then fails (cannot truncate a table referenced in a foreign key constraint), aborting the reseed before its fail-closed leak checks and leaving staging serving the unscrubbed prod database, every night, until the aside table is dropped. portfolio_position_snapshots_preswap is listed in scripts/ops/scrub-staging-pii.sql and in the reseed's leak check (to_regclass-guarded, so it is a no-op when absent), which fixes it properly; the pause is the belt-and-braces half. scripts/ops/scrub-staging.test.ts keeps the two lists in lockstep, and portfolio_flow_events_preswap was pinned alongside it pre-emptively — migration 072 now produces exactly that table, so the fix was already in place when the flow repartition landed rather than being rediscovered by a failed reseed.

Partition maintenance after 067. Runtime creation is a backstop, not the primary path: the 6h cron pre-creates the CURRENT and NEXT month's partition for the spine, so the rollover fires from a quiet non-request process rather than from whichever writer happens to produce the first row of a new month. CREATE TABLE … PARTITION OF takes ACCESS EXCLUSIVE on the parent and Postgres's lock queue is FIFO, so the DDL runs under a 2s lock_timeout — queued behind one slow reader it gives up rather than parking every later /portfolio SELECT behind itself, and the next writer retries. Nothing to do operationally.

Applying 067 under running writers. The 6h tick, the minutely drain's children and the app all cache "is this table partitioned?" per process, and every one of them answered no before you ran the migration. That answer is NOT pinned: a negative expires after 60s, so a process that started before the migration begins creating month partitions within a minute, without a restart. A 23514 no partition of relation from a write also invalidates the cache immediately, which shortens that window for the writers whose caller survives the error (a drain child moves to the next wallet in-process); the 6h tick instead ends on the throw, so there the TTL is the whole mechanism and the cost is at most one missed snapshot window. Nothing to do operationally; noted because the failure it prevents is quiet (a snapshot missing beside a live balance reads as yield).

Flow-ledger repartition: hash by wallet (migration 072)

belongs to: migration 072 (flow-ledger repartition, hash by wallet) — v0.27.0
executed: unverified

This procedure is spent: it targets portfolio_flow_events, which migration 095 dropped on prod on 2026-09-05 (the retirement runbook). 072 carries a -- DESTRUCTIVE tag on line 1, so the fixture build always skips it and migrate.sh skips it until a hand-run --allow-destructive — which nobody should now give it. What follows is the record of a repartition that already ran, kept for the next one: the deploy-order hazard and the pruning law are what generalise, not the steps.

What it did. 072 rebuilds onchain_credit.portfolio_flow_events as PARTITION BY HASH (wallet) with MODULUS 16 — 16 partitions (portfolio_flow_events_h00 … h15), created ONCE by the migration because a hash modulus is fixed at CREATE time. ~6,250 wallets per partition at the 100k target. It renames the live table aside to portfolio_flow_events_preswap (including its index names — same trap as 067: a table rename leaves index names behind and CREATE INDEX IF NOT EXISTS matches on name, so without that step the new parent would end up carrying nothing but its PK), builds the partitioned replacement with identical columns / PK / CHECKs / the five indexes / the 056 FK / grants, copies every row, and KEEPS _preswap for a manual DROP. Tagged -- DESTRUCTIVE, so the staging auto-apply and a plain migrate.sh SKIP it; it runs only under --allow-destructive.

⚠ DEPLOY + ROLLBACK WARNING: the order is ONE-WAY. Ship the code release first, then apply 072. That release is a rollback floor for as long as this table is hash-partitioned. The PK genuinely does not change — wallet is already the second column of (chain_id, wallet, tx_hash, log_index, leg), so the partition key is inside the unique constraint Postgres requires it in and all three writers' 5-column ON CONFLICT targets match it exactly, before and after — but that buys the forward direction only:

Forward (this release's code, table not yet partitioned) — FREE. The only code change in the release is the DELETION of the five ensureMonthPartitions calls naming this table — one in live.ts, one in refreshers/portfolio.ts and three in backfill.ts, all of them the flow writers of the day (every one of those writers has since been deleted outright with the old engine) — all no-ops while the table was un-partitioned, plus the helper's partstrat guard. Nothing new is referenced, so the new code is correct against the old table for as long as it takes to get to the migration.
Backward (the previous release's code, table already partitioned) — HARD-BROKEN. Every flow read and write is fine on its own; it names the parent, never a partition. What breaks is the vestigial DDL call in front of the write. The previous release's ensureMonthPartitions gates on relkind = 'p' alone, so 072 flipping the parent to 'p' stops those five call sites no-opping and starts them issuing CREATE TABLE … PARTITION OF <hash parent> FOR VALUES FROM (…) TO (…), which Postgres rejects: 42P16 invalid bound specification for a hash partition. It never self-heals (isMissingPartitionError requires 23514, so the self-heal path declines 42P16, and the positive relkind memo is permanent for the process lifetime; a restart re-probes and gets 'p' again). Verified end-to-end on PG 16 against the deployed helper.
Blast radius, loud on one half and silent on the other. This describes the ROLLBACK TARGET, i.e. the pre-072 build, whose tick committed its snapshots before the first writeFlows — so it advances balances and then throws on the flow write, the cron alert fires, and a balance standing with no flow behind it reads as yield (the M3 violation). (A current build cannot reach that half-state at all: the tick publishes both halves in one transaction, so a failing flow write rolls the snapshots back with it. The warning is about what redeploying an older build does, and that is unchanged.) live.ts writeFlowRows sits inside the JIT try whose catch only logs mini flow scan failed, so the JIT flow persist and the provisional self-clean fail with no alert on every page load. All three backfill.ts call sites are pool-side, so they throw before their transaction ever opens.

Same class as 044 (hard-coupled, though 044 was coupled in both directions); unlike 067, whose forward direction needed the helper's memo semantics. This matters operationally because the bad order is reachable without an operator decision: deploy.yml reverts the tree with git reset --hard $PREV on an npm ci / npm run build failure (a documented recurring flake here), which lands the pre-072 code on a hash parent by itself. If the accompanying deploy fails its build, roll forward (re-run the deploy) rather than leaving the tree on $PREV. This is the same rule as the general one in the runbook's migrations section: the deploy's code-rollback path does not roll back the DB.

Pick a quiet window as well: the copy is a single statement under migrate.sh's statement_timeout=300s, and the rename needs ACCESS EXCLUSIVE within lock_timeout=10s.

Runbook (staging first, then prod after a soak):

bash

# 0. PRECONDITION — the DEPLOYED code must already be the release that ships 072. The
#    previous release's flow writers call ensureMonthPartitions behind a relkind-only probe,
#    so the moment the parent becomes hash-partitioned they emit month bounds at it and
#    EVERY flow write fails 42P16 (see the warning above). Confirm the box is running the
#    right build BEFORE touching migrate.sh — the partstrat guard is the marker:
#      cd /opt/onchain-credit-staging && git log -1 --format='%h %s'
#      grep -c partstrat src/lib/portfolio/partitions.ts      # must be > 0
#      grep -c 'portfolio_flow_events' <(grep -rh 'ensureMonthPartitions(' \
#        src/lib/portfolio scripts/refreshers)                # must be 0
#    If that fails, deploy first. If a deploy build FAILED and deploy.yml reverted the tree
#    to $PREV, re-run the deploy (roll forward) — do not migrate against $PREV.
# 0a. PAUSE THE NIGHTLY RESEED for the whole verification window — same reason as 067
#     (the aside table inherits the 056 FK to accounts and breaks the scrub's TRUNCATE):
touch /root/.reseed-paused
# 0b. PRE-FLIGHT: how big is the copy? migrate.sh runs every file under
#     statement_timeout=300s and lock_timeout=10s, and the copy is ONE statement.
#     psql> SELECT count(*), pg_size_pretty(pg_total_relation_size('onchain_credit.portfolio_flow_events'))
#            FROM onchain_credit.portfolio_flow_events;
#     A few million rows copy well inside 300s; if it would not, do NOT split the migration
#     (its atomicity is the safety property) — run that one file by hand with a raised
#     timeout instead, then record it in the ledger:
#       sudo -u postgres env PGOPTIONS='-cstatement_timeout=1800s -clock_timeout=10s' \
#         psql -v ON_ERROR_STOP=1 -X -q -1 -d <db> -f scripts/sql/072-repartition-portfolio-flows.sql
#       sudo -u postgres psql -q -d <db> -c "INSERT INTO onchain_credit.schema_migrations(filename) \
#         VALUES ('072-repartition-portfolio-flows.sql')"
# 1. Staging — 072 is DESTRUCTIVE, so it needs the explicit flag:
cd /opt/onchain-credit-staging && \
  scripts/ops/migrate.sh "postgresql://onchain_credit_staging@127.0.0.1:5432/creddit_staging" --allow-destructive
# 2. Verify — counts, the 16 partitions, the SIX indexes (5 + PK), and an EXPLAIN prune:
#    psql> SELECT count(*) FROM onchain_credit.portfolio_flow_events;         -- new
#    psql> SELECT count(*) FROM onchain_credit.portfolio_flow_events_preswap; -- old (equal)
#    psql> SELECT count(*) FROM pg_class c JOIN pg_inherits i ON i.inhrelid = c.oid
#           WHERE i.inhparent = 'onchain_credit.portfolio_flow_events'::regclass;   -- expect 16
#          -- 072 now ASSERTS this itself (16 partitions, 16 distinct remainders, before the
#          -- copy, so a shortfall aborts the atomic file). This query is the belt-and-braces
#          -- half: the case it guards is a relation already occupying one of the
#          -- portfolio_flow_events_hNN names, which `CREATE TABLE IF NOT EXISTS ...
#          -- PARTITION OF` matches on NAME and would otherwise adopt-around silently.
#    psql> SELECT c.relname FROM pg_class c JOIN pg_index i ON i.indexrelid = c.oid
#           WHERE i.indrelid = 'onchain_credit.portfolio_flow_events'::regclass;
#          -- expect: portfolio_flow_wallet_block_idx, portfolio_flow_wallet_pos_idx,
#          --         portfolio_flow_asset_idx, portfolio_flow_pendle_ptkey_idx,
#          --         portfolio_flow_provisional_idx, portfolio_flow_events_pkey
#          --         (anything missing is missing FOREVER once _preswap is dropped:
#          --          migrate.sh has recorded 072, so it never re-runs to reconcile)
#    psql> EXPLAIN (COSTS OFF) SELECT tx_hash, log_index FROM onchain_credit.portfolio_flow_events
#           WHERE chain_id = 1 AND wallet = '<a real lower-cased wallet>'
#           ORDER BY block_number DESC, log_index DESC;
#          -- expect ONE partition in the plan (e.g. ..._h06) and NO Append node. An Append
#          -- over all 16 means the wallet predicate is not pruning and the repartition has
#          -- bought nothing.
#    psql> ANALYZE onchain_credit.portfolio_flow_events;   -- fresh stats on the new relations
#    then a /portfolio + /portfolio events spot-check on a whale fixture wallet.
#    psql> DROP TABLE onchain_credit.portfolio_flow_events_preswap;
# 3. Prod (gated on Fred's approval; quiet window). RE-CHECK step 0's precondition here:
#    prod migrations are a separate manual step from the prod deploy, so this is the one
#    place the wrong order is easy to reach by hand.
cd /opt/onchain-credit && \
  scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)" --allow-destructive
#    Verify + ANALYZE + DROP _preswap by hand exactly as on staging.
# 4. Only after _preswap is dropped on BOTH environments, resume the reseed:
rm -f /root/.reseed-paused

Because 072 renames + creates + copies inside migrate.sh's single transaction, a failure anywhere rolls the whole thing back and leaves the original table untouched — it is atomic. A re-run detects the table is already partitioned and no-ops.

Why step 0a exists is exactly the 067 story: RENAME carries a table's constraints, so portfolio_flow_events_preswap keeps migration 056's FK to accounts plus a full copy of every wallet's flow history. Prod backs up at 02:15 UTC and reseed-staging.sh restores that dump at 03:00, so an aside table left standing overnight rides into staging and the scrub's single TRUNCATE then fails, aborting the reseed before its fail-closed leak checks. portfolio_flow_events_preswap was added to scripts/ops/scrub-staging-pii.sql and the reseed leak check pre-emptively (both to_regclass-guarded), so the mechanism is already in place and scripts/ops/scrub-staging.test.ts now checks both ends of the contract against 072 itself. The pause is the belt-and-braces half.

Partition maintenance after 072: there is none. All 16 partitions exist by construction, so unlike the snapshot spine there is no month rollover, no runtime DDL, no ACCESS EXCLUSIVE parent lock on a write path, and no reason to hand the parent to the app role — it stays owned by postgres with the app holding DML grants only (the 16 partitions pick up the schema's ALTER DEFAULT PRIVILEGES grant at CREATE time, but nothing depends on that: privileges are checked on the relation a query names, and tuple routing does not re-check them). The writers' partition calls for this table are deleted, and three layers keep them gone: ensureMonthPartitions' table parameter no longer ADMITS the flow ledger (MonthPartitionedTable excludes it, so the copy-paste does not compile), the helper refuses any non-RANGE parent at runtime (it probes pg_partitioned_table.partstrat), and a source tripwire in src/lib/portfolio/partitions.test.ts catches a literal call. Without those, month bounds against a hash parent are rejected outright (invalid bound specification for a hash partition) — which is the whole reason the deploy order above is one-way.

If the 16 partitions ever need to become 32 (they will not at the 100k target): 16 is a power of two, and Postgres allows mixed moduli as long as each divides the next, so it is an incremental DETACH of one MODULUS 16 partition plus ATTACH of two MODULUS 32 partitions with remainders r and r+16, one at a time — not another full rebuild.

Flow basis provisional (migration 070, additive). Widens the portfolio_flow_events.basis CHECK to admit provisional and adds the partial index the two provisional deletes use (see database.md). Staging auto-applies it on deploy; prod is the usual gated manual migrate.sh run. Apply it BEFORE the code restart — not "before or with", and never after: until the constraint is widened, the JIT's flow persist fails on any flow landing in the last ~64 blocks (the whole persist transaction rolls back and the JIT logs mini flow scan failed). Nothing is corrupted (a rejected insert writes nothing) and the flow is not lost, but two details make the window worse than it looks and neither is self-announcing:

It does not heal on the next refresh. Every subsequent JIT refresh of that wallet fails on the same CHECK for as long as the flow sits in the reorg-exposed tail; only the next 6h cron tick — which writes live rows exclusively and so never touches the widened value — actually lands it. Up to 6h of "my deposit is not showing".
Nothing pages you. The failure is caught by the JIT and written with console.error to the app's pm2 log, not to run-cron.sh's log, and the cron alert needs a non-zero exit and a matching line. JIT flow persistence can be dead for hours in silence, so grep pm2 logs for mini flow scan failed if you ever run the code ahead of 070.

The cron's settle sweep is cheap without the index (it is bounded by the tick's wallet list and block window, so it can fall back to portfolio_flow_wallet_block_idx), but it runs once per tick against the whole flow ledger — one more reason not to leave the window open. Rollback-safe in the other direction: the previous release only ever writes live/backfill, which the widened CHECK still admits.

Migration `079` (exposure key gains the collateral address, additive)

belongs to: migration 079 (exposure key gains the collateral address) — v0.36.0
executed: unverified

Widens market_collateral_exposure's primary key with collateral_address, so two isolated markets on one deposit asset whose collateral shares a ticker are two rows instead of one contested key (database.md). Additive and idempotent, and the new key is a SUPERSET of the old one, so uniqueness only relaxes and no stored row moves: staging auto-applies it on deploy and prod is the usual gated manual migrate.sh run after the release.

No special ordering, unlike 044 / 070 / 072. The deploy ships code only and a prod migration is a manual step after it, so code-then-migration is the default order here, and for this file that order is the benign one. In that window a live ticker pair has the first market's INSERT commit and the second violate the still-narrow key; the second rolls back alone with a [morpho/fail] line, so one market publishes a fresh row and the other keeps its previous snapshot (before this release the writer withheld both). Which of the two publishes follows the registry read's row order, so it can differ between runs while the window is open; nothing is mislabelled, because every row carries its own collateral address. Run migrate.sh after the deploy and the next 6h refresh-assets.ts tick closes it. Rollback-safe in the other direction: the previous release's three writers bind exactly these columns and only produce rows the wider key still accepts.

The nightly reseed reverts it on staging. ops/reseed-staging.sh restores prod's schema and its schema_migrations ledger (§3.2), so until 079 is applied on PROD every 03:00 reseed drops it from staging and the next migrate.sh reports 0 pending. A next-morning validation pass would then run against the pre-079 schema, where the ticker-twin behaviour is simply absent rather than broken. Pause the reseed for a multi-day validation window (touch /root/.reseed-paused) or re-run the staging deploy before checking.

Migrations `082` / `083` (flow ledger + registries, additive)

belongs to: migrations 082 / 083 (flow ledger + registries) — v0.42.0
executed: unverified

082 creates the flow ledger portfolio_flow_events_v2 (empty), the chain-identity tables, the impairment record and the two registries; 083 seeds the registries (database.md). The release carrying them changes no application code, so nothing reads any of it and no served number can move. Both are additive: staging auto-applies them on deploy, prod is the usual gated manual migrate.sh run.

Quiesce before the prod run, including the ingester. 082 swaps a CHECK on raw_events, which takes ACCESS EXCLUSIVE on the parent and all 23 partitions for the duration of the catalogue update, while creddit-event-ingester writes that table on a ~60s loop and migrate.sh runs under lock_timeout = 10s. Failure is safe (the file rolls back whole under psql -1) but a runbook should not discover its own lock contention. In order: drain the backfill queue, comment out the five prod portfolio crontab lines (drain-portfolio-backfills.ts, refresh-portfolio.ts, refresh-portfolio-discovery.ts, refresh-portfolio-reconcile.ts, sync-portfolio-tokens.ts — one root crontab holds prod and staging, separated only by the /opt/onchain-credit vs /opt/onchain-credit-staging path prefix, so read the prefix on every line), then pm2 stop creddit-event-ingester. Restore all five lines and pm2 start creddit-event-ingester in the same window, before anything else begins. A release deploy inside the window leaves the stopped ingester stopped (the deploy's restart script skips a stopped process), and the hourly freshness alarm pages until it is started again.

That list is the complete set of writers. The app stays up and is a reader of both altered tables (raw_events on the JIT ledger path, portfolio_position_snapshots on every /portfolio render), so if the run dies on lock_timeout the likely holder is a page load, not a cron: retry, or stop onchain-credit for the window too. While the migration holds the lock, page loads block until it commits, which is seconds.

bash

cd /opt/onchain-credit
scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)"

Never hand-run psql -f scripts/sql/082-*.sql. It bypasses schema_migrations, so the ledger still reports the file pending and the next migrate.sh re-applies it. If a file genuinely must be hand-run, its INSERT INTO onchain_credit.schema_migrations(filename) VALUES (...) is part of the same procedure, not an afterthought.

Verification, after the run:

sql

-- 16 partitions, 16 distinct remainders, and the FK the user wipe depends on
SELECT count(*), count(DISTINCT pg_get_expr(c.relpartbound, c.oid))
  FROM pg_class c JOIN pg_inherits i ON i.inhrelid = c.oid
 WHERE i.inhparent = 'onchain_credit.portfolio_flow_events_v2'::regclass;   -- 16 | 16
\d onchain_credit.portfolio_flow_events_v2                                  -- pfe2_wallet_fk

-- The widened raw_events CHECK. Read the DEFINITION first, then the flag.
SELECT conname, convalidated, pg_get_constraintdef(oid) FROM pg_constraint
 WHERE conrelid = 'onchain_credit.raw_events'::regclass;
-- raw_events_stream_chk must list 'wallet-token'. If it does not, the DROP/ADD above did
-- not run and this is 064's narrow six-name constraint under the same name: stop.
-- With 'wallet-token' present, BOTH readings of convalidated can be correct, and which one
-- to expect is decided by a fact you can check on the box rather than by anything in this
-- query -- is scripts/sql/084-raw-events-stream-validate.sql part of THIS release?
--   * absent (082 shipping alone)      -> FALSE. The VALIDATE is a later release's file.
--   * present (082 and 084 together)   -> TRUE, and that is the fully correct end state:
--     one migrate.sh invocation applies every pending file, so a release carrying both
--     applies both. Confirm `APPLY: 084-raw-events-stream-validate.sql` in the run's
--     output and finish at 084's own acceptance query below, which reads the parent AND
--     every partition. Do not stop.
-- TRUE with the file ABSENT from the release is the one bad reading here: something
-- validated this constraint outside the runner. Stop.

-- Both snapshot CHECKs widened (under _chk2 on the servers), and the seeds present
SELECT conname, pg_get_constraintdef(oid) FROM pg_constraint
 WHERE conrelid = 'onchain_credit.portfolio_position_snapshots'::regclass
   AND conname ~ '(venue|basis)';                       -- must list 'escrow' / 'provisional'
SELECT count(*) FROM onchain_credit.reward_distributors;             -- 4
-- ORDER BY, so the expected triple below is read against a fixed order rather than
-- against whatever order the grouping happens to return:
SELECT kind, count(*) FROM onchain_credit.token_lineage GROUP BY 1 ORDER BY 1;
-- address_migration 2 | rebook 4 | wrapper_rotation 4

Where the other half went, and why this step reads a file listing before it reads a flag. The VALIDATE that flips raw_events_stream_chk to convalidated = true is its own file, Migration 084, which is also where the constraint's acceptance query lives. 082 was written expecting to reach prod one release ahead of it, and false is the correct reading in that case. But migrate.sh has no per-file selector and a release PR's diff is everything on staging, so whether the two files reach prod in one invocation or two is decided by merge order on staging, months before anyone reads this step, and it is not a thing the operator at the keyboard can verify. The presence of the file in the release is, so that is what the step keys on. Both orderings are supported; neither is a defect; the only bad reading is convalidated = true with no 084 in the release.

If 082's parent-level ADD … NOT VALID ever falls back to per-partition statements, the parent is part of the procedure. The file's header offers DROP / ADD … NOT VALID per partition should a future environment refuse the parent-level form. Two things about that fallback, both measured on PG 16.14: the partitions' copies are inherited, so ALTER TABLE onchain_credit.raw_events_p90 DROP CONSTRAINT raw_events_stream_chk fails outright (cannot drop inherited constraint) while the parent still carries 064's — the fallback is not applicable one partition at a time on its own. And if it is reached the long way round, by dropping on the parent first, the widened constraint has to be re-added on the parent as well: a parent left on 064's six-name CHECK rejects the first wallet-token row with 23514, and a name-only acceptance query reports that state as green. This is the same "stop before the parent" defect that 084's own fallback carries a correction for, and it is why 084's acceptance query matches the constraint's definition rather than its name.

The nightly staging reseed is part of this migration's blast radius, and the fix ships with it. portfolio_flow_events_v2 carries an FK to accounts, and the scrub truncates the whole user-data graph in ONE statement; a table missing from that statement makes the TRUNCATE fail, which aborts reseed-staging.sh under set -euo pipefail before every leak check below it and leaves staging serving an unscrubbed copy of the prod user-data graph. scripts/ops/scrub-staging-pii.sql, the reseed's leak check, scripts/ops/scrub-staging.test.ts and scripts/ops/reset-portfolio-users.ts are all updated in the same release. After 082 reaches prod, watch the first 03:00 reseed: it must exit 0.

Staging only: give the staging role runtime partition auto-create on raw_events, and again after every reseed. It takes BOTH statements.

bash

sudo -u postgres psql -d creddit_staging \
  -c "ALTER TABLE onchain_credit.raw_events OWNER TO onchain_credit_staging" \
  -c "GRANT CREATE ON SCHEMA onchain_credit TO onchain_credit_staging"

Staging connects as onchain_credit_staging, which neither owns the partitioned parent nor holds the schema CREATE grant, so runtime partition auto-create fails there. ensurePartitions issues CREATE TABLE IF NOT EXISTS onchain_credit.raw_events_pNN PARTITION OF …, and that one statement needs both privileges: ownership of the parent (or it fails must be owner of table raw_events) and CREATE on the schema (or it fails permission denied for schema onchain_credit). Owning the parent is necessary but not sufficient — 064's own header says so, which is why 064 also granted the prod role CREATE ON SCHEMA onchain_credit. Measured on a PG 16.14 scratch cluster reproducing staging's exact ACL ({postgres=UC/postgres,onchain_credit=UC/postgres,onchain_credit_staging=U/postgres}): the ALTER alone still fails permission denied for schema onchain_credit; the GRANT alone still fails must be owner of table raw_events; both together succeed. Note that ALTER TABLE … OWNER TO does not recurse to existing partitions, and does not need to: DML on those is covered by the reseed's GRANT … ON ALL TABLES, and only the parent's owner matters for creating a new one.

Both statements repeat after every reseed, for the same reason: reseed-staging.sh restores the prod dump with --no-owner (so the tables come back owned by postgres) and re-grants the staging role USAGE on the schema plus DML on all tables — never CREATE, and never ownership. A manual step repeated after every reseed is a step that will eventually be skipped, so the durable fix is to move both statements into the reseed's grant block, next to the GRANT USAGE. That is deliberately not done in this release: it hands the staging app role standing DDL rights on the schema, which is a call to make on purpose rather than as a side effect of a runbook edit, and nothing needs the privilege until the band is exhausted (below).

Do not read "no failure yet" as evidence this worked, and do not expect a failure yet either. Two measured facts combine here. First, the whole materialised band already exists: prod and staging each carry 23 raw_events partitions spanning p90[22,500,000, 22,750,000) through p112 [28,000,000, 28,250,000), because the one-time 064 backfill auto-created p90–p97 under the prod role and staging inherits the whole set from the dump. The rebuild's 22,527,558 history floor therefore lands in an existing partition. Second, CREATE TABLE IF NOT EXISTS … PARTITION OF checks the schema privilege before the "already exists, skipping" shortcut, so on staging today that statement raises permission denied for schema onchain_credit even for a partition that is already there [measured on PG 16.14] — and ensurePartitions catches it, re-checks existence with to_regclass, finds the partition present, and continues. So the current staging symptom is a swallowed error plus an existence probe per partition index, not a broken scan. The loud failure (ensurePartitions rethrows when the partition really is absent) arrives at the first index that does not exist, i.e. once the head crosses 28,250,000, ~2.4M blocks out. The step is cheap and belongs in the reseed routine precisely because the day it is needed is the day nothing else warns you. Verify it directly rather than by absence of an error:

sql

SELECT pg_get_userbyid(relowner)                                        -- onchain_credit_staging
  FROM pg_class WHERE oid = 'onchain_credit.raw_events'::regclass;
SELECT has_schema_privilege('onchain_credit_staging','onchain_credit','CREATE');   -- t

portfolio_flow_events_v2 needs no equivalent: it is hash-partitioned with all 16 partitions created up front by 082, so it has no runtime auto-create path at all.

Staging only: a nightly reseed REMOVES 082's objects until the next deploy.reseed-staging.sh restores staging from a prod dump and does not run migrate.sh (schema_migrations comes from the dump as well), while the staging deploy does run the runner. So in the window between merging this PR and 082 reaching prod, each 03:00 reseed drops the flow ledger, the chain-identity tables and the registries from staging, and the next staging deploy re-applies 082 and 083. Two consequences, both about not misreading staging in that window: (a) checking the 16 partitions or the seed counts on staging on a later morning finds them absent, and that is the reseed rather than a failed migration (re-deploy staging, or touch /root/.reseed-paused across a validation window, before reading anything into it); (b) a reseed that exits 0 in that window is not evidence the scrub fix works, because the table the scrub entry was added for is not in the dump yet. That is why the blast-radius note above waits for the first 03:00 reseed after 082 reaches prod.

Rollback: none needed. Both files are additive and unread; reverting the release's code leaves the tables in place, which is the expand half of expand/contract.

Migration `084` (the `raw_events` stream CHECK validation)

belongs to: migration 084 (the raw_events stream CHECK validation) — v0.42.0
executed: unverified

A later release than 082, deliberately, and the file is only on the box from that release onward. 082 widened raw_events' stream CHECK as DROP + ADD … NOT VALID, which is a catalogue update: the constraint rejects a bad stream name on every new row immediately, but reads convalidated = false because nothing has scanned the 7.8M stored rows yet. 084 is the matching VALIDATE CONSTRAINT, and it is one statement.

It is a separate release, not just a separate file, because migrate.sh has no per-file selector: one invocation applies every pending file in ls | sort order, so "apply 082, verify, then 084" is not expressible inside one release. Selection is by presence on disk. Had 084 shipped in 082's release, the same invocation would have applied it and 082's own acceptance query (convalidated = false) would have been false as written.

If the two do arrive together, nothing here breaks — only the intermediate reading disappears. Which release each file lands in is set by merge order on staging, so a prod release can carry 082, 083 and 084 at once. One invocation then applies all three, in ls | sort order, and prod goes straight to the end state: 082's step reads convalidated = true (correct, and its runbook entry says so), the "expected state before this file" below is never observable on prod, and this section's verification query is the one that decides whether the release is good. The quiesce below is 082's quiesce in that case, which is the stricter of the two, so follow that one.

bash

cd /opt/onchain-credit
scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)"

Quiesce is the same shape as 082's, and for a weaker reason. VALIDATE CONSTRAINT takes SHARE UPDATE EXCLUSIVE, which blocks neither reads nor writes on raw_events, so the ingester can keep running; what it does conflict with is another SHARE UPDATE EXCLUSIVE holder, i.e. a concurrent VACUUM or ALTER. With lock_timeout = 10s that is a failed acquisition, not an outage, and the fix is to retry.

Budget the scan, which is smaller than the table. VALIDATE CONSTRAINT reads the heap, and raw_events is 3,908 MB of heap inside 8,313 MB of total relation size — the difference is the indexes, which this statement does not touch [measured read-only on prod, 2026-08-18]. For scale: a full heap read of staging's copy of the same table (7,782,149 rows) returns in 1.4 s warm, so the statement_timeout = 300s is not a tight budget and a 57014 here means something is wrong rather than that the table is merely large. If it does time out, re-run migrate.sh — the file rolls back whole under psql -1, so a re-run is clean. If it times out twice, apply the same VALIDATE per partition, 23 statements, one at a time (each scan a 23rd of the table), and then run the parent statement once:

sql

ALTER TABLE onchain_credit.raw_events_p90 VALIDATE CONSTRAINT raw_events_stream_chk;
-- … p91 … p112, one statement at a time …
ALTER TABLE onchain_credit.raw_events     VALIDATE CONSTRAINT raw_events_stream_chk;  -- required

The last line is not a repeat of the 23 above it. Validating the partitions flips convalidated on the partitions only and leaves the parent row at false, which is the row the acceptance query reads, so a fallback that stops at the 23rd statement finishes with a red check and the constraint still unusable [measured on PG 16.14, three-partition replica, 3.1M rows]. With every partition already validated the parent statement does not re-scan: 1.3 ms against 945 ms for the full parent scan on that same replica. The lock profile is the same either way.

On staging, "once" means once per deploy until 084 reaches prod. The reseed interaction described for 082 above applies here too: each 03:00 reseed restores staging from a prod dump, schema_migrations included, so 082–084 disappear and the next staging deploy re-applies all three — meaning the full validation scan re-runs on every staging deploy in that window, not once. It stays inside the 300s budget, and reseed-staging.sh's own database-wide ANALYZE takes the same SHARE UPDATE EXCLUSIVE this section warns about, so a deploy landing during the reseed can fail its lock acquisition (55P03) and simply needs re-running.

It cannot fail on stored rows. The widened value set is a strict superset of the one the old CHECK validated, so every stored row already satisfies it. A 23514 here would mean something wrote a stream name outside both lists, and the response is to find that writer, not to widen the CHECK again.

Verification, after the run — this is the check 082's step deliberately expected false from. It reads the parent and every partition, because the per-partition fallback above is the one procedure that can leave the two disagreeing, and a partial validation must not read as success in either direction:

sql

SELECT count(*) AS relations,
       count(*) FILTER (WHERE NOT c.convalidated) AS not_validated
  FROM pg_constraint c
 WHERE c.conname = 'raw_events_stream_chk'
   AND pg_get_constraintdef(c.oid) LIKE '%wallet-token%'
   AND (c.conrelid = 'onchain_credit.raw_events'::regclass
        OR c.conrelid IN (SELECT inhrelid FROM pg_inherits
                           WHERE inhparent = 'onchain_credit.raw_events'::regclass));
-- 24 | 0   (the parent plus its 23 partitions, none left NOT VALID)

The wallet-token clause is what makes the reading mean anything, and relations is half the check. raw_events_stream_chk is 064's constraint name as much as 082's: 064 created it with the original six stream names, so a database that never ran 082 carries a constraint of exactly that name, already validated. Matched on the name alone this query returns 24 | 0 there — the success reading — on a database that admits none of the seven new streams, and 084 itself does not fail on such a box either: it re-validates 064's narrow constraint and returns. Both measured read-only on prod, 2026-08-18, with schema_migrations topping out at 081. With the definition clause the same box reads 0 | 0, and staging (where 082 is applied) reads 24 | 24 before the run. So read both numbers: relations = 0 means the widened constraint is absent, i.e. 082 has not run, and not_validated = 0 on its own certifies nothing.

24 is the box's partition census plus one, not a constant: 23 partitions on prod and staging today, 15 on a from-scratch fixture (16 | 0 there). Swap the aggregate for SELECT c.conrelid::regclass, c.convalidated … ORDER BY 1 to see which relation is lagging when not_validated is not 0. Match through pg_inherits rather than a name pattern: regclass renders unqualified when the schema is on search_path, so a LIKE 'onchain_credit.raw_events%' silently matches nothing on a session that has it there.

The reachable way to be shown a false green is the staging reseed window, not a mis-ordered runner: each 03:00 restore takes staging's schema_migrations back below 082 and restores 064's constraint with it, so a name-only query run on staging on a later morning reports 24 | 0 while none of the release's schema is present. The definition clause makes that state read as 0 | 0.

Rollback: none needed. Validating a constraint changes no row and no application behaviour; reverting the release's code leaves it validated.

Money market funds: first deploy (migration 086)

belongs to: migration 086 (money market funds) — v0.44.0
executed: unverified

/money-market-funds needs migration 086 plus one seeding run before the tab shows anything. Everything below is safe to run against staging; nothing here touches production without an explicit decision.

The page is prerendered, so its queries run during the BUILD that precedes the migration. Its reader probes information_schema first and renders an empty tab loudly rather than failing the deploy, which is what makes the pre-migration build safe. Keep that probe until 086 is on prod AND a prod dump has been reseeded to staging, because the nightly reseed rebuilds creddit_staging from the prod dump and would otherwise take the tables away every night.

Before the deploy: check the ledger does not already carry 086. The migration runner keys on FILENAME alone, with no checksum, and this file gained columns during review. Any database that ran an earlier copy of it gets SKIP and never receives them, while the reader names them all. On a database that already has the row, apply the ALTER TABLE ... ADD COLUMN IF NOT EXISTS block at the foot of scripts/sql/086-money-market-funds.sql by hand; it is a no-op where the columns exist.

sql

SELECT filename FROM onchain_credit.schema_migrations
 WHERE filename = '086-money-market-funds.sql';   -- expect NO row before the deploy

bash

# 1. Confirm the migration applied. A SKIP looks exactly like success in the log,
#    so check the ledger, not the deploy output.
sudo -u postgres psql -d creddit_staging -c \
  "SELECT filename, applied_at FROM onchain_credit.schema_migrations \
    WHERE filename = '086-money-market-funds.sql';"

# 2. Seed the registry: discover, confirm on chain, classify, propose.
#    Measured 13s against mainnet (928 vaults discovered, 116 chain-confirmed).
cd /opt/onchain-credit-staging
scripts/run-cron.sh sync-money-market-funds.ts --dry-run   # read the report first
scripts/run-cron.sh sync-money-market-funds.ts             # persist

sudo -u postgres psql -d creddit_staging -c \
  "SELECT status, count(*), round(sum(tvl_usd)/1e6) AS tvl_musd \
     FROM onchain_credit.money_market_fund_registry GROUP BY 1 ORDER BY 2 DESC;"
# Expected shape as of 2026-08-21: ~40 listed, ~37 proposed, the rest ineligible.
# The report prints an OFF PAR block (the two Metronome funds sit in it today)
# and a PAR UNMEASURED block (funds listed with the flag, none today).

# 3. First state and allocation read. Measured 23s for the ~40 listed funds.
scripts/run-cron.sh refresh-assets.ts
sudo -u postgres psql -d creddit_staging -c \
  "SELECT count(*) FILTER (WHERE withdrawable_now IS NOT NULL) AS with_exit, \
          count(*) AS total FROM onchain_credit.money_market_fund_state;"
sudo -u postgres psql -d creddit_staging -c \
  "SELECT count(*), count(DISTINCT fund_address) \
     FROM onchain_credit.money_market_fund_allocation;"

# The invariant that matters: no fund may report more exit liquidity than it has.
sudo -u postgres psql -d creddit_staging -c \
  "SELECT r.name, s.withdrawable_now, s.total_assets \
     FROM onchain_credit.money_market_fund_state s \
     JOIN onchain_credit.money_market_fund_registry r USING (chain_id, address) \
    WHERE s.withdrawable_now > s.total_assets * 1.0001;"   -- expect zero rows

# And the cross-check against Morpho's own published figure. A HANDFUL IS
# NORMAL: our figure is read at a pinned block and theirs seconds later, so a
# few funds land outside the 2% tolerance on any given tick and clear on the
# next. It is a log line for a human, never a gate. What is worth looking at is
# the SAME fund appearing tick after tick, or a sudden jump to most of the list.
sudo -u postgres psql -d creddit_staging -c \
  "SELECT count(*) FROM onchain_credit.money_market_fund_state \
    WHERE chain_api_delta IS NOT NULL;"   -- 0 to ~5 of 40 is a settled tick

# 4. Return history for the newly listed funds. LONG: budget 6-9 hours.
#    Resumable: a point already on record is KEPT, never rewritten, so an
#    interrupted run is re-run with the same command. token_yield_apy is also
#    what the portfolio values yield-token holdings from, which is why replacing
#    a covered window needs the explicit --overwrite and is not a rerun.
nohup scripts/run-cron.sh backfill-money-market-funds.ts 365 \
  >/dev/null 2>&1 &
tail -f /tmp/onchain-credit-cron/onchain-credit-backfill-money-market-funds.log

sudo -u postgres psql -d creddit_staging -c \
  "SELECT count(*) FILTER (WHERE y.a IS NOT NULL) AS with_history, count(*) AS listed \
     FROM onchain_credit.money_market_fund_registry r \
     LEFT JOIN (SELECT DISTINCT lower(token_address) a FROM onchain_credit.token_yield_apy) y \
            ON y.a = r.address \
    WHERE r.status = 'listed';"
# Target: with_history = listed.

# 5. One more tick, to confirm the share-rate snapshotter picked the new funds up
#    and did NOT push itself over the partial-failure floor.
scripts/run-cron.sh refresh-assets.ts
grep -E '\[ok\]|\[partial\]|\[fail\]' \
  /tmp/onchain-credit-cron/onchain-credit-refresh-assets.log | tail -20
# `[partial] token-yields` here is a BLOCKER, not a warning. That job carries five
# permanent failures which are held OUTSIDE its ratio (knownPermanent: 5, floor
# 3%, in refresh-assets.ts) precisely because the fund set roughly doubles its
# item count: inside a bare 10% ratio the same floor would have quietly tolerated
# eight new silent failures instead of about three.

# 6. Bad debt is read by MARKET ID, from the ids the previous tick recorded, so
#    the column is null on the first state read and populated on the second.
sudo -u postgres psql -d creddit_staging -c \
  "SELECT count(*) FILTER (WHERE bad_debt_usd IS NULL) AS unmeasured, \
          count(*) FILTER (WHERE bad_debt_usd > 0) AS carrying, count(*) AS rows \
     FROM onchain_credit.money_market_fund_allocation WHERE slot_kind = 'market';"
# After the SECOND refresh-assets.ts, `unmeasured` should be near zero. A run that
# could not reach the API logs "market bad debt unreadable this tick" and leaves
# every row null, which renders as no reading rather than as a clean bill.

# 7. Add the daily discovery cron, after the existing 3am block:
#    50 3 * * * /opt/onchain-credit-staging/scripts/run-cron.sh sync-money-market-funds.ts

The approvals are a human's. After step 2 the report's NEEDS A DECISION block lists the managers behind every proposal, collapsed one line per house, plus the funds with no readable manager (UNNAMED) and the funds whose manager was read off their own name (BY NAME). Nothing lists until someone runs --approve-manager <key>, and a BY NAME fund needs its own --bind-manager <slug> <key> on top, because a vault name is a string its deployer chose. That is the point; do not pre-approve.

A run that could not see the whole universe removes nothing. If step 2 prints INCOMPLETE DISCOVERY, the Morpho endpoint was degraded, not the fund set: the registry keeps everything it had and no vault is marked gone. Re-run it later rather than working around it.

Rollback. The migration is additive, so a code rollback needs no database change: the four tables sit unread. If the page itself has to go, revert the merge and redeploy; the tables stay and the next deploy picks them back up. Do not drop them.

Fluid per-vault rates: first deploy (migration 093)

belongs to: migration 093 (Fluid per-vault rates) — v0.55.0
executed: unverified

/carries prices every Fluid funding leg off fluid_vault_rates from this release on (issue #769). The table is additive and the readers withhold a snapshot they have no vault row for — never a fallback to the Liquidity-Layer token rate, never a zero — so between the deploy and the backfill a Fluid carry would show no funding figure at all. That is the whole reason this runbook orders the passes the way it does.

Prod, in this order. Steps 1 to 3 run BEFORE the merge, and that ordering is the point: the backfill needs nothing but DATABASE_URL and an archive RPC, so the first pass can run from the release branch's own script while the deployed build is still the previous release. Do it after the merge instead and every Fluid carry renders with no figures and no expansion for the deploy plus ~20 minutes.

bash

# 1. Take the migration, the backfill script and the resolver-era helper it
#    imports from the release branch WITHOUT deploying them. `staging`, not
#    `main`: the release PR has not been merged yet at this point, so main
#    carries none of the three.
#
#    The SQL file has to be fetched for the same reason the script does, and
#    that is why this step comes FIRST. `migrate.sh` applies the files present
#    in the checkout it is RUN FROM (`SQL_DIR="$(dirname "$0")/../sql"`), and
#    that checkout is the PREVIOUS release, whose highest file is 092. Run the
#    runner before this step and it prints `0 applied` and creates nothing.
#
#    pm2 serves the built .next bundle, so a source file is inert until the next
#    build, and step 4's deploy does `git reset --hard` over all three.
cd /opt/onchain-credit
git fetch "https://x-access-token:$GH_TOKEN@github.com/FredCoen/onchain-credit.git" staging
git checkout FETCH_HEAD -- scripts/sql/093-fluid-vault-rates.sql \
  scripts/refreshers/fluid-vault-rates.ts src/lib/portfolio/fluid-abi.ts

# 2. Apply the migration (additive; gated manual step on prod), then CHECK THE
#    LEDGER. Prod's deploy workflow never runs `migrate.sh` at all, so this run
#    is the only thing that ever creates the table — and a no-op run prints
#    `0 applied` and reads exactly like success.
sudo -u postgres psql -d creddit -c \
  "SELECT filename, applied_at FROM onchain_credit.schema_migrations \
    WHERE filename = '093-fluid-vault-rates.sql';"     # expect NO row first
scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)"
#    expect a `APPLY: 093-fluid-vault-rates.sql` line in that output
sudo -u postgres psql -d creddit -c \
  "SELECT filename, applied_at FROM onchain_credit.schema_migrations \
    WHERE filename = '093-fluid-vault-rates.sql';"     # expect ONE row now

# 3. FIRST PASS — the last 35 days, which is everything the screener's trailing
#    windows read. ~20 min for 21 vaults (2 archive reads per vault per 6h
#    window, one block-pair lookup per window). Run BEFORE the merge, so the
#    release builds against a table that already has the windows it reads.
scripts/run-cron.sh refreshers/fluid-vault-rates.ts --backfill \
  --from "$(date -u -d '35 days ago' +%Y-%m-%d)"

# 4. Merge the release, let the deploy finish.

# 5. SECOND PASS — the rest of the Fluid carry history depth (2025-05-21), so
#    the charts' long timeframes keep the range they have today. Idempotent, so
#    the overlap with pass 1 is harmless. Several hours; run it in tmux.
scripts/run-cron.sh refreshers/fluid-vault-rates.ts --backfill \
  --from 2025-05-21 --to "$(date -u -d '34 days ago' +%Y-%m-%d)"

# 6. Re-deploy (empty commit) so the prerendered /carries picks the long history
#    up immediately instead of at its next revalidate. See the ISR note below.

What a skipped step 2 looks like at step 3. Every write in the backfill opens with the 24h-anchor SELECT, so with no table each vault throws 42P01, is caught per vault, and the pass ends 0 written, N failed with a non-zero exit after the full ~20 minutes of archive reads. Read that as the table, not as a rate-limited RPC: re-run step 2 and check the ledger row before repeating the pass.

If the migration has NOT been applied when the deploy runs, the build no longer fails: apy.ts probes information_schema for the table once and, when it is absent, every Fluid carry withholds instead of throwing 42P01. That matters because a build precedes migrate.sh wherever the deploy runs it at all (deploy-staging.yml), so a hard failure there would roll the deploy back with the migration still unapplied and the next deploy would fail identically (the trap hit in PR #382). On prod the deploy runs migrate.sh nowhere, so the table arrives only through step 2 above and nothing downstream repairs a missed one: the degrade is a page with dashes on all 21 Fluid rows and no expansion, held until a human applies it. That is a page with no figures rather than wrong ones, and the ordering above is what avoids seeing it at all. Staging re-arms this nightly: reseed-staging.sh rebuilds creddit_staging from the prod dump, so a hand-applied 093 on staging is undone every night until 093 is on prod.

Why oldest-first, and why two passes. Each window's *_apy_24h is measured against the stored row 24 hours behind it, so a newest-first walk would leave every 24h column NULL until a second pass. The split is only about what the screener needs soonest: pass 1 restores the live figures, pass 2 restores the chart's depth.

Checks after the backfill.

sql

-- Every active Fluid vault has a row at the latest window.
SELECT count(*) FROM onchain_credit.fluid_vault_rates
 WHERE snapshot_ts = (SELECT max(snapshot_ts) FROM onchain_credit.fluid_vault_rates);
-- Expect the active Fluid registry count (21 as of 2026-09-03).

-- 1x parity: for every single-debt vault the new series must agree with the
-- Liquidity-Layer series for its debt token to annualisation noise. Anything
-- above a basis point means a vault has left 1x (real, and the alert will have
-- said so) or the series is wrong.
SELECT r.strategy_key,
       round(max(abs(v.borrow_apy - l.borrow_apy)) * 10000, 4) AS max_diff_bp
  FROM onchain_credit.fluid_vault_rates v
  JOIN onchain_credit.carry_registry r ON r.vault_address = v.vault_address
  JOIN onchain_credit.fluid_ll_apy l
    ON l.snapshot_ts = v.snapshot_ts
   AND l.token_address = lower(r.config->>'debtAddr')
 WHERE r.protocol = 'Fluid' AND r.status = 'active'
   AND r.config->>'kind' IN ('t1-single','t2-smart-col-debt')
   AND v.snapshot_ts >= now() - interval '30 days'
 GROUP BY 1 ORDER BY 2 DESC;

-- Smart-debt overlay: 0 everywhere today, and any non-zero row is a real
-- vault-level charge, not a defect.
SELECT vault_id, count(*) FILTER (WHERE borrow_apy <> 0) AS windows_with_overlay
  FROM onchain_credit.fluid_vault_rates
 WHERE is_smart_debt AND snapshot_ts >= now() - interval '30 days'
 GROUP BY 1 ORDER BY 1;

Approving a new Fluid carry after this. The refresher covers status IN ('active','proposed'), so a vault that was proposed already has history when it is approved. A vault that goes straight to active without ever sitting in proposed needs one catch-up pass, or its carry shows no funding figure until the 6h cron has filled 30 days. "No funding figure" is the whole behaviour, on every surface: the screener dashes the cells, the home page's rate tape skips the trade rather than ranking it, and the assistant reports nulls for the week. Nothing anywhere treats the missing reading as 0% funding.

bash

scripts/run-cron.sh refreshers/fluid-vault-rates.ts --backfill \
  --vault <0xvault> --from 2025-05-21

Rollback. Additive, so a code rollback needs no database change: the table sits unread and the previous readers go back to the Liquidity-Layer token rate. Do not drop it — the history is expensive to rebuild.

Retiring the old portfolio ledger

belongs to: migration 095 (retire the old portfolio ledger, -- DESTRUCTIVE) — v0.55.0
executed: 2026-09-05

The drop ran on prod with the v0.55.0 release on 2026-09-05: the owner recorded it in #787 ("now that migration 095 has run on prod") and #788 ("found while running the v0.55.0 release"). scripts/ops/scrub-staging-pii.sql and scripts/ops/reseed-staging.sh guarded the relations 095 dropped until the #787 clean-up (#896) removed them, which was safe once the nightly reseed had run against post-095 prod dumps — it had, about ten times. Neither file names the retired ledger now, scripts/ops/scrub-staging.test.ts pins that absence so the names cannot drift back in, and the repartition aside tables (067 / 072) keep the to_regclass pattern itself in use.

The release that deleted the last reader of the old flow ledger ships one hand-run migration and three lines to delete from .env.local. Nothing here is automatic: the migration destroys the only copy of a ledger that was the product for a year, and a code rollback does not roll a DROP back.

1. Deploy the release and watch two ticks go green before touching anything else. The migration is safe only once nothing reads the relations it drops, and "nothing reads them" is something the box tells you rather than something the diff proves:

one 6h portfolio tick logs its v2 dual-write: line naming the wallets and the range, with no [fail] beside it;
one reconciliation tick (refresh-ledger-reconcile.ts, 40 minutes after the portfolio tick) completes and reports no new disagreement.

Both are in pm2 logs onchain-credit and in the cron log. A tick that has not run yet is not a green tick: wait for it.

The first portfolio tick after this deploy can page on an in-transit position, and that page is not about the drop. The leg-disappearance tripwire (scripts/refreshers/portfolio.ts) now judges escrow legs for the first time — it used to hide them, because the engine served then had no receipt kind able to explain one closing and would have paged with the whole escrowed notional attached. A tracked leg that was in transit at the previous snapshot and whose closing receipt is not in the rebuilt ledger raises the alarm on this tick. Read the wallet and the leg off the line before calling the tick red.

2. Delete the three dead switches from .env.local, on prod and on staging:

PORTFOLIO_LEDGER_SOURCE
PORTFOLIO_LEDGER_WRITE
PORTFOLIO_LEDGER_MODE

They are inert once this release is deployed — no code in it reads them — and that qualifier is why they go after step 1 rather than before it. Until the new build is running, readLedgerSource() reads PORTFOLIO_LEDGER_SOURCE at call time and an unset value resolves to v1. Delete the line first and any restart of the OLD build in the window that follows — deploy.yml reverting to $PREV after a failed deploy, or an unrelated pm2 restart --update-env — serves /portfolio from portfolio_flow_events, a relation that has had no writer since the previous release. Every tracked wallet's history would freeze at that block with nothing to say so: the page looks normal, and the reconciliation alarm, which compares the rebuilt rows, keeps passing.

3. Apply the drop, by hand, once per environment.

bash

cd /opt/onchain-credit
scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)" --allow-destructive

095-drop-old-ledger.sql is tagged -- DESTRUCTIVE, so the ordinary runner and the fixture build both skip it and --allow-destructive is the only way it runs. It drops portfolio_flow_events (with its sixteen hash partitions), portfolio_flow_events_preswap, portfolio_parity_runs, fluid_event_log and fluid_event_coverage, and deletes the old scanner's seven chain_scan_cursors rows. Every statement is IF EXISTS and the delete is a no-op on a second run, so re-running it changes nothing.

Until it ran on prod, the nightly staging reseed brought the dropped tables back — the part of this worth carrying to the next retirement. reseed-staging.sh restores a prod dump, so staging's schema is prod's schema of the night before; scrub-staging-pii.sql therefore had to go on naming the old relations (its TRUNCATE accounts is illegal while a table with a foreign key to it exists, and a scrub that aborts leaves staging serving un-scrubbed production data). Those restored rows were read by nothing. Once the drop had run on prod the next reseed stopped carrying them, the guards became permanent no-ops, and #896 deleted them. The order is one-way: drop on prod, let a reseed prove the dump no longer carries the relations, then tidy the scrub — never the other way round.

And what binds afterwards is the DUMP, not the calendar. reseed-staging.sh takes an optional [path-to.dump] and backup-creddit.sh keeps 14 days, so dumps predating the drop stay restorable for two weeks after it. Reseed — or DR-restore — only from a dump taken after 2026-09-05. An older one still carries the retired ledger, and now that the scrub has stopped naming it the single TRUNCATE aborts on its FK to accounts after the restore and before every leak check, leaving staging serving un-scrubbed production data. Nothing pages when that happens: this cron is not wrapped by run-cron.sh, so it is a log line. The default newest-dump path is always safe; the named-dump argument is the one to think about. The constraint is general, not a detail of this retirement — it applies to every later drop of a relation the scrub used to name, and it expires only when the last pre-drop dump ages out.

Pendle redemption-index factor: first deploy (migration 101)

belongs to: migration 101 (Pendle redemption-index factor) — v0.59.0
executed: unverified

What changes for a reader: a Pendle principal token does not always redeem for one unit of the asset it settles into. Once the yield-bearing asset behind it is written down, the whole fall lands on the PT holder, and the size of that fall is f = min(1, SY.exchangeRate() / YT.pyIndexStored()). Until this release f was read live, used for the mark and thrown away. From here it is recorded on every market-state row, because a holder's accrual line needs f at each purchase block and a dated write-down event is by definition a step in f between two moments. Neither is reconstructible from a read of "now".

Prod, in this order. Nothing here is urgent and nothing here is bulk.

bash

# 1. Apply the migration (additive, two nullable columns; gated manual step on
#    prod). It cannot deadlock the deploy in either order — no prerendered page
#    reads the new columns — and the 6h refresher TOLERATES their absence: it
#    probes information_schema once per run and, until they exist, writes the
#    snapshot without the pair and says so on one line. So a tick that lands in
#    the gap costs the factor for that tick and nothing else. Run it in the same
#    post-deploy window anyway; every tick before it is a market-hour of factor
#    history nobody can get back without an archive read.
cd /opt/onchain-credit
sudo -u postgres psql -d creddit -c \
  "SELECT filename, applied_at FROM onchain_credit.schema_migrations \
    WHERE filename = '101-pendle-redemption-index.sql';"    # expect NO row first
scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)"
#    expect an `APPLY: 101-pendle-redemption-index.sql` line in that output
sudo -u postgres psql -d creddit -c \
  "SELECT filename, applied_at FROM onchain_credit.schema_migrations \
    WHERE filename = '101-pendle-redemption-index.sql';"    # expect ONE row now

# 2. Nothing else is required. The standing 6h refresher fills both columns for
#    every ACTIVE market from its next tick, in the same multicall batch and at
#    the same block as the rest of the row. Confirm after one tick:
sudo -u postgres psql -d creddit -c \
  "SELECT count(*) FILTER (WHERE redemption_index_factor IS NOT NULL) AS filled, \
          count(*) AS rows_today \
     FROM onchain_credit.pendle_market_state \
    WHERE snapshot_ts > now() - interval '7 hours';"

Historical rows stay NULL, deliberately, and are filled ON DEMAND. Every row written before the migration has no factor, and the read path withholds rather than assuming par. Fill a market's history only when a tracked wallet actually holds its PT:

SCOPE, SINCE THE AUTO-BACKFILL SHIPPED. The command below is now for markets the registry already carried when that change landed. A market discovered from then on has its history and factors filled by the 6h refresher itself, in the tick that inserts it (both discovery arms), capped at two markets a tick and run after the snapshot loop; a standing drain returns markets that still hold no API history at all, narrowed through portfolio_held_pts and bounded at the ledger derivation floor, rotating so no market can hold the head of the queue, so a deferred or failed pass retries without an operator. The drain does NOT fill pre-2026-01-01 display history, and it does not refill a market whose archive legs failed — both stay this command's job. So the manual pass is for the backlog, for a market whose automatic pass was short, and for --no-factors display history. It is unchanged, and still idempotent.

THIS BACKFILL IS WHAT TURNS THE ACCRUAL LINE ON FOR AN EXISTING HOLDING
Migration 101 is necessary and NOT sufficient, and the shape of the gap is worth stating before someone reports it as a bug. A PT lot locks its yield from the factor at its own fill block (M34), and factorAt will only answer from a stored row at or before that block, within its bound. The 6h refresher fills FORWARD from the deploy — so for a market whose history has not been backfilled, every lot bought before the deploy has no factor, opens unvalued, and withholds that leg's whole accrual line for as long as the lot is open. The position row says so rather than leaving it blank: markWithheld.redemption reads pt-factor-unavailable, whose remedy is exactly the per-market command below.
So for any wallet holding a PT bought before this release, the gated step below is not an optimisation — it is the step that makes the feature do anything at all for that holding. Cost is per market and stated below (~1,500 chain reads per market-year); run it for the markets the query at the end of this section returns and no others.
Nothing is at risk while it is outstanding. A withheld line is a dash with a named reason, never a wrong number, and the market line is unaffected throughout. Prod holds zero users after the 2026-09-10 wipe, so today the set is empty; it matters from the first wallet registered with pre-existing Pendle history.

bash

# Cost first, honestly. Per row: one block resolution — which is one DefiLlama
# call plus one or two eth_getBlockByNumber verification reads, or a full
# ~25-probe bisect when Llama is down — and one multicall of two archive calls.
# Call it 4 to 5 chain reads per row, so a market-year is ~1,500, not ~730.
# Preview it (the dry run probes ONE point live, so it also tells you the chain
# is reachable at that depth and whether the market is impaired):
scripts/run-cron.sh backfill-pendle-history.ts --market 0x8cef2919… --dry-run

# Then the real pass for that one market:
scripts/run-cron.sh backfill-pendle-history.ts --market 0x8cef2919…

Do NOT run it over all 76 markets. A bulk pass is on the order of 100k chain reads to fill history for markets nobody holds. Running the script without --market is still supported for its original job (display history for the /carries term surfaces); pair it with --no-factors there, which restores the API-only behaviour and skips the chain reads entirely.

It is safe to re-run. The conflict clause writes the pair only where the stored one is NULL and the read produced one, so a second pass over a finished market updates no row, a failed read never nulls a factor already banked, and no API-sourced column can overwrite a cron snapshot. A row that already carries a block anchor is read AT THAT BLOCK, so it keeps describing a single block.

Which markets are worth it. The ones a tracked wallet holds, matured or not — and a PT held as collateral counts. The four arms below are the repo's own held-by-anyone probe (PENDLE_PT_REFERENCED_SQL in scripts/refreshers/pendle-markets.ts): a directly held PT on either ledger, plus a PT sitting in any venue's accounting/asset slot, which is the arm that finds a levered PT loop on Morpho or Aave. Scanning only pendle:pt:% keys would miss exactly those legs, so their history would never be filled and their accrual line would be withheld indefinitely.

sql

WITH held AS (
  SELECT DISTINCT lower(split_part(s.position_key, ':', 3)) AS pt
    FROM onchain_credit.portfolio_position_snapshots s
   WHERE s.venue = 'pendle'
  UNION
  SELECT DISTINCT s.accounting_asset
    FROM onchain_credit.portfolio_position_snapshots s
   WHERE s.accounting_asset IS NOT NULL
  UNION
  SELECT DISTINCT f.asset
    FROM onchain_credit.portfolio_flow_events_v2 f
   WHERE f.asset IS NOT NULL
  UNION
  SELECT DISTINCT lower(split_part(f.position_key, ':', 3))
    FROM onchain_credit.portfolio_flow_events_v2 f
   WHERE f.venue = 'pendle'
)
SELECT p.market_address, p.pt_symbol, p.status,
       count(*) FILTER (WHERE s.redemption_index_factor IS NULL) AS unfilled_rows,
       count(*)                                                  AS total_rows
  FROM onchain_credit.pendle_markets p
  JOIN onchain_credit.pendle_market_state s
    ON s.chain_id = p.chain_id AND s.market_address = p.market_address
 WHERE p.pt_address IN (SELECT pt FROM held)
 GROUP BY p.market_address, p.pt_symbol, p.status
HAVING count(*) FILTER (WHERE s.redemption_index_factor IS NULL) > 0
 ORDER BY unfilled_rows DESC;

The bound, so a NULL stretch is read correctly. The reader takes the latest non-null factor at or before a moment and refuses to carry it beyond 18 hours for a 6h cron row or 48 hours for a daily backfill row. Two of the writer's own windows is room for one missed tick; the cron gets a third because its rows are stamped with the window floor but read at the head block minutes later, so at a flat 12 hours the replacement row after a single missed tick would arrive just after the previous one expired and a live figure would blink to a dash for those minutes. Past the bound the reader returns nothing and the value that depended on it is withheld. So a gap in this series shows up as a missing figure, never as a PT quietly marked at par through an impairment.

A matured market's series ends at maturity. The 6h snapshot arm visits status='active' markets only, so no new factor rows appear for a market once it matures. That is not a gap in the mark: at and after maturity the snapshot stores qty_underlying = qty x f, so the factor for a held leg is recoverable per spine row as qty_underlying / qty — never par. What the stored series alone cannot give is f at a flow block no snapshot covers, and the step that makes a dated write-down event. Widening the snapshot arm to held matured markets is a follow-up, deliberately not done here; the reader's staleness rule makes the absence explicit rather than extrapolating.

Rollback. Additive, so a code rollback needs no database change: the two columns sit unread. Do not drop them; a filled history is expensive to rebuild.

Pendle PT payout assets: first deploy (migration 102)

belongs to: migration 102 (the first three Pendle PT payout assets) — v0.59.0
executed: unverified

apply it after the deploy, not before. All three rows (USDat, trUSD, iUSD) carry wallet_tracked = true, so applied first against the old bundle the idle sweep would start reading these balances while the previously-compiled disclosed-holdings constant knows nothing about them. Nothing else is required: the three price tapes are bought automatically on the first standing refresh-assets tick after the DEPLOY (not after the migration), because the feed set answers from the compiled seed and the database is an overlay on top of it — docs/database.md's portfolio_tokens row carries that argument, the three 90-day executions and the ~12-credit one-off cost. scripts/backfill-token-price-bars.ts is a REPAIR tool here, not a gate.

The coverage rule: first deploy (migration 105)

belongs to: migration 105 (the coverage rule: 18 new rows, 11 rows swept, 11 rows retired) — v0.64.0
executed: 2026-09-17

Run 2026-09-17 (v0.64.0): applied after the deploy by the bare runner at 13:22:21Z, in one run with 106 and 107; the three counts read 18 | 11 | 25.

Apply it after the deploy, not before. 105 flips wallet_tracked on for eleven rows, four of which (WBTC, cbBTC, LBTC, eBTC) the OLD bundle reports through its compiled disclosed-holdings constant. Applied first, the old idle sweep would start reading those balances — not doubled (the old bundle filtered its disclosed list against the stored spine, so a holding the pipeline had started storing was reported once), but served into the wrong view: the old bundle's one branch for a book-NULL idle wallet leg put such a row inside the USD statement, so a bitcoin balance would have sat in the dollar view for the length of the window. Applied after the deploy there is no such window, because the new bundle serves a no-base leg into the All view only.

The three parts do not all take effect at the same moment, and it is worth knowing which is which while the window is open. The eighteen new rows are live at the deploy: the registry overlays the database on top of the compiled seed, and no row exists for any of them yet, so the seed's row stands and they are swept (and their feeds declared) from the first process start. The eleven tracking flips and the eleven retirements wait for this file: those addresses do have rows, and the sweep universe is a direct read of portfolio_tokens.wallet_tracked where status = 'active', so the database's value wins until the migration lands. Between the deploy and the migration a bare WBTC balance is in NEITHER place — not swept, and no longer disclosed — so it is absent rather than doubled, which is the smaller of the two failures and the one this order chooses. Do not leave the runbook half-run overnight.

bash

# PROD, after the release deploys.
cd /opt/onchain-credit && scripts/ops/migrate.sh "$DATABASE_URL"
# Expect 105-coverage-rule.sql in the ledger, and these three counts: 18 | 11 | 25
sudo -u postgres psql -d creddit -c \
  "SELECT count(*) FILTER (WHERE source = 'venue-live')                AS added_18,
          count(*) FILTER (WHERE status = 'retired')                   AS retired_11,
          count(*) FILTER (WHERE wallet_tracked AND book IS NULL)      AS no_base_tracked
     FROM onchain_credit.portfolio_tokens WHERE chain_id = 1;"

Nothing else is required for the PRICES. The nineteen rows declaring a dune_tape are in the standing feed set from the first process start after the code ships — the set answers from the COMPILED seed and the database is an overlay on top of it — so the mirror's auto-backfill on add buys 2026-01-01 → now on the next 0 */6 refresh-assets tick, in about three Dune executions per batch, bounded by DUNE_MAX_CREDITS_PER_RUN. scripts/backfill-token-price-bars.ts is a REPAIR tool here, not a gate.

The account wipe that goes with it is a separate step, listed under the gated repairs below: the sweep universe changes, so every stored history is a history of a narrower universe than the one the release reads.

Migration `106` (retire the discovery scope rows)

belongs to: migration 106 (the legacy-code cleanup, issue #909) — v0.64.0
executed: 2026-09-17

Run 2026-09-17 (v0.64.0): applied at 13:22:21Z; 11 portfolio:discover:% rows before, 0 after.

Deletes every chain_scan_cursors row under portfolio:discover:%: the per-wallet discovery watermarks that migration 061 superseded with the certificate on accounts. The release carrying it removes the last writer (the one-release dual-write) and the last readers (the pre-061 fallbacks), so nothing reads what it deletes. Prod held 11 such rows when the file was written; staging holds none (the nightly scrub deletes them).

Additive-shaped (a DELETE, no DDL, no -- DESTRUCTIVE tag), so staging auto-applies it. On prod run scripts/ops/migrate.sh after the deploy, in the same session as the other migrations of the release; order against the deploy does not matter for correctness:

before the deploy, the previous release re-creates a row on its next certificate stamp, and nothing reads it;
after a code rollback, the same: the previous release only ever wrote these rows on a database that has 061, and read them only when 061's columns were missing.

Verify with SELECT count(*) FROM onchain_credit.chain_scan_cursors WHERE scope LIKE 'portfolio:discover:%' (expect 0).

Follow-up (code, after this step): the staging scrub (scripts/ops/scrub-staging-pii.sql), the reseed leak check (scripts/ops/reseed-staging.sh), the user wipe (scripts/ops/reset-portfolio-users.ts) and scripts/ops/scrub-staging.test.ts keep the portfolio:discover:% prefix for this release, because a prod dump taken before 106 still carries prod's enrolled-wallet list under it. Once 106 has run on prod and a nightly reseed has restored a dump taken after it, drop the prefix from all four together.

Migration `107` (narrow the portfolio book CHECKs)

belongs to: migration 107 (the legacy-code cleanup, issue #909; specified as 078) — v0.64.0
executed: 2026-09-17

Run 2026-09-17 (v0.64.0): applied at 13:22:22Z with no replay running and no lock timeout; neither constraint names 'BTC' and no portfolio_pos_book_chk row exists.

Drops 'BTC' from portfolio_pos_book_chk2 (the partitioned snapshot spine; the parent carries it to every monthly partition, and a database where the snapshot repartition 067 never ran has the older name portfolio_pos_book_chk, which it drops too) and from portfolio_tokens_book_chk. Self-guarding: it RAISES, changing nothing, while any book='BTC' row survives in either table. Prod held none on 2026-09-17.

Untagged, so staging auto-applies it; on prod run scripts/ops/migrate.sh after the deploy. The validation scan holds ACCESS EXCLUSIVE on the spine for its duration (77 MB on prod when written: well under a second), so the 6h tick or a page load that arrives during it waits rather than fails.

The reverse wait can fail it. migrate.sh sets lock_timeout=10s, and a backfill replay holds its transaction on the spine for minutes, so 107 arriving during one aborts with canceling statement due to lock timeout. Nothing changes (the file is one transaction) and it is safe to re-run. The nightly 02:15 backup's pg_dump holds a lock on the table for the whole dump, so it causes the same abort: do not run 107 across that window. Before running it on prod, check that no replay is in flight:

sql

SELECT count(*) FROM onchain_credit.portfolio_backfill_state WHERE status = 'running';  -- expect 0

On staging the same abort turns the deploy job red after the build and before the restart; re-run the deploy workflow. Every staging deploy re-runs 107 until prod has applied it, because each nightly reseed restores prod's migration ledger. Verify:

sql

SELECT conrelid::regclass, pg_get_constraintdef(oid)
  FROM pg_constraint
 WHERE conname IN ('portfolio_pos_book_chk', 'portfolio_pos_book_chk2', 'portfolio_tokens_book_chk')
   AND conrelid::regclass::text NOT LIKE '%\_preswap';

Expect no 'BTC' in any definition and no portfolio_pos_book_chk row. (The filter leaves out 067's aside table, portfolio_position_snapshots_preswap, which keeps its old constraint wherever it still exists.) A code rollback needs nothing: the previous release cannot write 'BTC' either.

Migration `108` (drop the token_basis shadow table)

belongs to: migration 108 (the legacy-code cleanup, issue #909) — v0.64.0
executed: 2026-09-17

Run 2026-09-17 (v0.64.0): the only pending tagged file after 105–107; applied with --allow-destructive at 13:22:42Z, and to_regclass reads NULL.

Drops token_basis_dune_shadow, the scratch table of the one-time mirror rebuild of token_basis (see data-pipeline), which ran on prod on 2026-07-20. The two scripts that used it are deleted in the same release.

Tagged -- DESTRUCTIVE, so neither environment auto-applies it. On prod, after the deploy:

bash

cd /opt/onchain-credit && scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)" --allow-destructive

--allow-destructive applies every pending tagged file, so check first that 108 is the only one pending (ls scripts/sql against SELECT filename FROM onchain_credit.schema_migrations). Staging needs no step: it restores prod's dump nightly, so the table leaves staging with the first dump taken after the prod drop. Verify with SELECT to_regclass('onchain_credit.token_basis_dune_shadow') (expect NULL).

Staging env: the JIT positions fast path matches production (#909)

belongs to: the release carrying #909 (PR #912) — v0.64.0
executed: 2026-09-17

Production's .env.local sets JIT_LEDGER_LAG_BLOCKS=200, which turns the JIT positions fast path on; staging leaves it unset, so staging tests the default (off). Align staging once:

bash

# on the box
grep -q '^JIT_LEDGER_LAG_BLOCKS=' /opt/onchain-credit-staging/.env.local \
  || echo 'JIT_LEDGER_LAG_BLOCKS=200' >> /opt/onchain-credit-staging/.env.local
pm2 restart onchain-credit-staging --update-env

It is a configuration match rather than a live path: staging runs no ingester process, so its ledger tip trails head by far more than 200 blocks except right after an attended ingester hand-run, and a page load full-reads positions until then. See Environment variables.

The price-source rule: first deploy (migration 109)

belongs to: migration 109 (the price-source rule: the four rows 105 left unpriced) — v0.65.0
executed: 2026-09-18

Applied 2026-09-18 08:37:09Z on prod, after the v0.65.0 deploy (app + ingester restarted 08:36Z), with the backfill queue idle (done|4, empty|1). migrate.sh applied 109 alone; the checks read 22 for no-base tracked and the four rows exactly as expected (AA_FalconXUSDC USD (null) composed getter primary_buffer, BTC.b NULL llama_aggregate market (null) secondary, USDai USD llama_aggregate market (null) secondary, wFalconX USD (null) composed getter primary_buffer). The ingester freshness alarm read both arms ok at 08:37Z (13 feeds within the 6h bound, worst 13 min). The live sync (#919) shipped in the same release with no data step; no live tip existed on prod at 08:40Z (no page load since the deploy).

Apply it after the deploy, in the release's own migrate.sh run. It ships alone: 105 is already on prod — applied 2026-09-17 13:22:21Z alongside 106–108 with v0.64.0 — so this file is the only unapplied one the runner finds, and the state it lands on is the one that release left.

What it changes. 105 left four tracked rows with no price feed at all, valuing to a dash, and on prod today all four still read book NULL, feed NULL, valuation market. 109 gives each of them a price: BTC.b and USDai take feed = 'llama_aggregate', a fourth feed kind that fills the same hourly bar table from DefiLlama's Coins API; USDai also takes the dollar book; and the FalconX pair (AA_FalconXUSDC and its 1:1 wrapper wFalconX) becomes a composed dollar holding valued at the credit vault's own NAV. It adds no row, deletes none, and changes no quantity. It widens the feed CHECK to admit the fourth value, dropping and re-adding it by name so the file is idempotent.

Why after the deploy, and why it is sharper than 105's reason. All four rows already EXIST in portfolio_tokens, so unlike 105's eighteen new rows none of this takes effect at the deploy: the registry overlays the database on top of the compiled seed field by field, and a column the database answers — NULL included — wins. Between the deploy and this file the four rows behave exactly as they do today.

Applied FIRST, against the old bundle, three of the four rows go wrong and nothing is permanent. Every line below was read off the deployed loader and the deployed valuation path rather than reasoned from the column, which is the only way to get this right: the old bundle's loader admits three feed values (dune_tape, dune_dex_ratio, pool_quote) and maps anything else to NULL, and the overlay installs that NULL as the answer. To that bundle llama_aggregate is not "a standing feed" — it is no feed at all, and every set that decides a CSV, the spike gate or an alert is computed off the overlaid row.

BTC.b: nothing changes. Its feed arrives NULL, which is the state 105 left it in, so it is in no CSV, no standing set and on no alert, and it keeps the dash it has today. The two hazards the row invites you to assume are both impossible for that same reason: an old bundle can neither capture its hours into the Dune tape nor page for a dark feed, because each of those is judged off a feed value it reads as absent rather than as unrecognised.
USDai: the real hazard, and on its own the reason for the ordering. It arrives booked USD, class = 'par', valuation market, feed NULL — the shape booking it is only safe without because 109 declares the feed in the same file. That bundle has nowhere to buy the mark, so the leg is stored with its redemption line answering identity (quantity × 1, every holder, no read) and its market mark withheld: face value on one basis, a dash beside it on the other, and nothing flagged, because the incomplete flag reads the mark in force.
The FalconX pair: worse than a dash — the holding disappears. Both arrive booked USD and variable_rate with rate_kind = 'getter' and a getter kind that bundle has no reader for. An unknown kind logs one line and answers null; neither address has a stored share-rate series to fall back to (checked read-only on prod: no token_yield_apy row carries a share_rate for either), so the redemption resolver raises and the snapshot skips the leg (M9). No row is stored at all — the holding is in no band, no total, and not in the outside group either, since that group is where an EXCLUDED-book row goes and this row is no longer EXCLUDED. Nothing pages either: a withheld leg does not fail the run, so the only trace is one [rate-getters] no reader for rate_getter kind line in the cron log. Today the two are listed under "outside the yield book" with a dash; in that window they are not listed at all.
The widened CHECK is harmless in both directions. It only admits one more value, and the old bundle writes none of them.

So the order stands for (2) and (3) rather than for anything irreversible. No bar can be written under the wrong source — nothing puts these rows in a CSV — and a position snapshot taken inside the window is re-derivable like any other. What applying after the deploy buys is that no holder is shown a face value nothing can contradict, or a holding that vanished.

bash

# PROD, after the release deploys. 109 is the only file pending.
cd /opt/onchain-credit && scripts/ops/migrate.sh "$DATABASE_URL"
# Expect 109-price-source-rule.sql in the ledger, and these four rows:
#   BTC.b           book NULL | feed llama_aggregate | market   | liquidity secondary
#   USDai           book USD  | feed llama_aggregate | market   | liquidity secondary
#   AA_FalconXUSDC  book USD  | feed (null)          | composed | liquidity primary_buffer
#   wFalconX        book USD  | feed (null)          | composed | liquidity primary_buffer
sudo -u postgres psql -d creddit -c \
  "SELECT symbol, book, feed, valuation, rate_kind, liquidity
     FROM onchain_credit.portfolio_tokens
    WHERE chain_id = 1
      AND address IN ('0xb0f70c0bd6fd87dbeb7c10dc692a2a6106817072',
                      '0x0a1a1a107e45b7ced86833863f482bc5f4ed82ef',
                      '0xc26a6fa2c37b38e549a4a1807543801db684f99c',
                      '0x4614f7a56a3eb83b2ff9fa4b4b9575b28fb68644')
    ORDER BY symbol;"
# …and the CHECK now admits four values, not three:
sudo -u postgres psql -d creddit -c \
  "SELECT pg_get_constraintdef(oid) FROM pg_constraint
    WHERE conname = 'portfolio_tokens_feed_chk';"
# ONE of 105's three counts moves, and only after this file: the no-base one, 25 -> 22,
# because USDai and the FalconX pair stop being booked against nothing. The other two
# (18 added, 11 retired) are 105's and do not move. Read on prod 2026-09-17: 18 | 11 | 25.
sudo -u postgres psql -d creddit -c \
  "SELECT count(*) FILTER (WHERE wallet_tracked AND book IS NULL) AS no_base_tracked_22
     FROM onchain_credit.portfolio_tokens WHERE chain_id = 1;"

Nothing else is required for the PRICES. Declaring a feed enrols a token in the mirror's auto-backfill on add, and the aggregate leg loads its own floor: on the first 0 */6refresh-assets tick after this file is applied it buys 2026-01-01 → now for both rows — about 6,200 hours each, a week of hours per request, so roughly 38 keyless HTTP requests per token, paced 250ms apart, under a minute of wall clock and no Dune credits at all. There is no bulk tool to run and none is owed. The FalconX pair needs no price load whatever: it is composed, so its mark is the vault's NAV read at each leg's own block times USDC's existing bar.

The re-derive question is answered at run time, not by this page. 109 changes how four rows are VALUED, and a history stored before it keeps the old answer: a USDai leg derived while the row was booked against nothing carries book = 'EXCLUDED' and no mark, so once this file books it USD the same holding joins the dollar view part-way through its own history with nothing behind the join.

rows_on_the_four is the whole gate — not the account count. Whether prod is empty is a fact with a date on it and it decays between the day this page is written and the day it is run: the v0.64.0 account wipe emptied prod on 2026-09-17, and a read-only check during review later the same day found 5 accounts and 577 snapshot rows again, with rows_on_the_four still 0. That is the point rather than a footnote — a signed-in account holding none of the four assets owes this migration nothing, so the count that decides the branch is the second one and the first is only context for the run note. Read the scope immediately before applying:

sql

-- read-only, on prod, just before migrate.sh. `accounts` is CONTEXT — record it, do not
-- gate on it. `rows_on_the_four` is what decides the branch below, and it is normal for
-- `accounts` to be non-zero while it is 0.
SELECT (SELECT count(*) FROM onchain_credit.accounts) AS accounts,
       (SELECT count(*) FROM onchain_credit.portfolio_position_snapshots
         WHERE accounting_asset IN ('0xb0f70c0bd6fd87dbeb7c10dc692a2a6106817072',
                                    '0x0a1a1a107e45b7ced86833863f482bc5f4ed82ef',
                                    '0xc26a6fa2c37b38e549a4a1807543801db684f99c',
                                    '0x4614f7a56a3eb83b2ff9fa4b4b9575b28fb68644')) AS rows_on_the_four;

rows_on_the_four = 0 → record both figures in this step's run note and skip the re-derive, whatever accounts says — including a non-zero accounts, which is the expected reading now that prod takes sign-ins again: no stored leg is valued off any of the four rows, so there is nothing for a re-derive to restate.

rows_on_the_four > 0 → it is owed, in the same window as the migration. List the wallets (SELECT DISTINCT wallet FROM onchain_credit.portfolio_position_snapshots WHERE accounting_asset IN (…the same four…)) and re-derive each one RANGED from its own floor to its own cursor, the shape the forced-pull step spells out in full (scripts/build-shadow-ledger.ts --wallet <w> --from <floor> --to <cursor> --range 50000, floor and cursor read per wallet, never copied from a page). If the list is long enough that the campaign costs more than the history is worth, wipe instead, exactly as 105 did — but then it is a gated step of its own and needs Fred's approval before it runs.

No ISR step. /portfolio IS prerendered, but as a static shell with no revalidate window (next build lists it ○ with no timer, unlike /carries at 5m or /asset-profiles at 1h). Every figure a signed-in reader sees on it is fetched client-side from the portfolio APIs, so the new rows appear on the next request with no page to re-render and nothing to purge. No screener page reads portfolio_tokens.feed, so none of the timed routes is stale either.

And no prerender/migrate deadlock, the trap a registry migration has to clear before the order above is safe: 109 adds no column. It widens one CHECK and updates four rows, so the bundle the staging deploy prerenders against reads exactly the columns it already reads, whether or not this file has been applied yet.

Rollback. The UPDATEs are reversible by hand (set the four rows back to feed NULL / book NULL / valuation market). A code rollback needs that revert, in the same window: rolling the bundle back while these rows stand puts the product in exactly the state (2) and (3) above describe — USDai at an uncontradicted face value, the FalconX pair gone from the page — which is the same window this ordering exists to avoid, entered from the other end. The feed value itself is harmless to an older bundle (it reads as no feed); book and rate_kind are what have to go back. Leave the widened CHECK in place — it admits a superset and nothing breaks on it.

Hourly bars only, and the pool-quote feed retired (migration 110)

belongs to: migration 110 (pricing categories sub-PR A: hourly bars only, pool feeds and the aggregator tip removed) — v0.68.0
executed: 2026-09-23

Run 2026-09-23 (v0.68.0), 13:24Z on prod, after the 13:13Z deploy and after 111/112 had gone on at 13:06Z — the order this release's step list requires. Applied by hand with --allow-destructive, as the file's tag demands; the bare runner had already listed 111 and 112 before it was run. The window between the deploy and this file, in which PST and sUSDai read as having no standing feed, was eleven minutes. The hand-run tick at 13:26Z is what closed it.

It is tagged -- DESTRUCTIVE and it means it. scripts/ops/migrate.sh SKIPS a tagged file unless it is handed --allow-destructive, so neither the staging deploy nor the prod migrate.sh run applies this one: it is a hand step, on staging first and then on prod, AFTER the deploy.

What it deletes, and why deleting is the honest answer. Two populations, both of them rows no writer can ever replace (bars are inserted ON CONFLICT DO NOTHING and 055 REVOKEd DELETE from the app roles):

The fine-grained bars. 054 created the bar table on a 300-second grid, for a channel that bought named 5-minute bars at a wallet's flow instants. That channel had no caller after v0.55.0 and this release deletes it, so what is left of it is a population of off-hour rows the spike gate can never judge — it reads the hourly grid, because convicting a spiked hour needs its two NEIGHBOURS to agree and an off-hour row beside it defeats that test. A bad one among them can be served as a mark for ever. The largest block is one staging incident's 12,412 contiguous rows.
The pool-quote bars (source = 'pool:fluid_dex_t1'). The feed is retired, so leaving them would pin PST's and sUSDai's series to a feed nothing fills any more, permanently and invisibly.

The second deletion opens a hole, and the file closes the door that would have kept it open. The feeds PARTITION the rows, so while the pool feed ran it was the ONLY writer either of those two had: deleting its bars leaves a hole over exactly the span it covered. Two properties of the six-hourly sync would have made that hole permanent: floor_loaded is sticky (written existing OR EXCLUDED, and the retired pool leg set it true on both rows every tick, so the planner would never floor-load them again), and the cursor is monotonic (GREATEST(existing, EXCLUDED), so it still points at the top of the hole and the next incremental window would open above it). So the migration DELETES both rows' sync-state rows. 098 creates those lazily on a token's first sync attempt, so a row without one is exactly a row that has just joined the standing set: the planner bootstraps it from its stored bars instead of a cursor, and the next tick either floor-loads it from the 2026-01-01 mirror floor or catches it up from its oldest surviving bar. Either covers the span.

What that does not promise. The tape is quiet for these two assets — the whole reason they had a feed of their own — so the re-offered span may come back thin, and for PST close to empty. The plan accepts that (R9) and names DefiLlama's hourly history as PST's filler, which arrives with the live-price work. The guarantee here is that the span is ASKED FOR by every leg that can fill it, rather than skipped silently.

No wallet is repaired and no mark is restated, because the release that carries this deletes every tracked wallet: there is no stored position history for a deleted bar to change the value of.

What else it does. It verifies every remaining bar_ts is on the hour and RAISES naming the offenders if not (an off-hour row that is not from the deleted channel is a writer nobody has accounted for); replaces the 300-second grid CHECK with an hourly one; moves the two pool-quoted registry rows to feed = 'dune_tape' with a NULL feed_config; narrows the feed CHECK to the three remaining kinds; and drops the shape-CHECK clause that required a pool-quoted row to carry its wiring.

Why after the deploy, and why the window should be short. Between the deploy and this file the two rows still hold pool_quote, a kind this release's registry loader does not admit, so it installs NULL over it and both read as having NO standing feed: no bars are bought for them, the dark-feed alert does not watch them, and their marks walk back to what is already stored. Nothing is lost permanently — a bar not bought in that window is bought by the next tick's own gap-covering sync — but the shorter the window the fewer hours have to be caught up.

The rollback direction is safe for a different reason, not the same one. An older bundle reads dune_tape as the tape kind it already knows, so it does not see the two rows as unfed: its pool leg finds no pool-quoted row to serve, its partitioner puts both on the standard CSV, and its tape leg syncs them. Hourly bars satisfy the old 300-second CHECK, so nothing it writes is refused either. The one thing to watch is that the old bundle's COMPILED pool-quote defaults are overridden only by a registry read that SUCCEEDS: a rolled back process whose registry read fails runs its pool leg off the compiled list and writes fresh pool:fluid_dex_t1 bars back into the hours this file cleared, which nothing in the database refuses (the feed CHECK constrains portfolio_tokens, not token_price_bars.source). A rollback that lasts long enough to matter therefore ends with this file re-applied.

bash

# STAGING first, then PROD, after the release deploys.
cd /opt/onchain-credit && scripts/ops/migrate.sh "$DATABASE_URL" --allow-destructive
# Check what the flag would let through BEFORE running it: it is not scoped to 110.
#   psql> SELECT filename FROM onchain_credit.schema_migrations ORDER BY filename DESC LIMIT 5;
# If another DESTRUCTIVE file is still pending on this box, apply 110 BY HAND instead:
#   sudo -u postgres psql -v ON_ERROR_STOP=1 -X -q -1 -d <db> \
#     -f scripts/sql/110-hourly-bars-only.sql
#   sudo -u postgres psql -q -d <db> -c "INSERT INTO onchain_credit.schema_migrations(filename) \
#     VALUES ('110-hourly-bars-only.sql')"

If it fails on lock_timeout, re-run it. Adding a validated CHECK takes ACCESS EXCLUSIVE on token_price_bars for the whole validation scan, and migrate.sh runs under lock_timeout=10s: a live reader holding the table at that moment aborts the file. The file is idempotent, so the fix is to run it again when the box is quiet.

sql

-- Expect: zero off-hour bars, zero pool-sourced bars, the two rows on the tape, and no
-- sync-state row left for either of them (the next tick recreates both).
SELECT count(*) FROM onchain_credit.token_price_bars
 WHERE extract(epoch FROM bar_ts)::bigint % 3600 <> 0;                  -- 0
SELECT count(*) FROM onchain_credit.token_price_bars
 WHERE source IN ('dune:prices.minute', 'pool:fluid_dex_t1');           -- 0
SELECT symbol, feed, feed_config FROM onchain_credit.portfolio_tokens
 WHERE address IN ('0x22ae3d9a738471f405169af055d31c687087d4c7',
                   '0x0b2b2b2076d95dda7817e785989fe353fe955ef9');       -- both dune_tape, NULL
SELECT count(*) FROM onchain_credit.token_price_sync_state
 WHERE chain_id = 1
   AND token_address IN ('0x22ae3d9a738471f405169af055d31c687087d4c7',
                         '0x0b2b2b2076d95dda7817e785989fe353fe955ef9'); -- 0

sql

-- AFTER THE NEXT SIX-HOURLY TICK, on the same box: both rows are back, floor-loaded, with
-- a cursor. `last_bar_ts` may still be NULL for PST if the tape printed nothing for it,
-- which is the accepted R9 outcome and not a failure of this step.
SELECT token_address, last_bar_ts, floor_loaded, consecutive_failures
  FROM onchain_credit.token_price_sync_state
 WHERE chain_id = 1
   AND token_address IN ('0x22ae3d9a738471f405169af055d31c687087d4c7',
                         '0x0b2b2b2076d95dda7817e785989fe353fe955ef9');

Rollback. The deleted bars are gone: the purge is not reversible, and it is not meant to be (nothing reads them, and the release wipes every wallet). The CHECKs, the two registry rows and the two sync-state rows are all reproducible by hand, and an older bundle needs none of them reverted — it reads an hourly bar and a dune_tape feed exactly as this one does.

The declared route and the pricing categories (migrations `111` and `112`)

belongs to: migrations 111 (pricing categories sub-PR B: the declared route, the capacity table, five category moves and two feeds withdrawn) and 112 (sub-PR E: the reported-volume measurement kind) — v0.68.0
executed: 2026-09-23

Run 2026-09-23 (v0.68.0), 13:06Z on prod, BEFORE the 13:13Z deploy, by hand from the staging checkout's copy of the release and recorded in the ledger by hand — migrate.sh reads the SQL directory of the checkout it is invoked from, and /opt/onchain-credit did not hold these files until the deploy. COINGECKO_API_KEY and DUNE_QUERY_ID_DEX_MARKET=8808795 went into /opt/onchain-credit/.env.local in the same 13:06Z window, so the first process start after the deploy saw both. 112's widened CHECKs were in place before the weekly market job's first run at 13:23Z, which is the ordering that file exists for.

It is additive, but it goes on BEFORE the deploy and BY HAND — the opposite of the usual instruction for an additive file on both counts, and the reason is what the new code reads rather than what the file writes. registry-load.ts selects the six new columns by name from the first process start, and a SELECT naming a column that does not exist throws. initRegistries is fail-open, so nothing deadlocks; what happens instead is that every process on the box — the app and every cron — answers from the SHIPPED SEED until the file lands, and a row that exists only in the database (a stablecoin admitted with sync-portfolio-tokens.ts --approve, any operator row edit) has no seed to fall back to. It vanishes from the registry for that window, and a wallet holding it values it as uncovered on every page load.

What applying it early costs instead, stated because it is the asymmetry being traded: against the OLD bundle, sDAI, sUSDS, srUSDe and sUSDD stop being marked off their own bars and start composing — the right answer arriving a few minutes ahead of the code that explains it — and USD3 starts asking for a tape the old bundle can only fill from Dune, which prints nothing for it. sGHO and sUSDf lose their feed in the same file, and the old bundle still grades feed quality over the rows that compose WITHOUT a primary buffer, which both of them still are, so for the length of the window that grader is asked about two rows whose feed is now NULL; it grades the bars they already hold and nothing acts on the verdict. All of that is a hole, not a wrong number, and it closes on the first tick after the deploy. Minutes of it is strictly cheaper than a registry overlay that is dark for every process on the box.

112 GOES ON IN THE SAME STEP, immediately after it, for a different reason. It is additive too — it drops and re-adds the two CHECKs on token_pricing_measurements so the vocabulary admits reported_median_daily_volume, the median daily volume the price vendor reports across exchanges and DEXes, which is what the category test's volume bar is judged on. Nothing READS it early, so the registry-dark argument above does not apply to it; what does apply is the other direction. The weekly market job writes that kind from its first run after the deploy, and against an unwidened CHECK the database refuses the INSERT. Applying it with 111 costs nothing either way: the previous release never writes the kind, and its reader drops a row whose kind is outside its own list, so a reading of this kind sitting in the table is invisible to it rather than a surprise.

What it carries. Six nullable columns on portfolio_tokens (mint_terms, redeem_terms, capacity_reader, terms_verified, loopable, loopable_verified), the append-only token_pricing_measurements table, one new registry row (USDD 2.0, 0x4f8e5de4…cd1a, which is what sUSDD composes onto), and a generated block of UPDATEs: the declared route on every yield-bearing row, five category moves (sDAI, sUSDS, srUSDe and sUSDD to redemption-priced; USD3 to market-priced), and the two feeds withdrawn from sGHO and sUSDf, which change category and class neither of them and simply stop buying bars nothing marks off.

Why by hand, and why the standard runner CANNOT do this one. migrate.sh resolves its SQL_DIR from the checkout it is invoked from (scripts/ops/migrate.sh, the dirname "$0"/../sql line), and /opt/onchain-credit is a checkout tracking origin/main that receives this release only inside the deploy pipeline, at its own git fetch + git reset --hard FETCH_HEAD step (deployment §2.1). Before the deploy there is no scripts/sql/111-pricing-terms.sql on that box at all. Run there, the runner loops over the files that ARE there, finds nothing pending, prints migrate creddit: 0 applied, N total on disk and exits 0 — which is exactly what an already-applied migration looks like, so the step reports success for having done nothing, and 111 is then swept up by step 4's unscoped --allow-destructive run AFTER the deploy. That is the registry-dark window this entry exists to avoid, arrived at silently. So the file is applied by hand from the release's own copy, which is the same shape the 110 entry documents as its own by-hand fallback. Pointing the runner at a checkout that DOES hold the file is not the same step either: it would apply every file pending in that checkout rather than this one, which is a blast radius no release step should carry.

Where the release's copy is, before the deploy. /opt/onchain-credit-staging is on the same box, tracks origin/staging, and the release is the merge of staging into main (deployment §3) — so at release time its checkout holds this exact file, and staging's own deploy has already applied it to creddit_staging. Confirm that before trusting it: if the commit does not match the release, scp the file from a local checkout of the release commit instead.

bash

# PROD, BEFORE the release deploys. 1. the file must be THE RELEASE'S copy.
git -C /opt/onchain-credit-staging rev-parse HEAD     # = the release PR's head commit
sha256sum /opt/onchain-credit-staging/scripts/sql/111-pricing-terms.sql   # = your checkout's

# 2. apply it as the owner, in one transaction and under the runner's own timeouts (the
#    file carries no transaction control and is not tagged -- NO-TRANSACTION, so this is
#    exactly what migrate.sh would have done with it). If it aborts on lock_timeout a
#    live reader held the table; the file is idempotent, so run it again when it is quiet.
sudo -u postgres env "PGOPTIONS=-clock_timeout=10s -cstatement_timeout=300s" \
  psql -v ON_ERROR_STOP=1 -X -q -1 -d creddit \
  -f /opt/onchain-credit-staging/scripts/sql/111-pricing-terms.sql

# 3. and 112, which widens the measurement CHECKs 111 created. Same shape, same reasons,
#    and it must not be left for after the deploy: the weekly job writes the new kind from
#    its first run and the database would refuse the INSERT.
sha256sum /opt/onchain-credit-staging/scripts/sql/112-reported-volume-kind.sql  # = your checkout's
sudo -u postgres env "PGOPTIONS=-clock_timeout=10s -cstatement_timeout=300s" \
  psql -v ON_ERROR_STOP=1 -X -q -1 -d creddit \
  -f /opt/onchain-credit-staging/scripts/sql/112-reported-volume-kind.sql

# 4. record BOTH in the ledger, or step 4 of the release order will apply them a second
#    time. The files are idempotent, so a repeat is harmless; a MISSING ledger row is the
#    thing to notice, because it means the ordering this entry argues for did not happen.
sudo -u postgres psql -q -d creddit -c "INSERT INTO onchain_credit.schema_migrations(filename) \
  VALUES ('111-pricing-terms.sql'), ('112-reported-volume-kind.sql') ON CONFLICT DO NOTHING"
sudo -u postgres psql -d creddit -c "SELECT filename, applied_at FROM \
  onchain_credit.schema_migrations WHERE filename IN ('111-pricing-terms.sql', \
  '112-reported-volume-kind.sql')"   # 2 rows

sql

-- Expect: six columns present, the table created, USDD 2.0 seeded, the five moves applied
-- and the two feeds gone. Every one of these is idempotent, so a re-run changes nothing.
SELECT count(*) FROM information_schema.columns
 WHERE table_schema = 'onchain_credit' AND table_name = 'portfolio_tokens'
   AND column_name IN ('mint_terms','redeem_terms','capacity_reader','terms_verified',
                       'loopable','loopable_verified');                        -- 6
SELECT to_regclass('onchain_credit.token_pricing_measurements');               -- not null
SELECT symbol, name, book, feed FROM onchain_credit.portfolio_tokens
 WHERE address IN ('0x4f8e5de400de08b164e7421b3ee387f461becd1a',
                   '0x0c10bf8fcb7bf5412187a595ab97a3609160b5c6');
                                        -- USDD 2.0 (USD, dune_tape) and USDD (legacy)
SELECT symbol, valuation, feed, liquidity FROM onchain_credit.portfolio_tokens
 WHERE address IN ('0x83f20f44975d03b1b09e64809b757c47f942beea',   -- sDAI      composed, NULL
                   '0xa3931d71877c0e7a3148cb7eb4463524fec27fbd',   -- sUSDS     composed, NULL
                   '0x3d7d6fdf07ee548b939a80edbc9b2256d0cdc003',   -- srUSDe    composed, NULL
                   '0xc5d6a7b61d18afa11435a889557b068bb9f29930',   -- sUSDD     composed, NULL
                   '0x056b269eb1f75477a8666ae8c7fe01b64dd55ecc',   -- USD3      market,   dune_tape
                   '0xe1753f2e00940cc31213dd92013cf019dfe4ca1d',   -- sGHO      composed, NULL, secondary
                   '0xc8cf6d7991f15525488b2a83df53468d682ba4b0');  -- sUSDf     composed, NULL, secondary
SELECT count(*) FILTER (WHERE mint_terms IS NOT NULL) AS declared
  FROM onchain_credit.portfolio_tokens WHERE class = 'variable_rate' AND status = 'active';
                       -- 28 of the 46 active variable-rate rows. The other 18 are fund
                       -- shares, which are redemption-priced by definition and declare
                       -- no route.
-- And 112: the measurement vocabulary admits the reported-volume kind, on both CHECKs.
SELECT count(*) FROM pg_constraint
 WHERE conrelid = 'onchain_credit.token_pricing_measurements'::regclass
   AND conname IN ('token_pricing_measurements_kind_chk',
                   'token_pricing_measurements_anchor_chk')
   AND pg_get_constraintdef(oid) LIKE '%reported_median_daily_volume%';          -- 2

No data step follows it, and that is a decision rather than an omission. A redemption-priced row never consults its own bar, so the bars sGHO and sUSDf already hold decide nothing and are left where they are; their sync-state rows are simply never read again once the rows leave the standing set. The four rows that became composed are not re-marked either, because the release deletes every tracked wallet (below) and there is no stored position history for a changed method to restate.

Rollback. An older bundle selects its registry columns by name and never sees the six new ones, has no reason to read a table it does not know about, and can already express every one of the moved rows (composed with an underlying_address, market with a dune_tape feed, and — for sGHO and sUSDf after this file — composed with a NULL feed, which is the shape every fund share already has) — it simply values them the way the previous release decided rather than the way this one does. Nothing is dropped and nothing is deleted, so there is nothing to revert; a genuine reversal is a new migration setting the columns back.

The pricing categories release: prod steps, in order

belongs to: the pricing categories release (migrations 110, 111 and 112; plan docs/plans/pricing-categories-plan.md) — v0.68.0
executed: 2026-09-23

Run end to end on prod 2026-09-23. Step 1 (the two environment variables) 13:06Z; step 2 (111 + 112 by hand) 13:06Z; step 3 (the deploy, merge c667d43d) 13:13Z; step 5 (the wallet wipe) 13:14–13:19Z, 11 accounts and 5 tracked wallets deleted, data-only backup at /root/backup-users-2026-09-23.sql; step 6 (the weekly market measurement added to the crontab and run once) 13:23Z; step 4 (110 by hand, --allow-destructive) 13:24Z; step 7 watched on a hand-run tick at 13:26Z rather than waiting for 50 */6.
The run took 4 last, after 5 and 6, and that is a deviation worth stating rather than tidying away. What this list's ordering is actually load-bearing about held: 111 went on before the deploy, and the environment was set before the first process start. Nothing depends on 110 preceding either of the two steps that overtook it — the wipe leaves no stored history for a purged bar to have marked, and step 6's own constraint is 111, which was seventeen minutes old by then. The cost was the window in step 4's own note: PST and sUSDai read as having no standing feed for eleven minutes rather than for two, and the 13:26Z tick closed it.
5c had to be run twice (13:19Z). The step then deleted the wallet-token coverage rows BEFORE restarting the two processes, and the still-running ingester — holding the enrolled-wallet list it had started with — re-wrote 3 of the 11 rows seconds later, read them back after the restart and resumed backfilling three deleted wallets (enrolled wallets: 3). The step above has since been reordered so the restart comes first; that reordering is what the second pass is evidence for.

This is the running order for the whole release, and the order is load-bearing. Each step links to its own entry where it has one; what this entry adds is the sequence and why nothing may be reordered.

The environment, first, because the first process start after the deploy has to see it./opt/onchain-credit/.env.local on prod needs COINGECKO_API_KEY (the Demo key behind every live price and the hourly history that fills a hole the tape never printed; already set on staging) and DUNE_QUERY_ID_DEX_MARKET=8808795 (the weekly market measurement). The key is not in the repo and never will be. A missing COINGECKO_API_KEY is not an outage — the same host answers keyless at a much lower rate limit and a refused call degrades to DefiLlama and then to the stored bar — but a missing DUNE_QUERY_ID_DEX_MARKET is: without it the trading-day count never arrives and every pricing verdict stays null for ever, which is why the weekly job prints a [fail] line for it rather than a skip.
Setting the id is not the same as the query being right, because the saved query lives on Dune and no deploy can move it (scripts/dune/README.md: the vendored file is a copy, and changing the file changes nothing). 8808795 has to be at v2, the version that values a trade Dune could not price on its recognised dollar counter leg. At v1 it still answers, still exits zero, and reports USD3 as having traded on 0 of 30 days, which after fourteen days proposes pinning a market-priced asset to its redemption rate. Step 6 is where that is checked, against the data rather than against the id.
Migrations 111 and 112, BY HAND, BEFORE the deploy. Their own entry has the reasoning, the apply and the verification queries. Both are additive, but the standard runner cannot be the way they land here: migrate.sh reads the SQL directory of the checkout it is invoked from, and /opt/onchain-credit does not hold this release's files until step 3 puts them there. Run there now it prints 0 applied and exits 0, which is indistinguishable from "already applied" — and both would then be swept up by step 4, after the deploy, which is the one ordering 111 must not have. They are applied from the staging checkout's copy of the release and recorded in the ledger by hand. 112 widens the measurement CHECKs so the weekly job can write the reported-volume kind; against an unwidened CHECK the database refuses that INSERT from the first run onward.
Deploy the release the ordinary way (merge staging into main; the pipeline restarts the app and the ingester with the new build).
Migration 110, BY HAND, AFTER the deploy. It is tagged -- DESTRUCTIVE and migrate.sh skips it without --allow-destructive. Its own entry has the purge, the two sync-state deletions, the lock_timeout retry note and the verification queries. After the deploy, because between the deploy and this file PST and sUSDai read as having no standing feed at all: nothing is lost permanently, but the shorter the window the fewer hours the next tick has to catch up.
Read the ledger before running it, because its command is not scoped to 110: 111-pricing-terms.sql and 112-reported-volume-kind.sql must already be listed from step 2. If they are not, step 2 did not happen and this run is what applies them — after the deploy, with every process meanwhile answering from the shipped seed, which is exactly what step 2 is ordered first to avoid.

Delete every tracked wallet. No wallet is repaired and no history is rebuilt: this release changes how four assets are valued and where every live price comes from, so every stored history is a history of a different pricing regime from the one the release reads. Deleting is instant; re-deriving would take hours and produce a history nobody asked for. Run it between 6h ticks (the refresher runs at 50 */6 UTC), as the postgres user, after a data-only backup.

The restart comes BEFORE the two deletions, and that ordering is the step's one trap. A running ingester keeps writing the wallet-token coverage rows that 5d deletes: its in-flight enrolment backfill re-stamps them as it works. On 2026-09-23, with the deletes run first, three of the eleven rows were back seconds later; the restart then re-read the enrolment list, found those three rows and resumed backfilling three wallets that no longer existed (enrolled wallets: 3), and the delete had to be repeated. Restarting first kills that backfill, so the rows deleted after it stay deleted.

What the restart does NOT do is drop a list. The ingester re-reads its enrolled wallets every cycle, and that read (WALLET_ENROLMENT_SQL) unions the wallet-token coverage rows themselves — so between 5c and 5d it enrols the eleven wallets from those rows alone, with every account already gone. That is why 5e is read one cycle AFTER 5d and not before it.

bash

# PROD. 5a. back up what is about to be deleted. --table-and-children (pg_dump 16),
#          because -t on a partitioned table dumps the parent, which holds no rows.
sudo -u postgres pg_dump -d creddit --data-only \
  --table-and-children=onchain_credit.accounts \
  --table-and-children=onchain_credit.account_wallets \
  --table-and-children=onchain_credit.portfolio_position_snapshots \
  --table-and-children=onchain_credit.portfolio_flow_events_v2 \
  > /root/backup-users-$(date +%F).sql

# 5b. the wipe. --execute REQUIRES --confirm-db, matched against current_database().
sudo -u postgres bash -lc 'cd /opt/onchain-credit && \
  DATABASE_URL="postgres://postgres@/creddit?host=/var/run/postgresql" \
  npx tsx scripts/ops/reset-portfolio-users.ts --execute --confirm-db=creddit'

# 5c. rotate SESSION_SECRET in /opt/onchain-credit/.env.local so no old cookie resolves
#     to a deleted account, then restart both processes so they pick it up AND so the
#     in-flight enrolment backfill stops re-stamping the rows 5d is about to delete.
pm2 restart onchain-credit --update-env
pm2 restart creddit-event-ingester --update-env

# 5d. the two tables the reset script does not own. AFTER the restart, or the still-running
#     ingester re-writes the coverage rows it has just had deleted underneath it.
sudo -u postgres psql -d creddit -c \
  "DELETE FROM onchain_credit.event_coverage WHERE stream = 'wallet-token' AND address <> '*';"
sudo -u postgres psql -d creddit -c "DELETE FROM onchain_credit.portfolio_held_pts;"

# 5e. confirm the ingester enrolled nothing. WAIT ONE 60s CYCLE after 5d: the line prints
#     only when the count CHANGES, and the restart's own first read still sees the
#     coverage rows. The pass is the drop -- `enrolled wallets: 11 -> 0` (whatever N was).
#     `enrolled wallets: 11` alone is the pre-5d read, not a failure; wait for the arrow.
#     A count that settles above 0 means 5d landed while something was still writing.
sleep 70
pm2 logs creddit-event-ingester --lines 50 --nostream | grep 'enrolled wallets'

The ingester alarm's feed-lag arm starts paging about 6h later while no wallet is enrolled: with an empty list the wallet-token stream is not emitted, so its cursor stops while its rollout marker stays. The first sign-in clears it.

Add the weekly market measurement to the crontab. It is a manual add, and it must come after 111 because it writes token_pricing_measurements.
```
0 5 * * 1 /opt/onchain-credit/scripts/run-cron.sh refresh-token-market.ts
```
bash
```
# Confirm the line is there and is THIS one, the way the other manual adds are checked.
crontab -l | grep -c '/opt/onchain-credit/scripts/run-cron.sh refresh-token-market.ts'   # 1
# And run it once now rather than waiting for Monday.
/opt/onchain-credit/scripts/run-cron.sh refresh-token-market.ts
```
Run it once by hand the same day rather than waiting for Monday: the six-hourly verdict needs the market limb, and until the first run every candidate is null. One Dune execution over every token (about 0.06 credits) plus a pool-reserves walk of a few minutes keyed.
Then read USD3's row off that first run. It is the check that the saved query on Dune is the v2 this release was measured against (step 1): v1 drops every trade the vendor could not price, and USD3's only venue is a Curve pool against frxUSD where that is all of them.
bash
```
sudo -u postgres psql -d creddit -c \
  "SELECT kind, value_usd FROM onchain_credit.token_pricing_measurements
    WHERE chain_id = 1 AND token_address = '0x056b269eb1f75477a8666ae8c7fe01b64dd55ecc'
      AND kind IN ('dex_trading_days_30d', 'dex_median_daily_volume',
                   'reported_median_daily_volume')
    ORDER BY measured_at DESC, kind LIMIT 3;"
```
Expect 30 trading days and two medians in the low hundreds of thousands: the on-chain one read $318,427 on 2026-09-22 and the reported one $315,748 over the 30 days to 2026-09-23, which is the corroboration worth having — two vendors, two methods, the same market. 0 trading days is the v1 answer: fix the saved query on Dune before the fourteen-day hysteresis has anything to count, and the proposal line never appears. No row at all means the Dune limb did not run, which its own [fail] line in the job's output will have said. No reported_median_daily_volume row for ANY token means the volume limb is dark — a spent plan or a refused key — which has its own [fail] line too; every bar then falls back to the on-chain median, which understates a token that trades on exchanges.
Watch the first six-hourly tick (50 */6 UTC) for the lines this release adds. None of them is an outage by itself; they are the signals that say the new machinery is running:
- [token-pricing] proposal lines, [fail]-prefixed so a run that fails for another reason still quotes them, reporting a token whose category CANDIDATE has disagreed with its declared category for fourteen consecutive days. There can be none on the first tick: the hysteresis needs fourteen days of series. The run still exits zero, and applying a proposal is a registry edit in a pull request, never an automatic flip.
- [fail] dark feed for a standing feed with no accepted bar in 12 hours, now extended with the sources that were tried. PST is the row to read carefully: it has no CoinGecko listing, DefiLlama prices it, and most of its history will be DefiLlama-filled, which is the accepted outcome of retiring its pool feed rather than a fault.
- [fail] price disagreement where two vendors disagree beyond the band at the newest hour, and a withheld live price (an M9 dash) for an asset where both were outside it.
- [fail] the primary vendor answered for NONE of N asset(s), which is the CoinGecko door being shut rather than anything being mispriced: a spent monthly plan, a refused key or a vendor-wide outage. On the FIRST tick after the deploy read it as a key check — step 1 sets COINGECKO_API_KEY, and a box running keyless answers at a much lower rate limit, which is what this line looks like when the variable did not land.
- [partial] N row(s) have no tape bar in … while the vendors carry them. This one is NOT a failure and is deliberately not a [fail]: the hours were filled and nobody is mispriced. It says the Dune tape has gone quiet for rows the vendors still answer for, which is a question about the saved query's token coverage rather than about the product, and it is expected here for PST.
- [token-market] limb failures, if the weekly job has been run: one [fail] line per limb that wrote nothing at all, which is what tells a stopped limb from a quiet week. There are three limbs now — the Dune trades, the reported volume, and the pool depth.

What this release does NOT need, said so nobody goes looking for it: no re-mark, no wallet backfill, no history rebuild and no re-derive. The wipe in step 5 is what settles every one of those, and there is no stored position history left for a changed method to restate.

Rollback. Code first: redeploy the previous release. All three migrations are expand/contract and none needs reverting — the older bundle reads an hourly bar and a dune_tape feed exactly as this one does, ignores the six new columns, and values the four moved rows the way it always did. The one thing to watch is 110's own note: a rolled-back process whose registry read FAILS runs its pool leg off the compiled list and writes fresh pool:fluid_dex_t1 bars back into the hours 110 cleared, so a rollback that lasts long enough to matter ends with 110 re-applied.

The ledger-first release: prod steps, in order

belongs to: the ledger-first release (plan docs/plans/portfolio-ledger-first-plan.md §9 and §10; migrations 114, 115, 116 and 117; sub-PRs S0 #946, S1 #951, S3 #949, S6 #954, S2a #955, S2b #957, S5-lib #956, S4a #958, S4b #959 and S7 #960)
executed: 2026-09-26

The whole release is this one entry, and the order is load-bearing. The steps below are §9 of the plan with every sub-PR's server step folded in; the standing procedure each one leans on is in the runbook and is linked from the step. The order differs from §9 in one place, on purpose: the worker starts after the opening backfill (step 6, after steps 4 and 5), where §9 starts it before any backfill. The worker runs the reading audits, and an audit of a wallet enrolled before this release that runs before the wallet's openings exist books each missing opening as an unexplained correction (paged) and opens every unmoved holding a second time at a later reading (S4a's release blocker, plan §10). The opening backfill queues a re-audit that removes both either way, so the order costs pages, not data, but nothing is gained by taking them.

Nothing here is a wipe, a history rebuild or a re-mark, and no stored value is rewritten: the release is expand-only (R13), the old writers keep running beside the new ones for its whole life, and what a user sees moves only in the ways the comparator classes (plan §7.4: rows the ledger holds, a correction's display, dust, the retired W9 / W10 withholds, a value where the frozen one was null, late or corrected series data, and the "Updated" stamp, which S4b's amendment of §7.4 classes by name: the summary's readAt and readBlock and nothing else).

Before step 1: the staging comparison is finished (#960 review round 2, SF-1). Step 1 starts only once the staging order below is done through its step 5 (NEW captured), the plan §7.4 report on the integration PR classifies every difference between OLD and NEW (a difference in no class is a blocker on that PR, and the release waits for its fix), and the crontab restore (its step 6) is done or scheduled. That comparison is this release's approved-diff gate: run after the steps below, it can no longer stop anything.

Migrations 114, 115, 116 and 117, BY HAND, BEFORE the deploy, in that order (all four expand only; none writes a row). The new writers insert the two new kinds (114) and the rate facts (116), and the new loader selects both sets of columns from its first start: after the deploy without 114, the enrolment replay's first opening fails its INSERT and takes the replay's transaction with it, and without 116 every spine write and every page load fails. 115 creates the derivation queue, and 117 restates 115's CHECKs to admit the audit kind, so it must follow it: without it every audit enqueue is refused and no reading is ever audited. The standard runner cannot be the way they land: migrate.sh reads the SQL directory of the checkout it runs from, and /opt/onchain-credit does not hold these files until step 3. The release's copy of the files is the staging checkout's (staging's deploy has applied them to creddit_staging): confirm the commit before trusting it.
bash
```
git -C /opt/onchain-credit-staging rev-parse HEAD                      # = the release PR's head commit
git -C /opt/onchain-credit-staging status --porcelain -- scripts/sql   # prints nothing: the files are that commit's
for f in 114-ledger-first-kinds 115-derive-jobs 116-observed-rate-facts 117-audit-job-kind; do
  sudo -u postgres env "PGOPTIONS=-clock_timeout=10s -cstatement_timeout=300s" \
    psql -v ON_ERROR_STOP=1 -X -q -1 -d creddit -f /opt/onchain-credit-staging/scripts/sql/$f.sql \
  && sudo -u postgres psql -q -d creddit -c "INSERT INTO onchain_credit.schema_migrations(filename) \
       VALUES ('$f.sql') ON CONFLICT DO NOTHING" \
  || { echo "STOPPED at $f"; break; }
done
```
Each file is idempotent, so one that aborts on lock_timeout (a live reader held the table) is run again when it is quiet, and the loop resumed from it. Check: SELECT filename FROM onchain_credit.schema_migrations WHERE filename ~ '^11[4-7]-' lists all four; \d onchain_credit.portfolio_flow_events_v2 lists explain_status, cause and reading_block; \d onchain_credit.portfolio_derive_jobs exists and its pdj_kind_chk names audit; \d onchain_credit.portfolio_position_snapshots lists rate_raw and rate_source, and \d onchain_credit.portfolio_pt_prewindow_fills lists consideration_asset, consideration_raw and rate_facts.
The environment: nothing to add. No variable is required by this release. The worker and the producer have tuning knobs that default correctly and stay unset (INGEST_CONTINUOUS_BUDGET_MS, INGEST_CONTINUOUS_POPULATION_TTL_MS, and the alarm's LEDGER_*_MAX_MINUTES bounds, the ingester alarm); LEDGER_WORKER_EXIT_WHEN_IDLE is for a bounded hand-run only and must never be in .env.local, which the pm2 worker sources too.
The deploy (merge staging into main). The pipeline restarts the app, the ingester and, from this release on, the ledger worker (the deploy restarts both). On this deploy there is no worker yet, and that is expected: the restart step prints ::warning::No pm2 process named creddit-ledger-worker, … and the run stays green. Do not start the worker yet (step 6). (A roll-forward: this merge must carry a revert of the revert (roll-forward step 1), and a worker that already exists is stopped BEFORE this step, pm2 stop creddit-ledger-worker && pm2 save, so the deploy leaves it stopped.)
From here the ingester queues a continuous job for every tracked wallet a cycle sees move, and every stored reading (a page load, Synchronize, the 6h tick) queues an audit job; nothing runs either until step 6, which is intended. The page load, the tick and the enrolment replay keep deriving inline this release (WORKER_OWNS_DERIVATION is false), so nothing a user sees waits on the worker, and a wallet enrolled from now on gets its floor openings from its own replay. What moves at once:
- the values (the rate facts, 2): every row whose value reads no rate is computed from the first start and takes the bars that landed after its write;
- the "Updated" stamp: the block time up to which each wallet's movements are derived rather than its newest reading's time (the ingester's first cycle only starts the continuous producer, the cycle after it writes one follow record per wallet it follows, the follow record, and from each wallet's next 6h tick on a quiet wallet's stamp moves every ingester cycle);
- every holding that never moved inside its history window, for a wallet enrolled before this release: it has no ledger row until step 4, and is off the positions and the All view until then. So take step 4 straight after the deploy.
Read the ingester's log after its first two cycles: per-stream scanned lines, no cycle error, and no continuous enqueue error (#960 review round 3, SF-2). The continuous producer has no cursor on prod before this deploy (a roll-forward's has one: step 9.1), so its first run only starts it: it logs [ingester/derive-jobs] continuous derivation starts following at block …, queues no job and records no wallet. Every later cycle that ingests new blocks logs [ingester/derive-jobs] J continuous job(s) over [from,to] (J is 0 where no tracked wallet moved), and the first of them ends it with ; now following N more wallet(s), 0 fewer (step 9.1):
bash
```
pm2 logs creddit-event-ingester --lines 50 --nostream | grep -c 'cycle error'                # 0
pm2 logs creddit-event-ingester --lines 50 --nostream | grep -c 'continuous enqueue error'   # 0
```
The hourly alarm pages until step 6. No worker has ever written a heartbeat here, so every freshness run (5 * * * *) from the deploy on prints [fail] ledger-derive-lag: start the worker: pm2 start scripts/worker/run-worker.sh … It has never run here; N job(s) queued and exits 2, which pages (on a roll-forward, whose worker has run before, the line names its last heartbeat instead). That page is this order, not a second fault: N counts the jobs this step and step 4 queue, and the first run after step 6 reads ok. To take no page at all, deploy just after a :05 and reach step 6 before the next one.
So can the 6h reconciler, until step 4 (PR #961 final review, N4). A reconciler run (30 1,7,13,19 * * *) that falls between this deploy and step 4 reads every wallet enrolled before this release with no openings yet, so a holding whose first in-window record is a movement out of an unstated balance is W6 at its unstated opening, which pages at W6's production budget of zero (P3). That page is this order too, and the next run after step 4 does not repeat it. To take none, deploy and finish step 4 away from those half-hours.

The opening backfill, before the worker starts. One opening per holding each enrolled wallet's floor reading holds, where the ledger has none, a holding worth under a cent included (PR #963, the comparator run's F1: written like any other, and kept off the page while it is dust): dry run by default, insert only, idempotent, one transaction per wallet under the portfolio write lock, and one re-audit queued per wallet it wrote to (what it writes, and why).

bash

cd /opt/onchain-credit
set -a; . ./.env.local; set +a
# 1. Dry run: one line per opening it would write (wallet, venue, leg, block, the quantity in
#    its stored count and in the asset's units, its value, its key), one per wallet it skips,
#    and the summary. Writes nothing.
npx tsx scripts/ops/backfill-opening-rows.ts --db-url "$DATABASE_URL" | tee /tmp/opening-backfill-dry.log
# 2. Apply: the same lines, "wrote", and "re-audit queued for <wallet> over [<floor>,<newest>]".
npx tsx scripts/ops/backfill-opening-rows.ts --db-url "$DATABASE_URL" --apply | tee /tmp/opening-backfill-apply.log
# 3. Again: it must write nothing ("openings written 0", every row "present, left as is:",
#    "re-audits queued 0").
npx tsx scripts/ops/backfill-opening-rows.ts --db-url "$DATABASE_URL" --apply | tail -1

Record the three summary lines here (opening-backfill: … wallets N (skipped S); openings written W (of which dust, hidden on the surfaces, D), already present P; re-audits queued R). The dry run's to write must equal the apply's written, and its dust count the apply's. D is part of W: every sub-cent floor holding is written, none is left out (the line said dust (no row) D before PR #963, and those holdings got no row). A skipped wallet names why: no derive cursor (its build is in flight or was deferred, and its replay or its audits give it openings) or no stored reading at or after its history floor. Check:

sql

SELECT wallet, count(*) AS openings, min(block_number), max(block_number)
  FROM onchain_credit.portfolio_flow_events_v2
 WHERE chain_id = 1 AND kind = 'opening' GROUP BY wallet ORDER BY wallet;
SELECT kind, state, count(*) FROM onchain_credit.portfolio_derive_jobs
 WHERE chain_id = 1 GROUP BY 1, 2 ORDER BY 1, 2;

The rate-facts backfill: the rate facts, 3 (a dry run, --apply, and a second --apply that reports 0 write statement(s)). It needs no worker, and the two backfills are independent: this one never touches the audit's rows, which are valued from their readings.
Start the worker (the runbook, which has what healthy looks like). The kill timeout belongs to the process definition, so it is set here, once, and every later deploy's restart keeps it:
bash
```
cd /opt/onchain-credit
pm2 start scripts/worker/run-worker.sh --name creddit-ledger-worker --time --kill-timeout 120000
pm2 save
pm2 logs creddit-ledger-worker
```
It drains what steps 3 and 4 queued: the continuous jobs (priority 20), then the audit jobs (25). From here every prod deploy restarts it, and leaves it alone while it is stopped.
PT stamps (#950 review S16, plan §9 (3a)): a PT movement is valued at read from the rate stamped on it at its block (meta.ptRate / meta.ptFactor, since v0.69.0, #940), so a PT movement derived before the stamps is served stored until its wallet is re-derived. Count them, read-only. The count leaves the two audit kinds out: an opening or an adjustment on a PT leg carries no stamp (its meta holds the reading it was written from, or nothing) and is valued from its reading, and a re-derivation keeps every opening, so a count that took them in would name a wallet that held a PT at its floor for good (#960 review B2):
sql
```
SELECT f.wallet, count(*)
  FROM onchain_credit.portfolio_flow_events_v2 f
  JOIN onchain_credit.pendle_markets m ON m.chain_id = f.chain_id AND m.pt_address = f.asset
 WHERE NOT (f.meta ? 'ptRate' OR f.meta ? 'ptFactor')
   AND f.kind NOT IN ('opening', 'adjustment')
 GROUP BY f.wallet;
```
Expected: no rows (prod's accounts were enrolled after v0.69.0), and then this step is done. For every wallet it names, enqueue a rederive over the whole window (the hand-insert rules: from_block is the wallet's own floor, and to_block any block at or above the ingested tip, where the writer stops either way; the INSERT below takes the newest block the ingester has scanned), wait for done, and re-run the count, which must return no rows. The jobs are the worker's to run, so this step needs step 6 first. A re-derivation also stamps the re-derived movements' rate facts, keeps every opening and queues the re-audit of the readings it covers.
sql
```
INSERT INTO onchain_credit.portfolio_derive_jobs (chain_id, wallet, from_block, to_block, priority, kind)
SELECT 1, a.uid,
       CASE WHEN a.history_floor_ts IS NULL OR a.history_floor_block IS NULL THEN 24136053
            ELSE GREATEST(a.history_floor_block, 24136053) END,
       (SELECT max(last_scanned_block) FROM onchain_credit.chain_scan_cursors
         WHERE chain_id = 1 AND scope LIKE 'ledger:%'),
       30, 'rederive'
  FROM onchain_credit.accounts a
 WHERE a.uid IN ('<wallet the count named>', …);
```
They are done when SELECT state, count(*) FROM onchain_credit.portfolio_derive_jobs WHERE chain_id = 1 AND kind = 'rederive' GROUP BY 1 shows every one done (a partial one says in its error why it derived nothing: the hand-insert rules).

Crontab: no change this release (R13). The minutely drain and the 6h tick stay, and so does the tick's inline flow pass; the worker's alarm arms ride the existing hourly freshness line. Confirm the three lines are there and are THIS checkout's:

bash

crontab -l | grep -c '/opt/onchain-credit/scripts/run-cron.sh drain-portfolio-backfills.ts'   # 1
crontab -l | grep -c '/opt/onchain-credit/scripts/run-cron.sh refresh-portfolio.ts'           # 1
crontab -l | grep -c '/opt/onchain-credit/scripts/run-cron.sh check-ingester-freshness.ts'    # 1

Watch, in this order.
1. The first ingester cycle that sees a tracked wallet move queues a continuous job ([ingester/derive-jobs] … continuous job(s)), and the worker logs -> done for it a few seconds later. The follow records come from the cycle after the deploy's first (step 3: the first only starts the producer), whose line ends now following N more wallet(s), 0 fewer, and SELECT count(*) FROM onchain_credit.chain_scan_cursors WHERE chain_id = 1 AND scope LIKE 'portfolio:derive:v2:1:%:followed-after' is that N (the tick's population). On a roll-forward the producer's cursor is still there from the release's first deploy (neither the old code nor a user wipe touches it), so no cycle starts it: the first one scans on from where the revert stopped it (a stretch more than a day old is left to the 6h sweep, and its line names it), now following counts only the wallets with no record from before the revert, and the count is still the whole population.
2. The queue drains: the continuous jobs, then the audit jobs. Once it has:
  bash
```
pm2 logs creddit-ledger-worker --lines 5000 --nostream | grep -E 'job #[0-9]+ audit ' \
  | grep -oE -- '-> (done|partial|queued|failed)' | sort | uniq -c
pm2 logs creddit-ledger-worker --lines 5000 --nostream | grep '\[ledger-audit\]'
cd /opt/onchain-credit && ./scripts/run-cron.sh check-ingester-freshness.ts
grep 'env=onchain-credit$' /tmp/onchain-credit-cron/onchain-credit-check-ingester-freshness.log | tail -4
```
  sql
```
SELECT kind, explain_status, count(*), count(DISTINCT wallet) AS wallets
  FROM onchain_credit.portfolio_flow_events_v2
 WHERE chain_id = 1 AND kind IN ('opening', 'adjustment')
 GROUP BY 1, 2 ORDER BY 1, 2;
```
  Expected: every audit job done (a queued retry while a wallet's ledger catches up to its reading is healthy and spends no attempt), the freshness run's four ok lines (freshness, feed lag, derive lag, ledger audit), the openings step 4 wrote plus any a new enrolment wrote itself, and no unexplained adjustment (an accepted one, a native ether transfer, is fine and pages nothing). An unexplained one is a real finding: the worker's [ledger-audit] line names the wallet, venue, leg, block range and size, and the remedy is the ledger audit arm's (the runbook). Record the counts here. An audit job that ends failed (three attempts that threw) is not closed by the worker, and the derive-lag arm pages it hourly until somebody deals with it: read its error, then requeue it or close it as the runbook says (S4a).
  One cause of an unexplained correction is known and has no decoder yet: a WETH wrap or unwrap. A tracked wallet's own deposit() or withdraw() on WETH9 emits Deposit or Withdrawal and never a Transfer, and no stream ingests either event. So from this release each one books an unexplained WETH correction at the wallet's next reading: a page, a "Balance adjustment ±X WETH, cause under review" line on the activity statement, and the stretch up to that reading not measured; its ether side is an accepted native-ether correction. Its cause reads … log(s) found, … on no ingested stream, never no chain evidence: the explanation found the wrap and cannot book it. This is #956's F1, still open as Fred's decision (the contract release). For one of these, record the wallet and the transaction here and take no other step.
3. The worker's first day (its health check): the page load's skipped merges (grep -c 'writer lock busy; v2 persist skipped' /root/.pm2/logs/onchain-credit-out.log, a handful a day, compared with the same window before the deploy); the worker's verdicts, mostly -> done and no -> failed (a -> queued naming another writer is the stale-merge fence and is healthy when occasional); no continuous enqueue error; at most one open continuous job per wallet (the worker's health check).
4. Synchronize on a tracked wallet (Fred's): the response says refreshed: true (or refreshed: false with skippedReason: "nothing-newer" in the ~13 minutes after a 6h tick, when there is nothing newer to read), and the summary's readBlock is at or above the live tip it stored, unless the wallet still owes an earlier stretch: its pending marker, which SELECT last_scanned_block FROM onchain_credit.chain_scan_cursors WHERE chain_id = 1 AND scope = 'portfolio:derive:v2:1:<wallet, lowercase>:jit-pending' returns when there is one. A page load's reading counts only below that marker (#959 review round 4, B3), so readBlock can stay below the tip it just stored until the next 6h tick, or the wallet's next continuous job, derives the stretch. readAt is readBlock's own time on any explorer.
5. The first 6h tick after the start (50 */6): its audit jobs done within minutes and no unexplained correction on a clean wallet; from then on a quiet wallet's summary readBlock follows the ingester, a minute or two behind the chain (one that owes a stretch, as in 4, once that stretch is derived).
6. The first 6h reconciler run after the start (30 1,7,13,19): its values= line (the rate facts, 5): differ above zero is expected, factless=r/m/f is recorded here.
The node provider's key: Fred decides (#954 follow-up 1, its second half; #960 review round 2, SF-4). This step is Fred's decision, not the operator's: record what he decided here, and take only that. From step 3 the ingester's start line redacts its endpoints, but every ingester start before it printed the live and archive RPC URLs whole, key included, into /root/.pm2/logs/creddit-event-ingester-out.log. The reference log rotation does not cover that file (log rotation), so unless the box's copy was extended, every such line since the ingester's first start is still in it. The S6 window's staging hand-run logs (/tmp/staging-ingester-2026-09-24T*-s6.log) held the key too; #954 redacted them in place and set them to 0600 (plan §10), so check them rather than assume. Count the lines that hold the archive URL, key included, without printing it (the same with ETHEREUM_RPC_URL where that URL carries a key):
bash
```
(cd /opt/onchain-credit && set -a && . ./.env.local && set +a && test -n "$ETHEREUM_ARCHIVE_RPC_URL" && \
  zgrep -c -F -e "$ETHEREUM_ARCHIVE_RPC_URL" /root/.pm2/logs/creddit-event-ingester-out.log*)
(cd /opt/onchain-credit-staging && set -a && . ./.env.local && set +a && test -n "$ETHEREUM_ARCHIVE_RPC_URL" && \
  grep -c -F -e "$ETHEREUM_ARCHIVE_RPC_URL" /tmp/staging-ingester-2026-09-24T*-s6.log)
```
The remedies, either or both:
- Rotate the key at the provider, put the new one in /opt/onchain-credit/.env.local (and in staging's, where it is the same key), and restart what read the old one: pm2 restart onchain-credit --update-env, the ingester and the worker (restart by hand), and onchain-credit-staging if staging's file changed; the crons read the file on each run. A rotated key makes every copy on disk worthless.
- Empty the log: truncate -s 0 /root/.pm2/logs/creddit-event-ingester-out.log (pm2 appends, so the ingester goes on writing to the emptied file), and delete any rotated copy the count named. The ingester's log from before this release goes with it.
On a roll-forward the step applies again: the reverted code's ingester printed the key in use at its start during the revert.
Record every step's summary lines in this entry, and set executed: to the date. The same follow-up PR sets the plan's header (docs/plans/portfolio-ledger-first-plan.md, PR #961 final review B2) to status: built-with-deviations, with landed: naming the release version and current: naming the deviations: the worker started after the backfills, R2's never-read leg (a leg the chain holds that no reading has read is served nowhere, §11), and the decision still open with Fred on incoming ether in the fold (§11); then cd docs && npm run plans:index.

Executed 2026-09-26 (Fred: "Go no need to rotate the key"), summary lines per step.

Migrations 114–117 applied by hand from /opt/onchain-credit-staging at c0ac76b6 (14:0xZ); all four in schema_migrations; every column and the pdj_kind_chk check present.
No environment change.
Release PR #964 merged; deploy.yml run 36247157828 green; prod head 5e3923df (v0.71.0); ingester restarted 14:03:42Z, continuous derivation starts following at block 26062036, next cycle now following 5 more wallet(s), 0 fewer; 0 cycle error, 0 continuous enqueue error.
opening-backfill: APPLIED; wallets 4 (skipped 0); openings written 63 (of which dust, hidden on the surfaces, 2), already present 0; re-audits queued 4; re-run: written 0, already present 63, re-audits queued 0 (0xd775 43, 0xe51d 15, 0xea88 4, 0xef08 1).
rate-facts: APPLIED; readings: scanned 2594, lacking a fact 455, filled 455 (facts: 455 chain); movements: scanned 168, lacking a fact 7, filled 7; pre-window fills: scanned 6, lacking a fact 6, filled 6; 141 chain read(s), 12 write statement(s); re-run 0 write statement(s).
Worker started 14:05:18Z (creddit-ledger-worker, kill timeout 120000, pm2 save); the 4 continuous jobs done; audits #1 done (0 booked), #3 done (2 accepted), #2 and #4 partial (8 + 3 accepted; wallet-token legs unread at two past readings, no page).
PT-stamp count: no rows.
Crontab lines 1 / 1 / 1.
Freshness arms (direct run, 14:0xZ): ingester ok, feed-lag ok, ledger-derive-lag ok (heartbeat, 0 queued, 2 partial in 24h), ledger-audit ok (no unexplained adjustment, no leg unread twice). Ticks 18:00Z, 00:00Z, 06:00Z, 12:00Z wrote their checkpoints; by 2026-09-27 16:30Z: 18 audits done, 3 partial, 27 continuous done, 20 accepted native-eth-transfer corrections, 0 unexplained. Fred's wallet 0xe159…6b0b enrolled 19:21Z, replay done, its page-load reading at block 26063745 audited done. The "Updated" stamp trails the chain by the ingester's 64-block finality margin (~13 min): issue #965 (serve the tip, pending finality).
Key rotation: Fred decided NOT to rotate (2026-09-26).
This record. Staging's reseed pause (/root/.reseed-paused) removed 14:1xZ.

The rate facts: migration `116` and its backfill

From this release the money columns are computed at read from each row's facts, and a row whose value reads a rate no fact answers is served its stored columns (the loader rule). These steps (S2b) take the environment from "every rate-reading row served stored" to "every row computed that a fact can compute"; the list above says where each one falls in the release's order.

Migration 116, before the deploy (step 1 applies it with the other three). Expand only: nullable columns and two CHECKs, and it writes no row. The new writers insert into the columns and the new loader selects them from its first start, so a deploy that lands before 116 fails every spine write and every page load until it is applied.
What the deploy moves (step 3). From here every new reading, movement and fill carries its facts. What a user sees moves at once in three places, all of them the served number catching up with the price series (plan §7.4 d1), none of them a rate:
- A row whose value reads NO rate is computed from the first start, written before 116 or not: every par, identity and excluded leg (USDC, USDT, DAI, GHO, WETH, WBTC, XAUt, …) and every movement priced from bars alone. Such a row now takes the bar covering its own moment, so it moves wherever a bar landed after its write or was corrected since (the 6h tick prices with the bars the last sync had written, routinely hours old). A live tip written before this release that stored a vendor level where no bar stood is valued from the bars (or "not measured" where none stands) until the next tick supersedes it.
- A row whose value reads a rate, written before 116, is served stored: what it served before the release.
- So a wrapper leg's history mixes the two where its last stored reading meets its first computed one: the return over that interval carries the difference between the bar its stored value was struck at and the covering bar, until the backfill (3) computes the older rows too. A row the backfill cannot fill keeps that seam.
The rate-facts backfill (step 5; the runbook): a dry run, then --apply, then a second --apply that must report 0 write statement(s). Record each run's summary line here, with the rows it left and why. It fills a fact (the chain's answer at the row's block, or the fallback its writer had) only where the served read then reproduces the stored value of every row the fact turns computed: the row it fills, and every other leg of the same reading that reads the same rate, since the read pools a reading's facts over its legs (review S8 of #957). Where one of those legs does not reproduce, that rate is filled nowhere in the reading. So every row the backfill turns computed serves what it served before, up to late or corrected series data or a hole it fills, and every row it leaves is served exactly as before (a row written before 116 that no fact reaches stays served stored). The summary counts apart, as computed through another leg's fact, the legs it does not fill that the read computes through a fact another leg of their reading holds (typically a PT leg over a composed payout asset); they are not among the rows left.
bash
```
cd /opt/onchain-credit
set -a; . ./.env.local; set +a
npx tsx scripts/ops/backfill-rate-facts.ts --db-url "$DATABASE_URL" | tee /tmp/rate-facts-dry.log
npx tsx scripts/ops/backfill-rate-facts.ts --db-url "$DATABASE_URL" --apply | tee /tmp/rate-facts-apply.log
npx tsx scripts/ops/backfill-rate-facts.ts --db-url "$DATABASE_URL" --apply | tail -1   # 0 write statement(s)
```
PT stamps (step 7): a PT movement derived before v0.69.0's stamps is served stored until its wallet is re-derived; a re-derivation also stamps the re-derived movements' rate facts.
The check (step 9). The next 6h reconciler run prints, per wallet and summed, differ= and factless=r/m/f on its values= line (the acceptance reconciler). Expect differ above zero: it counts the computed rows whose value moved off the stored one by a bar that landed or was corrected after the write (2), and scripts/ops/value-parity.ts --db-url … classes every such line as late or corrected series data; a line it leaves as RESIDUE is the finding to chase. factless is what is still served stored: the rows the backfill left (its summary says why), a PT movement with no stamp (4), and the two classes the writers keep producing that no fact column holds (a reading over wFalconX-style two-level compositions, a PT leg over a composed payout asset: the loader rule). Record the numbers here. The contract release that drops value_market / value_redemption requires 0/0/0, so it owes a decision on those two classes first (below).

Rollback, and a roll-forward after it

Rollback.

Stop the worker first: pm2 stop creddit-ledger-worker && pm2 save. It runs this release's code, and once the release is reverted nothing it would still do is wanted (the old code produces no job and never reads the queue). Stopped before the revert, it writes nothing over the old code's rows in the minutes the revert takes to deploy. Until the revert's deploy lands, the hourly freshness check is still this release's, so every run (:05) more than ten minutes after the stop pages for the stale heartbeat until then: once, for a revert that lands within the hour. That page is this step; the old check has no worker arm.
Revert the release commit on main. The release landed as its release PR's merge commit (the Merge pull request #… from FredCoen/release/… that shipped it, in git log --first-parent origin/main): revert it with git revert -m 1 <that commit> on a branch off main, and merge that as a PR into main. The revert's own deploy runs the workflow as it stood before this release: it restarts the app and the ingester on the old code, and does not know the worker, which stays stopped. Keep the revert off staging while a roll-forward is intended: back-merged alone (the runbook's hotfix rule), it would take the release out of staging's next deploy too, and a roll-forward brings it back only with a revert of the revert (roll-forward step 1).
Delete the two new kinds as soon as the revert's deploy is green, before the next 6h tick (50 */6) or reconciler run (30 1,7,13,19). Not optional: the old loader counts opening and adjustment rows and skips them, but three other parts of the old code do not tolerate them (#960 review B1):
- The events list fails. /api/portfolio/events reads every kind of a wallet's ledger and throws on a kind outside its twelve, so it answers 500 ("Could not load events.") for any page that holds an opening or an adjustment: a wallet's last page, since its floor openings are its oldest rows, and every page of a wallet whose ledger is all openings (in the staging population, 0xd775, whose 43 legs never moved, fails on its first).
- The 6h tick refuses its merges and pages [partial] every 6h. The old range merge requires its wallet-token universe to hold every asset its range's stored wallet-leg rows name, and it counts rows of every kind, while that universe holds only tokens a Transfer moved in the range (it has no held-now term). An opening or a correction on a wallet leg nothing moved in the range falls outside it: the merge refuses (unresolvable-universe) and marks the range pending, and the same row refuses the range again at every later tick and page load, so it never clears. The routine case is the accepted native-ether correction: gas moves native ether without a log, so every wallet that transacts has them, and each sits at a reading block above the tick cursor (the tick derives to 64 blocks below the block it reads at). This release's copy of the check leaves the two kinds out (S4a); the old code's does not.
- The reconciler pages P4. It carries a kind it does not know through to its reducer on purpose, so a window the old engine booked with an opening or an adjustment inside it reports completeness none ([fail] ledger-reconcile unexplained, exit 2).
The old code writes neither kind, so nothing re-creates them once they are gone. Record the per-wallet counts here first (the roll-forward reads them), then delete, then check:
sql
```
SELECT wallet, kind, explain_status, count(*)
  FROM onchain_credit.portfolio_flow_events_v2
 WHERE chain_id = 1 AND kind IN ('opening', 'adjustment')
 GROUP BY 1, 2, 3 ORDER BY 1, 2, 3;
DELETE FROM onchain_credit.portfolio_flow_events_v2
 WHERE chain_id = 1 AND kind IN ('opening', 'adjustment');
SELECT count(*) FROM onchain_credit.portfolio_flow_events_v2
 WHERE chain_id = 1 AND kind IN ('opening', 'adjustment');   -- 0
```
A range a page load's merge refused between the deploy and the delete is marked pending, and the next tick derives it, now that nothing refuses it.
No migration is rolled back. All four are expand only, and the old code never names their columns. The queue is left as it stands: the old code never reads it.

A roll-forward after it.

Bring the release back with a revert of the revert (#960 review round 3, SF-1). Once main holds rollback step 2's revert, git treats the release's commits as merged: a later merge of staging, or of a branch cut from it, brings only what was committed since, never what the revert took out. Step 3 as written (a release PR from staging) would ship the old code with any fix on top, release.yml would tag that with the bumped version, and step 4 would fail, its tool not being on main. So the roll-forward's merge carries a revert of the revert, one of two ways:
- On a branch off main: git revert <the commit rollback step 2 made>, or git revert -m 1 <the revert PR's merge commit>, then git merge origin/staging when the roll-forward carries a fix made on staging, opened as a PR into main, whose merge is step 3's deploy. Afterwards bring staging level with main: git push origin origin/main:staging, or, where staging has moved on, a PR merging main into it.
- On staging, then the usual release PR: on a branch off staging, git merge origin/main (the revert, which takes the release out of the branch) and then the same git revert (which puts it back), merged into staging as one PR. Staging's code does not change, and its history now holds the revert and its revert, so step 3's release PR from staging carries the release again. Take this way before a fix to the release's own files lands on staging: after that, the back-merge meets the fix as a conflict (a file the revert deletes and the fix changes), and the first way is the one that works.
Either way staging ends up carrying the release again, as it must: the next release PR ships from it.
A fix the roll-forward carries that moves a served number takes the precondition again (before step 1): the plan §7.4 report is re-run over the surfaces the fix changes, with NEW captured with the fix (plan §7.3), every difference is classed on the fix's PR, and step 3 waits for it; a difference in no class blocks the roll-forward as it blocked the release. A fix that moves none (its goldens byte-identical, plan §6) keeps the report the release was approved on.
Clear the rate facts the revert made stale. The old code re-reads a reading at an existing label (the 6h tick's same-window re-read) and upserts its new values, block and updated_at, but keeps the fact of the earlier read, from which the new code would compute. So before the roll-forward's deploy, clear the facts of every reading written during the revert; step 5 fills them again where their stored values reproduce:
sql
```
UPDATE onchain_credit.portfolio_position_snapshots
   SET rate_raw = NULL, rate_source = NULL
 WHERE chain_id = 1 AND updated_at >= '<the revert deploy, UTC>' AND rate_raw IS NOT NULL;
```
A movement the old code re-derived is rewritten with its whole meta, which drops its facts: it is served stored until the backfill fills it, and needs nothing by hand.
Take steps 3 to 11 again, with the worker stopped until step 6. The deploy leaves a stopped worker stopped. The opening backfill (step 4) writes only what is missing, so it restores every opening rollback step 3 deleted, and the full-window re-audit it queues for each wallet it writes to is what books the corrections again, at the readings that found them. An unexplained one is a new row then, so the ledger audit arm pages for it again: expect that for every unexplained count the rollback recorded. A wallet it writes nothing to (its floor reading holds nothing) gets no re-audit: for every wallet the counts recorded at rollback step 3 name and the apply's log names in no [backfill-opening-rows] re-audit line, enqueue a whole-window rederive (step 7's INSERT) before step 6. A re-derivation queues the re-audit of the readings it covers, which books that wallet's corrections again. What the queue kept from before the revert drains with the rest: a continuous job starts at its wallet's tick floor as it stands when it runs, and an audit past its six-hour horizon ends partial by itself. Except the jobs of a wallet deleted during the revert: that delete went through the old tool and runbook, which do not know the queue, and a job left for a wallet no account holds fails on the ledger's foreign key and pages once the worker starts. So before step 6, delete them (the user wipe's own statement):
sql
```
DELETE FROM onchain_credit.portfolio_derive_jobs j
 WHERE NOT EXISTS (SELECT 1 FROM onchain_credit.accounts a WHERE a.uid = j.wallet);
```

Staging: the order before the comparator's NEW capture

The comparator's window on staging (plan §7.3, S6 preparation #954) is open until the NEW capture, and its rules and state are recorded in plan §10, where each step below is logged when it runs. Staging runs no ingester and no worker, so nothing there drains the queue by itself. The prod steps above wait for this order (their precondition).

One hand-run at a time, each with a small pool. Steps 2 to 4 are hand-runs against creddit_staging, whose role is capped at 20 connections. In the S6 window seven runs that overlapped a hand-run ingester cycle and the minutely drain failed on that cap (too many connections for role "onchain_credit_staging") and sent seven cron-failure pages (#954 follow-up 3). So run each step alone, never beside another hand-run, and with PG_POOL_MAX=4 in its environment (the cap the deploy's build uses), which bounds the shared pool a run opens beside its own connections.

Staging's own commands, never prod's (#960 review round 2, SF-2). Steps 2 to 4 run from the staging checkout, with staging's environment, over the population alone, and no command in them names pm2. Prod's worker runs from /opt/onchain-credit on the same box, so the prod steps' commands copied here (cd /opt/onchain-credit, its .env.local, and the pm2 stop / pm2 start around a worker hand-run) would stop prod's worker and drain prod's queue. Each of the three steps runs in a shell set up like this, its checks first:

bash

cd /opt/onchain-credit-staging
set -a; . ./.env.local; set +a
printf '%s\n' "$DATABASE_URL" | grep -c '/creddit_staging'   # 1: this shell writes staging's database
grep -c PG_POOL_MAX .env.local                              # 0: run-worker.sh sources .env.local AFTER the
                                                            #    command line, so a value there would win
W=$(node -p 'require("./tests/comparator/old/manifest.json").wallets.join(",")')
echo "$W" | tr ',' '\n' | grep -c '^0x[0-9a-f]\{40\}$'       # 12: the population, from OLD's manifest
umask 077                                                   # the logs below are readable by root alone

Deploy the integration branch to staging, schema only. Its migrations (114, 115, 116 and 117) add columns, a table and CHECKs, and write no row; the staging deploy applies them through migrate.sh. A file that writes rows waits until NEW is captured.
The rate-facts backfill of the population (the rate facts, 3): the 3,336 readings and 225 receipts, in place, at each row's own block. One of the window's two row-writing exceptions: it writes the fact columns only and leaves updated_at alone, so NEW's readings digests stay equal.
bash
```
date -u
PG_POOL_MAX=4 npx tsx scripts/ops/backfill-rate-facts.ts --db-url "$DATABASE_URL" --wallets "$W" | tee /tmp/staging-rate-facts-dry.log
PG_POOL_MAX=4 npx tsx scripts/ops/backfill-rate-facts.ts --db-url "$DATABASE_URL" --wallets "$W" --apply | tee /tmp/staging-rate-facts-apply.log
PG_POOL_MAX=4 npx tsx scripts/ops/backfill-rate-facts.ts --db-url "$DATABASE_URL" --wallets "$W" --apply | tail -1   # 0 write statement(s)
date -u
```
Check: the apply fills what the dry run said it would, and its summary lists the rows it left and why; the second apply reports 0 write statement(s). Log in plan §10 the commands, the start and end, and the three summary lines.
The opening backfill of the population (the tool of prod step 4): the other exception, which S4a's follow-up 1 added to the window's order. S4a's release blocker applies to NEW as it does to prod (without it, a population wallet's holdings that never moved are off NEW's surfaces), and the openings come from each wallet's stored floor reading, never from a new replay. It writes opening rows and queues one re-audit per wallet it wrote to; NEW's pin check reports the ledger rows' counts and digests as moved, which step 5 expects.
bash
```
date -u
PG_POOL_MAX=4 npx tsx scripts/ops/backfill-opening-rows.ts --db-url "$DATABASE_URL" --wallets "$W" | tee /tmp/staging-opening-backfill-dry.log
PG_POOL_MAX=4 npx tsx scripts/ops/backfill-opening-rows.ts --db-url "$DATABASE_URL" --wallets "$W" --apply | tee /tmp/staging-opening-backfill-apply.log
PG_POOL_MAX=4 npx tsx scripts/ops/backfill-opening-rows.ts --db-url "$DATABASE_URL" --wallets "$W" --apply | tail -1
date -u
sudo -u postgres psql -d creddit_staging -c "SELECT wallet, count(*) AS openings FROM onchain_credit.portfolio_flow_events_v2 WHERE chain_id = 1 AND kind = 'opening' GROUP BY wallet ORDER BY wallet"
```
Check: the dry run's to write equals the apply's written, and the apply logs one re-audit queued for <wallet> … line per wallet it wrote to; the second apply writes nothing (openings written 0, re-audits queued 0); 0xd775, whose 43 legs never moved, now holds 43 openings (its 1e-7 WETH included). On a re-run over the state the 2026-09-26 run left (its 97 openings in place, the population not reseeded): the dry run and the apply write the floor's sub-cent holdings that run left unopened (its dry run said dust (no row) 7; now openings written 7 (of which dust, hidden on the surfaces, 7), already present 97) and queue one re-audit per wallet written to. Step 4's drain runs those re-audits: 0x53e2's REMOVES the +0.0001 USDT correction the first run's audit booked (… reading @25857182: … 1 removed) and pages nothing, and the others are expected to book nothing (a dust leg that grew with no recorded movement would be a break, explained first, and is named in the job's reason if one does). Log it in plan §10 as for step 2.

rederive the 12 wallets (§7.3), then drain the queue: the re-derivations, and the audits that step 3 and the re-derivations queued. The INSERT is prod step 7's, over the population and against creddit_staging (psql prints INSERT 0 12):

bash

sudo -u postgres psql -d creddit_staging -v ON_ERROR_STOP=1 -c "
  INSERT INTO onchain_credit.portfolio_derive_jobs (chain_id, wallet, from_block, to_block, priority, kind)
  SELECT 1, a.uid,
         CASE WHEN a.history_floor_ts IS NULL OR a.history_floor_block IS NULL THEN 24136053
              ELSE GREATEST(a.history_floor_block, 24136053) END,
         (SELECT max(last_scanned_block) FROM onchain_credit.chain_scan_cursors
           WHERE chain_id = 1 AND scope LIKE 'ledger:%'),
         30, 'rederive'
    FROM onchain_credit.accounts a
   WHERE a.uid = ANY ('{$W}'::text[])"

Then the queue, and a bounded worker hand-run with no pm2:

bash

sudo -u postgres psql -d creddit_staging -c "SELECT wallet = ANY ('{$W}'::text[]) AS population, kind, state, count(*) FROM onchain_credit.portfolio_derive_jobs WHERE chain_id = 1 GROUP BY 1, 2, 3 ORDER BY 1, 2, 3"
PG_POOL_MAX=4 LEDGER_WORKER_EXIT_WHEN_IDLE=1 bash scripts/worker/run-worker.sh 2>&1 | tee -a /tmp/staging-worker-s7.log

Before the first hand-run every row of the query is population = t (a job of another wallet was queued outside this order: find out why before draining). Repeat both, a minute apart, until the query shows no queued and no running row: the hand-run exits once nothing is claimable, and a job sent back (an audit waiting for an earlier reading's audit, say) is claimable again only after its backoff. The hand-run's worker lock is taken in creddit_staging, where nothing else holds one (prod's worker holds its own in creddit), so it neither waits for prod's worker nor stops it. Check: every rederive ends done, and grep -c -- '-> failed' /tmp/staging-worker-s7.log prints 0. Log the job counts in plan §10.

Capture NEW: npx tsx scripts/ops/golden-portfolio.ts --db-url … --pin-from tests/comparator/old/manifest.json --label "NEW …" --out tests/comparator/new. It refuses if a wallet's newest reading moved (--override records it) and reports every mark that moved. Expected to move: both tables' column lists (the new columns), the ledger rows' counts and digests (the receipts' rate facts, rederive, the opening rows) and the derive cursors. A readings digest that moved means a stored reading was rewritten, and §7.4 must explain it.
Restore the crontab: all three PAUSED-S6 lines at once, from a file and never through a pipe (in crontab -l | sed … | crontab -, a failing crontab -l installs an EMPTY crontab, prod lines included). Verbatim from #954:
bash
```
ts=$(date -u +%Y%m%dT%H%M%SZ)
crontab -l > /root/crontab.pre-restore-s6-$ts && test -s /root/crontab.pre-restore-s6-$ts
sed 's/^# PAUSED-S6 2026-09-24 ([^)]*) //' /root/crontab.pre-restore-s6-$ts > /root/crontab.restore-s6-$ts
test -s /root/crontab.restore-s6-$ts && diff /root/crontab.pre-restore-s6-$ts /root/crontab.restore-s6-$ts
# the diff must show exactly the three PAUSED-S6 lines losing the marker, and nothing else
crontab /root/crontab.restore-s6-$ts && crontab -l | diff - /root/crontab.restore-s6-$ts && echo restored
```
The dry run on 2026-09-24 21:36Z proved it exact: the marker stripped from the live crontab reproduced /root/crontab.bak-s6-20260924T195921Z, the crontab from before the window, byte for byte. The next nightly reseed then removes the hand-inserted wallet-token rollout marker again (plan §10); whether the scrub should keep it is carried in plan §11 (#954 follow-up 2).
What the restore turns back on (#960 review round 2, N4): the 03:00 reseed, which restores prod's schema and runs no migrate.sh. Until the release's step 1, prod's schema lacks 114 to 117, so from that reseed until one restores a dump that has them, or a staging redeploy applies them, every spine write and page load of staging's app fails. So release before the next reseed, or redeploy staging after it.

The contract release: what this one leaves for the next

This release is expand only (R13): the old writers run beside the new ones, the stored value columns stay filled, and nothing the old code needs is removed. The contract release after it, once the audits have run clean for a week (§9), takes these, each its own reviewed change and, where it touches prod data, its own entry in this log:

Drop value_market / value_redemption from the readings (DESTRUCTIVE, -- DESTRUCTIVE, hand-run). Only once the reconciler's factless=0/0/0 across the population, which first needs a decision (Fred's) on the two classes the writers keep producing that no fact column holds: a reading over a two-level chain-rated composition, and a PT leg over a composed payout asset (the loader rule).
Flip WORKER_OWNS_DERIVATION: the page load, the registration replay and the 6h tick become enqueue-only (sync, enrol, sweep), and the worker is the only derivation writer besides the offline history builder. The flip's own gates, from #949:
- the stale-merge fence's four Postgres cases (round-6 SF-1): a row rewritten in place, a row deleted, a delete and an insert in one transaction, and the identical re-upsert as the control, each through the fence on a fixture clone. Today's cases all insert, so a fence that compared the row count alone would pass them;
- a hand-typed sync or enrol row (SF-A, carried follow-up 9): handMadeKindRefusal returns null once the constant flips, so the flip has to tell a producer's row from a typed one (a producer-stamped column the kind's CHECK requires, or each kind re-checking its producer's guard when it runs);
- the fence stays on every tick-range kind, sweep and sync included (true today, pinned); the re-derivation trigger moves onto rederive jobs (carried 1); a completed sweep notes its scanned scope (carried 2); the [partial] paging contract carries onto partial jobs (carried 4); failed sweep jobs fold into one standing row as continuous ones do; sync jobs are deduplicated per wallet (F1); settleSweep counts only the jobs still in the table (F2); and a chronically partial wallet holding the tick cursor is a decision for Fred (S8c);
- one lane (carried 8, a decision for the flip): a sync job waits behind a running whole-window enrol, so once the page load's derivation is a sync job, a page load during a large wallet's enrolment times out to derivation-pending. A second lane for sync, or sync claimed between an enrolment's chunks;
- the per-job reads, measured again at the flip's volume (carried 7): every job re-reads the scope registries, the rolled-out set, the ingested tip and, when it inserted legs, the PT universe, and the fence adds its two fingerprint reads (about 32 ms on a 100,000-row wallet, round-6 N1). A short per-loop cache is the easy win; a per-wallet ledger version that every writer bumps is the fence's constant-time form;
- base defect 5's mirrored order stays open for the history builder. A whole-window merge re-derives only through the tip it read when it began, so rows a tick-range merge commits above that tip meanwhile keep their old anchors, below the tick cursor. Carried 1 closes it for the population's tick; the offline history builder stays a direct writer after the flip, so a history build keeps the order open until it too runs through the worker;
- round 5's two nits: the SF-B fixture recipe gains the head-bound case (to_block the chain head with the arm above the ingested tip, which only the tip guard catches), and historyVerdict's empty-range reason stops saying "deferred" for a rederive (only an enrol writes the deferral marker): it says the job derived nothing and when to enqueue it again;
- the deploy's restart turns a missing worker into a red run by itself: restart-ingester.sh reads the constant from the checkout it deploys. Until the worker restarts on the flip's deploy, an old-code worker refuses the new page load's sync jobs (partial, so refreshed: false), for the minute or two the restarts take. Restarting the worker before the app on that one deploy closes it.
Capacity and the soak, before an outside cohort (#949's later waves; fine at today's population): an old pending marker widens every continuous job for its wallet (a job starts at its wallet's pending marker when that is below the tick cursor); NAMED_ADDRESSES_SQL binds the whole population as one parameter every cycle; and the worker soak runs no whole-window writer (the fence's fixture cases pin that class; an inline replay or trigger merge fired at random in the soak would cover its interleavings).
Retire the minutely drain line and the tick's inline flow pass (§9, R13): after the flip the drain's derivation and the tick's are the worker's enrol and sweep jobs, so the crontab's * * * * * … drain-portfolio-backfills.ts line and the tick's own flow pass go, once whatever else the drain still runs (the registration replay's spine half, the re-derivation trigger until it is a rederive producer) has a home. A crontab change, recorded here when made.
The explanation corpus's missing slots (#956 round-2 SF-3 and N4): a routed Morpho Blue WithdrawCollateral (Bundler3's adapter as receiver), one Blue Liquidate, and an ERC-4626 Deposit log, so that dropping any of them from its filter fails a recorded case. (The same review's SF-1, the checksummed-copy cell, is fixed in S7.)
Delete the W9 / W10 residue (§9): whatever still names the two shapes this release retired.
Decide WETH wrap and unwrap (#956 F1, Fred's decision; asked for before S4 shipped, and open when this entry was written). WETH9 moves a balance by Deposit and Withdrawal, never by a Transfer, and no stream ingests either, so every direct wrap or unwrap by a tracked wallet books an unexplained WETH correction (step 9.2): a page, a "cause under review" line and a stretch that is not measured. Either ingest and decode the two events (a stream, a decoder, and a re-derivation of the wallets that wrapped, which replaces each correction with the movement), or accept a cause for them in accepted-causes.ts (no page, and the line joins the accepted corrections' summary line, R5c). Decided 2026-09-27 (issue #966): ingest and decode, with native ether's own receipts, in the native-ether receipts release; this item leaves the contract release's list when that entry has run.

Everything else the sub-PRs carried, which is neither a step of this release nor the contract release's (writer checks before an outside cohort, the fixture and the harness, the adapter contract's routing, docs, staging), is listed in plan §11.

The native-ether receipts release (migration `118`): prod steps, in order

belongs to: issue #966 (native ether's transfers, fees and WETH9 wraps as ledger receipts; the native-eth block feed; the audit's transfer listing), migration 118
executed: pending

What it changes for a user, in one line: native ether stops producing a balance adjustment at almost every reading. Its transfers, its network fees and its wraps and unwraps become proper lines, and ether a contract pays a wallet is found at the next reading and recorded as a transfer. The existing accepted adjustments are replaced by those lines when step 7 re-derives each wallet. Expand only: nothing is wiped, and a code rollback leaves native-eth rows that no previous-release read names (every raw_events read names its streams, and the previous release's derivation list does not name this one).

Staging first. The acceptance issue #966 asks for (the feed's provider calls measured with the 12 comparator wallets, and their accepted native-ether corrections falling to zero after a re-derive) is taken on staging BEFORE this release, by the attended order below; steps 3 and 7 here repeat its two checks on prod. Take this release only once that order's step 5 reads clean, or once Fred has decided to go without it (its preconditions name the case: staging's archive endpoint not answering the listing).

Migration 118, BY HAND, BEFORE the deploy. It adds 'native-eth' to the two stream CHECKs (raw_events_stream_chk as NOT VALID, catalogue only, the 082 pattern; event_coverage's inline and validated). The new ingester's first cycle writes rows and coverage under the new name; without 118 both writes are refused (the feed step fails soft every cycle and holds its cursor, so nothing is lost, but nothing is read either). As for 114 to 117, the file comes from the staging checkout, whose deploy has applied it to creddit_staging:
bash
```
git -C /opt/onchain-credit-staging rev-parse HEAD                      # = the release PR's head commit
git -C /opt/onchain-credit-staging status --porcelain -- scripts/sql   # prints nothing
f=118-native-eth-stream
sudo -u postgres env "PGOPTIONS=-clock_timeout=10s -cstatement_timeout=300s" \
  psql -v ON_ERROR_STOP=1 -X -q -1 -d creddit -f /opt/onchain-credit-staging/scripts/sql/$f.sql \
&& sudo -u postgres psql -q -d creddit -c "INSERT INTO onchain_credit.schema_migrations(filename) \
     VALUES ('$f.sql') ON CONFLICT DO NOTHING"
```
Idempotent; one that aborts on lock_timeout is run again when quiet. Check: SELECT pg_get_constraintdef(oid) FROM pg_constraint WHERE conname IN ('raw_events_stream_chk', 'event_coverage_stream_chk') names native-eth in both (the file's own acceptance block raises otherwise).
The environment: nothing to add, on a pay-as-you-go plan or above. Production's ETHEREUM_RPC_URL and ETHEREUM_ARCHIVE_RPC_URL are Alchemy's, which serves eth_getBlockReceipts (the feed) and alchemy_getAssetTransfers (the audit's listing). The feed's knobs default correctly and stay unset (INGEST_NATIVE_FEED, INGEST_NATIVE_MAX_BLOCKS, INGEST_NATIVE_CONCURRENCY, and INGEST_NATIVE_CU_PER_SEC, the feed's throughput budget: 2,000 Alchemy throughput CU a second, a fifth of pay-as-you-go's 10,000 cap; the table). Read the plan on Alchemy's dashboard first: on a plan whose per-second cap is below pay-as-you-go's, set INGEST_NATIVE_CU_PER_SEC in .env.local under that cap with the ingester's other calls left room (the sizing), and give step 6 a matching --cu-per-sec.
The deploy (merge staging into main), which restarts the ingester and the worker. The ingester's first cycle starts the native-ether feed AT the scan tip (a stream with no cursor starts there; history is step 6's) and stamps one native-eth coverage row per enrolled wallet at that block. Read its log after two cycles:
bash
```
pm2 logs creddit-event-ingester --lines 80 --nostream | grep 'ingester/native-eth'
# [ingester/native-eth] scanned [A,B] wrote N row(s) · C call(s), X CU, Y throughput CU · W tracked wallet(s) · coverage begun for W   (first cycle)
# [ingester/native-eth] scanned [A,B] wrote N row(s) · C call(s), X CU, Y throughput CU · W tracked wallet(s)                          (after)
```
What moves at once, and why step 7 should not wait long. The ether leg is an ordinary leg from this deploy on, booked by the law rather than 0 by construction (CN-3 is gone from every coverage report). So until step 7 re-derives a wallet, each ACCEPTED native-ether correction it already carries reads as what any correction reads as: its stretch of the ether leg is "not measured" (W11) instead of folded under CN-3. No figure is invented either way, and step 7's re-audits replace those corrections with the movements, which is what ends the stretches.
C is at most 2 × (B − A + 1) and X at most 40 × (B − A + 1), whatever W is: at most two calls per block (one where no tracked wallet took part). A feed error line is a failed read the next cycle retries; one that repeats every cycle is an outage (a refused method prints a one-time does not serve eth_getBlockReceipts warning and falls back instead); a line naming blocks read "with their transactions' own receipts after their block receipts failed the check" is the feed getting past a block the provider answers inconsistently, not a failure. Record the first cycles' blocks, C, X, Y and W in the PR beside staging's measurement (the estimate: at most 40 CU a block, about 8.6M CU a month).

Stamp the native-eth rollout marker, AFTER step 3's first cycle wrote the stream's cursor. From the marker on, the ingested tip the continuous producer and the served stamp follow waits for the feed's cursor too, so no range is derived ahead of the ether rows in it. Before the cursor exists the marker would read it as 0 and pin the tip there, which is why this is not earlier:

bash

sudo -u postgres psql -d creddit -c "SELECT last_scanned_block FROM onchain_credit.chain_scan_cursors WHERE chain_id = 1 AND scope = 'ledger:native-eth'"   # one row
sudo -u postgres psql -d creddit -c "
  INSERT INTO onchain_credit.event_coverage (chain_id, stream, address, from_block, to_block, status, updated_at)
  VALUES (1, 'native-eth', '*', 22527558, 22527558, 'live', now())
  ON CONFLICT (chain_id, stream, address) DO UPDATE SET status = 'live', from_block = LEAST(onchain_credit.event_coverage.from_block, 22527558), updated_at = now()"

The marker is the on-switch only (claim (a), the rollout marker): native-eth is in no venue's required scopes, so it certifies nothing and withholds no wallet.

Reset wallet-token's per-wallet certificates (WETH9's Deposit and Withdrawal joined its topics, so every row below the change was earned without them). Stop the ingester for the delete, so no in-flight catch-up re-stamps a row underneath it, then start it: each enrolled wallet's catch-up re-earns its row over its window with the widened topics, three wallets a cycle.

bash

pm2 stop creddit-event-ingester
sudo -u postgres psql -d creddit -c \
  "DELETE FROM onchain_credit.event_coverage WHERE chain_id = 1 AND stream = 'wallet-token' AND address <> '*';"
pm2 start creddit-event-ingester && pm2 save
# wait for the catch-ups: no wallet left without a live row
sudo -u postgres psql -d creddit -c "
  SELECT count(*) AS enrolled_without_row FROM onchain_credit.account_wallets w
   WHERE NOT EXISTS (SELECT 1 FROM onchain_credit.event_coverage c
                      WHERE c.chain_id = 1 AND c.stream = 'wallet-token' AND c.address = lower(w.wallet) AND c.status = 'live')"

While a wallet's row is missing, its derivation defers (the enrolment gate), exactly as at a signup; the query reaching 0 ends it. Take step 7 only after it does.

The native-ether history run, once, after step 3 (the run refuses while an enrolled wallet has no coverage start) and before step 7: the same two reads per block over the enrolled wallets' window, for all of them at once, lowering each wallet's coverage start over what it read (how). Its --from is the oldest reading any enrolled wallet has:
bash
```
cd /opt/onchain-credit
set -a; . ./.env.local; set +a
FROM=$(sudo -u postgres psql -At -d creddit -c "SELECT min(block_number) FROM onchain_credit.portfolio_position_snapshots WHERE chain_id = 1")
# 1. dry run: the blocks and the most calls and CU (two and 40 per block); reads and writes nothing
npx tsx scripts/ops/backfill-native-eth.ts --db-url "$DATABASE_URL" --from "$FROM"
# 2. apply: resumable and idempotent, one line per 500-block window (its calls, CU, throughput CU)
npx tsx scripts/ops/backfill-native-eth.ts --db-url "$DATABASE_URL" --from "$FROM" --concurrency 16 --apply | tee /tmp/native-history.log
```
About 216,000 blocks for a 30-day window (at most 432,000 calls and 8.6M CU, once). It spends at most --cu-per-sec throughput CU a second (3,000 by default, beside the live feed's 2,000, under pay-as-you-go's 10,000), so the width only keeps enough blocks in flight to use that budget: with today's population most blocks need no receipt (20 throughput CU each) and the run is bound by the endpoint's latency, which --concurrency 16 spreads. Check: its last line reads APPLIED, and SELECT count(*) FROM onchain_credit.event_coverage WHERE chain_id = 1 AND stream = 'native-eth' AND address <> '*' AND from_block > $FROM is 0.

GATED: re-derive every enrolled wallet over its window, so the ether rows steps 3 to 6 wrote become ledger rows, and the re-audits the re-derivations queue replace the existing accepted native-ether adjustments with them (each reading's audit restates its rows to the residue that is left). A prod write: take it with explicit approval, after step 5's query reached 0 and step 6 applied.

bash

sudo -u postgres psql -d creddit -v ON_ERROR_STOP=1 -c "
  INSERT INTO onchain_credit.portfolio_derive_jobs (chain_id, wallet, from_block, to_block, priority, kind)
  SELECT 1, a.uid,
         CASE WHEN a.history_floor_ts IS NULL OR a.history_floor_block IS NULL THEN 24136053
              ELSE GREATEST(a.history_floor_block, 24136053) END,
         (SELECT max(last_scanned_block) FROM onchain_credit.chain_scan_cursors
           WHERE chain_id = 1 AND scope LIKE 'ledger:%'),
         30, 'rederive'
    FROM onchain_credit.accounts a
   WHERE EXISTS (SELECT 1 FROM onchain_credit.portfolio_flow_events_v2 f
                  WHERE f.chain_id = 1 AND f.wallet = a.uid)"

The population is every wallet the ledger holds rows for, each over its own window (its account row's recorded floor, the hand-insert rules); to_block is the newest block any stream has scanned, where the writer stops either way.

The worker runs them (and the audits they queue). Done when SELECT kind, state, count(*) FROM onchain_credit.portfolio_derive_jobs WHERE chain_id = 1 AND kind IN ('rederive', 'audit') GROUP BY 1, 2 shows no queued or running row. Then the acceptance, read before and after:

sql

-- the accepted native-ether corrections: expected to fall to (near) zero
SELECT count(*) FROM onchain_credit.portfolio_flow_events_v2
 WHERE kind = 'adjustment' AND explain_status = 'accepted' AND cause = 'native-eth-transfer';
-- the ether receipts: transfers, fees and wraps now on the ledger
SELECT kind, count(*) FROM onchain_credit.portfolio_flow_events_v2
 WHERE position_key = 'wallet:token:0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee' GROUP BY 1 ORDER BY 1;

What pages now: a native-ether residue left after the feed and the listing both answered for its range is unexplained (a missed movement or a source defect, never routine), and the audit's page names it. An accepted one is left only where a source could not speak (a listing outage, a block the feed did not read).

Staging, before the release: the acceptance issue #966 asks for

Two checks, on the 12 comparator wallets (plan §7.1, tests/comparator/old/manifest.json): the feed's provider calls per cycle, measured, and those wallets' accepted native-ether corrections (45 on 2026-09-26, plan §10) falling to zero after a re-derive. Staging runs no ingester and no worker (the hand-run rule), so every step is an attended hand-run from the staging checkout, one at a time, with PG_POOL_MAX=4, over the population alone, exactly as the ledger-first staging order ran; no command here names pm2 or touches creddit. Each step's command, start and end and summary lines go into the PR's comment.

0. The shell, and the preconditions, read rather than assumed. This PR merged into staging and deployed (the staging deploy's migrate.sh applies 118, an expand-only file); the population still enrolled (the 03:00 reseed truncates the accounts graph: /root/.reseed-paused present, or the population re-enrolled first as plan §7.2 did); and staging's endpoints answering the two methods, the live one eth_getBlockReceipts (the feed) and the archive one alchemy_getAssetTransfers (the audits' listing). The two probes print the start of each answer, never the URL, which carries the key:

bash

cd /opt/onchain-credit-staging
set -a; . ./.env.local; set +a
printf '%s\n' "$DATABASE_URL" | grep -c '/creddit_staging'   # 1: this shell writes staging's database
grep -c PG_POOL_MAX .env.local                              # 0: a value there would win over the command line's
sudo -u postgres psql -At -d creddit_staging -c "SELECT count(*) FROM onchain_credit.schema_migrations WHERE filename = '118-native-eth-stream.sql'"   # 1
W=$(node -p 'require("./tests/comparator/old/manifest.json").wallets.join(",")')
sudo -u postgres psql -At -d creddit_staging -c "SELECT count(*) FROM onchain_credit.accounts WHERE uid = ANY ('{$W}'::text[])"   # 12
umask 077
curl -s "${ETHEREUM_RPC_URL:-https://ethereum-rpc.publicnode.com}" -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"eth_getBlockReceipts","params":["latest"]}' | head -c 100; echo   # a "result":[{ …
curl -s "${ETHEREUM_ARCHIVE_RPC_URL:-https://eth.drpc.org}" -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"alchemy_getAssetTransfers","params":[{"fromBlock":"0x0","toBlock":"0x1","category":["external"],"maxCount":"0x1"}]}' | head -c 100; echo   # a "result":{"transfers" …

If the second probe answers method not found (staging's archive endpoint is not Alchemy's), every audit's listing is an unavailable source and the corrections cannot fall to zero here: stop after step 2 (the measurement stands on its own), and the choice between an Alchemy archive endpoint for staging and taking the second check on prod alone (step 7 above) is Fred's. Staging's key may be production's own Alchemy app: the throughput budgets below are what leave production its headroom.

1. Before. The population's corrections as they stand:

bash

sudo -u postgres psql -d creddit_staging -c "
  SELECT explain_status, cause, count(*) FROM onchain_credit.portfolio_flow_events_v2
   WHERE chain_id = 1 AND kind = 'adjustment' AND wallet = ANY ('{$W}'::text[])
   GROUP BY 1, 2 ORDER BY 1, 2"

2. The feed, measured: an attended ingester hand-run. Its first cycle starts the feed at the scan tip (a stream with no cursor starts there) and stamps one native-eth coverage row per enrolled wallet; every cycle prints the step's line.

bash

date -u
for i in 1 2 3 4 5; do PG_POOL_MAX=4 INGEST_SINGLE_CYCLE=1 scripts/ingester/run-ingester.sh; sleep 60; done \
  2>&1 | tee /tmp/staging-native-ingester-$(date -u +%FT%H%M%SZ).log
date -u
grep -h 'ingester/native-eth' /tmp/staging-native-ingester-*.log
# [ingester/native-eth] scanned [A,B] wrote N row(s) · C call(s), X CU, Y throughput CU · W tracked wallet(s) …

The measurement: for each cycle, the blocks B − A + 1, C, X, Y and W. It holds when C ≤ 2 × (B − A + 1) and X ≤ 40 × (B − A + 1) at the W staging tracks, and the calls do not depend on W by construction: they are one body per block and the receipts the tracked wallets need, which the SCALE cells in native-feed.test.ts and native-eth.test.ts hold equal at 5 wallets and at 50,000. Tear down as the hand-run rule says (no process left with the staging cwd).

3. The history run over the population's window, the tool of prod step 6, restricted to the population, from its oldest reading:

bash

FROM=$(sudo -u postgres psql -At -d creddit_staging -c "SELECT min(block_number) FROM onchain_credit.portfolio_position_snapshots WHERE chain_id = 1 AND wallet = ANY ('{$W}'::text[])")
date -u
PG_POOL_MAX=4 npx tsx scripts/ops/backfill-native-eth.ts --db-url "$DATABASE_URL" --from "$FROM" --wallets "$W" | tee /tmp/staging-native-history-dry.log
PG_POOL_MAX=4 npx tsx scripts/ops/backfill-native-eth.ts --db-url "$DATABASE_URL" --from "$FROM" --wallets "$W" --concurrency 16 --apply | tee /tmp/staging-native-history-apply.log
date -u
sudo -u postgres psql -At -d creddit_staging -c "SELECT count(*) FROM onchain_credit.event_coverage WHERE chain_id = 1 AND stream = 'native-eth' AND address = ANY ('{$W}'::text[]) AND from_block > $FROM"   # 0

Check: the apply's last line reads APPLIED with its calls, CU and throughput CU (record them), and no population wallet's coverage starts above $FROM.

4. Re-derive the population, then drain the queue: the INSERT of the ledger-first staging order's step 4 (psql prints INSERT 0 12), then bounded worker hand-runs, a minute apart, until the queue shows no queued and no running row. The worker's audits are what ask the listing, on staging's archive endpoint.

bash

sudo -u postgres psql -d creddit_staging -c "SELECT kind, state, count(*) FROM onchain_credit.portfolio_derive_jobs WHERE chain_id = 1 AND wallet = ANY ('{$W}'::text[]) GROUP BY 1, 2 ORDER BY 1, 2"
PG_POOL_MAX=4 LEDGER_WORKER_EXIT_WHEN_IDLE=1 bash scripts/worker/run-worker.sh 2>&1 | tee -a /tmp/staging-native-worker.log

Check: every rederive ends done, and grep -c -- '-> failed' /tmp/staging-native-worker.log prints 0.

5. The acceptance. Step 1's query again, and the ether receipts the ledger now holds:

bash

sudo -u postgres psql -d creddit_staging -c "
  SELECT explain_status, cause, count(*) FROM onchain_credit.portfolio_flow_events_v2
   WHERE chain_id = 1 AND kind = 'adjustment' AND wallet = ANY ('{$W}'::text[])
   GROUP BY 1, 2 ORDER BY 1, 2"
sudo -u postgres psql -d creddit_staging -c "
  SELECT kind, count(*) FROM onchain_credit.portfolio_flow_events_v2
   WHERE chain_id = 1 AND wallet = ANY ('{$W}'::text[])
     AND position_key = 'wallet:token:0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee'
   GROUP BY 1 ORDER BY 1"
sudo -u postgres psql -d creddit_staging -c "
  SELECT wallet, reading_block, qty_delta FROM onchain_credit.portfolio_flow_events_v2
   WHERE chain_id = 1 AND kind = 'adjustment' AND wallet = ANY ('{$W}'::text[])
     AND (cause = 'native-eth-transfer' OR (explain_status = 'unexplained' AND position_key LIKE 'wallet:token:0xeeee%'))
   ORDER BY 1, 2"

It holds when the accepted native-eth-transfer corrections are 0 and no unexplained one sits on the ether leg; the ether leg's receipts are transfer_in, transfer_out, cost and the wraps' internal_in / internal_out. A correction the last query still lists is either an accepted one, where a source could not speak for its reading's range (a listing that failed, a block the history run did not reach), which is a finding for the PR with the reading named, or an unexplained one, which is a missed movement and blocks the release until it is explained.

One-time repairs and rebuilds (gated)

Each of these is a prod write that needed explicit approval before it started. None is part of any standing procedure, and none is re-runnable for effect: they restate rows the chain has since settled differently. Ordered as the releases that needed them shipped.

GATED STEP: relabel the receipts stranded below the tick cursor (#777)

belongs to: the #777 settle-line release (v0.53.0)
executed: unverified

Prod held zero users after the full wipe of 2026-09-10 (docs/plans/pt-coverage-811-plan.md §0), so this step has had no population to act on since that date. That is not the same as having run: confirm against prod before deciding it is finished.

Run once, on prod, after the release that carries the settle-line fix, and only there (staging reseeds from prod's dump nightly, so a repair applied to staging is gone by morning). It is one UPDATE. It changes no amount, no quantity and no valuation — only the confidence label a movement is shown with.

What it repairs. Until this release the composition judged a row's basis against the top of the range it was handed rather than against the writer's own settle line, so each 6h tick wrote roughly 48 to 64 blocks of receipts as provisional below the cursor it then saved. Nothing re-derives below a cursor, so those rows kept that label for ever, and the Activity feed renders provisional as Confirming and withholds the stated total of any group containing one. The fix stops new rows landing there; it cannot reach the rows already written.

Read it first — the query is the issue's own (#777):

sql

SELECT count(*) AS stuck_provisional, min(block_number), max(block_number)
  FROM onchain_credit.portfolio_flow_events_v2
 WHERE chain_id = 1 AND basis = 'provisional'
   AND block_number <= COALESCE(
         (SELECT last_scanned_block FROM onchain_credit.chain_scan_cursors
           WHERE chain_id = 1 AND scope = 'portfolio:v2:tick'), -1);

COALESCE(…, -1) is load-bearing on a box whose first tick has not run yet: the scalar subquery is NULL there, block_number <= NULL is NULL, and the query would report a clean zero for the wrong reason. With the fallback it reports zero because the bound is below every block, which is the same answer honestly arrived at. Expect a count in the low thousands per month of history on a tracked population, with max(block_number) at or just under the current cursor.

Then write. Every row it touches is at or below a cursor the tick advanced only over finalised blocks, so it is settled by construction:

sql

UPDATE onchain_credit.portfolio_flow_events_v2
   SET basis = 'live'
 WHERE chain_id = 1 AND basis = 'provisional'
   AND block_number <= COALESCE(
         (SELECT last_scanned_block FROM onchain_credit.chain_scan_cursors
           WHERE chain_id = 1 AND scope = 'portfolio:v2:tick'), -1);

Re-running is a no-op (the predicate no longer matches). It is safe beside a live tick: the tick's own writes land strictly above the cursor this statement bounds on, and the merge restates rows on their primary key rather than reading basis back. Nothing needs to be stopped for it.

Verify by re-running the count above (zero), and on /portfolio's Activity feed: a group whose total previously read "One movement here is still confirming beside a settled one, so no total is stated yet" now states its amount. A band of genuinely recent movements — the ones above the cursor, which the next tick re-derives — correctly keeps the Confirming tag.

GATED STEP: the Fluid tick-padding re-mark

belongs to: the Fluid tick-padding fix (v0.53.0)
executed: unverified

Prod held zero users after the full wipe of 2026-09-10 (docs/plans/pt-coverage-811-plan.md §0), so this step has had no population to act on since that date. That is not the same as having run: confirm against prod before deciding it is finished.

Runs once, on prod, after the release that carries it, and only there: staging reseeds from prod's dump nightly, so a repair applied to staging is gone by morning. It restates every stored Fluid debt-leg snapshot from the position's net borrowing, dropping the dustBorrow tick padding the wallet does not owe (M14, The tick padding).

Dry-run first, always. The dry-run is read-only, prints the read plan before it issues a single chain call, and names every row it would touch.

Through run-cron.sh, like every other scripts/repair/* job, and not with a bare npx tsx: the wrapper is what sources .env.local, takes the per-script single-flight lock, honours the deploy-window toolchain guard and writes the [fail] line the cron alert greps. It appends every run to /tmp/onchain-credit-cron/onchain-credit-remark-fluid-dust-debt.log, so that is where the dry-run's report is read, not the terminal.

bash

cd /opt/onchain-credit                  # ETHEREUM_ARCHIVE_RPC_URL must be in .env.local:
                                        # a historical position read needs an archive node

# 0. size the run without paying for it
scripts/run-cron.sh repair/remark-fluid-dust-debt.ts --plan-only

# 1. the dry-run: every row, every delta, every skip class
scripts/run-cron.sh repair/remark-fluid-dust-debt.ts

# 2. write
scripts/run-cron.sh repair/remark-fluid-dust-debt.ts --execute

# the report of each of those runs:
tail -n 80 /tmp/onchain-credit-cron/onchain-credit-remark-fluid-dust-debt.log

Judge the run against its OWN derived population, never against a number on this page. The corpus moves on its own, and it has moved twice in 24 hours: one Fluid borrowing is still open, so the 6h refresher adds a padded row to that leg on every tick until the release lands, and a tracked wallet's historical backfill can write the first snapshot of a leg nobody had seen before (that is exactly how a fifteenth leg and a fifth wallet appeared on 2026-09-02). Every count in the run is therefore derived by the script from the rows that exist at the moment it runs, and the judgement is made on the shape and on those three lines reading consistently:

--plan-only prints the scope it derived (rows, legs, wallets) and the reads it will pay for;
the dry-run prints would repair N rows on M legs, the per-leg padding in bp, and every skip class on one line, skipped: unparseable=… no-read=… no-leg=… unrecognised=… already-net=… no-amount=…;
--execute prints executed: N rows updated.

Read them against each other, then against the reconciler's own before and after. The --plan-only and dry-run scopes must agree; executed: N must equal the would repair N of the dry-run immediately before it; and a second dry-run after the write must classify every row it just wrote already-net and report would repair 0 rows (the repair is idempotent).

STOP conditions. All of them are shape, none of them is a count (report, and do not proceed):

a leg outside the Fluid debt class on the scope line. Everything in scope is a fluid:vault:…:debt key, so anything else means the scope reached rows it must not.
unrecognised above 0. That is a stored quantity the script could not reproduce as borrow + dust, so it was refused rather than rewritten. A refusal is the script saying a row was not written by the convention this repair reverses.
unparseable above 0: a stored key the repair cannot name at all.
no-read, no-leg or no-amount above 0. Every row in scope sits far above the pinned resolver's deployment, so a withheld read is new information, not routine.
any positive delta. A debt leg can only shrink when the padding comes off.
per-leg padding outside the vault's tick spacing (roughly 0 to 15 bp of gross debt). The padding is one tick; a leg quoting more than a tick is not padding.
executed: differing from the would repair line of the dry-run immediately before it.

already-net is the one class expected to be non-zero: it counts whatever the live 6h refresher wrote net between the release landing and the run, and those rows are correct and left alone. No skip is ever written as a zero. A row the script cannot reproduce as the old convention is refused (unrecognised), and a positionByNftId that does not land is withheld (no-read); neither is guessed at, which is why a non-zero count in either class is information rather than noise.

Illustrative only, measured read-only on 2026-09-03, and stale by design: 316 rows across 15 legs and 5 wallets, costing 234 positionByNftId reads; sum |delta value_market| around 10,050 and every delta negative, with per-leg padding running 0.83 bp (NFT 18415) to 13.77 bp (NFT 17423) of gross debt. The same measurement read 307 rows / 14 legs / 4 wallets / 225 reads / 5,861.67 the day before. These figures are here to say what the output looks like, not to be matched: the run's real ones come from its own --plan-only line, which is correct by construction.

What it deliberately does not touch, and why that population is a DIFFERENT SET.portfolio_flow_events_v2 carries the same convention in to_balance (and in qty_delta on the state-read rows the venue witnesses). Those rows are DERIVED, so they are corrected by a ranged re-derivation, never patched. They are also not the legs the re-mark can see: a snapshot row exists only where the 6h refresher or a backfill read the position, while a v2 flow row exists wherever the wallet's Fluid receipts were derived. A wallet whose Fluid borrowings are all closed can therefore keep every one of its ledger rows while holding no Fluid snapshot for the re-mark to find, and which legs are in that state changes whenever a backfill runs.

Both populations move, and the gap between them moves independently of either. Illustrative, measured read-only on 2026-09-03: the ledger holds 39 rows on 16 legs across 5 wallets, 24 of them carrying a non-zero to_balance (a live borrowing, and so that block's padding), against a re-mark scope of 15 legs on 5 wallets, with exactly one leg unseen (nft:18380 on wallet 0x5d09098f…, 3 rows of which 2 carry a balance). The day before, two legs were unseen: the same wallet's nft:17764 had no snapshot until its historical backfill wrote seven, which moved that leg into the re-mark's own scope without moving the ledger's counts at all. Whatever the split is on the day, those padded ledger rows are served numbers, and the padding at a leg's last mark before its close is what lands as the phantom exit gain this whole step exists to remove: $460.27 on NFT 17764 (4.48 bp of gross debt at block 25,191,053) and $408.10 on NFT 18380 (14.16 bp at 25,392,968), against the $202 that opened the finding. On 2026-09-03 the first of those sits inside the re-mark's own scope and only the second is unseen; read the run's own list rather than assuming that split.

So step 3 is scoped from the LEDGER, never from the re-mark's own output and never from this page. The script prints that population on every run, --plan-only and a refusal to run included (it costs two SELECTs and no chain read): the row, leg and wallet counts, every leg the re-mark itself cannot see, the wallet list, and the command below with its --to already filled in. Take both the --to and the wallet list from the LAST run of the script, which is the dry-run whose write you have just accepted, and run it unscoped (no --wallet: narrowing the build to one wallet is what leaves another one padded). Only rows written before the release can be padded at all (after it the live writers use the fixed reader), so a bound taken from a run made after the release reaches every row that needs re-deriving, and re-deriving the net ones costs nothing.

bash

set -a; . ./.env.local; set +a          # the builder is not a cron job: it needs the env here
# 3. re-derive, RANGED (a bare sweep resumes cursors and derives nothing) and UNSCOPED.
#    The bounds below are ILLUSTRATIVE (2026-09-03): copy the command the LAST run printed.
npx tsx scripts/build-shadow-ledger.ts --from 24136053 --to 25859296
# 4. then the reconciler, and re-state the acceptance figures

--to is max(recorded arm block, highest Fluid debt ledger row), and the arm block by itself is not enough. The re-derivation trigger derives [floor, arm_block] and the derive cursor is a GREATEST upsert, so nothing ever revisits a row written above the arm. That costs nothing only while the highest Fluid debt row sits below it (25,847,926 against an arm of 25,859,296, both still true on 2026-09-03), and NFT 19028 is still open: one operate before the release puts a padded row above the arm that neither this step nor the live writers would ever correct. The script computes the bound from the data, so the printed command is right even on the day that happens. If the bound it prints is above the arm, let that block finalize before the build runs: the history build declares the settled basis and writes no provisional row, so a range reaching into the unfinalized window would leave the reorg repair no handle on the rows it rewrites there.

Step 3 is the same command the Fluid opening-day step runs, and that step is the one that owns the expectations for it: it is scoped from the same ledger, over the same wallets, with the same bound. Where both are outstanding, run this one first and read the two sections' expectations together — the ranged build restates the debt marks AND lands the birth markers in one pass, so neither section's "what moves" list is complete on its own.

GATED STEP: mark the Fluid births, so a loop's opening day is on the books

belongs to: the Fluid opening-day fix (v0.53.0)
executed: unverified

Prod held zero users after the full wipe of 2026-09-10 (docs/plans/pt-coverage-811-plan.md §0), so this step has had no population to act on since that date. That is not the same as having run: confirm against prod before deciding it is finished.

Runs once, on prod, after the release that carries the opening-day fix (issue #739, PR #792), and only there: staging reseeds from prod's dump nightly.

Run it AFTER the Fluid tick-padding re-mark, never before, and after that section's own step 3 — the ranged ledger re-derivation it defers the debt marks to. The command below is the same ranged unscoped build, so if the padding step's ledger half has not run, this build restates the Fluid debt marks as well and the "only meta moves" expectation below is simply false. Check that first; if both are outstanding, the expectations are the padding step's plus this one's, not this one's alone.

What it repairs. A Fluid position's opening movement carries two quantities that are struck against different things: the amount the venue's event names, and the balance the venue itself reports at the end of that block. On a borrowing they are not the same number — the vault rounds the drawn debt up to a price tick, and on a two-token position the balance is struck at the pool's composition. The reader used to decide whether a holding was brand new by comparing those two, so a rounding of a billionth of the position made it conclude the holding pre-existed: it invented a balance for every day before the position was created, marked all of them unvaluable, and reported nothing for the opening day itself. The venue publishes the creation (NewPositionMinted), so the derivation now stamps that fact into meta on each leg's opening row and the reader answers from it. Nothing else in the ledger changes.

Part of this lands at DEPLOY, ahead of this step, and that is expected. The second half of the fix — a two-token position's opening day reported at position scope — reads the same "did this leg exist" question, and for a position all of whose pool-token legs have agreeing quantity columns the arithmetic already answers it without any marker. Exactly one position on prod is in that state: 0xef08c6a4…'s NFT 18415. Its ETH-book day cell 2026-06-27 → 06-28 moves from 0 to about −0.055 total return / +0.003 accrual the moment the release deploys, and to about −0.034 / −0.017 after this step adds its debt leg. Take that wallet's "before" ETH figures from a measurement made before the release, or a correct repair reads as a failed one.

The earlier "expected 4" was one wallet's legs mistaken for the corpus. Issue #739 and PR #792's original body both said this step removes exactly four W1 unread-endpoint rows, naming NFTs 17548, 17549, 18415 and 17671. All four are 0xef08c6a4…'s. The real population is every Fluid wallet, and on 2026-09-05 that is 5 wallets, 14 positions, 35 legs, of which 11 carry the positive difference that produces a W1. Derive the population from the ledger on the day, and do not re-derive that error from this page: every count below is stated so it can be checked leg by leg, and the queries are the authority.

1. Scope and size it from the ledger, read-only, before anything else.

sql

-- every Fluid leg whose opening row's two quantity columns disagree. A POSITIVE difference is a
-- W1 carrier (the leg reads as occupied before it existed); a NEGATIVE one already reads as
-- unoccupied and files no W1, but its at-entry basis still moves. On 2026-09-05: 11 positive,
-- 10 negative.
WITH first_row AS (
  SELECT DISTINCT ON (wallet, position_key)
         wallet, position_key, block_number, qty_delta, to_balance
    FROM onchain_credit.portfolio_flow_events_v2
   WHERE chain_id = 1 AND venue = 'fluid'
   ORDER BY wallet, position_key, block_number, log_index, seq
)
SELECT wallet, position_key, block_number,
       to_balance - qty_delta AS fabricated_pre_birth
  FROM first_row
 WHERE to_balance IS DISTINCT FROM qty_delta
 ORDER BY (to_balance - qty_delta) DESC;

-- the wallets to re-derive: every wallet with a Fluid vault position, not only the carriers
SELECT DISTINCT wallet FROM onchain_credit.portfolio_flow_events_v2
 WHERE chain_id = 1 AND venue = 'fluid' ORDER BY 1;

-- the upper bound, from THIS population
SELECT GREATEST(
         (SELECT last_scanned_block FROM onchain_credit.chain_scan_cursors
           WHERE chain_id = 1 AND scope = 'portfolio:v2-campaign:arm-block'),
         (SELECT max(block_number) FROM onchain_credit.portfolio_flow_events_v2
           WHERE chain_id = 1 AND venue = 'fluid')
       ) AS to_block;

Re-take the bound immediately before the run. A Fluid position minted between the query and the build lands above --to and stays unmarked, and the derive cursor is a GREATEST upsert so nothing ever revisits it. If the bound the query returns sits above the arm block, let that block finalize first, for the reason the padding step gives. On 2026-09-05 the highest Fluid row was 25,847,926 against an arm of 25,859,296, so the bound was the arm; the ledger's highest row over all venues was 25,904,143, and leaving those above --to costs nothing because no non-Fluid row can carry the marker.

It is a multi-hour campaign holding PORTFOLIO_WRITE_LOCK_SQL. Drain the portfolio cron first, as the other from-the-floor steps on this page do, and plan for the wall clock of a full replay over every Fluid wallet rather than a tick.

2. Re-derive, RANGED and UNSCOPED. Ranged because a bare sweep resumes cursors and derives nothing; unscoped because narrowing to one wallet is what leaves another one unmarked, and re-deriving a wallet the marker does not touch costs nothing.

bash

cd /opt/onchain-credit
set -a; . ./.env.local; set +a          # the builder is not a cron job: it needs the env here
npx tsx scripts/build-shadow-ledger.ts --from 24136053 --to <to_block, re-taken now>

3. Acceptance — shape first, and every count checkable leg by leg.

No row is added and none is deleted. meta is the only column that moves (plus updated_at), given the precondition above. A changed row count means something else moved and is a stop.
One marked row per Fluid leg — every Fluid position on prod has a NewPositionMinted inside the range, so the count is the leg count, 35 on 2026-09-05:
sql
```
SELECT wallet, count(*) AS marked_legs
  FROM onchain_credit.portfolio_flow_events_v2
 WHERE chain_id = 1 AND venue = 'fluid'
   AND meta @> '{"birth": true}'::jsonb
 GROUP BY 1 ORDER BY 1;
```
and it is on each leg's first row only: re-run query 1 and confirm no later row of the same leg carries it. The predicate is a containment test rather than a cast, deliberately — a cast reads the string "yes" as true.
W1 unread-endpoint falls by exactly 9, and no new W1 appears anywhere. The count is engine output, not a column, so it comes from the read-only reconciler (below). The nine are all debt legs: 17548 GHO, 17549 GHO, 18415, 17671 (0xef08c6a4…); 17303, 17438, 17423 (0xe1590894…); 17764, 18380 (0x5d09098f…). A tenth, or a different leg, is a stop. Two of the eleven positive carriers deliberately do not move: NFT 18348 (0xaae6ae86…) sits in an EXCLUDED book, where a leg books 0 under a coverage note and files no withhold at all; and NFT 19028 (0xb2ee508c…) was minted before that wallet's first snapshot, so it has no pre-birth interval to withhold. "No new W1" is measured rather than hoped: every one of the eleven is read by the spine at every grid point it is held at, so none acquires a later stretch.
W9 ghost-row stays 0. This is the one hazard the change creates rather than removes: a leg's pre-birth points now read as ledger-empty, so a spine row at such a point would become a ghost. None exists — checked directly, zero snapshot rows precede any of these legs' births — and the budget is zero, so a single row is a stop.
W4 unresolvable-composition keeps its row count; its BIRTH rows move from unreconciled to attributed, joining the closes. No withhold shape appears that was not there before.
The at-entry basis and the vintage move on all 21 disagreeing legs — the 11 positive and the 10 negative — and that is expected, not the stop condition. Each stops having its at-entry basis reconstructed from the first snapshot the leg appears in and starts having it read off the opening movement, and its "since" date moves from that snapshot to the mint. That is the position row's at-entry column and the left-hand side of its secondary-market dislocation P&L. Measured on 0xef08c6a4…'s May pair: −3.8599 → −16.7786, −3.2215 → −5.7567, −4.5241 → −0.1632, −3.7449 → −6.3931, vintages 05-12 00:00 → 05-11 19:25 / 19:35. A leg outside query 1's list whose at-entry column moves IS a stop.
sql
```
-- the legs whose at-entry basis stops being reconstructed: query 1's set, said the other way
SELECT wallet, position_key
  FROM onchain_credit.portfolio_flow_events_v2
 WHERE chain_id = 1 AND venue = 'fluid'
   AND meta @> '{"birth": true}'::jsonb
   AND to_balance IS DISTINCT FROM qty_delta
 ORDER BY 1, 2;
```

These day cells move, and no others. Each is the interval containing a position's birth. A cell outside this table is a new shape and a stop.

wallet	book	cell	driver
`0xef08c6a4…`	ETH	2026-06-27 → 06-28	NFT 18415 — partly at deploy, see above
`0xef08c6a4…`	USD	2026-05-11 → 05-12	NFTs 17548 + 17549; books −17.81 / +35.68, the close stays −15.19 / −0.73, and the pair's life foots to +26.48 / +32.10
`0xef08c6a4…`	USD	2026-05-17 → 05-18	NFT 17671
`0xe1590894…`	USD and ETH	2026-04-29 → 04-30	NFT 17302 (USD) and NFT 17303 (ETH)
`0xe1590894…`	USD and ETH	2026-05-06 → 05-07	NFT 17423 (USD) and NFT 17438 (ETH)
`0x5d09098f…`	USD	2026-05-21 → 05-22	NFTs 17764 + 17761
`0x5d09098f…`	none — the leg has no snapshot row	2026-06-25 → 06-26	NFT 18380
`0xaae6ae86…`	`EXCLUDED`	—	nothing moves
`0xb2ee508c…`	—	—	nothing moves

The list is wider than query 1's leg list on purpose: a two-token position's birth cell also moves where its own opening rows agree, because the position-scope reporting is what changed there. 0xe1590894…'s lifetime figures for NFT 17302 should land at −21.48 / −2.66 and for NFT 17423 at +50.06 / +86.70.

A Fluid position minted AND unwound between two grid points reconciles rather than paging. The reader files nothing at position scope there — the day's total is right, but the shares a joint figure would be split by are unobservable at both ends of that day — and the guard that makes it file nothing is what keeps the reconciler from throwing. No such two-token position exists on prod today (the three positions whose whole life sits inside one interval all have a single-token shape), so this is protection against the first same-day loop rather than a live fault. Confirm it anyway: a [fail] … the engine filed 0 rows for 1 inputs means the guard did not land.

sql

-- read-only. Any row with grid_pts_inside_life = 0 AND smart_legs >= 2 is the shape.
WITH life AS (
  SELECT wallet,
         split_part(position_key,':',1)||':'||split_part(position_key,':',2)||':'||
         split_part(position_key,':',3)||':'||split_part(position_key,':',4)||':'||
         split_part(position_key,':',5) AS scope,
         count(DISTINCT position_key) FILTER (
           WHERE array_length(string_to_array(position_key,':'),1) >= 7) AS smart_legs,
         min(ts) AS born, max(ts) AS last_row
    FROM onchain_credit.portfolio_flow_events_v2
   WHERE chain_id = 1 AND venue = 'fluid'
   GROUP BY 1,2)
SELECT l.wallet, l.scope, l.smart_legs,
       (SELECT count(DISTINCT s.snapshot_ts) FROM onchain_credit.portfolio_position_snapshots s
         WHERE s.chain_id = 1 AND s.wallet = l.wallet
           AND s.snapshot_ts > l.born AND s.snapshot_ts < l.last_row) AS grid_pts_inside_life
  FROM life l ORDER BY 4, 3 DESC;

Reconciler, read-only — this is also where the W1 and W4 counts come from. Without --marker the run writes nothing at all:
bash
```
cd /opt/onchain-credit && DATABASE_URL="$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)" \
  npx tsx scripts/ops/reconcile-ledger.ts
```
Zero identity failures at leg and book scope, zero roll-up skips, zero unshaped declines. The withhold budget must be re-set from this run rather than carried forward: nine legs lose their pre-birth stretches and the W4 birth rows begin pricing a real joint residual, so the before and after numbers are not comparable.
Ghost adjudicator, read-only: npx tsx scripts/ops/adjudicate-ghost-rows.ts — ledger-hole and spine-stale at their standing values and no new ghost.

No migration. No new table or column — the marker rides in meta, which every row already has, and the reader projects it from columns that exist today. The code is inert until the data lands for every position except NFT 18415, named above, so there is no window in which new code meets old data in a broken state.

GATED STEP: re-derive every wallet that holds a fixed-rate principal token

belongs to: the #767 principal-token wallet-leg fix (v0.54.0)
executed: unverified

Prod held zero users after the full wipe of 2026-09-10 (docs/plans/pt-coverage-811-plan.md §0), so this step has had no population to act on since that date. That is not the same as having run: confirm against prod before deciding it is finished.

Runs once, on prod, after the release that carries the principal-token repair, and only there: staging reseeds from prod's dump nightly.

Read this first: the release itself already moves one account's published figure, before you run anything. The repair has two halves and they land at different moments. The debit — the wallet's principal-token holding losing what it posted as collateral — is derivation-side and waits for the run below. The bridge guard is read-path: it runs inside the engine every time the page is served, so the moment the release deploys it declines the phantom on 0x5d09098f…'s dollar chart and the account's served lifetime jumps by the declined amount. Measured read-only on the 2026-09-03 rows, that account moves from about minus $4,541,714 before the release to about plus $417,133 at deploy, and its charted-nowhere curve from about plus $1,019,624 to about minus $90,239. Nothing else in the corpus moves and no accrual figure moves anywhere.

That intermediate is not a halfway house: the guard declines the minus $4.96 million leg of the phantom pair and leaves its plus $380,932 mirror standing, so between deploy and this run the account publishes roughly plus $417k of lifetime return on about $1.5M of equity where the honest figure is about plus $31k. Take the "before" figure from a measurement made BEFORE the release, not from the page after it, and expect the observed move across this step to be downward — about minus $386k — rather than the plus $4.573M restatement measured against the pre-release ledger. An operator judging this step against the pre-release number after deploy reads a correct repair as a failed one.

Every number this step is judged on is derived by a query at the moment you run it. A pre-written count made a correct repair look like a stop condition once already, so the checks below are identities between two populations the database itself produces.

1. Scope it from the ledger, not from this page.

sql

-- read-only; run before anything else and use ITS output as the wallet list
SELECT DISTINCT wallet
  FROM onchain_credit.portfolio_flow_events_v2
 WHERE position_key LIKE 'pendle:pt:%'
 ORDER BY 1;

2. Take the upper bound from the ledger too, and from THIS population. --from is the derivation floor 24136053, a constant of the system that the run refuses to go below (exit 2). --to is max(recorded arm block, the highest ledger row on the wallets step 1 returned). That is not the pair the Fluid re-mark step above prints: that script computes max(arm, highest FLUID DEBT row), which is Fluid-scoped by construction and sits below the arm today, so copying its command lands on a bound that is too low. Nothing else prints the right bound for this population, so derive it:

sql

-- read-only; its single value is the --to below
WITH pt_wallets AS (
  SELECT DISTINCT wallet FROM onchain_credit.portfolio_flow_events_v2
   WHERE position_key LIKE 'pendle:pt:%'
)
SELECT GREATEST(
         (SELECT last_scanned_block FROM onchain_credit.chain_scan_cursors
           WHERE chain_id = 1 AND scope = 'portfolio:v2-campaign:arm-block'),
         (SELECT max(block_number) FROM onchain_credit.portfolio_flow_events_v2
           WHERE wallet IN (SELECT wallet FROM pt_wallets))
       ) AS to_block;

A bound that is too low is permanent, not transient, which is why it gets its own query. The re-derivation deletes and rewrites only block_number BETWEEN --from AND --to; rows above the bound are untouched, the derive cursor is a GREATEST upsert so nothing revisits them, and the next anchor for the holding is taken from the newest stored balance below the range. A row left above the bound therefore keeps its double-counted balance and hands it forward for good. On 2026-09-03 the arm block was 25,859,296 while this population's own highest row was 25,889,096, and one of the rows in between was a principal-token holding standing at 5,566,010.631614 — the double count exactly, to the wei. If the bound the query returns sits above the arm, let that block finalize before the build runs.

It has to be a re-derivation from the FLOOR, not a recent range. A holding's balance is carried forward from the balance the previous run stored, so a range that starts above the damage inherits the impossible starting point and adds correct movements on top of it. Worse here than merely wrong: the widened refusal now refuses any range that inherits a negative principal-token balance, so a narrow re-run reports a refusal rather than a repair.

bash

cd /opt/onchain-credit
set -a; . ./.env.local; set +a          # the builder is not a cron job: it needs the env here
# ranged, and scoped to the wallets query 1 returned; <to> is query 2's single value
npx tsx scripts/build-shadow-ledger.ts --from 24136053 --to <to> --wallet <wallet>
# ...once per wallet, then the reconciler

What to expect, judged as SHAPE first.

No refusal. A refused range names a holding, a balance and a transaction, and means the derivation still produces a negative principal-token balance: the repair did not land. Do not re-run scoped; report it.
No negative balance survives anywhere. Zero is the floor, whatever the count was before: SELECT count(*) FROM onchain_credit.portfolio_flow_events_v2 WHERE to_balance < 0;

The two sides of every collateral movement are now both recorded, and the check is an identity rather than a count. Every venue row whose asset is a principal token must have exactly one holder-side mirror, and every mirror must carry a pointer to the row it settles. Both columns come out of the same statement, so nobody has to remember a number:

sql

WITH pt AS (
  SELECT DISTINCT replace(position_key, 'pendle:pt:', '') AS token
    FROM onchain_credit.portfolio_flow_events_v2
   WHERE position_key LIKE 'pendle:pt:%'
)
SELECT wallet,
       count(*) FILTER (WHERE venue <> 'pendle')                              AS venue_side,
       count(*) FILTER (WHERE venue = 'pendle' AND kind LIKE 'transfer%')     AS holder_side,
       count(*) FILTER (WHERE venue = 'pendle' AND kind LIKE 'transfer%'
                          AND meta->>'settlementOf' IS NULL)                  AS unbound
  FROM onchain_credit.portfolio_flow_events_v2
 WHERE asset IN (SELECT token FROM pt)
 GROUP BY 1 ORDER BY 1;

holder_side = venue_side on every row, and unbound = 0. (This identity holds for the venue family whose own event names the collateral token, which is every principal-token position the corpus has. A venue that names its own receipt token instead would add holder-side rows with no venue row of the same asset beside them, and the identity would then read as an excess on the holder side rather than a shortfall.)

The highest principal-token row is at or below --to, and its balance is the chain's. Take the newest pendle:pt: row, read PT.balanceOf(wallet) at its block, and expect the two to agree to the unit. That is the one check that says the bound was high enough.
0x5d09098f…'s dollar chart lands at about plus $31,200, from the plus $417k the deploy left it at, and its charted-nowhere curve at about minus $90,500, which is where the guard had already put it. Its accrual line does not move at any point in this step. The two days that carried the phantom collapse: 2026-07-10 to 07-11 to about minus $2,529 and 2026-07-13 to 07-14 to about minus $5,057 — both negative, both small; a positive few thousand on either day means the repair did not land.
0xaae6ae86… does not move, and not for the reason "it has no price". Its principal token IS priced (it is in the fixed-rate market registry, and its stored rows carry real marks). It does not move because its two acquisitions are unpriced market buys whose interval is withheld on both lines, and because the credit-back mirror and the redemption carry the identical mark and cancel exactly. The holding stops being overstated, which is the point of re-deriving it. Do not read "nothing moved" as "this wallet is inert": a future movement on it will move figures.
Reconciler: the withhold budget must be re-set from this run, not carried forward. The affected holdings lose their long unread stretches and gain per-movement unpriced-input withholds on the accrual line, which is the honest count for a fixed-rate holding, so the before-and-after numbers are not comparable and the new run is the baseline.
Comparator: the unattributed cells on 0x5d09098f… sit on its charted-nowhere curve and are the measure of the other open defect (its collateral is not valued at all). Expect that count to be broadly unchanged by this release and to fall only when that one lands. The dollar curve's own cells should stop carrying the two multi-million differences.

Do not run the two repairs in the other order. If the collateral-valuation fix lands first, its repair pass restates a collateral holding on a wallet whose own principal-token holding is still counted twice, and both numbers move again when this one runs. That repair is the step below.

GATED STEP: re-lay the wallets holding a Pendle PT as Morpho collateral

belongs to: the #754 Morpho PT collateral fix (v0.54.0)
executed: unverified

Prod held zero users after the full wipe of 2026-09-10 (docs/plans/pt-coverage-811-plan.md §0), so this step has had no population to act on since that date. That is not the same as having run: confirm against prod before deciding it is finished.

Runs once, on prod, after the release that carries the Morpho PT-collateral valuation fix (issue #754), and only there: staging reseeds from prod's dump nightly.

Run it AFTER the principal-token re-derive, never before. That section's closing paragraph is about this step by name: if the collateral-valuation repair lands first, it restates a collateral holding on a wallet whose own principal-token holding is still counted twice, and both numbers move again when the other one runs.

sql

-- read-only ordering check. Expect 0.
SELECT count(*) FROM onchain_credit.portfolio_flow_events_v2 WHERE to_balance < 0;

That check is a PROXY, and a one-directional one. Negative derived principal-token balances were the fingerprint of the defect #pt-wallet-leg-rederive repairs, so a non-zero result means that step has certainly not run and this one waits. A zero does not prove it ran: a wallet whose principal-token leg was double-counted without ever going negative passes it. The unambiguous signal is the settlement rows the #767 rungs write — a pendle:pt:… leg carrying transfer_in/transfer_out rows whose meta->>'settlementOf' names a morpho:market:…:collateral leg. Their absence on a wallet that has posted PT collateral means the re-derive is still owed.

Why history does not heal itself, and neither does the stored tip. The code fix makes every new FULL read carry the leg. The 6h tick does not do a full read for a quiet wallet: it RECOMPOSES from the wallet's stored legs, and a leg that has never been stored is never recomposed. A Morpho PT holder with no ledger movement is in no always-dirty set, so its snapshot tip stays collateral-less until something re-reads it. The replay below covers the tip and the history in one pass.

1. Scope it from the registries.

sql

-- read-only; use ITS output as the wallet list. The JOIN is the scope, NOT the `morpho-pt-%`
-- strategy_key prefix: on 2026-09-05 the join returns 73 markets and the prefix only 28, so
-- scoping by the name silently drops 45 markets' holders.
WITH pt_markets AS (
  SELECT lower(r.market_id) AS market_id
    FROM onchain_credit.morpho_market_registry r
    JOIN onchain_credit.pendle_markets p
      ON p.chain_id = r.chain_id AND lower(p.pt_address) = lower(r.collateral_address)
   WHERE r.chain_id = 1
)
SELECT DISTINCT wallet FROM (
  SELECT wallet, position_key FROM onchain_credit.portfolio_flow_events_v2
  UNION ALL
  SELECT wallet, position_key FROM onchain_credit.portfolio_position_snapshots
) x
 WHERE split_part(position_key, ':', 1) = 'morpho'
   AND split_part(position_key, ':', 3) IN (SELECT market_id FROM pt_markets)
 ORDER BY 1;

Take each wallet's before figures now, from the served endpoints, so the move across this step is measured rather than remembered:

bash

curl -s "https://creddit.xyz/api/portfolio/positions?wallet=<wallet>" \
  | jq '{outside: [.outside[] | {groupKey, reason}],
         usd: [.books[] | select(.book=="USD") | .rows[] | {positionKey, valueMarket}]}'

2. Re-lay each wallet's history. --fresh is the destructive full replay: it revokes the coverage anchor up front, replaces the wallet's snapshot history, and re-derives its v2 ledger whole-history in the same call. A gap patch is not enough — it would faithfully preserve the missing leg for every block below the anchor.

bash

cd /opt/onchain-credit
set -a; . ./.env.local; set +a          # not a cron job: it needs the env here
# once per wallet from query 1, serially
npx tsx scripts/backfill-portfolio-wallet.ts --uid <wallet> --fresh

--fresh lays tier 1 (a ninety-day window, span-bounded) and re-queues the wallet for the backward extension down to the history floor; the minutely drain completes that, so the script's exit is not the end of the repair. Wait for status='done' and a floor_ts at the floor:

sql

SELECT uid, status, floor_ts, covered_through_ts
  FROM onchain_credit.portfolio_backfill_state WHERE uid IN (<wallets>);

⚠ A position closed more than ninety days before the run is pruned from the replayed window until the queued deepening reaches it. That is --fresh's documented behaviour, not a fault of this repair, but it means the wallet's history is briefly shallower than it was.

3. Acceptance — identities the database produces, not counts written here.

Every READING that has the debt has the collateral. Per reading, not per wallet: a count-vs-count comparison over the whole history hides a reading missing its collateral whenever another reading carries a spare.

sql

WITH pt_markets AS (
  SELECT lower(r.market_id) AS market_id
    FROM onchain_credit.morpho_market_registry r
    JOIN onchain_credit.pendle_markets p
      ON p.chain_id = r.chain_id AND lower(p.pt_address) = lower(r.collateral_address)
   WHERE r.chain_id = 1
)
SELECT count(*) FROM (
  SELECT wallet, split_part(position_key,':',3) AS market, snapshot_ts,
         bool_or(position_key LIKE '%:debt')       AS has_debt,
         bool_or(position_key LIKE '%:collateral') AS has_collateral
    FROM onchain_credit.portfolio_position_snapshots
   WHERE venue='morpho-blue'
     AND split_part(position_key,':',3) IN (SELECT market_id FROM pt_markets)
   GROUP BY 1,2,3
) x WHERE has_debt AND NOT has_collateral;

Expect 0, with one legitimate exception to check before calling it a failure. The reader emits a collateral leg only when the on-chain collateral is > 0, so a market left with bad debt after a FULL liquidation genuinely has debt readings and no collateral reading. Any row this returns should be matched against a seizure on that market at that block; what must not survive is a reading with live collateral on chain and no row for it.

A booked collateral leg carries a value, and the gate is ZERO.
sql
```
SELECT count(*) FROM onchain_credit.portfolio_position_snapshots
 WHERE venue='morpho-blue' AND position_key LIKE '%:collateral'
   AND book <> 'EXCLUDED' AND value_market IS NULL;
```
Each such reading understates the published book value by the entire collateral — the group is structurally a carry whatever its values, so it is included and charted, and the leg with no mark contributes nothing while its borrow contributes in full. That is not "the odd isolated block": one reading is one wrong headline. If the count is not zero, enumerate the affected (wallet, snapshot_ts) and look at what the chart does at those points before accepting the run.

index_raw stays NULL on every Morpho collateral row (a non-zero count means something fabricated an index for a venue that has none).

sql

SELECT count(*) FROM onchain_credit.portfolio_position_snapshots
 WHERE venue='morpho-blue' AND position_key LIKE '%:collateral' AND index_raw IS NOT NULL;

The served response. The market leaves outside (it was {reason: "cross-book"}, "Debt without matched collateral") and the USD book carries two carry_trade rows, the collateral named by its PT ticker rather than by an address.
Expect the market-mark book value and the total-return line to RISE by the collateral being counted. HISTORICAL NOTE, superseded by #811 C: when this step was written a PT had no redemption mark at all, so the accrual line did not move and the redemption-mark book value went NEGATIVE on a levered wallet by the size of the borrow. A PT leg now carries a read-time accrual value (M34) and a financed group publishes both sides of a line or neither, so the accrual line moves WITH the market one and the redemption-mark book value is collateral − debt.
Reconciler: the withhold budget gains one W2 unpriced-input row per collateral reading on the accrual line, which is the honest count for a principal-token holding. Re-set the budget from this run rather than carrying the previous one forward; the two are not comparable.

No migration. No new table or column.

GATED STEP: the historical ledger build

belongs to: the rebuilt ledger's first build, per environment (v0.42.0 onward)
executed: unverified

Prod held zero users after the full wipe of 2026-09-10 (docs/plans/pt-coverage-811-plan.md §0), so this step has had no population to act on since that date. That is not the same as having run: confirm against prod before deciding it is finished.

Runs once per environment, after every stream's history is backfilled, and it is a gated step: it is long, it is per wallet, and its wall-clock is the number the rest of the campaign is planned from.

bash

cd /opt/onchain-credit && DATABASE_URL="$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)" \
  npx tsx scripts/build-shadow-ledger.ts

It writes flow rows and each wallet's derive cursor; it does not touch the position history, the backfill queue or the chart's building state. It is not a safe no-op for the product: the rows it writes are the served ledger, so a wallet it re-derives changes what that wallet's page shows. Mechanics, the two invocation modes and what complete means are in data-pipeline.

The build starts on 1 January 2026, and that is not the same number as the event store's depth. Two floors are in play through this whole campaign and confusing them is the easiest way to spend a night on the wrong thing:

	block	what it means
raw ingestion floor	22,527,558 (2025-05-21)	how deep `raw_events` and its per-stream certificates go. Every stream sweep, every `from_block`, every rollout marker is written against this and none of it changes
derivation floor	24,136,053 (2026-01-01)	where the portfolio view's history begins. Every whole-history walk bottoms out here: the history build, the re-derivation trigger, the registration replay, the completeness certificate and the price-mirror preflight

The portfolio view supports everything from 1 January 2026 onward; positions held only before that are out of scope. Keeping the store deeper than the served window costs nothing (it is already ingested) and means widening the window later is a constant plus a rebuild, not a re-ingestion. It also removes a whole class of failure outright: the Fluid position resolvers went live at blocks 23,881,723 and 23,881,747, both a quarter of a million blocks below the derivation floor, so no supported range can ask a venue contract that does not yet exist.

Before starting:

Pause the minutely drain first, and leave it paused through the reconciliation. An un-paused drain sweeps the same wallets from the other end — harmless in itself, because the merge is idempotent and both writers serialise on the write lock, but it doubles archive reads and makes this step's wall-clock measurement worthless.
Read the entry check. It refuses if any tracked wallet already carries a derive cursor, which is the mechanical evidence that the pause held. There are exactly two legitimate non-zero causes and they are different: declare --re-entry when this step died and is being resumed, and --checkpoint-restored when a restored ledger checkpoint brought the cursors back. Record which in the run log. A restored checkpoint must be re-run with an explicit range (or its cursors cleared first), because a resume over restored cursors derives nothing and then passes every downstream check vacuously. The check binds this step — the sweep that measures the campaign. A run given an explicit range prints the count and proceeds, because every re-derivation targets a wallet that is already stamped past the range it is given; that is what the range is for.
--dry-run prints, per wallet, the cursor it found and the block it would start at, and writes nothing.

Afterwards, confirm every wallet is complete and that no [shadow-build/fail] line was printed. A wallet that derived rows and could not certify them is reported as a failure, not a success, and the run exits non-zero.

When a wallet refuses

A [shadow-build/fail] line means the run kept that wallet's rows rather than replacing them, and stopped walking it. Nothing is lost and nothing is wrong with the ledger; what is blocked is the claim that the wallet is derived, so every check that reads the completeness condition refuses until it clears. Do not re-run the sweep and hope. Read which refusal it is:

the line says	what happened	next step
`refused its delete … terminal receipt … would be DELETED`	the run would have removed a receipt that closed a position without producing a replacement for the same position — because it did not cover that venue, or that venue's history is not certified over the whole range	complete the missing stream backfill for that venue, then re-run this wallet alone with an explicit range. If the venue's history is already certified, the derivation genuinely cannot restate that close and this is a defect, not an operational step
`unresolvable-universe … declared venue(s) … but supplied no series universe`	the derivation could not say which legs it was able to resolve on a venue it claims to have covered	a defect in the derivation, not an operator step. Do not clear it by narrowing the venue list
`unresolvable-universe … NOT a superset of the leg(s) its stored rows already name`	a vault or market the wallet's stored rows name has left the registry, so the run cannot re-derive it	re-list that vault or market in the registry and re-run the wallet with an explicit range. If it is genuinely gone, see the box below — this one does not clear itself
`adapter-error … at least one transaction in this range produced no rows`	one transaction's decode threw, so that range was read incompletely. The delete and the stamp are both withheld rather than treating the unread rows as phantoms	fix the decode (it names the transaction) and re-run the wallet with an explicit range
`rows written, cursor NOT stamped — uncovered: …`	the rows are in, the claim is withheld until the named streams certify back to the derivation floor (24,136,053). A stream certified from the deeper raw floor already satisfies it	wait for that backfill. For a wallet registered minutes ago this is the ordinary state and it clears itself

Two of these can never clear by waiting, and the runbook must not send an operator to wait for them

A de-registered venue and native ETH a wallet has since spent to zero cannot re-enter a replay universe: the sentinel enters only from a live non-zero balance, and a delisted vault is not in the registry to be found. The refusal is still correct — the run genuinely cannot re-derive those rows — but "re-run after the backfill" is telling that operator to wait for something that cannot arrive.

A refused range also stops that wallet's walk, by design — a later range must not commit above a hole and certify across it — so the wallet stays short of the arm block and uncertified for as long as the refusal stands. There are exactly two ways out and both are decisions, Fred's rather than the operator's:

Re-list the venue in the registry. The run can resolve it again, the refusal clears on the next re-derivation, and nothing was lost.
Decide the rows may go, and remove them deliberately. That is precisely what the merge refused to do on its own, and no tool here will do it for you: it is a recorded act, naming the wallet, the venue, the block range and the reason in the campaign's run log, after which the re-derivation has nothing left to protect and proceeds normally.

Never widen the delete to clear a refusal. The refusal is the only thing standing between a close the run cannot restate and a position that never ends.

Re-deriving one wallet later — after a stream is enabled post-build, or to repair a single wallet — takes an explicit range and derives that whole range, cursor or no cursor:

bash

npx tsx scripts/build-shadow-ledger.ts --wallet 0x… --from 24136053 --to <recorded arm block>

Passing --from is what selects that behaviour. Omitting it resumes from the wallet's cursor, which for an already-certified wallet derives nothing and exits clean — a repair that never happened, reported as one.

--from below 24136053 is refused, and the old command is the reachable way in

Releases up to v0.46.0 printed this command with --from 22527558 — the raw ingestion floor, which is where the event store starts, not where the portfolio view does. That number is still in operator scrollback and in every published copy of this page from those releases, so running it against current code is the likely mistake, not a hypothetical one.

The run now refuses it and exits 2 before deriving anything. It is not silently raised to the derivation floor: the ranged mode exists precisely so an operator's explicit range is never shortened behind their back, so a range that cannot be honoured is rejected out loud instead. Re-run with --from 24136053.

Obeying it would have been worse than the wasted hours: the run derives roughly 1.6M blocks below the supported window, marks the wallet as covering its whole history, stamps its cursor and exits 0 — leaving that one wallet's history deeper than every other wallet's, decided by which command someone happened to paste.

GATED STEP: reset the two Pool streams' coverage certificates

belongs to: the widened Aave/Spark Pool event set (v0.54.0)
executed: unverified

Prod held zero users after the full wipe of 2026-09-10 (docs/plans/pt-coverage-811-plan.md §0), so this step has had no population to act on since that date. That is not the same as having run: confirm against prod before deciding it is finished.

aave-pool and spark-pool used to request two events (LiquidationCall, UserEModeSet). They now request eight — Supply, Withdraw, Borrow, Repay and the two Deficit events joined them. Their event_coverage rows did not change, because a coverage row certifies a stream NAME over a block range and knows nothing about the event set that was being requested. Measured read-only on prod, both rows read ('*', 22527558, live) while six of the eight topics have no row anywhere:

sql

SELECT stream, topic0, count(*), min(block_number)
  FROM onchain_credit.raw_events
 WHERE stream IN ('aave-pool','spark-pool') GROUP BY 1,2;
-- two topic0s per stream, from 22,527,761 up. The other six: nothing.

So the rows are a completeness claim over an index that holds a quarter of what the claim now covers — see the rule in the schema doc.

Do not run this at the release deploy. Today the only served read of these two streams binds a single scalar topic0 = LiquidationCall, which IS indexed to the floor, so the certificate is still true for everything that reads it. Withdrawing it early would take every wallet holding an Aave or SparkLend position off the ledger path and onto the chain fallback (the JIT gate and the historical backfill gate both require these two rows to reach back to their read's start block) for no gain at all.

Run it as the FIRST step of the backfill that re-earns them, and before any verification step that counts coverage rows — a stale row is exactly what makes such a check pass vacuously:

bash

# PROD. Immediately before backfilling the two Pool streams over the widened event set.
sudo -u postgres psql -d creddit -c \
  "DELETE FROM onchain_credit.event_coverage WHERE chain_id = 1 AND stream IN ('aave-pool','spark-pool') AND address = '*';"

Expect the always-on ingester to re-insert both rows within a cycle at its current tip; that is correct, and the backfill is what walks from_block back down to 22,527,558. Do not try UPDATE … SET status = 'backfilling' instead: the live scan re-stamps live every cycle and from_block merges with LEAST, so the update is inert.

GATED STEP: reset the two Fluid streams' coverage certificates

belongs to: the widened Fluid event set (v0.54.0)
executed: unverified

Prod held zero users after the full wipe of 2026-09-10 (docs/plans/pt-coverage-811-plan.md §0), so this step has had no population to act on since that date. That is not the same as having run: confirm against prod before deciding it is finished.

Same rule, same shape, two more streams, and one of them fixes a class of loss that is currently invisible.

fluid-operate used to request two vault events. It now requests three: the event a Fluid vault emits when it writes a position off entirely joined them. That event is the ONLY trace such a write-off leaves — the vault returns before emitting its ordinary liquidation event — so a pipeline that never requested it cannot see an absorbed position at all, and 13 of Fluid's 179 live vaults carry absorbed bad debt today. fluid-nft used to request the position transfer alone and now also requests the factory's mint signal, which is what lets a position opened inside a replay window anchor without an archive read.

Do not run this at the release deploy, for exactly the reason above: today's served read of fluid-operate binds the two original topics, which ARE indexed to the floor, so the certificate is still true for everything that reads it. Withdrawing it early puts every Fluid wallet on the chain fallback for no gain.

Run it as the FIRST step of the backfill that re-earns them, and before any verification step that counts coverage rows:

bash

# PROD. Immediately before backfilling the two Fluid streams over the widened event set.
sudo -u postgres psql -d creddit -c \
  "DELETE FROM onchain_credit.event_coverage WHERE chain_id = 1 AND stream IN ('fluid-operate','fluid-nft') AND address = '*';"

The same DELETE-not-UPDATE caveat applies, and for the same reason.

erc4626 widened too, and owes NOTHING — checked, not assumed. It gained Morpho Vaults V2's forced-deallocation event, without which a forced-deallocation penalty is indistinguishable from a queued exit and opens a position waiting on a payout that can never arrive. It is exempt because the stream has never been scanned: on 2026-08-21, read-only on both prod and staging, it held zero event_coverage rows and zero raw_events rows. There is no range below the change to invalidate, so there is no delete to run and none should be scheduled — its one-time backfill earns the certificate over the full event set from the start. Re-run the check below before trusting this paragraph; once that backfill has run, a further widening owes the reset in full, on 546 addresses over 3.2M blocks.

sql

-- Both must be 0 for the exemption to hold.
SELECT (SELECT count(*) FROM onchain_credit.event_coverage WHERE stream = 'erc4626') AS coverage_rows,
       (SELECT count(*) FROM onchain_credit.raw_events     WHERE stream = 'erc4626') AS stored_logs;

GATED STEP: re-derive the holder whose redemptions shared a transaction with a forced pull

belongs to: the #889 Vaults V2 forced-pull redemption fix — v0.64.0
executed: 2026-09-17

Run 2026-09-17 (v0.64.0): nothing to re-derive. The same release carried the coverage-rule account wipe, run at 13:22Z, which deleted the one affected holder along with every other account; step 1's scope query returns no row, so per step 1 this is recorded rather than run. The holder was tracked until the wipe (floor 25,742,367, derive cursor 25,991,218). Any history the wallet gets from here is derived by the fixed code.

Runs once, on prod, after the release that carries the fix, and only there. Staging cannot even be a rehearsal: its nightly reseed restores prod's dump and then scrubs it, and the scrub truncates the whole user-data graph — accounts, tracked wallets and the flow ledger alike — so staging holds no wallet for this to act on and never will.

#889 ships in two halves and this step serves both. The first restores the redemptions the transaction-keyed rule deleted; the second books the forced-pull fee itself as a row. The run is the same command either way and it is idempotent over the range, so running it after the first half and again after the second is harmless — it simply re-derives, and the second pass adds the five fee rows. If both halves ship in one release, one run does everything.

Why the release is not enough on its own. The fix changes what the derivation does with a transaction that carries a forced liquidity pull and the holder's own withdrawal — the shape Morpho's app builds whenever an exit needs more than the fund's idle cash. It is forward-looking only. Nothing re-derives below a wallet's derive cursor: the 6h tick and the JIT persist both resume at cursor + 1, so every redemption already dropped stays dropped, and the position keeps publishing the loss. One tracked holder is affected and no other wallet has ever had a forced pull naming it.

1. Scope it from the ledger, not from this page. The wallet may have been unlinked, and its floor and cursor are both read at the moment you run, never copied from here.

sql

-- read-only. Expect ONE row. An empty result means the holder is no longer tracked and this
-- step has nothing to act on; record that rather than hunting for the wallet.
SELECT aw.wallet,
       a.history_floor_block                    AS from_block,
       c.last_scanned_block                     AS to_block
  FROM onchain_credit.account_wallets aw
  JOIN onchain_credit.accounts a  ON a.uid = aw.account_uid
  LEFT JOIN onchain_credit.chain_scan_cursors c
         ON c.chain_id = 1
        AND c.scope = 'portfolio:derive:v2:1:' || aw.wallet
 WHERE aw.wallet = '0x68e7e72938db36a5cbbca7b52c71dbbaadfb8264';

The issue recorded a floor of 25771074 for this wallet on 2026-09-16; read both values again rather than reusing that one. --from must be the wallet's own floor, not the global derivation floor and not a recent block: the run certifies what it derives only when its bottom is at or below the wallet's floor or exactly at cursor + 1, and a range that starts above the damage inherits the balance the damaged rows left behind.

If from_block comes back NULL, the account has no floor pair stored — a boundary block that never resolved, or a row written before migration 104's grandfather. That is an ordinary state, not a fault, and the engine already has an answer for it: UNRESOLVED_HISTORY_FLOOR, which is the derivation floor 24136053. Pass that as --from. It is below everything this wallet has, so it certifies and costs only walk time. A NULL to_block is different: it says the wallet has never been derived on this chain, which for this holder cannot be true — stop and report it rather than substituting the chain tip.

2. Take the "before" figures first, because the repair restates them and there is then nothing to compare against.

sql

-- read-only. The leg's stored history, and what it currently says the holder holds.
SELECT count(*)                                   AS rows_stored,
       count(*) FILTER (WHERE kind = 'withdraw')  AS withdrawals,
       max(block_number)                          AS highest_row,
       (SELECT to_balance FROM onchain_credit.portfolio_flow_events_v2
         WHERE chain_id = 1 AND wallet = '0x68e7e72938db36a5cbbca7b52c71dbbaadfb8264'
           AND position_key = 'vault:0x093272c07700d3ca5301c3bf9b3a392624179e2f'
         ORDER BY block_number DESC, log_index DESC LIMIT 1) AS last_to_balance
  FROM onchain_credit.portfolio_flow_events_v2
 WHERE chain_id = 1 AND wallet = '0x68e7e72938db36a5cbbca7b52c71dbbaadfb8264'
   AND position_key = 'vault:0x093272c07700d3ca5301c3bf9b3a392624179e2f';

3. Re-derive, RANGED and scoped to the one wallet. Ranged because a bare sweep resumes at the cursor and derives nothing, which is the silent no-op this mode exists to prevent; scoped because exactly one wallet is affected and re-deriving the rest would be a multi-hour campaign for no change.

bash

cd /opt/onchain-credit
set -a; . ./.env.local; set +a          # the builder is not a cron job: it needs the env here
npx tsx scripts/build-shadow-ledger.ts \
  --wallet 0x68e7e72938db36a5cbbca7b52c71dbbaadfb8264 \
  --from <query 1's from_block> --to <query 1's to_block> --range 50000

--range 50000, not the 200,000-block default the builder falls back to. The prod role carries statement_timeout = 30s, and a 200,000-block range's reads do not finish inside it: the run is cancelled mid-range and writes nothing (measured 2026-09-05). This is the only step on this page that passes --range — take the width from here, not from a precedent.

4. Then the acceptance reconciler, read-only, on the same wallet.

bash

cd /opt/onchain-credit && DATABASE_URL="$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)" \
  npx tsx scripts/ops/reconcile-ledger.ts --wallet 0x68e7e72938db36a5cbbca7b52c71dbbaadfb8264

What moves, judged as SHAPE first.

Withdrawal rows APPEAR; nothing that should survive is deleted. Fourteen were missing inside the wallet's stored history window on 2026-09-16 — 15,363,939.48 USDC and 14,542,419.87 shares between 2026-08-18 and 2026-09-16, the two largest being 5,611,414.579986 USDC on 09-07 and 5,328,792.398150 USDC on 08-30. Re-run query 2: withdrawals rises by the number the range actually held, and rows_stored rises by the same amount — plus the five fee rows below, which are withdraw rows too. No deposit moves: none of this wallet's deposits shares a transaction with a forced pull, so a changed deposit count means the range or the scope is wrong.
The leg's last stored balance is the chain's — and AFTER the run that is a verdict, not a hope. Re-run query 2 for highest_row and last_to_balance, read IVaultV2(0x093272c0…).balanceOf(0x68e7e729…) at that block, and the two must agree to the unit. The same pass that inserts the rows recomputes every to_balance from a fresh archive anchor, so agreement is what a landed run produces. A disagreement is a STOP: do not run the reconciler (step 4) and do not re-run the builder; capture both figures, the block they were read at, and the run's [shadow-build] lines, and escalate. The run wrote rows whose balances are not the chain's, and a second pass over the same range would only write them again.
Whether the BEFORE figure moves is a separate question, and it is measured rather than predicted. Take the same reading before the run too: the issue recorded a stored 9,763,396,524,832,301,595,415,391 against about 826,737.79 shares on chain on 2026-09-16, but an offline replay of the same logs closed on the chain's figure with the old code as well, so how far the stored number moves depends on how the stored series was built. Nothing about the step is judged on that; the acceptance is the bullet above, plus the row count and the withdrawal amounts.
The position CLOSES mid-history and REOPENS, twice, and that is the chain. Two of the restored redemptions are FULL exits rather than partial ones — block 25,864,842 (2026-08-30 01:37:23Z) and block 25,923,774 (2026-09-07 06:45:59Z) — and after each one the holder deposits again. Every time quoted here is the BLOCK's own timestamp, like the fee table below; the feed prints the stamp stored with the event, which is interpolated on historically swept rows and can sit a minute or more off the block's (issue #749), so identify a row by its block and not by its clock. Worked through on the second exit, which is the one the issue is about: balanceOf(0x68e7e729…) is 5,307,605,297,946,978,707,439,278 at block 25,923,773, exactly the shares that withdrawal burns, and 0 at 25,923,774; the next deposit lands at block 25,925,046 (2026-09-07 11:01:11Z, 254,252.686546 USDC for 240,478.632551637336367068 shares, tx 0xec922516… log 231) and opens a SECOND series on the same leg. So the repair writes a terminal row in the middle of the wallet's stored history and the position reads closed-then-reopened on that date, on the Activity feed and in the history chart alike. Expect it: restoring the row that ENDED the position is the point of the repair, and a close followed by a reopen mid-history is otherwise a shape worth being suspicious of. (Reading further out, balanceOf is 826,737,792,636,922,870,855,503 at 25,989,485 — the holder is still in.)
The position's "Yield earned" stops carrying the redemptions. It read about −$15.36M before; afterwards it is the small figure the position actually earned. The USD book's total-return and accrual lines restate on the next page load — they are read-path, so no second step is owed for them.

The five forced-pull FEES appear too, as withdrawals of nothing. The release that carries the second half of #889 books the fee itself as a row — amount_raw = 0, because none of it reached the holder, and qty_delta = −shares, because the shares were burned — so the run inserts these five as well as the fourteen redemptions. Verified log for log on chain 2026-09-17, each fee's Withdraw immediately followed by its ForceDeallocate:

block	UTC	log	`qty_delta` (shares)	fee
25,946,586	2026-09-10 11:07:47	478	−3024861209714764938	3.199930 USDC
25,953,043	2026-09-11 08:44:11	571	−4716201562560842585	4.989951 USDC
25,965,468	2026-09-13 02:17:59	595	−1417297029749754137	1.499986 USDC
25,974,486	2026-09-14 08:25:59	1549	−463583795728652513	0.490738 USDC
25,988,038	2026-09-16 05:46:59	605	−2984623896167379201	3.160410 USDC

That is 12.606567493921393374 shares and $13.341015 in total. Each row carries meta.penalty = true and the fee on meta.penaltyAssets, so they are separable from the fourteen redemptions in one query; none of them adds to any flow figure, because their amount is zero.

Expected anomalies: NONE. Under the first half of #889 alone each of the five fees was a quantity-drift note saying the leg lost quantity with NO ROW to carry it — the designed report at that point. With the fee booked as a row there is nothing left to report: the leg's own rows add up to the balance stored on them. A drift note on this leg after the run, or a withdrawal still missing, is a stop.
No other wallet moves. Nothing else has ever had a forced pull naming it, so a second wallet's figures changing means the run was not scoped.

Cautions. Never point backfill-portfolio-wallet.ts --fresh at this: it is a different tool, and a refused delete there parks the wallet and needs a hand restore. The builder's own refusal is loud and harmless by comparison — it withholds the stamp and exits non-zero, which means the run did not certify and must be reported rather than re-run narrower. And do not run this on staging: the nightly reseed takes it back.

The coverage rule: wipe every account (no history rebuild)

belongs to: migration 105 / the coverage rule — v0.64.0
executed: 2026-09-17

Run 2026-09-17 (v0.64.0), 13:22Z, after 105–108: 11 accounts, 5 tracked wallets, 1,728 snapshot rows, 241 flow rows, 11 wallet-token coverage rows and 1 held-PT row deleted; the '*' marker kept; SESSION_SECRET rotated; app and ingester restarted. The backup in step 1 misses the partitions: -t on a partitioned table dumps the parent only, which holds no rows. The run used --table-and-children (pg_dump 16) for every table the wipe touches. --execute also needs --confirm-db=creddit, which the command above omits; without it the script refuses. And the ingester alarm's feed-lag arm starts paging about 6h later while no wallet is enrolled: with an empty list the wallet-token stream is not emitted, so its cursor stops while its rollout marker stays (see the ingester alarm). The first sign-in clears it.

The standing procedure is now deleting portfolio users in the runbook: this recipe, plus (from the ledger-first release on) the ledger worker's stop before the wipe and the derivation queue's jobs of the deleted wallets cleared after the ingester's restart. The commands below are this entry's own, as written for 2026-09-17 (the note above says where that run differed from them).

Fred's decision: every wallet is deleted and NO history is rebuilt. The sweep universe changes with this release — eleven bare tokens start being read, eighteen assets are added, eleven are retired — so every stored history is a history of a different universe from the one the release reads. Re-deriving would take hours and produce a history nobody asked for; deleting is instant and the next six-hourly tick starts a clean one.

Run it between 6h ticks (the refresher runs at 50 */6 UTC), as the postgres user, and take a data-only backup first.

bash

# PROD. 1. back up what is about to be deleted.
sudo -u postgres pg_dump -d creddit --data-only \
  -t onchain_credit.accounts -t onchain_credit.account_wallets \
  -t onchain_credit.portfolio_position_snapshots -t onchain_credit.portfolio_flow_events_v2 \
  > /root/backup-users-$(date +%F).sql

# 2. the wipe itself, as the postgres user (the script writes through the socket).
sudo -u postgres bash -lc 'cd /opt/onchain-credit && \
  DATABASE_URL="postgres://postgres@/creddit?host=/var/run/postgresql" \
  npx tsx scripts/ops/reset-portfolio-users.ts --execute'

# 3. rotate the session secret, so no old cookie resolves to a deleted account,
#    and restart both processes so they pick it up AND drop the enrolled-wallet
#    list they are still holding.
#    (edit SESSION_SECRET in /opt/onchain-credit/.env.local first)
pm2 restart onchain-credit --update-env
pm2 restart creddit-event-ingester --update-env

# 4. the two tables the reset script does not own. AFTER the restart, or the
#    still-running ingester re-writes the coverage rows underneath the DELETE.
sudo -u postgres psql -d creddit -c \
  "DELETE FROM onchain_credit.event_coverage WHERE stream = 'wallet-token' AND address <> '*';"
sudo -u postgres psql -d creddit -c "DELETE FROM onchain_credit.portfolio_held_pts;"

# 5. confirm the ingester enrolled nothing. Wait one 60s cycle after step 4: the line prints
#    only when the count CHANGES, so the pass is the drop -- `enrolled wallets: 11 -> 0`.
sleep 70
pm2 logs creddit-event-ingester --lines 50 --nostream | grep 'enrolled wallets'

Restart before the deletions, not after. A running ingester keeps writing the wallet-token coverage rows step 4 deletes — its in-flight enrolment backfill re-stamps them as it works. Run 2026-09-23 with the old ordering and three of the eleven rows came back seconds after the DELETE; the restart then re-read the enrolment list, found those rows and resumed backfilling three deleted wallets (enrolled wallets: 3), so the deletion had to be repeated. The restart is what stops that backfill; it does not drop a list. The ingester re-reads its enrolled wallets every cycle, and the read unions the wallet-token coverage rows themselves, so between steps 3 and 4 it enrols the eleven wallets from those rows alone with every account already deleted — which is why step 5 is read AFTER step 4, one cycle later.

The wallet-token coverage rows go because the stream's universe widened: a certificate earned over the old token set is not true of the new one, and DELETE rather than UPDATE is what makes the next enrolment re-earn it. portfolio_held_pts goes because it is a per-wallet index of a set that no longer has any wallets in it.

GATED STEP: give the running ingester a 120s kill timeout

belongs to: the ingester feed-lag alarm + kill-safe reorg repair release (issues #860 / #858) — v0.64.0
executed: 2026-09-17

Run 2026-09-17 (v0.64.0): read (unset -> pm2 default 1600) before, 120000 after the first form (pm2 restart … --kill-timeout 120000 --update-env + pm2 save), so the box's pm2 merges the flag on restart. The next hand-run of the alarm printed no grace line.

docs/deployment.md now starts the ingester with --kill-timeout 120000, so a restart lets the current cycle finish instead of SIGKILLing it after pm2's default 1.6 seconds. That is a change to the documented start command only. scripts/ops/restart-ingester.sh — which is what every prod deploy runs — issues pm2 restart, and a restart reuses the stored process definition, so the ingester already running on prod keeps whatever kill_timeout it was created with until somebody changes it here.

Run it once, on prod, at any quiet moment (it restarts the ingester, which every deploy does anyway). Read the value back before and after. Current pm2 does merge a restart's own CLI options into the stored process (God.restartProcessId → Utility.extendExtraConfig copies the invocation's config over pm2_env), so the first form should suffice — but the version installed on the box is what decides, and the read is what settles it.

bash

# PROD. 1. what the running process has today (prints ONE number, never the process list --
#           `pm2 jlist` carries every process's environment, `.env.local` secrets included).
pm2 jlist | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{
  const p=JSON.parse(s).find(x=>x.name==="creddit-event-ingester");
  console.log(p?.pm2_env?.kill_timeout ?? "(unset -> pm2 default 1600)")})'

# 2. set it, and persist the definition so a reboot keeps it.
pm2 restart creddit-event-ingester --kill-timeout 120000 --update-env
pm2 save

# 3. read it back. Expect 120000.
pm2 jlist | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{
  const p=JSON.parse(s).find(x=>x.name==="creddit-event-ingester");
  console.log(p?.pm2_env?.kill_timeout)})'

If step 3 does not print 120000, this pm2 does not merge the flag on restart. Re-create the process from the runbook's start command instead — the ingester is down for the seconds between, which is the same gap every deploy's restart opens:

bash

# PROD, only if the read above disagreed.
cd /opt/onchain-credit
pm2 delete creddit-event-ingester
pm2 start scripts/ingester/run-ingester.sh --name creddit-event-ingester --time --kill-timeout 120000
pm2 save
pm2 logs creddit-event-ingester --lines 30      # expect a `[ingester] starting` line

From here the hourly alarm confirms it for you. check-ingester-freshness.ts reads kill_timeout back on every tick and prints a non-paging note while it is not 120,000 ms, so the step is settled by reading the log rather than by another SSH round trip — and a later pm2 resurrect from a stale dump that drops the field announces itself the same way.

Read the NEWEST TICK, not the whole file. The log is append-only and never rotated, so every grace note written between this release's deploy and this step stays in it for ever: grepping for the note alone always finds those and reads as a failure on a step that succeeded. Anchor on this checkout's label instead — prod and staging share the file, which is keyed by script — and look at the last tick's two or three lines for the ABSENCE of a grace note among them.

bash

grep 'env=onchain-credit$' /tmp/onchain-credit-cron/onchain-credit-check-ingester-freshness.log | tail -3
# after the next tick: a freshness `ok:` line and a feed-lag line, and NO `grace period` line
# beside them. A grace line among the newest names the value the box actually has.

Verification that it did what it is for, next time the ingester is restarted: the log ends with [ingester] SIGTERM received; finishing the current cycle then exiting followed by [ingester] stopped. Under the 1.6s default the second line is missing, because the process never reached it. Read the last lines rather than counting matches — a count over the retained window includes every earlier graceful stop and so proves nothing about this restart.

bash

pm2 logs creddit-event-ingester --lines 5 --nostream

GATED STEP: rebuild the wallets whose window was cut at their first move

belongs to: the held-at-floor window fix (issue #852, branch fix/held-at-floor-window) — the next release
executed: 2026-09-22

Recorded 2026-09-22, not run: there is nothing left to rebuild. The nine wallets this step was written for were all test wallets, and Fred had them deleted on 2026-09-22 14:14Z, before the release: accounts, tracked-wallet rows, spine, flow ledger, wallet index, backfill state, their wallet-token coverage rows and their portfolio:derive:v2: cursors, in one transaction (data-only dump first, /root/backup-users-20260922.sql; SESSION_SECRET rotated and the app and ingester restarted, per the reset recipe). Step 1's scope query returned no row afterwards, so per its own rule this is recorded rather than run. Every wallet registered from the release on is built by the fixed code; a wallet that shows the defect again is a NEW instance and takes this step as written.
That delete was a hand recipe, and it is now written down, corrected, as a standing procedure: deleting portfolio users. Its cursor selector matched the scope's last 42 characters, which catches a wallet's bare derive cursor and misses its suffixed records (:jit-pending, :replay-deferred); the runbook's selects by the scope's own wallet part, and adds the ledger worker's stop and the derivation queue. Any suffixed record that run left was deleted by the full wipe of 2026-09-23 (the reset tool deletes the whole portfolio:derive:v2:% prefix), so nothing of it remains on prod.

Runs once, on prod, after the release that carries the fix, and only there. Staging is not a rehearsal: its nightly reseed restores prod's dump and then scrubs the whole user-data graph, so it holds no wallet for this to act on.

Why the release is not enough on its own. The fix changes where a REPLAY opens, and a replay runs once, when the wallet is registered. Nothing re-opens a window afterwards: the 6h tick and the JIT persist both append above the wallet's tip, and a re-add takes the gap-patch path, which by design never touches anything at or below the coverage anchor. So every wallet already built under the unconditional raise keeps the cut chart it was given — the fix only spares the wallets registered from here on. Putting the dropped days back means re-running the build for each affected wallet, which is what this step is.

What was wrong, in one line. The replay opened at the wallet's stored floor RAISED to its first curve-starting activity, always. That raise exists for a wallet that held nothing at its floor, whose chart would otherwise open on a flat-zero lead-in. Applied to a wallet that already held a position there, it dropped every day before that wallet's first in-window move — 25 days on 0xea88d13e…7974, which held a Pendle PT and an sGHO balance right through its floor — and where the first move was the EXIT of one of those positions, it opened the chart on the day the wallet closed something.

1. Scope it from the database, not from this page. Read-only, on prod. The wallets to rebuild are the ones whose chart start sits ABOVE their stored floor; the set moves with every signup, so take it at the moment you run.

sql

-- read-only. One row per wallet whose window was cut, deepest cut first.
SELECT a.uid,
       a.history_floor_ts,
       b.floor_ts                                          AS chart_start,
       (b.floor_ts::date - a.history_floor_ts::date)        AS days_cut
  FROM onchain_credit.accounts a
  JOIN onchain_credit.portfolio_backfill_state b USING (uid)
 WHERE b.status = 'done'
   AND b.floor_ts > a.history_floor_ts
 ORDER BY days_cut DESC, a.uid;

On 2026-09-22 this listed nine wallets, and eight of the nine are the defect. The ninth, 0xef08c6a4…07c5, held idle USDC and ETH at its chart start and nothing else, which is the shape the raise exists for, so it is expected to keep its 2026-09-16 start and to still appear in this query after the rebuild. Treat that as an expectation and not as a check: what decides the window is what the wallet held at its floor (2026-08-18), a month below the day that observation was made, and only the rebuild's own floor read answers that. If its start moves down, the wallet held something curve-starting at its floor after all and it was a ninth instance of the defect, not a regression. Record the rows as they come back rather than assuming that list — wallets added since will be in it, and a wallet may have been unlinked.

2. Take the "before" figures, per wallet, because the rebuild restates them — and because the second query is what a refusal is undone from (below).

sql

-- read-only. Substitute the uid. The chart's first day and how much history it holds.
SELECT min(snapshot_ts) AS first_point,
       max(snapshot_ts) AS last_point,
       count(*)         AS rows_stored
  FROM onchain_credit.portfolio_position_snapshots
 WHERE wallet = '<uid>';

sql

-- read-only, and KEEP THIS OUTPUT. These four values are the wallet's whole build state;
-- a refused run revokes the anchor, and this is the only record of what it was.
SELECT status, floor_ts, covered_through_ts, covered_through_block
  FROM onchain_credit.portfolio_backfill_state
 WHERE uid = '<uid>';

3. Rebuild, one wallet at a time. The history writers serialise on one advisory lock, so running these in parallel only queues them behind each other; the drain keeps running throughout, since it claims queued rows and these are done.

bash

# PROD. Per uid from step 1, one at a time.
cd /opt/onchain-credit
scripts/run-cron.sh backfill-portfolio-wallet.ts --uid <uid> --fresh

Output lands in /tmp/onchain-credit-cron/onchain-credit-backfill-portfolio-wallet.log. The file is prod's own — the log name carries the checkout — but it is still append-only across every run of this job, so anchor on the run you just started. The line to read is the probe's: held at floor: yes (N curve-starting leg(s) @block B) for a wallet this step is meant to fix, followed by a replay whose grid opens at the wallet's stored floor.

--fresh is DESTRUCTIVE by design, and it is the only mode that can do this: it replaces the wallet's whole history, which is the point — the cut chart has to go. Two consequences an operator must know before running it.

The wallet's 6h resolution is thinned to daily, over its WHOLE history, permanently. The replay deletes every snapshot at or below its own last grid point — the 6h seam at ~now, not the signup date — and re-lays daily UTC midnights plus that one seam. So the 6h live points written since the wallet was added are not merely joined by a daily pre-signup span: they are replaced by it. The wallet's chart reaches weeks further back and draws one point per day where it drew four, and the 6h cadence only resumes from the next tick after the rebuild. This is the consequence to weigh before running a destructive command against a live user's portfolio, and it applies to every wallet in step 1, not only to the ones the fix moves.
A position that has dropped out of the window since the original build is pruned, because a fresh run re-derives the replayed set at run time.

Both are the documented behaviour of a fresh replay, not side effects of this step. The run re-earns the coverage anchor when it completes.

If a rebuild REFUSES, stop. A --fresh run revokes the wallet's coverage anchor BEFORE its first destructive write, deliberately, so that a run dying mid-wipe cannot leave its retry dispatched to the gap path. The §4.8 universe-superset refusal is taken after that revoke and before the wipe, so a refused run writes and deletes nothing and still ends with the anchor gone. That combination is terminal: the drain claims queued rows only, so it never retries an error row of its own accord, and re-adding the wallet re-queues it onto the fresh path where the same refusal fires again. This is not hypothetical — it happened on prod on this exact command during the Morpho PT relay step (#807), on a wallet whose stored rows named assets it had since disposed of, and the recovery was a hand restore of the row.

What it looks like. The CLI exits non-zero and run-cron.sh raises the ordinary cron alert (a [fail] line and a non-zero exit are both halves the alert needs). In the log: [fail] backfill <uid>: flow wipe at/below <block> REFUSED: N stored wallet asset(s) missing from the replay universe (…), then [backfill <uid>] REFUSED before the first destructive write: status 'error', no coverage anchor, nothing written.
What it leaves. status='error' with the verdict on the row, floor_ts untouched, the wallet's OLD history and flow rows fully intact, and covered_through_ts / covered_through_block NULL. The user's chart is unchanged, still cut; nothing is lost.
Restore it by hand, from the four values step 2 recorded for that wallet. Nothing else will: UPDATE onchain_credit.portfolio_backfill_state SET status = 'done', error = NULL, covered_through_ts = '<recorded>', covered_through_block = <recorded> WHERE uid = '<uid>'; That puts the wallet back exactly where it was before the run, cut chart and all, and its 6h ticks resume on the gap path.
Then stop and decide, do not run the other seven. The refusal says the replay universe no longer covers assets the wallet's stored rows name, which is a property of that wallet's holdings and not of this fix; whether the remaining wallets share it is a question to answer before spending another destructive run. The verdict line's own remedy says which case it is: an ERC-20 miss clears once the wallet-token backfill covers the wallet, and a native-ETH miss cannot clear at all while the live balance reads zero.

4. Re-run step 1. A wallet that held a curve-starting position at its floor now reports floor_ts = history_floor_ts and has dropped out of the result. 0xef08c6a4…07c5 is expected to remain, for the reason step 1 gives — but its floor read is what decides that, so a start that moves down on that wallet is a ninth instance of the defect and not a regression. Record before/after per wallet under this step's executed: line and set the date.

5. The ledger merge runs inside each rebuild, and its line should read ledger-v2 … [<floor>, <tip>] without DEFERRED: these wallets' bare-token coverage is certified down to their floors already. A DEFERRED line is not a failure of this step — the drain re-checks every tick and derives once the enrolment lands — but note which wallet it happened on, because until it derives that wallet has a spine and no receipts.

Rollback. There is none to take, and none is needed. The rebuild produces the wallet's history under the corrected rule, which is more history than it had and the same data where they overlap; a code revert would not shorten a chart already rebuilt, and shortening one by hand would be re-introducing the defect on that wallet.

The PT row's pre-window fills: first deploy (migration 113)

belongs to: migration 113 (a held PT's real pre-window router fills, docs/plans/pt-row-attribution-plan.md D2c) — the release carrying feat/pt-row-attribution
executed: 2026-09-23

Applied 2026-09-23 17:57:56Z on prod (v0.69.0), after the deploy, by migrate.sh; schema_migrations records it (read back 2026-09-24 during the v0.70.0 release).

113 is ADDITIVE: one new table, portfolio_pt_prewindow_fills, FK-cascaded from accounts, with grants for both app roles. Staging auto-applies it on the merge deploy. On prod it is the usual gated manual step, and it may run AFTER the deploy: the code tolerates the table's absence on both sides (the wallet build reports table absent (migration 113 not applied), nothing stored and the loader reads no fills), so the only cost of a late apply is that wallets built in between keep the stand-in lot for PTs held since before their window until their next build.

bash

ssh -i ~/.ssh/hetzner_ed25519 root@dexhq.io
cd /opt/onchain-credit && scripts/ops/migrate.sh "$(grep ^DATABASE_URL= .env.local | cut -d= -f2-)"
sudo -u postgres psql -d creddit -At -c "SELECT to_regclass('onchain_credit.portfolio_pt_prewindow_fills'), has_table_privilege('onchain_credit', 'onchain_credit.portfolio_pt_prewindow_fills', 'INSERT')"
# expect: onchain_credit.portfolio_pt_prewindow_fills|t

No backfill and no re-derive. Everything else in the release is read-time or forward-only: the PT receipt stamps (meta.ptFactor / ptRate / ptMarkValue) are written by every derivation from the deploy on, and a row derived before them reads exactly as it did (the lot book falls back to the stored factor series, and to the coin count or the row's dollar value for the price). What that means on the page until the wallet's next full build re-derives its ledger: the PT row's open position is stated in full except that its mark to market stands unsplit (a lot bought before the deploy has no market yield at its purchase on file), and a purchase paid in a coin other than the payout coin keeps its rate struck from its dollars at $1, a few basis points off. Nothing reads as a dash for want of a stamp. A wallet added after the deploy gets all of it on its first build.

GATED STEP: re-judge the stored price history against both vendors

belongs to: the re-judge repair (scripts/repair/rejudge-price-bars.ts, branch feat/price-rejudge) — the release carrying it
executed: 2026-09-24

Run end to end on prod 2026-09-24 (v0.70.0, merge 8e465826, deployed 16:27Z), on Fred's go in session. Step 2 was taken FIRST, read-only with the cron still running (16:29-16:39Z; END 2026-09-22T16:00Z; 94 CoinGecko + 1,006 DefiLlama requests, 0 Dune credits), so that the cron was drained only for the writes: nothing the cron writes can reach an hour at or before END. Step 1 drained the 18 prod lines at 16:39:37Z, and step 2 re-run from the cache with the cron drained planned the identical 1,875 retractions, 4,148 fills and 1,738 basis rows. That is the reviewed report's plan exactly, bar one extra spike-refused UNI candidate, which writes nothing. Step 4 (dump /root/rejudge-pre-20260924T165025.sql) committed 52 tokens with no rollback: 1,875 retracted, 1,884 retracted rows replaced, 4,148 bars inserted, 1,738 token_basis rows rewritten, 0 errored; one wallet (0xe51d…3ed9, via AUSD) queued. Step 5: the re-run plans 0 / 0 with nothing owed; AUSD Sep 3-10 now tops out at $1.00032 (181 coingecko:hourly + 11 tape bars); apyUSD Aug 27 - Sep 15 reads $1.350-$1.383; 4 convicted hours stay unfilled (vendor split or dark). Step 7 ran BEFORE step 6, with the cron still drained as the re-mark's own procedure requires, and scoped to --wallet=0xe51d…3ed9 and AUSD's three consumer assets (the --tokens=AUSD dry run's list) rather than the full report's 185-asset union, which covers every changed token although only AUSD's hours were read by a stored row: 7 snapshot rows (Sep 4-10) went from $65.1M-$154.6M to $3.69-$4.20, and a second run repaired 0. Step 6 restored the crontab at 16:51:36Z (identical to the backup; drained for 12 minutes), and the wallet's cursor was back at 16:52Z (launched=1 stamped=1 failures=0). The state files, the dump and every log are in /root.

Runs once, on prod, after the release that carries the script, and only on Fred's explicit go on the dry-run report posted to the pull request. Staging can rehearse it — its bars are prod's as of the nightly reseed — and the rehearsal is wiped by the next reseed; remove its state with it (rm /var/tmp/rejudge-price-bars/*-creddit_staging.jsonl /root/rejudge-end-creddit_staging), or a later staging run replays follow-up for writes the reseed undid.

Why the release is not enough on its own. The six-hourly leg judges a bar as it arrives and fills an hour that goes missing, inside its own 48-hour window, and it has only run since v0.68.0. Every bar accepted before that was never judged, so the stretches found on 2026-09-21 were still stored and served: AUSD at $17.7M-$36.8M for 181 hours (Sep 3-10), apyUSD at $0.85-$1.09 against about $1.36 (Aug 27 - Sep 15, partly retracted on 09-23), FBTC (Aug 17-20, Sep 18-19), eBTC (Aug 19), ETHx (Aug 28) and osETH (Jun 27-29). Any wallet whose 30-day window reaches one of them marks off it. The rules and what the repair does are in Re-judging the stored history.

1. Drain the cron. Every prod run-cron.sh line, because three of them write what this step writes or reads (the six-hourly refresh-assets writes bars and retracts, the portfolio tick and the minutely drain mark off bars and stamp derive cursors), and a plan read while one of them runs is refused token by token. Staging's lines, the backup and the reseed are left alone.

bash

crontab -l > /root/crontab.bak-rejudge
crontab -l | sed 's|^\([^#].*/opt/onchain-credit/scripts/run-cron.sh .*\)$|# REJUDGE \1|' | crontab -
crontab -l | grep '/opt/onchain-credit/scripts/run-cron.sh' | grep -vc '^# REJUDGE'   # expect 0
pgrep -af '/opt/onchain-credit/scripts/' || echo idle   # wait until nothing of prod's is running

2. Dry run, read-only, with a fixed end and a vendor cache. The end and the cache are what make step 4 write exactly what step 3 reads: the execution plans from the same stored bars (the cron is drained) and the same vendor answers (the cache), so a vendor restating history in between cannot change what is written. The end is written to a file the first time this block runs and read back every time after, so re-running the block, in any shell, on any day, judges the same window.

bash

cd /opt/onchain-credit
[ -s /root/rejudge-end-creddit ] || date -u -d '48 hours ago' +%Y-%m-%dT%H:00:00Z > /root/rejudge-end-creddit
END=$(cat /root/rejudge-end-creddit); echo "$END"
mkdir -p /var/tmp/rejudge-cache && chmod 1777 /var/tmp/rejudge-cache
(set -a; . ./.env.local; set +a; \
  PGOPTIONS='-c default_transaction_read_only=on -c statement_timeout=600000' \
  node_modules/.bin/tsx scripts/repair/rejudge-price-bars.ts --to="$END" \
  --vendor-cache=/var/tmp/rejudge-cache) 2>&1 | tee /root/rejudge-dryrun-$(date -u +%Y%m%dT%H%M%S).log

It takes about half an hour (roughly 19 keyless DefiLlama requests a token, and CoinGecko only over spans holding a suspect bar or a gap), buys no Dune credits, and writes nothing. The statement timeout turns a hung query into a failure rather than a wait.

3. Read the magnitude. If it is not what the report on the pull request said, STOP. Per token: bars judged, bars to retract, hours to fill and what each replaces, and every retraction with its hour and the three prices. The plain stables the tape and the vendors agree about (USDC, USDT, DAI, …) must show retract 0. A [partial] line means a vendor could not be asked about some span; re-run step 2's block (the cache keeps every answer already had, and the end is read back from its file) until it is gone.

4. Back up, then execute as the table OWNER (a refill deletes the retracted row it replaces, and 055 revoked DELETE from the app role), over the same end and cache, --offline so nothing is asked twice. Nothing else in this file runs tsx as postgres, so the first line checks that it can, before anything is written:

bash

cd /opt/onchain-credit && sudo -u postgres node_modules/.bin/tsx --version   # must print a version
END=$(cat /root/rejudge-end-creddit); echo "$END"                          # step 2's end
sudo -u postgres pg_dump -d creddit --data-only -t onchain_credit.token_price_bars \
  -t onchain_credit.token_basis -f /var/tmp/rejudge-pre-$(date -u +%Y%m%dT%H%M%S).sql
sudo -u postgres env DATABASE_URL="postgres://postgres@/creddit?host=/var/run/postgresql" \
  PGOPTIONS='-c statement_timeout=600000' \
  node_modules/.bin/tsx scripts/repair/rejudge-price-bars.ts --execute --confirm-db=creddit \
  --to="$END" --vendor-cache=/var/tmp/rejudge-cache --offline \
  2>&1 | tee /root/rejudge-execute-$(date -u +%Y%m%dT%H%M%S).log
cp /var/tmp/rejudge-price-bars/rejudge-*-creddit.jsonl /var/tmp/rejudge-pre-*.sql /root/

Each token is one transaction. Before it commits, the rows a refill deletes are appended to /var/tmp/rejudge-price-bars/rejudge-deleted-creddit.jsonl and the hours it wrote to rejudge-changes-creddit.jsonl beside it; a failure to write either rolls the token back. Both are named by the database and moved by no flag, so every later run on this database finds them. They are written under /var/tmp because the execution runs as postgres, which cannot write /root, and /var/tmp is swept after 30 days, so the block's last line copies them and the dump to /root, where the logs already are. A [fail] line names a token that rolled back (its stored rows moved since the plan, because something wrote while the cron was meant to be drained, or it threw): nothing of that token was written. Find the cause, then re-run step 4's block without --offline (the end is read back from its file; the cache still answers every span it holds, and only what the repaired history now asks differently is fetched; each run keeps its own dump). A re-run plans nothing for a token already repaired, finishes the basis recompute and the wallet step over every logged write that step has not yet completed, and never repeats a step that completed, so a wallet already queued is not queued a second time.

5. Verify. Re-run step 2's block (it reads the same end back): every token must now plan retract 0 and fill only what it could not fill before (a vendor split, a dark hour), and no [partial] line may say the change log still owes follow-up. Then the headline stretch:

sql

-- read-only. The AUSD September stretch: no bar above a dollar and a cent remains accepted.
SELECT source, rejected, count(*), max(price_usd)
  FROM onchain_credit.token_price_bars
 WHERE chain_id = 1 AND token_address = '0x00000000efe302beaa2b3e6e1b18d08d69a9012a'
   AND bar_ts >= '2026-09-03' AND bar_ts < '2026-09-11'
 GROUP BY 1, 2;

6. Restore the cron. crontab /root/crontab.bak-rejudge. The report's wallet line has two halves. The wallets it queued are re-derived by the minutely drain's re-derivation trigger, two a minute: each had its derive cursor removed (only after the trigger's own gates said it would come back for it), and while it has none its ledger is withheld from the page (minutes). Watch /tmp/onchain-credit-cron/onchain-credit-drain-portfolio-backfills.log and confirm each cursor is back (SELECT scope FROM onchain_credit.chain_scan_cursors WHERE scope LIKE 'portfolio:derive:v2:1:%'). The wallets it named as not queued kept their cursors, because the trigger would not have come back for them (the reason is printed beside each): re-derive each by hand, ranged from its own floor to its own cursor, in the shape the forced-pull step spells out.

7. The snapshot half. If the report named wallets, their stored SNAPSHOT marks still carry the old prices, which the re-derivation does not restate. Run it BEFORE step 6, while the cron is still drained (the re-mark's own procedure drains first). The report's --assets= list is the union over every changed token; narrow it to the tokens printed beside the named wallets: a --tokens=<those> dry run over the same end and cache prints their list. Then run the category-flip re-mark once per named wallet, dry run first: scripts/run-cron.sh repair/remark-snapshot-marks.ts --wallet=<w> --assets=<that list>, then --execute; a second dry run must repair 0. If WETH is among those tokens, every ETH-book asset's snapshots are in scope too.

8. Record the run here: the date in executed:, and the summary lines of step 4's log. Then rm /root/rejudge-end-creddit; keep the /root copies of the two state files and the dump until the release after next.

Rollback. The deleted rows are in rejudge-deleted-creddit.jsonl, the hours written in rejudge-changes-creddit.jsonl, and the whole of both tables in the step-4 dump, all three copied to /root at the end of step 4 (the originals under /var/tmp are swept after 30 days). To undo a token by hand, as the owner: delete its coingecko:hourly / llama:chart bars at the logged hours, re-insert the backed-up rows, and set rejected = false on the bars the run retracted (reject_reason LIKE 'vendor-disagreement-rejudge%'); then restore token_basis for those hours from the dump. There should be no reason to: every retraction is two vendors agreeing, and step 3 is where to refuse one.

Release steps (one-time) ​

Reading the executed: field ​

Per-migration first-deploy steps ​

AI assistant migrations 038 / 039, and the 044 coordinated migration ​

048-account-wallets.sql (portfolio multi-wallet) ​

Portfolio taxonomy: first deploy (migrations 049 / 050 / 052) ​

Migration 061 (discovery completeness certificates) ​

Migrations 062 / 063 (coverage hardening, additive) ​

Portfolio 100k scale: Phase C envs + the snapshot repartition (migration 067) ​

Flow-ledger repartition: hash by wallet (migration 072) ​

Migration 079 (exposure key gains the collateral address, additive) ​

Migrations 082 / 083 (flow ledger + registries, additive) ​

Migration 084 (the raw_events stream CHECK validation) ​

Money market funds: first deploy (migration 086) ​

Fluid per-vault rates: first deploy (migration 093) ​

Retiring the old portfolio ledger ​

Pendle redemption-index factor: first deploy (migration 101) ​

THIS BACKFILL IS WHAT TURNS THE ACCRUAL LINE ON FOR AN EXISTING HOLDING ​

Pendle PT payout assets: first deploy (migration 102) ​

The coverage rule: first deploy (migration 105) ​

Migration 106 (retire the discovery scope rows) ​

Migration 107 (narrow the portfolio book CHECKs) ​

Migration 108 (drop the token_basis shadow table) ​

Staging env: the JIT positions fast path matches production (#909) ​

The price-source rule: first deploy (migration 109) ​

Hourly bars only, and the pool-quote feed retired (migration 110) ​

The declared route and the pricing categories (migrations 111 and 112) ​

The pricing categories release: prod steps, in order ​

The ledger-first release: prod steps, in order ​

The rate facts: migration 116 and its backfill ​

Rollback, and a roll-forward after it ​

Staging: the order before the comparator's NEW capture ​

The contract release: what this one leaves for the next ​

The native-ether receipts release (migration 118): prod steps, in order ​

Staging, before the release: the acceptance issue #966 asks for ​

One-time repairs and rebuilds (gated) ​

GATED STEP: relabel the receipts stranded below the tick cursor (#777) ​

GATED STEP: the Fluid tick-padding re-mark ​

GATED STEP: mark the Fluid births, so a loop's opening day is on the books ​

GATED STEP: re-derive every wallet that holds a fixed-rate principal token ​

GATED STEP: re-lay the wallets holding a Pendle PT as Morpho collateral ​

GATED STEP: the historical ledger build ​

When a wallet refuses ​

GATED STEP: reset the two Pool streams' coverage certificates ​

GATED STEP: reset the two Fluid streams' coverage certificates ​

GATED STEP: re-derive the holder whose redemptions shared a transaction with a forced pull ​

The coverage rule: wipe every account (no history rebuild) ​

GATED STEP: give the running ingester a 120s kill timeout ​

GATED STEP: rebuild the wallets whose window was cut at their first move ​

The PT row's pre-window fills: first deploy (migration 113) ​

GATED STEP: re-judge the stored price history against both vendors ​

Release steps (one-time)

Reading the `executed:` field

Per-migration first-deploy steps

AI assistant migrations `038` / `039`, and the `044` coordinated migration

`048-account-wallets.sql` (portfolio multi-wallet)

Portfolio taxonomy: first deploy (migrations 049 / 050 / 052)

Migration `061` (discovery completeness certificates)

Migrations `062` / `063` (coverage hardening, additive)

Portfolio 100k scale: Phase C envs + the snapshot repartition (migration 067)

Flow-ledger repartition: hash by wallet (migration 072)

Migration `079` (exposure key gains the collateral address, additive)

Migrations `082` / `083` (flow ledger + registries, additive)

Migration `084` (the `raw_events` stream CHECK validation)

Money market funds: first deploy (migration 086)

Fluid per-vault rates: first deploy (migration 093)

Retiring the old portfolio ledger

Pendle redemption-index factor: first deploy (migration 101)

THIS BACKFILL IS WHAT TURNS THE ACCRUAL LINE ON FOR AN EXISTING HOLDING

Pendle PT payout assets: first deploy (migration 102)

The coverage rule: first deploy (migration 105)

Migration `106` (retire the discovery scope rows)

Migration `107` (narrow the portfolio book CHECKs)

Migration `108` (drop the token_basis shadow table)

Staging env: the JIT positions fast path matches production (#909)

The price-source rule: first deploy (migration 109)

Hourly bars only, and the pool-quote feed retired (migration 110)

The declared route and the pricing categories (migrations `111` and `112`)

The pricing categories release: prod steps, in order

The ledger-first release: prod steps, in order

The rate facts: migration `116` and its backfill

Rollback, and a roll-forward after it

Staging: the order before the comparator's NEW capture

The contract release: what this one leaves for the next

The native-ether receipts release (migration `118`): prod steps, in order

Staging, before the release: the acceptance issue #966 asks for

One-time repairs and rebuilds (gated)

GATED STEP: relabel the receipts stranded below the tick cursor (#777)

GATED STEP: the Fluid tick-padding re-mark

GATED STEP: mark the Fluid births, so a loop's opening day is on the books

GATED STEP: re-derive every wallet that holds a fixed-rate principal token

GATED STEP: re-lay the wallets holding a Pendle PT as Morpho collateral

GATED STEP: the historical ledger build

When a wallet refuses

GATED STEP: reset the two Pool streams' coverage certificates

GATED STEP: reset the two Fluid streams' coverage certificates

GATED STEP: re-derive the holder whose redemptions shared a transaction with a forced pull

The coverage rule: wipe every account (no history rebuild)

GATED STEP: give the running ingester a 120s kill timeout

GATED STEP: rebuild the wallets whose window was cut at their first move

The PT row's pre-window fills: first deploy (migration 113)

GATED STEP: re-judge the stored price history against both vendors