Deep-history signup build

Built in three stages on feat/deep-history. Approved 2026-08-10, base origin/staging (v0.34.1). Prior context: fix-wave-v0321-plan.md (P5a's design-only feasibility notes), closed-in-window-replay-plan.md (the D2 design and the cost estimate that gated it), nav-rebuild-plan.md.

The normative plan is reproduced verbatim below, followed by an as-built section recording every place the implementation departed from it, and why. Descriptive reference for the behaviour itself lives in Data pipeline → Registration backfill, Metrics and Portfolio; this page is the decision record, not the documentation.

The plan, as approved

D1 History floor: 2026-01-01 (config constant), replacing signup−90d

• New constant PORTFOLIO_HISTORY_FLOOR_TS = 2026-01-01T00:00Z replacing DEFAULT_LOOKBACK_DAYS=90 as the replay start (still clamped by the ledger floor 2025-05-21 and per-token coverage certificates — verify each replayed group's window against event_coverage, never assume).
• Daily pre-signup grid unchanged; live 6h unchanged.

D2 Closed-in-window positions replay (kills the phantom-deposit defect)

• The replay set becomes: every position group open at ANY point in [floor, signup], discovered from the wallet's rows in the raw-events ledger (transfers + venue streams; the wallet's own topics), NOT from current holdings. open-set.ts / backfill.ts's hasCurrentPositions gate is the seam; its own comments anticipate this.
• A closed group replays only over its own lifespan (birth→death inside the window). Its exit flows are ordinary position events, so the cash side no longer books as an external deposit. ACCEPTANCE (real case): test wallet 0x12d2c3e2… must show its 2026-07-04 vault redemption (28,832.81 USDC) as a position exit, and NO +$28.8k external-deposit marker.
• M21/coverage/certificate invariants hold for closed groups (death explained by its own exit flows). Aggregate merge and trimIdleLeadIn must tolerate groups that are entirely historical.

D3 Tiered build (fast first paint, deepen in background)

• Tier 1: replay [signup−90d, signup] first → chart appears in ~1-2 min as today. Tier 2: extend BACKWARD to the floor as lower-priority queue work, same per-wallet building/progress machinery; the served series simply lengthens as tiers land. The building state/progress UX already exists — reuse, do not invent. Backward extension writes must respect the coverage anchor rules (never stamp unscanned ranges; the anchor semantics in backfill.ts:782 area are the guide) and must not fight the 6h cron.
• RPC discipline: batched multicall reads as existing; serial per wallet; Dune throttle stands. No new provider.

D4 Bundled: show cross-asset Fluid liquidation magnitudes

• Drop the read-path withholding (api-data.ts ~:959 area) — BUT only behind a one-time re-derivation: server step re-derives ALL currently-registered wallets from the new floor (full replay; the anchor-clearing recipe is in memory/project_v0320_prod_validation.md — requeue alone no-ops on fresh coverage). After that every stored row is post-fix and trustworthy. The wire loss field stays nullable (other withheld shapes remain).
• ACCEPTANCE: 0x1ec6942b…'s 9 markers serve real magnitudes summing ≈1.335856 ETH on the wire.

D5 Bundled: unvaluable-debt liquidation = withhold + alert

• When a liquidation's debt side cannot be valued (unregistered debt reserve), book the penalty NULL (withheld; marker draws with no magnitude) instead of full seizure, and emit an ops alert line that run-cron's alert grep will catch (exit!=0 not required for the write path — use the established alert pattern; check reference: alert needs a grep-matching line). Never a wrong number, never silent.

D6 Tests / docs / process (house rules, all binding)

• Unit tests incl. mutation-tested acceptance cases; fixture: add a closed-in-window position wallet to the seed (own commit, existing figures untouched); e2e: phantom-deposit absence, deep-chart tiers, Fluid magnitude shown; register new test files in package.json's explicit list; full e2e on ISOLATED ports with SKIP count read.
• docs/metrics.md + portfolio.md + data-pipeline.md updated same PR; docs build green. Migration only if truly needed — NEVER number 075; 076+.
• PR body: product terms, before/after, Server steps: (1) no migration expected, (2) post-deploy one-time re-derivation of all registered wallets from the new floor (serial), which simultaneously activates D4.
• Staging validation after merge: re-register the campaign account (recipe at scratchpad/prod-test-account/, adapt origin+basic-auth; MAX_TRACKED_WALLETS=3 cap — use addWallet's own statements minus the cap check), verify: charts reach back to 2026-01-01 where the wallet has history; D2/D4 acceptance numbers; no regression on the 14 fix-wave checks (spot-check the liquidation magnitudes and loop TR).

As built

Every departure from the text above, in the order the stages made them. Nothing here is a silent change: each one is also carried in the code it affects.

Stage 1 — D1 and D2

1. D2 discovery is NOT gated on PORTFOLIO_LEDGER_MODE, unlike the existing ledger-served flow detection. Flow derivation REPLACES an authoritative scan, so a ledger miss silently drops a real capital movement and must wait for the flag; discovery only ever WIDENS the read universe, and a group the wallet never held reads a zero balance at every grid point, which is no row, no flow and no chart. The per-row coverage certificate is the gate instead. Gating on the flag would have made the whole D2 fix a no-op wherever the flag currently sits.

2. Group lifespans are observed and recorded but do NOT bound the per-grid-point read window, which is one reading of "a closed group replays only over its own lifespan". Two measured shapes make bounding unsafe: for a group opened BEFORE the window and closed inside it the ledger's FIRST sighting is its EXIT, so a lifespan-bounded read would skip the opening balance and the exit would then book as a fabricated withdrawal; and a Fluid position NFT is not burned when its position closes, so its last factory-Transfer sighting is usually its mint. Natural reads already produce rows only across a group's real lifespan (outside it the balance reads zero, which is no row), so the served data matches the plan's intent. The lifespan map is the honest observation record and the input the tiering prioritises by.

3. MAX_GRID_POINTS raised 120 → 500. Not in the plan text but forced by it: a daily grid from 2026-01-01 is ~222 points and growing, and the old cap would have silently clamped the floor away, defeating D1 entirely. computeGapWindow shares the constant; with GAP_SEGMENT_DAYS=30 its grid is ~31 points, so raising the cap only makes its clamp-throw less likely. Post-review: the old cap was also the de-facto bound on the fresh replay's single-transaction lock hold, which is why tier 1 now carries a span bound of its own (note 19).

4. DEFAULT_LOOKBACK_DAYS=90 was NOT deleted. It survives as the operator's --days bound, which now WINS over the floor so a fast run stays fast, and as the "deeper of the two" rule that stops an account registered before 2026-01-01 losing history it already had. The plan said the floor "replaces" it; keeping it as a floor-versus-lookback max is strictly additive and is unit-tested both ways.

5. An unresolvable discovered group is DROPPED with a log line, not thrown on, unlike the gap patch's GapGroupUnresolvableError. Parking a registering wallet because a vault it touched months ago has since been delisted would be worse than the pre-D2 behaviour for that one group; dropping returns exactly that group to pre-D2 behaviour and leaves everything else fixed.

6. Fluid group membership tests the NFT id, not a 5-part group prefix. Required because a factory ERC-721 Transfer carries no vault address, so a discovered Fluid group has no prefix to match on. Safe because the single Fluid VaultFactory mints every vault's position NFTs, so ids are globally unique. A strict generalisation of the old behaviour.

Stage 2 — D3, D4 and D5

7. D3 needed a MIGRATION after all: 076-backfill-deep-extend.sql adds portfolio_backfill_state.deep_extend_ts. Nothing else can say "this wallet is only PARTLY built": floor_ts is legitimately above the deep target whenever the wallet's own first activity is later than the target, so a derived "floor_ts > target" test would re-queue such a wallet for ever. Reads and writes are 42703-tolerant, so with the column absent nothing is deferred. Never 075, per the rule. Post-review: "replays the whole window in one pass" was the wrong fallback and "deploy order does not matter" was false because of it — pre-tiering the single pass was NINETY DAYS, and a deep single pass does not fit the drain's 30-minute child SIGKILL, which is not retried. A wallet registering in a deploy-before-migrate window (the normal prod order, since prod migrations are a gated manual step) would have force-parked on 'error' with no history and no retry. The fallback now replays the ordinary bounded ninety days and simply defers nothing, which IS the pre-tiering behaviour and makes the order genuinely immaterial.

8. Tier 1 is the fresh path bounded through backfillCandidateStartSec, and an explicit --days now DISABLES the tiering entirely. Scheduling a deepening behind an operator's bound would silently undo the bound, which is the one thing --days exists to provide. Post-review: not scheduling one was not enough — a cursor from a PREVIOUS run survived the bounded run untouched and the next re-queue deepened back to the floor anyway. The fresh path now writes the cursor unconditionally from one pure decision (deepeningAfterFreshRun), so "nothing owed" means CLEARED.

9. The path choice is a new PURE function, backfillPath(forceFresh, anchor, deepTargetSec, nowSec) → 'fresh' | 'gap-patch' | 'extend', rather than inline branching. This is the decision where a wrong branch costs DATA (a fresh replay taken where a gap patch was owed deletes preserved history), so it is unit- and mutation-tested instead of being untestable I/O-bound control flow.

10. A wallet queued for tier 2 is excluded from the 6h SNAPSHOT tick (ELIGIBLE_WALLETS_SQL excludes queued/running). Mitigated rather than removed: the dispatch gives the TIP priority once the anchor ages past one 6h window, so the forward patch reclaims the wallet and writes the window the tick skipped. Its FLOWS are never at risk (the flow-scan population's 48h arm covers any wallet with a recent snapshot whatever its backfill status).

11. The extension certifies memberships from its CURRENT read alone, deliberately: a membership bounds the 6h tick's read universe, so folding in groups the deep replay found would tax every future tick with vaults the wallet left months ago. Tier 1 still folds its own replayed groups.

12. D5 was implemented as the GENERAL rule "a liquidation whose debt side was not valued books no magnitude", not only the flagged unregistered-reserve case. The same fall-through also books the full seizure when the seized collateral is EXCLUDED. Every liquidation repays something, so an absent debt context can never mean "the repayment was zero". Post-review: the generalised arm was SILENT — debtUnvaluable is unset when the reserve IS registered, so nothing selected the EXCLUDED-collateral case and its remedy (give that token a book) never surfaced. Each withheld row now carries WHICH cause it was (penaltyWithheld), both reach the [fail] line and the Telegram alert, and each gets the remedy that matches it. D5's "never silent" now holds for both arms.

13. D5's Telegram alert fires from the 6h refresher only (scripts/ops/alert.ts is scripts-only by design and src/ cannot import it). Every write path prints the [fail]-tagged line, checked in-test against run-cron.sh's own grep pattern and cut width. KNOWN LIMIT: a withholding that happens only inside a backfill child on a run that then succeeds does not page (run-cron needs a non-zero exit); it lands in the cron log.

14. The existing cross-asset withholding test was rewritten rather than added to. D4 makes serving the magnitude the correct behaviour, so the old assertion was the thing being changed. Post-review: D4 was dropped UNCONDITIONALLY, with an operator's memory of a manual SSH step as the only gate while the deploy itself is automatic. What a stale row actually holds is worse than "the old cross-currency figure": on the measured NFT 8643 shape it is max(33.06 − 48,941.07, 0) = 0, i.e. the wire stating that a seizure which cost 1.23 ETH cost nothing — the one statement api-types.ts says the null exists to prevent. The read path now checks the ROW (portfolio_flow_events.updated_at vs CROSS_BOOK_LIQUIDATION_FIX_TS, the instant the corrected writer went live) and withholds anything older, so a wallet the sweep misses — or one un-tracked before it and re-added after — degrades to the old honest behaviour instead. The gate is scoped to CROSS-ASSET rows, so a same-book figure (always correct, even pre-fix) is never over-withheld. Delete it a release after the sweep is provably complete.

Stage 3 — D6

15. The fixture gained TWO wallets, not one. The plan asked for a closed-in-window position wallet; the three e2e subjects the same section asks for do not fit on one. The closed position and the cross-currency seizure share a wallet (different views, no shared figure); the still-deepening state cannot join them, because a wallet mid-deepening has a SHALLOW floor by definition and the closed position sits below it. Both are on their own accounts, so no existing figure or count moves.

16. The e2e covers the READ path only, and says so. An offline fixture cannot run the reconstruction, so the specs assert what the served chart does with the rows the fixed reconstruction writes. That is the honest boundary of an offline harness, and it is still falsifiable: seeding only the cash side of the redemption (the pre-fix world) brings the phantom deposit straight back, and five cases fail on it. The write path is covered by the unit suite.

17. docs/processes.md was updated too, beyond the three pages the plan names: it is the page that documents the fixture and the e2e harness, which is what D6 changed.

18. No new test FILE was created in stages 2 or 3, so package.json's explicit list is unchanged since stage 1 registered src/lib/portfolio/historical-groups.test.ts.

Stage 4 — what the independent review changed

The review ran three lenses (financial, integration, quality) over the merged branch. Notes 3, 7, 8, 12 and 14 above carry its outcome where it amended a decision already recorded; the rest are new.

19. Tier 1 is bounded by SPAN, not only by start. [signup − 90d, now] is ninety days only for an account registering today: measured against the real helpers, an account created 2025-07-01 got 448 grid points and one created 2025-11-01 got 374, and neither earned a deepening (its deep target already equalled its own tier-1 floor), so the whole depth ran as ONE unsegmented pass. That pass holds the single global write lock from its first grid flush to COMMIT, and the drain SIGKILLs a child at 30 minutes without retrying. The PR's own required sweep runs --fresh over exactly that population. Tier 1 now starts at max(signup − 90d, now − 90d) (tierOneStartFloorSec), and the extension carries everything below in 30-day segments that commit and release the lock one at a time. Final depth unchanged; only which pass pays for it.

20. Fluid closed positions are discovered from the RESOLVER, not the ledger. D2's Fluid arm read factory ERC-721 Transfers only, and a Fluid position NFT is NOT burned when the position closes — the close emits a LogOperate, whose zero indexed params put it out of reach of every wallet-topic query. So a position opened BEFORE the window and closed INSIDE it was invisible to discovery AND to the current read, and the phantom deposit survived on the one venue creddit's carry users live on. readFluidOwnedNftIds (the positionsNftIdOfUser call the position read already makes, which lists closed NFTs) supplies those ids alongside the ledger's. It also removes a tier SEAM: both tiers ask the same resolver at their own head block, so the deep pass cannot admit a Fluid group the recent pass missed and leave its leg dying at the stored floor. The seam tripwire's "should never fire" is now true rather than aspirational, and its comment says why.

21. The Fluid EVENT read is skipped for a wallet with no Fluid position, and an under-seeded cache scans only the uncovered prefix. fluid_event_log was seeded for the trailing ninety days and its coverage start is monotone-down, so D1's fixed floor put every replay window below it — and planFluidEventRead answered that by chain-scanning the WHOLE window, chain-wide (LogOperate has zero indexed params), for every wallet, on every deepening segment. Three changes: the read is not issued at all when the replay universe holds no Fluid NFT (byte-identical, since the open-set filter would discard every row); below the coverage start only the uncovered prefix is scanned and the cached middle is still read from the table; and a re-seed to the history floor is now a REQUIRED server step, stated in the PR body and in docs/data-pipeline.md next to the seed.

22. The D2 discovery reads are SEGMENTED. They spanned the full deep window (~1.5M blocks at the floor, ~3.5M at the coverage floor) in one statement each, against a role carrying statement_timeout = 30s, and migration 064 indexes topic0/topic1/topic2 but NOT topic3 — so the Morpho read's topic2 OR topic3 arm cannot BitmapOr and degrades to a scan. The module deliberately propagates anything but "the ledger is not deployed", so a 57014 would have failed the whole registration and parked the wallet. They now run on the same block-segment constant the ledger flow reads use; the fold is incremental, so the answer is segment-invariant.

23. The extension's no-stored-floor early return writes a terminal status. It cleared the cursor and returned, leaving the drain's claim at 'running' until the 60-minute reaper — excluded from every 6h tick in the meantime. It now matches its two sibling early returns.

24. Two docs corrections. The published "tracked since" formula was max(history floor, coverage floor, first activity), which can never return anything below 2026-01-01 and was therefore wrong for every account on prod today; the rule takes the DEEPER of the history floor and signup − 90d. And docs/processes.md section D still documented the pre-PR replay (90 days, open-only, closed groups pruned), which is the page an operator reads before running the repair CLI.

25. Refuted, with the preventing code. "Tier 2 discovers groups tier 1 did not, for ERC-20-backed venues": unreachable — a group HELD at the tier-1 floor and closed later has its exit Transfer inside tier 1's own window, so tier 1 finds it too. The only systematic case was Fluid, closed by note 20. And the e2e phantom-deposit spec's second assertion was genuinely dead code; the pair is reordered so the magnitude check (the one that fires under the defect) runs first.

Still open after the merge

Both acceptance numbers are stated against LIVE wallets and neither can be confirmed before the server step runs. They are the first thing to check on staging:

• D2: 0x12d2c3e2…'s 2026-07-04 vault redemption of 28,832.81 USDC shows as a position exit with NO +$28.8k deposit marker. Encoded as a unit-level acceptance case in historical-groups.test.ts and, in fixture form, as the sixth wallet's whole story.
• D4: 0x1ec6942b…'s 9 markers serve real magnitudes summing ≈1.335856 ETH on the wire.

Neither holds until the one-time re-derivation in the PR's Server steps has run. D4's magnitudes additionally will not appear until then even if the sweep is skipped, by construction (note 14): the read path withholds any row it can see was written before the corrected writer went live, so a skipped sweep now shows the old absent magnitude rather than a fabricated zero.

Also worth a check on staging, and NOT covered by any offline test:

• the Fluid event cache reaches the history floor before the sweep runs (note 21). SELECT start_block, start_ts FROM onchain_credit.fluid_event_coverage; — a start_ts above 2026-01-01 means every Fluid wallet's replay chain-scans the difference. Re-seed with --from-block <block at 2026-01-01>.
• price bars reach the floor for every tracked token. A deep grid point's MARKET mark comes from the price mirror, and a token added to the mirror recently has bars only from its add date; below that the mark is absent and the market line simply has no point there (the redemption line is unaffected). The sweep is destructive and one-time, so it is worth knowing before rather than after: compare min(bar_ts) per tracked token against the floor.