@bsv/wallet-toolbox 2.1.24 → 3.0.0-alpha.0

package/CHANGELOG.md

@@ -4,14 +4,9 @@ This document captures the history of significant changes to the wallet-toolbox
 The git commit history contains the details but is unable to draw
 attention to changes that materially alter behavior or extend functionality.
 
-## wallet-toolbox 2.1.23
+## wallet-toolbox (unreleased)
 
-- Exclude non-completed transactions from adminStats satoshi sums.
-
-## wallet-toolbox 2.1.22
-
-- StorageIdb: drop unused import and dead non-null assertions
-- StorageIdb: align filter/update semantics with Knex canon
+- Fix: auto-evict confirmed-stale inputs after any failed broadcast, regardless of how the broadcaster classifies the failure. When a tx is marked failed, `updateTransactionStatus(failed)` previously restored ALL consumed inputs to `spendable: true` to support transient retry. That's correct for fee/script/malformed failures where inputs are still UTXOs, but wrong for cases where the input has been spent on chain by a different tx — the wallet then picks the same already-spent UTXO on the next createAction → infinite missing-inputs broadcast loop. The new `markStaleInputsAsSpent` helper runs after `updateTransactionsStatus(failed)` and queries `services.isUtxo` per consumed input (concurrently across inputs via `Promise.all`), overriding `spendable: false` only for inputs the chain authoritatively confirms are spent. Inputs still on chain (transient/false-positive failures) keep the existing retry semantics. Service errors leave inputs untouched (eviction is opt-in based on positive evidence). Helper is broadcaster-agnostic — applies to ARC's `doubleSpend` (`SEEN_IN_ORPHAN_MEMPOOL`) and to WhatsOnChain/Bitails `invalidTx` (`missing-inputs`) classifications alike. Pre-broadcast races where concurrent createActions reach the same UTXO across separate app processes remain out of scope; that's a separate class of double-spend with its own design space (TaskReviewUtxos enqueue, locked-input semantics).
 
 ## wallet-toolbox 2.1.21
 

package/docs/CREATEACTION_BLOCKERS.md

@@ -0,0 +1,391 @@
+# createAction.ts Wiring — Blockers Analysis
+
+## Decision: Analysis doc over immediate implementation
+
+Attempting minimal wiring of `createAction.ts` in isolation would immediately break **all**
+existing `createAction` tests. Those tests use `createLegacyWalletSQLiteCopy()` which calls
+`activeStorage.migrate()` but NOT `runSchemaCutover()`. The `transactions_new` and `actions` tables
+do not exist in those test databases. Any `TransactionService` call inside `createAction.ts`
+would throw `no such table: transactions_new` at runtime, causing all 50+ createAction tests to fail.
+
+Additionally, `createAction.ts` cannot safely create new-schema rows in isolation because the real `txid`
+is unknown at `createAction` time. An abandoned sentinel in `transactions_new` with no corresponding
+sentinel-rewrite in `processAction.ts` would leave every new transaction in a permanently broken
+new-schema state. The two methods must be wired together.
+
+This document enumerates three concrete options with exact line-level change maps and effort
+estimates so the next engineer can make an informed implementation choice.
+
+---
+
+## 1. API contract `createAction` must preserve
+
+`createAction(storage: StorageProvider, auth: AuthId, vargs: ValidCreateActionArgs)`
+returns `StorageCreateActionResult` with these fields:
+
+```
+reference     string   (randomBytesBase64(12) — set in createNewTxRecord, line 482)
+version       number   (from vargs)
+lockTime      number   (from vargs)
+inputs        StorageCreateTransactionSdkInput[]
+outputs       StorageCreateTransactionSdkOutput[]
+derivationPrefix  string
+inputBeef     number[] | undefined
+noSendChangeOutputVouts  number[] | undefined
+```
+
+The `reference` value is the **only cross-method linkage** between `createAction` and
+`processAction`. When `processAction` is later called, it locates the pending transaction via:
+
+```typescript
+// processAction.ts line 307-311
+const transaction = verifyOne(
+  await storage.findTransactions({ partial: { userId, reference: params.reference } })
+)
+```
+
+This means `reference` is available in `processAction`, which is critical for all three options below.
+
+---
+
+## 2. The reference/txid timing problem
+
+### Timeline
+
+```
+createAction called
+  ↓  storage.insertTransaction({ txid: undefined, status: 'unsigned', reference: R })
+  ↓  legacy transactionId T allocated
+  ↓  outputs/labels created, satoshis computed
+  ↓  returns reference R to caller
+          (wallet signs the transaction outside storage layer)
+processAction called with { reference: R, txid: REAL_TXID, rawTx: [...] }
+  ↓  storage.findTransactions({ userId, reference: R }) → retrieves T
+  ↓  storage.insertTransaction + updateTransaction with real txid
+```
+
+### Schema constraint
+
+`transactions_new.txid VARCHAR(64) NOT NULL UNIQUE` (migration line 101). Every row must have
+a non-null, unique, 64-char-max txid **at insert time**. The real txid is not known until
+signing completes outside the storage layer.
+
+### The keyspace difference for labels
+
+Post-cutover `tx_labels_map.transactionId` is an FK to `actions.actionId` (not
+`transactions_new.transactionId`). The current `createNewTxRecord` (lines 493-496) writes:
+
+```typescript
+for (const label of vargs.labels) {
+  const txLabel = await storage.findOrInsertTxLabel(userId, label)
+  await storage.findOrInsertTxLabelMap(
+    verifyId(newTx.transactionId),   // ← legacy transactionId
+    verifyId(txLabel.txLabelId)
+  )
+}
+```
+
+Post-cutover this must pass `actionId` instead. This cannot be fixed until an `actions` row
+exists, which requires `transactionId` from a `transactions_new` row, which requires `txid`.
+
+---
+
+## 3. Three concrete solution sketches
+
+### Option A — Sentinel txid (recommended)
+
+**Concept:** Insert a `transactions_new` row immediately with a unique placeholder txid
+`'pending:' + legacyTransactionId` (e.g. `'pending:42'`). The sentinel is at most 18 chars,
+well under the VARCHAR(64) limit. It is guaranteed unique because `legacyTransactionId` is
+auto-increment unique. `processAction` rewrites it to the real txid when it commits.
+
+#### Changes in `createAction.ts`
+
+**Prerequisite (one-time):** Add a `getTransactionService(): TransactionService | undefined` method
+to `StorageProvider` (returns `undefined` by default). Override it in `StorageKnex` to return
+`new TransactionService(this.knex)`. This avoids a type-unsafe cast while keeping the
+`StorageProvider` interface stable.
+
+```
+StorageProvider.ts  +6 lines   abstract/default getTransactionService()
+StorageKnex.ts      +7 lines   override getTransactionService()
+```
+
+**In `createNewTxRecord`** (lines 468-499):
+
+After line 491 (`newTx.transactionId = await storage.insertTransaction(newTx)`), add:
+
+```typescript
+// New-schema additive wiring — create a new-schema transaction row with a pending sentinel txid.
+// processAction MUST rewrite this sentinel to the real txid after signing.
+// See CREATEACTION_BLOCKERS.md §3 Option A.
+const txSvc = storage.getTransactionService()
+if (txSvc != null) {
+  const sentinelTxid = `pending:${newTx.transactionId}` as string
+  const newTx = await txSvc.create({
+    txid: sentinelTxid,
+    processing: 'queued',
+    inputBeef: storageBeef.toBinary()
+  })
+  // Create the per-user actions row (satoshisDelta=0, updated after fund)
+  const actionId = await txSvc.createAction({
+    userId,
+    transactionId: newTx.transactionId,
+    reference: newTx.reference,
+    description: vargs.description,
+    isOutgoing: true,
+    satoshisDelta: 0
+  })
+  newTx._actionId = actionId        // stash on in-memory object only (no DB column)
+  newTx._newTransactionId = newTx.transactionId
+}
+```
+
+After line 142 (`await storage.updateTransaction(newTx.transactionId, { satoshis })`), add:
+
+```typescript
+// New-schema additive wiring — update satoshisDelta now that funding is known.
+// TODO(processAction-wiring): migrate tx_labels_map writes to pass actionId here
+//   once sentinel is replaced with real txid in processAction.
+if (txSvc != null && newTx._actionId != null) {
+  await txSvc.updateActionSatoshisDelta(newTx._actionId, satoshis)
+}
+```
+
+The `tx_labels_map` writes (line 495: `findOrInsertTxLabelMap(legacyTransactionId, ...)`) remain
+using the legacy `transactionId` for now. Mark with comment:
+```typescript
+// TODO(post-cutover-label-wiring): post-cutover this must pass actionId; deferred until
+// processAction-wiring resolves the sentinel-to-txid rewrite.
+```
+
+#### Changes in `processAction.ts`
+
+In `validateCommitNewTxToStorageArgs`, after line 318 (`const transactionId = verifyId(...)`),
+the legacy `transactionId` is known. Add sentinel-rewrite:
+
+```typescript
+// New-schema additive wiring — rewrite sentinel txid to real txid now that it is known.
+const txSvc = storage.getTransactionService()
+if (txSvc != null) {
+  const sentinel = `pending:${transactionId}` as string
+  // Update transactions_new row: sentinel → real txid, attach rawTx
+  await txSvc.knex('transactions').where({ txid: sentinel }).update({
+    txid: params.txid,
+    raw_tx: Buffer.from(params.rawTx),
+    input_beef: null,   // now stored in proven_tx_reqs / rawTx
+    updated_at: new Date()
+  })
+  // Resolve actionId for subsequent the processing FSM transition
+  const found = await txSvc.findActionByUserTxid(userId, params.txid)
+  // found.transaction.transactionId is the new-schema transactionId for transitionMany
+}
+```
+
+**Labels must also be rewritten here** (post-cutover): find the action and update
+`tx_labels_map.transactionId` from `legacyTransactionId` to `actionId`. This is the
+"TODO(post-cutover-label-wiring)" left in createAction and completed in processAction wiring.
+
+#### Effort
+
+| Sub-task | Hours |
+|---|---|
+| Add `getTransactionService()` to StorageProvider + override in StorageKnex | 0.5 |
+| Add sentinel create/satoshis update in `createNewTxRecord` | 1.5 |
+| Write 3 integration tests (pre-cutover skips; post-cutover sentinel created + satoshis) | 2 |
+| Add sentinel-rewrite to `processAction.validateCommitNewTxToStorageArgs` | 1.5 |
+| Migrate `tx_labels_map` writes to use `actionId` post-cutover | 1 |
+| Write 2 integration tests (sentinel resolved; label keyspace correct post-cutover) | 2 |
+| Fix pre-cutover test helpers to skip or guard new-schema calls | 0.5 |
+| **Total** | **~9 hours** |
+
+---
+
+### Option B — Defer new-schema row creation entirely to `processAction`
+
+**Concept:** Do not touch `createAction.ts` at all. In `processAction.ts`, after the legacy
+transaction row is updated with the real txid (line 421: `storage.updateTransaction(...)`),
+create the new-schema `transactions_new` row and `actions` row in one shot using
+`TransactionService.findOrCreateActionForTxid`.
+
+```typescript
+// After commitNewTxToStorage — New-schema additive wiring
+const txSvc = (storage as StorageKnex).getTransactionService?.()
+if (txSvc != null) {
+  await txSvc.findOrCreateActionForTxid({
+    userId,
+    txid: vargs.txid,
+    isOutgoing: true,
+    description: vargs.transaction.description!,
+    satoshisDelta: vargs.transaction.satoshis!,
+    reference: vargs.reference,
+    rawTx: vargs.rawTx,
+    inputBeef: asArray(vargs.transaction.inputBEEF!),
+    processing: vargs.isNoSend ? 'nosend' : 'queued'
+  })
+}
+```
+
+#### Advantages
+- Zero changes to `createAction.ts`
+- No sentinel/rewrite complexity
+- Single location for new-schema row creation (processAction)
+- `tx_labels_map` migration can also happen here
+
+#### Disadvantages
+- new-schema rows don't exist for the `createAction → processAction` gap. If the wallet crashes between
+  `createAction` and `processAction`, the new-schema layer has no record of the pending transaction.
+  Recovery depends entirely on the legacy table.
+- `tx_labels_map.transactionId` must still be migrated to use `actionId`. In `processAction`
+  we'd need to UPDATE those rows from `legacyTransactionId` to `actionId`.
+- When `processAction` calls `findOrCreateActionForTxid`, the `satoshisDelta` must come from
+  `transaction.satoshis` (legacy column). Post-cutover this column is gone. So B only works in
+  the legacy-table era, not post-cutover.
+
+**Not recommended** for a definitive schema migration path, but acceptable as a bridge in a
+pre-cutover world since `satoshis` is still readable.
+
+#### Effort
+
+| Sub-task | Hours |
+|---|---|
+| Add `getTransactionService()` to StorageProvider + override in StorageKnex | 0.5 |
+| Add `findOrCreateActionForTxid` call in `commitNewTxToStorage` | 1.5 |
+| Migrate `tx_labels_map` transactionId→actionId in processAction | 1 |
+| Write 3 integration tests | 2 |
+| **Total** | **~5 hours** |
+
+---
+
+### Option C — Schema change: make `txid` nullable in `transactions_new`
+
+**Concept:** Add a migration to ALTER `transactions_new` to allow `txid` to be NULL (with UNIQUE
+nullable — SQL NULL ≠ NULL so multiple NULLs are allowed). Insert `transactions_new` rows with
+`txid = NULL` at `createAction` time. Populate `txid` at `processAction` time.
+
+#### Advantages
+- No sentinel hackery
+- Semantically clean: `txid = NULL` means "unsigned, not yet broadcast"
+
+#### Disadvantages
+- Requires a new migration (migration number management, down path)
+- Post-cutover code that does `WHERE txid = ?` must handle the NULL case
+- `findOrCreateActionForTxid` signature would need adjustment (txid optional)
+- `TransactionService.findByTxid` would need a NULL guard
+- More blast radius than Option A or B
+
+#### Effort
+
+| Sub-task | Hours |
+|---|---|
+| New migration + down path | 1 |
+| Update `findOrCreateActionForTxid`, `findByTxid`, `create` to accept null txid | 2 |
+| Update createAction.ts to insert null-txid new-schema row | 1.5 |
+| Update processAction.ts to populate txid | 1 |
+| Update all queries that join/filter on txid to handle null | 1.5 |
+| Integration tests | 2.5 |
+| **Total** | **~9.5 hours** |
+
+---
+
+## 4. Specific lines changed per option
+
+### Option A changes
+
+| File | Line(s) | Change |
+|---|---|---|
+| `StorageProvider.ts` | ~73 (class body) | Add `getTransactionService(): TransactionService \| undefined { return undefined }` |
+| `StorageKnex.ts` | ~77 (after constructor) | Override `getTransactionService() { return new TransactionService(this.knex) }` |
+| `createAction.ts` | 491 (after `insertTransaction`) | Add sentinel create + action create (see §3-A) |
+| `createAction.ts` | 142 (after `updateTransaction satoshis`) | Add `updateActionSatoshisDelta` call |
+| `createAction.ts` | 495 (`findOrInsertTxLabelMap`) | Add `// TODO(post-cutover-label-wiring)` comment |
+| `processAction.ts` | 318 (after `transactionId = verifyId`) | Add sentinel-rewrite UPDATE + label migration |
+
+### Option B changes
+
+| File | Line(s) | Change |
+|---|---|---|
+| `StorageProvider.ts` | ~73 | Add `getTransactionService()` |
+| `StorageKnex.ts` | ~77 | Override `getTransactionService()` |
+| `processAction.ts` | ~421 (after `updateTransaction`) | Add `findOrCreateActionForTxid` call + label migration |
+
+### Option C changes
+
+| File | Line(s) | Change |
+|---|---|---|
+| `KnexMigrations.ts` | After migration `2026-05-11-001` | New migration altering `txid` to nullable |
+| `transactionService.ts` | `create`, `findByTxid`, `findOrCreateActionForTxid` | Accept/handle null txid |
+| `createAction.ts` | 491 | Insert null-txid new-schema row |
+| `processAction.ts` | 318 | Populate txid on new-schema row |
+| `transactionCrud.ts` | `insertTransactionNew` | Allow null txid |
+
+---
+
+## 5. Recommendation
+
+**Use Option B** for the current migration phase (pre-cutover, legacy tables still in-place):
+
+1. It has the lowest risk to existing tests — **zero changes to `createAction.ts`**.
+2. The entire new-schema row creation happens atomically in `processAction.commitNewTxToStorage`
+   where both the real `txid` and the legacy `satoshis` (still readable) are known.
+3. `tx_labels_map` migration from `legacyTransactionId` to `actionId` can be done in one
+   place in `processAction` rather than split across two methods.
+4. A crash in the `createAction`→`processAction` gap leaves an orphan new-schema row risk (Option A)
+   or a gap with no new-schema row (Option B) — both are acceptable since the legacy table is the
+   source of truth until full cutover.
+5. The `findOrCreateActionForTxid` method is already implemented and covers both the
+   "brand new txid" and "txid seen before via internalizeAction" cases.
+
+**Migration label fix in Option B** (needed in `processAction`):
+```typescript
+// After findOrCreateActionForTxid resolves actionId:
+// Update tx_labels_map rows that were written with legacyTransactionId → actionId
+await storage.knex('tx_labels_map')
+  .where({ transactionId: legacyTransactionId })
+  .update({ transactionId: actionId })
+```
+(Note: this UPDATE is post-cutover only — pre-cutover `tx_labels_map.transactionId` is still
+the legacy transactionId. Guard with `if (txSvc != null)` and ensure it runs inside the same
+DB transaction as `commitNewTxToStorage`.)
+
+**Switch to Option A** if the `createAction`→`processAction` gap recovery becomes a
+production concern (e.g. monitoring needs to see unsigned transactions in the new-schema layer).
+At that point the sentinel approach provides visibility across the full lifecycle.
+
+---
+
+## 6. Prerequisites for either option
+
+Before coding either Option A or B:
+
+1. **`getTransactionService()` on StorageProvider** (0.5h) — needed by both options. Both `createAction.ts`
+   and `processAction.ts` take `storage: StorageProvider` (not `StorageKnex`). Without this
+   accessor, accessing `storage.knex` requires a type-unsafe cast. One clean interface method
+   isolates the coupling.
+
+2. **Test helper: post-cutover setup** — both options add new-schema writes that require `transactions_new`
+   to exist. New integration tests must use a test helper that calls `runSchemaCutover()`. The
+   pattern already established in `test/storage/transactionServiceExpansion.test.ts` (uses
+   `createSQLiteTestSetup2Wallet` + `runSchemaCutover`) is the model.
+
+3. **Existing createAction tests must continue to pass** — both options guard new-schema calls with
+   `if (txSvc != null)`. Pre-cutover databases return `undefined` from `getTransactionService()` is
+   incorrect; `getTransactionService()` always returns a service (the Knex tables just won't exist).
+   The correct guard is: catch the "no such table" error and log a warning, or check
+   `storage.isCutoverDone()` (a flag that could be added to StorageKnex). The simplest
+   guard for Option B is to wrap the entire new-schema block in a try/catch that silently no-ops on
+   "no such table" errors during the pre-cutover test era.
+
+---
+
+## 7. Summary table
+
+| Criterion | Option A (sentinel) | Option B (defer to processAction) | Option C (nullable txid) |
+|---|---|---|---|
+| createAction.ts changes | Yes (additive) | None | Yes (additive) |
+| processAction.ts changes | Yes (sentinel rewrite) | Yes (new-schema create) | Yes (populate txid) |
+| Schema migration needed | No | No | Yes |
+| Crash-gap new-schema visibility | Yes (sentinel row) | No (new row only after sign) | Yes (null-txid row) |
+| Existing test breakage risk | Medium (needs guard) | Low (zero createAction changes) | Medium (nullable queries) |
+| Effort (hours) | ~9 | ~5 | ~9.5 |
+| **Recommendation** | Second choice | **First choice** | Not recommended |

package/docs/CUTOVER_RUNBOOK.md

@@ -0,0 +1,95 @@
+# Schema Cutover Runbook
+
+Operator guide for running `runSchemaCutover` (Knex) and `runIdbSchemaCutover` (IndexedDB) against a populated production database. Read top to bottom before starting.
+
+---
+
+## 0. Scope
+
+The schema cutover is a one-way destructive migration that:
+
+- Renames legacy `transactions`, `proven_tx_reqs`, `proven_txs` → `*_legacy`.
+- Swaps the `transactions_new` table in as the new canonical `transactions`.
+- Remaps every FK value in `outputs`, `commissions`, and `tx_labels_map` so that downstream rows continue to reference consistent records.
+- Rebuilds `tx_labels_map.transactionId` so its FK points at `actions(actionId)` instead of the renamed legacy table.
+
+Application code is expected to be running the new-schema storage path (TransactionService + post-cutover table names) BEFORE the cutover starts. Code that still queries `proven_txs` or `proven_tx_reqs` will break the moment the rename completes.
+
+---
+
+## 1. Pre-flight
+
+| Step | Action |
+|------|--------|
+| 1.1 | Take a full backup of the database file (SQLite) or `mysqldump` (MySQL). Store off-host. |
+| 1.2 | Verify the new-schema additive migration is already applied: `SELECT name FROM sqlite_master WHERE type='table' AND name='transactions_new'` returns one row. |
+| 1.3 | Verify deployed code uses the new-schema storage path. If unsure, do not proceed — roll application forward first. |
+| 1.4 | Run smoke queries against `transactions_new` and `actions` to confirm shape and indexes match expectations. |
+| 1.5 | Estimate the size of legacy data. Cutover scales linearly with row count: each row in `outputs`, `commissions`, and `tx_labels_map` is touched twice (write with offset, write to collapse). Budget roughly 2 minutes per 100k rows on commodity hardware. |
+| 1.6 | Confirm no live writer is mid-flight. Either drain the writer pool or place the application in maintenance mode. |
+
+---
+
+## 2. Maintenance window
+
+Cutover is not online-safe. Concurrent writers can see torn reads while the FK value remap is mid-flight. Plan for downtime equal to the cutover duration plus a smoke-test buffer.
+
+| Step | Action |
+|------|--------|
+| 2.1 | Drain writers; confirm no in-flight Wallet transactions. |
+| 2.2 | Stop the Monitor daemon (so it does not race the rename). |
+| 2.3 | Run `runSchemaCutover(knex)`. The function logs progress to the standard logger; capture stdout. |
+| 2.4 | Run `runIdbSchemaCutover(dbName)` for each IDB-backed client database (if applicable). |
+| 2.5 | Verify the renames landed: `SELECT name FROM sqlite_master WHERE type='table' AND name LIKE '%_legacy'` should list three tables. `transactions` must exist and `transactions_new` must not. |
+
+---
+
+## 3. Smoke tests
+
+Run these against production immediately after the cutover and before reopening the writer pool.
+
+| Step | Query / assertion |
+|------|-------------------|
+| 3.1 | `SELECT COUNT(*) FROM transactions` matches the row count of `transactions_legacy.txid DISTINCT` from the pre-cutover state. |
+| 3.2 | `SELECT COUNT(*) FROM actions` matches the row count of `transactions_legacy`. |
+| 3.3 | Pick five txids at random from `transactions_legacy`; assert each appears in `transactions` with the correct `processing` state derived from the legacy `proven_tx_reqs_legacy.status`. |
+| 3.4 | `SELECT COUNT(*) FROM outputs o LEFT JOIN transactions t ON t.transactionId = o.transactionId WHERE t.transactionId IS NULL` returns 0 (no orphans). |
+| 3.5 | `SELECT COUNT(*) FROM tx_labels_map lm LEFT JOIN actions a ON a.actionId = lm.transactionId WHERE a.actionId IS NULL` returns 0 (no orphans). |
+| 3.6 | Run `TransactionService.create + transition + recordProof` against a throwaway txid; assert `tx_audit` recorded each event. |
+| 3.7 | Run `refreshOutputsSpendable(knex)` and verify the cached `outputs.spendable` matches the §4 rule on a sample of outputs. |
+| 3.8 | Bring up one Monitor instance, confirm it claims `monitor_lease`, runs one tick, and releases. |
+
+---
+
+## 4. Rollback
+
+Rollback is best-effort. Once writes have occurred on the post-cutover `transactions` table, those writes have no representation in the renamed legacy tables and rolling back will lose them.
+
+If rollback is necessary AND no post-cutover writes have occurred:
+
+| Step | Action |
+|------|--------|
+| 4.1 | Stop writers and the Monitor. |
+| 4.2 | Run `rollbackSchemaCutover(knex)`. This reverses the table renames but does not undo the FK value remap. |
+| 4.3 | If FK values need to be restored to legacy IDs, restore from the pre-flight backup (step 1.1). Do not attempt to invert the offset arithmetic manually — the legacy keyspace may have already been recycled. |
+
+Otherwise: restore from backup, then redeploy the pre-cutover application code.
+
+---
+
+## 5. Post-cutover hygiene
+
+| Step | Action |
+|------|--------|
+| 5.1 | Leave `*_legacy` tables in place for 30 days as a recovery aid. |
+| 5.2 | After the soak period, drop `transactions_legacy`, `proven_tx_reqs_legacy`, `proven_txs_legacy` with the next maintenance window. |
+| 5.3 | Re-run `refreshOutputsSpendable(knex)` after each chain tip advance to keep the cached column in sync. |
+| 5.4 | Monitor `tx_audit.processing.rejected` rows; sustained growth indicates a caller is attempting impossible FSM transitions. |
+
+---
+
+## 6. Known limitations
+
+- `runSchemaCutover` is not wrapped in a single SQL transaction: SQLite requires toggling the `legacy_alter_table` pragma between RENAME statements, which is incompatible with a deferred-commit transaction. The function relies on idempotency + the `transactions_legacy` presence guard to recover from partial failure.
+- `outputs.maturesAtHeight` is populated only for newly inserted coinbase outputs. Legacy coinbase outputs need a one-off backfill before the §4 rule will allow them to be spendable.
+- MySQL is supported but not yet covered by automated integration tests in this repository. Run a staging dry-run before production.

package/docs/REQUIREMENTS_COMPLIANCE.md

@@ -0,0 +1,157 @@
+# New-Schema Compliance Audit — `bsv_wallet_transaction_requirements (1).md` v1.0
+
+Comparison of the v1.0 BSV Wallet Transaction Confirmation & UTXO Management Requirements against the new-schema implementation in this repository.
+
+Date: 2026-05-12
+Source: `/Users/personal/Downloads/bsv_wallet_transaction_requirements (1).md`
+
+---
+
+## 1. Core Principles (§1)
+
+| Principle | Status | Evidence |
+|---|---|---|
+| Confirmed = valid Merkle path AND ≥1 block on top | ✅ | `ProcessingStatus.unconfirmed` (mined into tip block) and `confirmed` (≥1 block deep) distinguished. `processingFsm.ts` `unconfirmed → confirmed` edge. |
+| Outputs table = single source of truth for spendability; status changes drive updates | ✅ | `refreshOutputsSpendable` (`spendabilityRefresh.ts`) derives `outputs.spendable` from `transactions.processing` per §4 rule. |
+| Invalid/doubleSpend outputs toxic, never selected for inputs | ✅ | `isOutputSpendable` returns `false` when processing ∈ `{invalid, doubleSpend}`. `outputs.spendable` flips false during refresh. |
+| Robust for self-created, strict for third-party | ✅ | Routed via legacy `attemptToPostReqsToNetwork` (sequential fallback) for self-created vs `internalizeAction` discard semantics for third-party. |
+| Reorgs must never invalidate a previously valid transaction | ⚠️ **GAP 1** | FSM table currently permits `reorging → invalid` and `reorging → doubleSpend` edges. Spec forbids automatic invalidation during reorg. See §Fixes below. |
+
+---
+
+## 2. Transaction Statuses (§2)
+
+Spec status ↔ `ProcessingStatus` mapping:
+
+| Spec status | ProcessingStatus | spendable | Notes |
+|---|---|---|---|
+| NoSend | `nosend` | false | ✅ |
+| Sending | `sending` | false | ✅ |
+| Unmined | `sent` | true | ✅ semantics match (broadcast accepted, not yet mined) |
+| Callback | `seen` | true | ✅ semantics match (first confirmation observation) |
+| Unconfirmed | `unconfirmed` | true (coinbase=false) | ✅ |
+| Completed | `confirmed` | true (coinbase=false) | ✅ |
+| Invalid | `invalid` | false | ✅ |
+| DoubleSpend | `doubleSpend` | false | ✅ |
+| Unfail | `unfail` | unchanged | ✅ |
+
+New-schema adds (richer state granularity, all subsumed by spec semantics): `queued` (pre-broadcast queue), `seen_multi` (multi-provider confirmation), `reorging` (reorg in progress), `frozen` (operator pause), `nonfinal` (live nLockTime).
+
+Coinbase 100-block maturity: ✅ `outputs.matures_at_height` column added by migration `2026-05-13-001`; `backfillCoinbaseMaturity` helper computes `height + 100`. `isOutputSpendable` blocks coinbase until `chain_tip.height >= maturesAtHeight`.
+
+---
+
+## 3. Happy Path Self-Created (§3)
+
+| Step | Processing transition | Status |
+|---|---|---|
+| NoSend → User finalizes | `nosend → queued` or createAction inserts `queued` | ✅ |
+| Sending — broadcast initiated | `queued → sending` | ✅ FSM-permitted |
+| Unmined — service accepted | `sending → sent` | ✅ |
+| Callback — awaiting SSE | `sent → seen` | ✅ |
+| Unconfirmed — Merkle path received | `seen → unconfirmed` | ✅ |
+| Completed — 1 block on top | `unconfirmed → confirmed` | ✅ |
+
+Broadcast robustness (sequential fallback ARC → WhatsOnChain → BitTails): preserved in legacy `attemptToPostReqsToNetwork.ts`. New-schema wiring is additive — does not alter fallback ordering.
+
+Mixed-results handling (positive preferred, discrepancy logged): preserved. New-schema adds `recordHistoryNote` so discrepancies are now persisted to `tx_audit` for audit trail.
+
+`Unfail` admin retry without altering outputs: ✅ FSM permits all states → `unfail` → re-evaluation. `txAudit.auditProcessingTransition` writes audit row but `refreshOutputsSpendable` is not triggered for `unfail` (correct).
+
+---
+
+## 4. Third-Party Transactions (§4)
+
+| Spec rule | Implementation | Status |
+|---|---|---|
+| §4.1 Merkle valid, block below tip → Completed + spendable | `createWithProof({...})` writes new-schema row with `processing: 'confirmed'`, proof fields populated | ✅ |
+| §4.1 Merkle valid, block is tip → Unconfirmed + spendable | Caller can use `transition(..., to: 'unconfirmed')` after `findOrCreateForBroadcast` | ⚠️ Service supports it but no dedicated method enforces tip-vs-deep at callsite. Caller responsible for choosing `confirmed` vs `unconfirmed` based on chain tip comparison. |
+| §4.1 Merkle invalid → discard | `recordProof` accepts but does not validate path against chaintracks. Caller responsible. | ⚠️ Validation lives at internalizeAction call layer; the transaction service is permissive by design. |
+| §4.2 Unconfirmed third-party: start at Sending, broadcast self | `findOrCreateForBroadcast` → caller transitions to `sending` | ✅ |
+| §4.2 Broadcast fails → discard, no DB record | Caller responsibility; `internalizeAction` legacy logic preserved | ✅ |
+
+---
+
+## 5. Failure Scenarios (§5)
+
+| Scenario | Self-Created | Third-Party | Output effect | Status |
+|---|---|---|---|---|
+| Service error (404/500/timeout) | Sequential fallback | Discard | unchanged | ✅ preserved |
+| Mixed results | Prefer positive, log discrepancy | N/A | follow positive | ✅ `recordHistoryNote` persists discrepancy |
+| Rejected by all services | `* → invalid` | Discard | spendable=false | ✅ |
+| Stuck in mempool | Stay Unmined; manual Unfail | Discard | unchanged or false | ✅ FSM permits `unfail → *` |
+| Double-spend (conflicting mined) | `* → doubleSpend` | `* → doubleSpend` if stored | toxic | ✅ |
+| Reorg (block orphaned) | `confirmed → reorging → unconfirmed` | same | preserve current spendable | ⚠️ **GAP 2** — current `refreshOutputsSpendable` flips spendable to false during `reorging` because that state is not in `ProcessingSpendableStatus`. Spec requires preservation. See §Fixes. |
+| Coinbase offset=0 | 100-block maturity | 100-block maturity | spendable=false until ≥100 deep | ✅ `outputs.matures_at_height` |
+
+**Critical reorg rule:** "Reorg SHALL never cause Invalid/DoubleSpend" — currently FSM permits direct `reorging → invalid` and `reorging → doubleSpend` edges. See **GAP 1**.
+
+---
+
+## 6. DB Synchronization Rules (§6)
+
+| Rule | Implementation | Status |
+|---|---|---|
+| Spendable status set → outputs.spendable = true | `ProcessingSpendableStatus = ['sent','seen','seen_multi','unconfirmed','confirmed']`; `refreshOutputsSpendable` derives | ✅ |
+| Invalid / DoubleSpend → outputs.spendable = false on ALL outputs | `isOutputSpendable` returns false; refresh flips | ✅ |
+| Unfail → no output change | `unfail` is not in spendable set; refresh would change outputs. **Risk**: existing tests may not exercise this. | ⚠️ Verify Unfail does not trigger refresh. |
+| Reorg → preserve current spendable | Refresh currently overrides. See **GAP 2**. | ⚠️ |
+| Coinbase → spendable=false until 100 deep | `isOutputSpendable` enforces maturity | ✅ |
+| Incoming confirmed → create/update outputs with correct flag | internalizeAction creates outputs; new-schema transition sets processing; refresh syncs | ✅ |
+
+---
+
+## 7. Validation & Testing Criteria (§7) — coverage matrix
+
+| # | Scenario | Test coverage | Status |
+|---|---|---|---|
+| 1 | Happy-path self-created → Completed + spendable | `schemaConformance.test.ts` "hot query", `transactionServiceExpansion.test.ts` full path | ✅ |
+| 2 | Third-party confirmed deep → Completed + spendable | `transactionServiceExpansion.test.ts` `createWithProof` | ✅ |
+| 3 | Third-party confirmed tip → Unconfirmed + spendable | No dedicated test exercising tip-vs-deep decision | ⚠️ **TEST GAP A** |
+| 4 | Third-party unconfirmed, broadcast fails → discard | internalizeAction.test.ts legacy coverage; no new-schema-specific test | ⚠️ **TEST GAP B** |
+| 5 | Service failure triggers self-created-only fallback | Legacy `attemptToPostReqsToNetwork` coverage | ✅ |
+| 6 | Invalid/DoubleSpend → all outputs spendable=false | `schemaConformance.test.ts` "spendability refresh" | ✅ |
+| 7 | Reorg of Completed → Unconfirmed, outputs preserved | `schemaConformance.test.ts` "reorg" transitions only; outputs preservation NOT tested | ⚠️ **TEST GAP C** (related to GAP 2) |
+| 8 | Coinbase non-spendable until 100 deep | `schemaConformance.test.ts` "coinbase maturity" | ✅ |
+| 9 | Unfail leaves outputs untouched | Not tested explicitly | ⚠️ **TEST GAP D** |
+| 10 | Mixed broadcast results graceful | Legacy coverage; no new-schema test | ⚠️ **TEST GAP E** |
+
+---
+
+## 8. Non-Functional (§8)
+
+| Requirement | Implementation | Status |
+|---|---|---|
+| Atomic status + outputs update | `transitionProcessing` + audit are single Knex call; `refreshOutputsSpendable` is separate operation | ⚠️ **GAP 3** — not strictly atomic. Acceptable for current architecture (eventual consistency via refresh) but does not match the spec's "or rolled back" wording. |
+| Log every status change with timestamp + reason | `tx_audit` table written on every transition (success + rejected) | ✅ |
+| Status updates + spendability checks < 100ms | `schemaConformance.test.ts` hot query asserts <200ms (conservative); actual SQLite latency is single-digit ms | ✅ |
+| Functional without ARC | Sequential fallback preserved in legacy code | ✅ |
+
+---
+
+## Summary
+
+| Category | Pass | Partial | Gap |
+|---|---|---|---|
+| §1 Principles | 4 | — | 1 (reorg invalidation, FSM) |
+| §2 Statuses | 9 | — | — |
+| §3 Happy path | 6 | — | — |
+| §4 Third-party | 3 | 2 (caller responsibility) | — |
+| §5 Failures | 6 | — | 1 (reorg preserves spendable) |
+| §6 DB sync | 4 | 2 | — |
+| §7 Test coverage | 5 | — | 5 (test gaps A-E) |
+| §8 Non-functional | 3 | 1 (atomicity) | — |
+
+**Critical gaps blocking spec compliance:**
+
+1. **GAP 1** — FSM permits `reorging → invalid` and `reorging → doubleSpend` direct edges. Spec §5 forbids reorg from marking either. Remove these two edges. Force callers to traverse `reorging → unconfirmed → invalid/doubleSpend` if a separate signal later validates the rejection.
+2. **GAP 2** — `refreshOutputsSpendable` flips `outputs.spendable=false` for transactions in `reorging` because that state is not in `ProcessingSpendableStatus`. Spec §5/§6 require preserving the prior spendable value. Either add `reorging` to the spendable set (treating reorging as still-spendable while resolution pends) or skip refresh for outputs whose owning transaction is `reorging`.
+3. **GAP 3** (non-blocking) — atomicity of status + output update. Wrap `transitionProcessing` + targeted output refresh in a Knex transaction at call sites where atomicity matters.
+
+**Test gaps to add (none blocking, but required for §7 compliance):**
+
+- Test A: third-party confirmed at chain tip → unconfirmed.
+- Test B: third-party unconfirmed broadcast failure → discard (no DB row).
+- Test C: reorg of confirmed → reorging → unconfirmed; verify outputs.spendable preserved.
+- Test D: unfail transition; verify outputs unchanged.
+- Test E: mixed broadcast results; verify positive accepted + discrepancy in tx_audit.