@bsv/overlay 2.1.0 → 2.2.0

package/docs/BRC-136-BASM.md

@@ -184,11 +184,16 @@ that observed the reorg. The engine reconciles this automatically.
 
 ### Chaintracks is the reorg authority
 
-Reorg detection is **not** reinvented here. The production chain tracker is
-Arcade, which wraps go-chaintracks; its reorg SSE is the source of truth:
+Reorg detection is **not** reinvented here. The production chain tracker should
+be a go-chaintracks compatible service. Arcade exposes go-chaintracks under
+`/chaintracks/v2`, and a standalone go-chaintracks deployment may expose the same
+API at `/v2`:
 
-- **`GET /v2/reorg/stream`** emits `data: <JSON>\n\n` frames (no `event:`/`id:`
-  lines; `: keepalive` comments between events). Each frame is a `ReorgEvent`:
+- **Arcade-mounted Chaintracks:** `GET /chaintracks/v2/reorg/stream`
+- **Standalone go-chaintracks:** `GET /v2/reorg/stream`
+
+The stream emits `data: <JSON>\n\n` frames (no `event:`/`id:` lines;
+`: keepalive` comments between events). Each frame is a `ReorgEvent`:
 
   ```jsonc
   {
@@ -199,8 +204,8 @@ Arcade, which wraps go-chaintracks; its reorg SSE is the source of truth:
   }
   ```
 
-- The same go-chaintracks service answers the merkle-root checks
-  (`isValidRootForHeight`) the engine already trusts for all SPV verification.
+- The same go-chaintracks-compatible service answers the merkle-root checks
+  (`isValidRootForHeight`) the engine already trusts for SPV verification.
 
 `packages/overlays/overlay-express/src/ReorgStream.ts` consumes this stream and
 maps each event to the engine: `orphanedHashes` → blocks to reconcile,
@@ -246,20 +251,53 @@ Reorg recovery and lookup-layer removal are independent and do not interfere:
 - Reorg demotion touches only the admitted set (proven → unproven) via the chain
   tracker; it never consults the ban list or lookup index.
 
-A demoted transaction follows the engine's existing unproven lifecycle: if it is
-re-mined, Arcade/ARC re-notifies `/arc-ingest` → `handleNewMerkleProof` re-proves
-it at its new height and the anchor includes it again; if it is never re-mined,
-`evictUnprovenTransactions` removes it after the threshold. The "we received it"
-record survives until one of those resolves.
+A demoted transaction follows the engine's unproven lifecycle. The preferred
+maintenance path is refresh-before-evict:
+
+1. If the transaction is re-mined and a provider calls `/arc-ingest` with a
+   proof, `handleNewMerkleProof` re-proves it at its new height and the anchor
+   includes it again.
+2. If no callback arrives, `refreshUnprovenTransactionProofs` asks configured
+   proof providers such as Arcade for a fresh proof.
+3. `maintainUnprovenTransactions` refreshes proofs first, then calls
+   `evictUnprovenTransactions` for rows that are still unproven past the
+   configured threshold.
+
+The "we received it" record survives until a proof, a terminal provider
+invalidation, or age-based eviction resolves it.
+
+### Provider invalidation and double spends
+
+Provider callbacks can also report terminal rejection. When `/arc-ingest`
+classifies a callback as double spend or another terminal invalid outcome, the
+Express layer evicts the applied transaction immediately through
+`Engine.evictAppliedTransaction`. This removes the transaction from the admitted
+set and notifies lookup services through their `outputEvicted` path. It is more
+important to stop serving rejected data than to wait for the normal unproven
+eviction threshold.
 
 ### Configuration
 
 ```ts
-server.configureReorgStream('https://arcade.example/v2/reorg/stream', 3)
+server.configureChaintracks('https://arcade.example', {
+  apiPrefix: '/chaintracks/v2',
+  reorgStream: true,
+  scanDepth: 3
+})
+
+server.configureEnableBASMSync(true)
+server.configureUnprovenMaintenance({
+  thresholdBlocks: 144,
+  intervalMs: 60 * 60 * 1000
+})
 ```
 
-- `reorgStreamUrl` — when set, the SSE adapter reconciles reorgs in real time.
-- `reorgScanDepth` — sweep depth from tip (default `3`).
+- `apiPrefix` — `/chaintracks/v2` for Arcade-mounted Chaintracks; `/v2` for many
+  standalone go-chaintracks deployments.
+- `reorgStream` — when enabled, the SSE adapter reconciles reorgs in real time.
+- `scanDepth` — sweep depth from tip (default `3`).
+- `thresholdBlocks` — how old an unproven row must be before maintenance tries
+  proof refresh and eviction.
 - The block poll (`basmBlockPollIntervalMs`) runs the sweep as a fallback even
   when no stream is configured.
 
@@ -281,5 +319,9 @@ token, yet can still prove it was received and admitted.
    `firstSeenHeight` retained) and the affected `topic_block_anchors` rows now
    carry the canonical block hashes with a recomputed `tac`.
 5. Confirm `TAC(topic, tip)` reconverges with peers that observed the same reorg.
-6. If the orphaned transaction is re-mined, confirm `/arc-ingest` re-proves it and
-   the anchor re-includes it; otherwise confirm it is eventually evicted as unproven.
+6. If the orphaned transaction is re-mined, confirm `/arc-ingest` or
+   `refreshUnprovenTransactionProofs` re-proves it and the anchor re-includes it.
+   Otherwise confirm `maintainUnprovenTransactions` eventually evicts it as
+   unproven.
+7. If a provider reports a double spend, confirm the applied transaction is
+   evicted immediately and lookup services no longer return it.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bsv/overlay",
-  "version": "2.1.0",
+  "version": "2.2.0",
   "type": "module",
   "description": "BSV Blockchain Overlay Services Engine",
   "main": "dist/cjs/mod.js",
@@ -55,7 +55,7 @@
   "homepage": "https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay#readme",
   "devDependencies": {
     "@types/jest": "^30.0.0",
-    "@types/node": "^25.9.3",
+    "@types/node": "^26.0.0",
     "jest": "^30.4.2",
     "ts-jest": "^29.4.11",
     "ts-standard": "^12.0.2",
@@ -65,11 +65,11 @@
     "@bsv/sdk": "^2.1.3"
   },
   "dependencies": {
-    "@bsv/gasp": "^1.2.4",
+    "@bsv/gasp": "^1.3.0",
     "knex": "^3.2.10"
   },
   "peerDependencies": {
-    "@bsv/sdk": "^2"
+    "@bsv/sdk": "^2.1.6"
   },
   "peerDependenciesMeta": {
     "@bsv/sdk": {

package/src/Engine.ts

@@ -593,8 +593,26 @@ export class Engine {
     // ===================================================================
     // PHASE 2: BROADCAST (before any mutations)
     // ===================================================================
+    // Only broadcast when at least one topic actually accepted the
+    // transaction. For a non-failed topic, acceptance means: previously
+    // accepted (dupe / client retry), outputs admitted, coins retained, or
+    // previously-admitted coins consumed (e.g. a consume-only deletion such
+    // as a KVStore remove, even one that retains nothing). A topic manager
+    // REJECTS by throwing from identifyAdmissibleOutputs (tracked in
+    // failedTopics). A transaction every topic rejected must never reach the
+    // network: submitters treat an empty STEAK as a rejection and
+    // abort/release their held inputs, so broadcasting it anyway would
+    // desync their wallets from the chain.
+    const anyTopicAccepted = validations.some(v =>
+      !failedTopics.has(v.topic) && (
+        v.isDupe ||
+        v.admissibleOutputs.outputsToAdmit.length > 0 ||
+        v.admissibleOutputs.coinsToRetain.length > 0 ||
+        v.previousCoins.length > 0
+      )
+    )
     this.startTime(`broadcast_${txid.substring(0, 10)}`)
-    if (mode !== 'historical-tx' && this.broadcaster !== undefined) {
+    if (mode !== 'historical-tx' && this.broadcaster !== undefined && anyTopicAccepted) {
       try {
         let response: BroadcastResponse | BroadcastFailure
         if (tx.merklePath !== undefined) {
@@ -1595,6 +1613,135 @@ export class Engine {
     }
   }
 
+  async refreshUnprovenTransactionProofs(options: {
+    topic?: string
+    thresholdBlocks?: number
+    proofProvider: (txid: string) => Promise<{ merklePath: MerklePath, blockHeight?: number } | undefined>
+  }): Promise<{
+      cutoffHeight: number
+      candidates: number
+      refreshedTransactions: number
+      missingProofs: number
+      failedProofs: number
+      failures: Array<{ txid: string, error: string }>
+    }> {
+    if (typeof this.storage.findUnprovenAppliedTransactions !== 'function') {
+      throw new TypeError('Storage does not support unproven transaction lookup')
+    }
+    if (this.chainTracker === 'scripts only') {
+      throw new Error('Unproven proof refresh requires a ChainTracker to determine block age')
+    }
+
+    const thresholdBlocks = options.thresholdBlocks ?? this.unprovenEvictionBlocks
+    const currentHeight = await this.chainTracker.currentHeight()
+    const cutoffHeight = currentHeight - thresholdBlocks
+    const candidates = await this.storage.findUnprovenAppliedTransactions(cutoffHeight, options.topic)
+    const txids = [...new Set(candidates.map(candidate => candidate.txid))]
+    let refreshedTransactions = 0
+    let missingProofs = 0
+    let failedProofs = 0
+    const failures: Array<{ txid: string, error: string }> = []
+
+    for (const txid of txids) {
+      try {
+        const proof = await options.proofProvider(txid)
+        if (proof === undefined) {
+          missingProofs++
+          continue
+        }
+        await this.handleNewMerkleProof(txid, proof.merklePath, proof.blockHeight)
+        refreshedTransactions++
+      } catch (error) {
+        failedProofs++
+        failures.push({
+          txid,
+          error: error instanceof Error ? error.message : String(error)
+        })
+      }
+    }
+
+    return {
+      cutoffHeight,
+      candidates: candidates.length,
+      refreshedTransactions,
+      missingProofs,
+      failedProofs,
+      failures
+    }
+  }
+
+  async maintainUnprovenTransactions(options: {
+    topic?: string
+    thresholdBlocks?: number
+    proofProvider: (txid: string) => Promise<{ merklePath: MerklePath, blockHeight?: number } | undefined>
+  }): Promise<{
+      refresh: {
+        cutoffHeight: number
+        candidates: number
+        refreshedTransactions: number
+        missingProofs: number
+        failedProofs: number
+        failures: Array<{ txid: string, error: string }>
+      }
+      eviction: {
+        cutoffHeight: number
+        candidates: number
+        evictedTransactions: number
+        evictedOutputs: number
+      }
+    }> {
+    const refresh = await this.refreshUnprovenTransactionProofs(options)
+    const eviction = await this.evictUnprovenTransactions({
+      topic: options.topic,
+      thresholdBlocks: options.thresholdBlocks
+    })
+    return { refresh, eviction }
+  }
+
+  async evictAppliedTransaction(txid: string, options: {
+    topic?: string
+    reason?: string
+  } = {}): Promise<{
+      txid: string
+      reason?: string
+      evictedTransactions: number
+      evictedOutputs: number
+    }> {
+    if (typeof this.storage.deleteAppliedTransaction !== 'function') {
+      throw new TypeError('Storage does not support applied transaction eviction')
+    }
+
+    const outputs = await this.storage.findOutputsForTransaction(txid)
+    const filtered = options.topic === undefined
+      ? outputs
+      : outputs.filter(output => output.topic === options.topic)
+    const topics = [...new Set(filtered.map(output => output.topic))]
+    let evictedOutputs = 0
+
+    for (const output of filtered) {
+      for (const service of Object.values(this.lookupServices)) {
+        try {
+          await service.outputEvicted(output.txid, output.outputIndex)
+        } catch (error) {
+          this.logger.debug(`outputEvicted notification failed for ${output.txid}.${output.outputIndex}: ${error}`)
+        }
+      }
+      await this.storage.deleteOutput(output.txid, output.outputIndex, output.topic)
+      evictedOutputs++
+    }
+
+    for (const topic of topics) {
+      await this.storage.deleteAppliedTransaction(txid, topic)
+    }
+
+    return {
+      txid,
+      reason: options.reason,
+      evictedTransactions: topics.length,
+      evictedOutputs
+    }
+  }
+
   /**
    * Given a GASP request, create an initial response.
    *

package/src/__tests/Engine.test.ts

@@ -167,6 +167,70 @@ describe('BSV Overlay Services Engine', () => {
     }
   })
 
+  it('refreshes old unproven transaction proofs before eviction', async () => {
+    const storage = {
+      ...mockStorageEngine,
+      findUnprovenAppliedTransactions: jest.fn(async () => [
+        {
+          txid: exampleTXID,
+          topic: 'Hello',
+          firstSeenHeight: 799000,
+          outputs: [{ txid: exampleTXID, outputIndex: 0 }]
+        }
+      ])
+    }
+    const engine = new Engine(
+      { tm_helloworld: mockTopicManager },
+      { ls_helloworld: mockLookupService },
+      storage,
+      mockChainTracker,
+      'https://example.com'
+    )
+    const merklePath = {} as any
+    ;(engine as any).handleNewMerkleProof = jest.fn(async () => undefined)
+
+    const report = await engine.refreshUnprovenTransactionProofs({
+      thresholdBlocks: 144,
+      proofProvider: jest.fn(async () => ({ merklePath, blockHeight: 799900 }))
+    })
+
+    expect(report.refreshedTransactions).toBe(1)
+    expect(report.missingProofs).toBe(0)
+    expect((engine as any).handleNewMerkleProof).toHaveBeenCalledWith(exampleTXID, merklePath, 799900)
+  })
+
+  it('evicts provider-invalidated applied transactions', async () => {
+    const deleteAppliedTransaction = jest.fn(async () => undefined)
+    const deleteOutput = jest.fn(async () => undefined)
+    const engine = new Engine(
+      { tm_helloworld: mockTopicManager },
+      { ls_helloworld: mockLookupService },
+      {
+        ...mockStorageEngine,
+        findOutputsForTransaction: jest.fn(async () => [
+          {
+            ...mockOutput,
+            topic: 'Hello'
+          }
+        ]),
+        deleteOutput,
+        deleteAppliedTransaction
+      },
+      mockChainTracker,
+      'https://example.com'
+    )
+
+    const report = await engine.evictAppliedTransaction(exampleTXID, {
+      reason: 'DOUBLE_SPEND_ATTEMPTED'
+    })
+
+    expect(report.evictedTransactions).toBe(1)
+    expect(report.evictedOutputs).toBe(1)
+    expect(mockLookupService.outputEvicted).toHaveBeenCalledWith(exampleTXID, 0)
+    expect(deleteOutput).toHaveBeenCalledWith(exampleTXID, 0, 'Hello')
+    expect(deleteAppliedTransaction).toHaveBeenCalledWith(exampleTXID, 'Hello')
+  })
+
   it('Uses SHIP sync configuration by default if no syncConfiguration was provided', () => {
     const engine = new Engine(
       { tm_helloworld: mockTopicManager },
@@ -440,6 +504,88 @@ describe('BSV Overlay Services Engine', () => {
         topics: ['Hello']
       })).rejects.toHaveProperty('message', 'Invalid merkle path for transaction 3ecead27a44d013ad1aae40038acbb1883ac9242406808bb4667c15b4f164eac')
     })
+    describe('Broadcast gating (PHASE 2)', () => {
+      const makeBroadcaster = (): { broadcast: jest.Mock } => ({
+        broadcast: jest.fn(async () => ({ status: 'success', txid: exampleTXID, message: 'ok' }))
+      })
+      const makeEngine = (broadcaster: any): Engine => new Engine(
+        { Hello: mockTopicManager },
+        {},
+        mockStorageEngine,
+        mockChainTracker,
+        undefined,
+        undefined,
+        undefined,
+        broadcaster
+      )
+
+      it('broadcasts when at least one topic admits outputs', async () => {
+        const broadcaster = makeBroadcaster()
+        const engine = makeEngine(broadcaster)
+        await engine.submit({ beef: exampleBeef, topics: ['Hello'] })
+        expect(broadcaster.broadcast).toHaveBeenCalledTimes(1)
+      })
+
+      it('never broadcasts a transaction every topic manager rejected', async () => {
+        mockTopicManager.identifyAdmissibleOutputs = jest.fn(async () => ({
+          outputsToAdmit: [],
+          coinsToRetain: []
+        }))
+        const broadcaster = makeBroadcaster()
+        const engine = makeEngine(broadcaster)
+        await engine.submit({ beef: exampleBeef, topics: ['Hello'] })
+        expect(broadcaster.broadcast).not.toHaveBeenCalled()
+      })
+
+      it('never broadcasts when topic validation throws, even with tracked coins consumed', async () => {
+        // The rejected tx spends previously-admitted coins — a throw must
+        // still gate the broadcast (this is the rejected-transfer shape).
+        mockStorageEngine.findOutput = jest.fn(async () => mockOutput)
+        mockTopicManager.identifyAdmissibleOutputs = jest.fn(async () => {
+          throw new Error('rule violation')
+        })
+        const broadcaster = makeBroadcaster()
+        const engine = makeEngine(broadcaster)
+        await engine.submit({ beef: exampleBeef, topics: ['Hello'] })
+        expect(broadcaster.broadcast).not.toHaveBeenCalled()
+      })
+
+      it('broadcasts a consume-only transaction that retains nothing (history purge)', async () => {
+        // e.g. a KVStore remove that also purges history: previously-admitted
+        // coins are consumed, nothing admitted, nothing retained. That is an
+        // acceptance, not a rejection (rejection = throw).
+        mockStorageEngine.findOutput = jest.fn(async () => mockOutput)
+        mockTopicManager.identifyAdmissibleOutputs = jest.fn(async () => ({
+          outputsToAdmit: [],
+          coinsToRetain: []
+        }))
+        const broadcaster = makeBroadcaster()
+        const engine = makeEngine(broadcaster)
+        await engine.submit({ beef: exampleBeef, topics: ['Hello'] })
+        expect(broadcaster.broadcast).toHaveBeenCalledTimes(1)
+      })
+
+      it('still broadcasts a duplicate (previously accepted) transaction', async () => {
+        mockStorageEngine.doesAppliedTransactionExist = jest.fn(async () => true)
+        const broadcaster = makeBroadcaster()
+        const engine = makeEngine(broadcaster)
+        await engine.submit({ beef: exampleBeef, topics: ['Hello'] })
+        expect(broadcaster.broadcast).toHaveBeenCalledTimes(1)
+      })
+
+      it('broadcasts when outputs are consumed but none admitted (coinsToRetain only)', async () => {
+        mockStorageEngine.findOutput = jest.fn(async () => mockOutput)
+        mockTopicManager.identifyAdmissibleOutputs = jest.fn(async () => ({
+          outputsToAdmit: [],
+          coinsToRetain: [0]
+        }))
+        const broadcaster = makeBroadcaster()
+        const engine = makeEngine(broadcaster)
+        await engine.submit({ beef: exampleBeef, topics: ['Hello'] })
+        expect(broadcaster.broadcast).toHaveBeenCalledTimes(1)
+      })
+    })
+
     describe('For each topic being processed', () => {
       it('Checks for duplicate transactions', async () => {
         const engine = new Engine(