@bsv/overlay 2.0.4 → 2.1.1

package/docs/BRC-136-BASM.md

@@ -184,11 +184,16 @@ that observed the reorg. The engine reconciles this automatically.
 
 ### Chaintracks is the reorg authority
 
-Reorg detection is **not** reinvented here. The production chain tracker is
-Arcade, which wraps go-chaintracks; its reorg SSE is the source of truth:
+Reorg detection is **not** reinvented here. The production chain tracker should
+be a go-chaintracks compatible service. Arcade exposes go-chaintracks under
+`/chaintracks/v2`, and a standalone go-chaintracks deployment may expose the same
+API at `/v2`:
 
-- **`GET /v2/reorg/stream`** emits `data: <JSON>\n\n` frames (no `event:`/`id:`
-  lines; `: keepalive` comments between events). Each frame is a `ReorgEvent`:
+- **Arcade-mounted Chaintracks:** `GET /chaintracks/v2/reorg/stream`
+- **Standalone go-chaintracks:** `GET /v2/reorg/stream`
+
+The stream emits `data: <JSON>\n\n` frames (no `event:`/`id:` lines;
+`: keepalive` comments between events). Each frame is a `ReorgEvent`:
 
   ```jsonc
   {
@@ -199,8 +204,8 @@ Arcade, which wraps go-chaintracks; its reorg SSE is the source of truth:
   }
   ```
 
-- The same go-chaintracks service answers the merkle-root checks
-  (`isValidRootForHeight`) the engine already trusts for all SPV verification.
+- The same go-chaintracks-compatible service answers the merkle-root checks
+  (`isValidRootForHeight`) the engine already trusts for SPV verification.
 
 `packages/overlays/overlay-express/src/ReorgStream.ts` consumes this stream and
 maps each event to the engine: `orphanedHashes` → blocks to reconcile,
@@ -246,20 +251,53 @@ Reorg recovery and lookup-layer removal are independent and do not interfere:
 - Reorg demotion touches only the admitted set (proven → unproven) via the chain
   tracker; it never consults the ban list or lookup index.
 
-A demoted transaction follows the engine's existing unproven lifecycle: if it is
-re-mined, Arcade/ARC re-notifies `/arc-ingest` → `handleNewMerkleProof` re-proves
-it at its new height and the anchor includes it again; if it is never re-mined,
-`evictUnprovenTransactions` removes it after the threshold. The "we received it"
-record survives until one of those resolves.
+A demoted transaction follows the engine's unproven lifecycle. The preferred
+maintenance path is refresh-before-evict:
+
+1. If the transaction is re-mined and a provider calls `/arc-ingest` with a
+   proof, `handleNewMerkleProof` re-proves it at its new height and the anchor
+   includes it again.
+2. If no callback arrives, `refreshUnprovenTransactionProofs` asks configured
+   proof providers such as Arcade for a fresh proof.
+3. `maintainUnprovenTransactions` refreshes proofs first, then calls
+   `evictUnprovenTransactions` for rows that are still unproven past the
+   configured threshold.
+
+The "we received it" record survives until a proof, a terminal provider
+invalidation, or age-based eviction resolves it.
+
+### Provider invalidation and double spends
+
+Provider callbacks can also report terminal rejection. When `/arc-ingest`
+classifies a callback as double spend or another terminal invalid outcome, the
+Express layer evicts the applied transaction immediately through
+`Engine.evictAppliedTransaction`. This removes the transaction from the admitted
+set and notifies lookup services through their `outputEvicted` path. It is more
+important to stop serving rejected data than to wait for the normal unproven
+eviction threshold.
 
 ### Configuration
 
 ```ts
-server.configureReorgStream('https://arcade.example/v2/reorg/stream', 3)
+server.configureChaintracks('https://arcade.example', {
+  apiPrefix: '/chaintracks/v2',
+  reorgStream: true,
+  scanDepth: 3
+})
+
+server.configureEnableBASMSync(true)
+server.configureUnprovenMaintenance({
+  thresholdBlocks: 144,
+  intervalMs: 60 * 60 * 1000
+})
 ```
 
-- `reorgStreamUrl` — when set, the SSE adapter reconciles reorgs in real time.
-- `reorgScanDepth` — sweep depth from tip (default `3`).
+- `apiPrefix` — `/chaintracks/v2` for Arcade-mounted Chaintracks; `/v2` for many
+  standalone go-chaintracks deployments.
+- `reorgStream` — when enabled, the SSE adapter reconciles reorgs in real time.
+- `scanDepth` — sweep depth from tip (default `3`).
+- `thresholdBlocks` — how old an unproven row must be before maintenance tries
+  proof refresh and eviction.
 - The block poll (`basmBlockPollIntervalMs`) runs the sweep as a fallback even
   when no stream is configured.
 
@@ -281,5 +319,9 @@ token, yet can still prove it was received and admitted.
    `firstSeenHeight` retained) and the affected `topic_block_anchors` rows now
    carry the canonical block hashes with a recomputed `tac`.
 5. Confirm `TAC(topic, tip)` reconverges with peers that observed the same reorg.
-6. If the orphaned transaction is re-mined, confirm `/arc-ingest` re-proves it and
-   the anchor re-includes it; otherwise confirm it is eventually evicted as unproven.
+6. If the orphaned transaction is re-mined, confirm `/arc-ingest` or
+   `refreshUnprovenTransactionProofs` re-proves it and the anchor re-includes it.
+   Otherwise confirm `maintainUnprovenTransactions` eventually evicts it as
+   unproven.
+7. If a provider reports a double spend, confirm the applied transaction is
+   evicted immediately and lookup services no longer return it.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bsv/overlay",
-  "version": "2.0.4",
+  "version": "2.1.1",
   "type": "module",
   "description": "BSV Blockchain Overlay Services Engine",
   "main": "dist/cjs/mod.js",
@@ -55,19 +55,27 @@
   "homepage": "https://github.com/bsv-blockchain/ts-stack/tree/main/packages/overlays/overlay#readme",
   "devDependencies": {
     "@types/jest": "^30.0.0",
-    "@types/node": "^25.9.3",
+    "@types/node": "^26.0.0",
     "jest": "^30.4.2",
     "ts-jest": "^29.4.11",
     "ts-standard": "^12.0.2",
     "ts2md": "^0.2.0",
     "tsconfig-to-dual-package": "^1.2.0",
-    "typescript": "^6.0.3"
+    "typescript": "^6.0.3",
+    "@bsv/sdk": "^2.1.3"
   },
   "dependencies": {
-    "@bsv/gasp": "^1.2.3",
-    "@bsv/sdk": "^2.1.4",
+    "@bsv/gasp": "^1.3.0",
     "knex": "^3.2.10"
   },
+  "peerDependencies": {
+    "@bsv/sdk": "^2.1.6"
+  },
+  "peerDependenciesMeta": {
+    "@bsv/sdk": {
+      "optional": false
+    }
+  },
   "scripts": {
     "test": "npm run build && jest",
     "test:watch": "npm run build && jest --watch",

package/src/Engine.ts

@@ -1595,6 +1595,135 @@ export class Engine {
     }
   }
 
+  async refreshUnprovenTransactionProofs(options: {
+    topic?: string
+    thresholdBlocks?: number
+    proofProvider: (txid: string) => Promise<{ merklePath: MerklePath, blockHeight?: number } | undefined>
+  }): Promise<{
+      cutoffHeight: number
+      candidates: number
+      refreshedTransactions: number
+      missingProofs: number
+      failedProofs: number
+      failures: Array<{ txid: string, error: string }>
+    }> {
+    if (typeof this.storage.findUnprovenAppliedTransactions !== 'function') {
+      throw new TypeError('Storage does not support unproven transaction lookup')
+    }
+    if (this.chainTracker === 'scripts only') {
+      throw new Error('Unproven proof refresh requires a ChainTracker to determine block age')
+    }
+
+    const thresholdBlocks = options.thresholdBlocks ?? this.unprovenEvictionBlocks
+    const currentHeight = await this.chainTracker.currentHeight()
+    const cutoffHeight = currentHeight - thresholdBlocks
+    const candidates = await this.storage.findUnprovenAppliedTransactions(cutoffHeight, options.topic)
+    const txids = [...new Set(candidates.map(candidate => candidate.txid))]
+    let refreshedTransactions = 0
+    let missingProofs = 0
+    let failedProofs = 0
+    const failures: Array<{ txid: string, error: string }> = []
+
+    for (const txid of txids) {
+      try {
+        const proof = await options.proofProvider(txid)
+        if (proof === undefined) {
+          missingProofs++
+          continue
+        }
+        await this.handleNewMerkleProof(txid, proof.merklePath, proof.blockHeight)
+        refreshedTransactions++
+      } catch (error) {
+        failedProofs++
+        failures.push({
+          txid,
+          error: error instanceof Error ? error.message : String(error)
+        })
+      }
+    }
+
+    return {
+      cutoffHeight,
+      candidates: candidates.length,
+      refreshedTransactions,
+      missingProofs,
+      failedProofs,
+      failures
+    }
+  }
+
+  async maintainUnprovenTransactions(options: {
+    topic?: string
+    thresholdBlocks?: number
+    proofProvider: (txid: string) => Promise<{ merklePath: MerklePath, blockHeight?: number } | undefined>
+  }): Promise<{
+      refresh: {
+        cutoffHeight: number
+        candidates: number
+        refreshedTransactions: number
+        missingProofs: number
+        failedProofs: number
+        failures: Array<{ txid: string, error: string }>
+      }
+      eviction: {
+        cutoffHeight: number
+        candidates: number
+        evictedTransactions: number
+        evictedOutputs: number
+      }
+    }> {
+    const refresh = await this.refreshUnprovenTransactionProofs(options)
+    const eviction = await this.evictUnprovenTransactions({
+      topic: options.topic,
+      thresholdBlocks: options.thresholdBlocks
+    })
+    return { refresh, eviction }
+  }
+
+  async evictAppliedTransaction(txid: string, options: {
+    topic?: string
+    reason?: string
+  } = {}): Promise<{
+      txid: string
+      reason?: string
+      evictedTransactions: number
+      evictedOutputs: number
+    }> {
+    if (typeof this.storage.deleteAppliedTransaction !== 'function') {
+      throw new TypeError('Storage does not support applied transaction eviction')
+    }
+
+    const outputs = await this.storage.findOutputsForTransaction(txid)
+    const filtered = options.topic === undefined
+      ? outputs
+      : outputs.filter(output => output.topic === options.topic)
+    const topics = [...new Set(filtered.map(output => output.topic))]
+    let evictedOutputs = 0
+
+    for (const output of filtered) {
+      for (const service of Object.values(this.lookupServices)) {
+        try {
+          await service.outputEvicted(output.txid, output.outputIndex)
+        } catch (error) {
+          this.logger.debug(`outputEvicted notification failed for ${output.txid}.${output.outputIndex}: ${error}`)
+        }
+      }
+      await this.storage.deleteOutput(output.txid, output.outputIndex, output.topic)
+      evictedOutputs++
+    }
+
+    for (const topic of topics) {
+      await this.storage.deleteAppliedTransaction(txid, topic)
+    }
+
+    return {
+      txid,
+      reason: options.reason,
+      evictedTransactions: topics.length,
+      evictedOutputs
+    }
+  }
+
   /**
    * Given a GASP request, create an initial response.
    *

package/src/__tests/Engine.test.ts

@@ -167,6 +167,70 @@ describe('BSV Overlay Services Engine', () => {
     }
   })
 
+  it('refreshes old unproven transaction proofs before eviction', async () => {
+    const storage = {
+      ...mockStorageEngine,
+      findUnprovenAppliedTransactions: jest.fn(async () => [
+        {
+          txid: exampleTXID,
+          topic: 'Hello',
+          firstSeenHeight: 799000,
+          outputs: [{ txid: exampleTXID, outputIndex: 0 }]
+        }
+      ])
+    }
+    const engine = new Engine(
+      { tm_helloworld: mockTopicManager },
+      { ls_helloworld: mockLookupService },
+      storage,
+      mockChainTracker,
+      'https://example.com'
+    )
+    const merklePath = {} as any
+    ;(engine as any).handleNewMerkleProof = jest.fn(async () => undefined)
+
+    const report = await engine.refreshUnprovenTransactionProofs({
+      thresholdBlocks: 144,
+      proofProvider: jest.fn(async () => ({ merklePath, blockHeight: 799900 }))
+    })
+
+    expect(report.refreshedTransactions).toBe(1)
+    expect(report.missingProofs).toBe(0)
+    expect((engine as any).handleNewMerkleProof).toHaveBeenCalledWith(exampleTXID, merklePath, 799900)
+  })
+
+  it('evicts provider-invalidated applied transactions', async () => {
+    const deleteAppliedTransaction = jest.fn(async () => undefined)
+    const deleteOutput = jest.fn(async () => undefined)
+    const engine = new Engine(
+      { tm_helloworld: mockTopicManager },
+      { ls_helloworld: mockLookupService },
+      {
+        ...mockStorageEngine,
+        findOutputsForTransaction: jest.fn(async () => [
+          {
+            ...mockOutput,
+            topic: 'Hello'
+          }
+        ]),
+        deleteOutput,
+        deleteAppliedTransaction
+      },
+      mockChainTracker,
+      'https://example.com'
+    )
+
+    const report = await engine.evictAppliedTransaction(exampleTXID, {
+      reason: 'DOUBLE_SPEND_ATTEMPTED'
+    })
+
+    expect(report.evictedTransactions).toBe(1)
+    expect(report.evictedOutputs).toBe(1)
+    expect(mockLookupService.outputEvicted).toHaveBeenCalledWith(exampleTXID, 0)
+    expect(deleteOutput).toHaveBeenCalledWith(exampleTXID, 0, 'Hello')
+    expect(deleteAppliedTransaction).toHaveBeenCalledWith(exampleTXID, 'Hello')
+  })
+
   it('Uses SHIP sync configuration by default if no syncConfiguration was provided', () => {
     const engine = new Engine(
       { tm_helloworld: mockTopicManager },