@bsv/overlay 0.6.0 → 0.6.1

package/docs/Synchronization.md

@@ -0,0 +1,203 @@
+## About
+
+**GASP (Graph Aware Sync Protocol)** is the synchronization protocol overlay nodes use to replicate overlay-relevant transaction data between peers in a way that is **verifiable**, **complete**, and **bandwidth-efficient**. Instead of just sending raw TXIDs or copying a flat UTXO list, GASP reconciles **transaction graphs** by exchanging transactions/outputs with proof material and then **recursively requesting any missing input transactions** needed to validate what was received.
+
+At a high level, GASP is built around:
+
+- **Legitimacy:** Nodes only finalize data they can validate by anchoring it back to the blockchain (e.g., using merkle proofs / SPV-style verification).
+- **Completeness:** If a transaction depends on other transactions, the protocol recursively fetches all required inputs so the end result isn’t partial or broken.
+- **Efficiency:** Nodes only sync what they don’t have, reducing duplicates and bandwidth.
+- **Redundancy / availability:** Multiple nodes can converge on the same overlay view over time, improving uptime and reducing single-host dependency.
+
+## Importance
+
+Overlays are useful because they let you track only the topic you care about. But once an overlay is distributed (multiple nodes serving the same topic), nodes need a path to **catch up** and **stay consistent** over time.
+
+GASP matters because it gives you:
+
+- **Redundancy and uptime:** If one overlay host goes offline, other synced hosts can still serve the same topic, reducing single points of failure.
+- **Fast bootstrap:** A fresh node can synchronize overlay state from peers instead of re-ingesting all previous transactions from scratch.
+- **Lower-trust sync (for correctness):** Peers exchange transactions with proofs and recursively prove inputs, so received data can be validated instead of trusted blindly. *(This doesn’t replace your auth policy; it reduces trust needed for correctness.)*
+- **Scaling the ecosystem:** As more apps and topics exist, GASP enables horizontal growth (more hosts for the same topic) instead of centralizing into one massive indexer.
+
+## How GASP Works
+
+One party initiates sync by summarizing what it currently has, then the peer responds with what’s missing, and both sides iterate until they converge.
+
+A typical flow looks like:
+
+1) **Summarize local state**  
+   The initiator summarizes known spendable outpoints (commonly via a Bloom filter over TXID+VOUT).
+
+2) **Responder sends “missing” inventory**  
+   The responder identifies items the initiator likely doesn’t have and returns inventory entries containing the output/transaction plus proof material (and optional metadata).
+
+3) **Recursive completion**  
+   If the initiator is missing any input transactions required to validate what it received, it requests them. This repeats recursively until the needed graph is complete.
+
+4) **Verification + finalize**  
+   Anything that can’t be validated/anchored is ignored rather than partially imported.
+
+## Usage
+
+There are two main ways to activate and run GASP sync:
+
+- **With CARS (cloud / managed):** Use CARS menus to enable sync options and deploy.
+- **Without CARS (direct / self-managed):** Run your own Overlay Express server and trigger sync using the admin endpoints.
+
+> **Important:** For GASP sync to work, *both* parties must have GASP enabled and must be configured to sync the same topic(s).
+
+---
+
+### With CARS
+
+> If your project has a package.json script called `cars`, you can use `npm run cars`. Otherwise you typically run `cars` directly.
+
+### Using CARS (high level)
+
+- `npm run cars` *(or `cars`)*
+- Manage Projects
+- Edit Advanced Engine Config
+- Choose correct CARS config
+- Toggle `gaspSync`
+- Edit `syncConfiguration`
+- Add your topic manager name(s) (e.g. `tm_example`)
+- Back → Done
+- Back to main menu
+- Build Artifact → Auto-create new release and upload latest artifact now
+
+### Side notes
+
+- Simply toggling `gaspSync` is not enough if your node is not “interested” in syncing your topic.  
+  You must also ensure your topic manager name is included in `syncConfiguration` so the engine knows which topic(s) to sync.
+
+---
+
+### Without CARS
+
+Without CARS, you run your own Overlay Express server and configure the overlay engine so it:
+1) **discovers peers / publishes ads** (via an advertiser), and
+2) knows **which topic managers to sync** (via `syncConfiguration`), and
+3) has **GASP enabled**.
+
+## Minimal engine config you must have
+
+The most important lines are:
+
+```ts
+server.configureEngineParams({
+  advertiser: wa,
+  syncConfiguration: {
+    'tm_plite': 'SHIP',
+    'tm_blockbeta': 'SHIP',
+  },
+  logTime: false,
+  logPrefix: '[OVERLAY] ',
+  throwOnBroadcastFailure: false,
+  suppressDefaultSyncAdvertisements: true,
+})
+// ^ tells the engine which topic(s) you want to sync, and the discovery mechanism to use (e.g. SHIP)
+
+server.configureEnableGASPSync(true)
+// ^ enables GASP sync in Overlay Express
+```
+
+## **Overlay Express Setup Example**
+
+```ts
+import { WalletAdvertiser } from '@bsv/overlay-discovery-services'
+import OverlayExpress from '@bsv/overlay-express'
+import { config } from 'dotenv'
+import packageJson from '../package.json'
+import PollrTopicManager from './services/pollroverlay/PollrTopicManager'
+import PollrLookupServiceFactory from './services/pollroverlay/PollrLookupServiceFactory'
+import ForumTopicManager from './services/blockitoverlay/ForumTopicManager'
+import ForumLookupService from './services/blockitoverlay/ForumLookupServiceFactory'
+
+config()
+
+const main = async () => {
+  const server = new OverlayExpress(
+    process.env.NODE_NAME!,
+    process.env.SERVER_PRIVATE_KEY!,
+    process.env.HOSTING_URL!,
+    process.env.ADMIN_TOKEN! // your chosen admin token to use the admin API
+  )
+
+  const wa = new WalletAdvertiser(
+    process.env.NETWORK! as 'main' | 'test',
+    process.env.SERVER_PRIVATE_KEY!,
+    process.env.WALLET_STORAGE_URL!,
+    process.env.HOSTING_URL!
+  )
+
+  await wa.init()
+
+  server.configureEngineParams({
+    advertiser: wa,
+    syncConfiguration: {
+      'tm_plite': 'SHIP',
+      'tm_blockbeta': 'SHIP',
+    },
+    logTime: false,
+    logPrefix: '[OVERLAY] ',
+    throwOnBroadcastFailure: false,
+    suppressDefaultSyncAdvertisements: true,
+  })
+
+  server.configureArcApiKey(process.env.ARC_API_KEY!)
+  server.configurePort(8080)
+
+  await server.configureKnex(process.env.KNEX_URL!)
+  await server.configureMongo(process.env.MONGO_URL!)
+
+  server.configureTopicManager('tm_plite', new PollrTopicManager())
+  server.configureLookupServiceWithMongo('ls_plite', PollrLookupServiceFactory)
+
+  server.configureTopicManager('tm_blockbeta', new ForumTopicManager())
+  server.configureLookupServiceWithMongo('ls_blockbeta', ForumLookupService)
+
+  server.configureEnableGASPSync(true)
+
+  await server.configureEngine()
+
+  server.app.get('/version', (req, res) => res.json(packageJson))
+
+  await server.start()
+}
+
+main()
+```
+
+# **Validation**
+
+A practical way to validate sync is to compare per-topic records in your SQL database before and after sync.
+
+Example (adapt the table/query to your schema):
+### **Before sync**
+
+```
++-------------+----+
+| topic       | n  |
++-------------+----+
+| tm_plite    |  2 |
+| tm_blockbeta|  1 |
+| tm_ship     |  2 |
+| tm_slap     |  2 |
++-------------+----+
+```
+
+### **After sync**
+
+```
++-------------+------+
+| topic       | n    |
++-------------+------+
+| tm_plite    |   10 |
+| tm_blockbeta|   17 |
+| tm_ship     | 2177 |
+| tm_slap     |    2 |
++-------------+------+
+```
+
+**What you’re looking for:** the topic(s) you care about (e.g. tm_plite, tm_blockbeta) should move toward the same counts/data across nodes after sync, and your lookup results for identical queries should converge as well.

package/docs/concepts/00-overview.md

@@ -0,0 +1,85 @@
+# Overlay Use and Optimization in the BSV Ecosystem
+
+[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)
+
+**Navigation:** [Overview](./00-overview.md) | [Best Practices](./01-best-practices.md) | [Query Performance](./02-query-performance.md) | [Database Monitoring](./03-database-monitoring.md) | [Pagination Example](./04-pagination-example.md) | [Recommendations Summary](./05-recommendations-summary.md)
+
+---
+
+## Introduction
+
+Overlays are distributed application layers built on top of Bitcoin SV (BSV). They allow applications to publish structured data and then query it efficiently through **Lookup Services** and **Topic Managers**. This pattern enables developers to build scalable, high‑performance, privacy‑aware applications without needing to index or scan the entire blockchain themselves.
+
+Applications such as **Convo Messenger**, **Tempo**, and **MetaMarket** use overlays to organize messages, content references, metadata, and other application‑specific data in a predictable and queryable way.
+
+This documentation is written for **developers who want to build overlays for their own applications**. It explains how overlays work, what must be stored, how to structure queries, and how to keep overlay implementations efficient and maintainable.
+
+---
+
+## Purpose of This Documentation
+
+Developers implementing overlays commonly need guidance on:
+
+* What data a Lookup Service should store
+* How to design efficient queries for retrieving overlay data
+* How to index and structure Lookup Service databases
+* How to keep overlays healthy and performant as they scale
+
+This documentation focuses on **practical guidance** that helps developers design overlays correctly and avoid common pitfalls. Examples from real applications—such as Convo Messenger—are used when relevant.
+
+This documentation does **not** cover internal BSVA system design, Overlay Express internals, or future platform development proposals.
+
+---
+
+## What This Series Covers
+
+Each document in this series focuses on a key aspect of overlay implementation.
+
+### 1. Best Practices for Overlay Design
+
+* What Lookup Services are required to store
+* What additional metadata overlays often include
+* What kinds of data should *not* be stored (e.g., large payloads)
+* How to use UHRP references for external content
+
+### 2. Query Optimization and Performance
+
+* How to design efficient queries
+* How to choose indexes for common access patterns
+* How to avoid unbounded or slow query patterns
+
+### 3. Monitoring and Debugging Overlays
+
+* How to monitor your own overlay deployment
+* How to inspect stored data safely
+* How to track ingestion issues from your Topic Manager
+
+### 4. Pagination and Practical Examples
+
+* Why pagination is critical for performance
+* How to implement pagination in Lookup Services and clients
+* Examples drawn from live overlay behavior (e.g., Convo Messenger)
+
+### 5. Summary of Recommendations
+
+* A concise reference of best practices
+* Quick reminders for developers building overlays
+
+---
+
+## Deliverables
+
+This documentation includes:
+
+* **Markdown files** that explain how to build, index, and monitor overlay implementations
+* **Examples** demonstrating common patterns such as pagination and indexed lookup queries
+* **Guidelines** that help ensure overlays remain scalable and consistent across different applications
+
+---
+
+## Next Steps
+
+1. Read the general best practices for overlay storage (`01-best-practices.md`).
+2. Review query performance and indexing fundamentals (`02-query-performance.md`).
+3. Explore examples using Convo Messenger (`04-pagination-example.md`).
+4. Refer to the summary document for quick reminders or cross-references.

package/docs/concepts/01-best-practices.md

@@ -0,0 +1,202 @@
+# Overlay Best Practices for Developers
+
+[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)
+
+**Navigation:** [Overview](./00-overview.md) | [Best Practices](./01-best-practices.md) | [Query Performance](./02-query-performance.md) | [Database Monitoring](./03-database-monitoring.md) | [Pagination Example](./04-pagination-example.md) | [Recommendations Summary](./05-recommendations-summary.md)
+
+---
+
+## Overview
+
+This document provides **best practices for designing and maintaining overlay services** in the BSV ecosystem. It focuses on the practical concerns of developers who are building their **own overlays**—not on BSVA internals or modifications to Overlay Express.
+
+The goal is to help developers make consistent and scalable decisions when implementing **Topic Managers**, **Lookup Services**, and associated **MongoDB storage layers**. Examples reference **Convo Messenger**, but the principles apply to all overlay-based systems.
+
+---
+
+## 1. Data Storage Principles
+
+Overlay Lookup Services should store **small, structured metadata**, not entire payloads. The goal is fast, predictable querying.
+
+### 1.1 Required Fields for Any Overlay
+
+Every Lookup Service must store enough information to uniquely reference an on-chain output. This includes:
+
+* **`txid`** — transaction ID containing the output
+* **`outputIndex`** — index of the admitted output
+* **`protocol`** — identifies which overlay protocol this entry belongs to
+
+These fields ensure that applications can always locate the exact on-chain output referenced by the overlay.
+
+### 1.2 Common Metadata Fields
+
+In addition to required fields, overlays typically store small metadata extracted from PushDrop fields or derived from application logic.
+
+Examples:
+
+* **`timestamp` / `createdAt`** — when the message or record was created
+* **`sender`** — sender's pubkey or DID
+* **`threadId`** — group/thread identifier
+* **`parentMessageId`** (optional) — for reply messages
+* **reaction fields** (when applicable)
+
+Example (Convo Messenger):
+
+```ts
+{
+  txid: "abc123...",
+  outputIndex: 0,
+  protocol: [2, "convo"],
+  threadId: "thread-xyz",
+  sender: "035a1b...",
+  createdAt: 1730000000000,
+  encryptedPayload: [...], // ciphertext array
+  header: [...],           // CurvePoint header
+  uniqueId: "optional-value",
+  parentMessageId: "optional-parent-txid"
+}
+```
+
+**Note:** Convo stores *encryptedPayload* and *header* because these remain small arrays needed by the client. Overlays must avoid storing anything large (e.g., full media files).
+
+### 1.3 When to Use UHRP
+
+If an overlay needs to handle large data (text bodies, images, files), Lookup Services should **only store a UHRP reference**:
+
+* Never store full file contents
+* Never store large plaintext or ciphertext blobs
+* Let the client fetch large data from UHRP when needed
+
+Lookup Services should remain lightweight.
+
+---
+
+## 2. What Lookup Services Should *Not* Store
+
+To maintain performance and predictable scaling, Lookup Services must avoid:
+
+* Full PushDrop payloads or full scripts
+* Large encrypted or plaintext message bodies
+* Media files or binary attachments
+* Identity certificates
+* Redundant or duplicate on-chain data
+
+**Rule of thumb:**
+If it cannot be indexed efficiently, it does not belong in the Lookup DB.
+
+---
+
+## 3. Structuring Lookup Queries
+
+### 3.1 Use Indexed, Selective Fields
+
+All queries should filter using indexed, selective fields such as:
+
+* `threadId`
+* `sender`
+* `parentMessageId`
+* `createdAt`
+
+Examples from Convo’s actual Mongo indexes:
+
+```js
+db.convoMessages.createIndex({ threadId: 1 })
+db.convoMessages.createIndex({ parentMessageId: 1 })
+db.convoMessages.createIndex({ threadId: 1, createdAt: -1 })
+db.convoReactions.createIndex({ threadId: 1 })
+```
+
+### 3.2 Use Pagination Everywhere
+
+List endpoints must support:
+
+* **`skip`** — starting offset
+* **`limit`** — number of items
+
+Typical defaults:
+
+* `limit = 50`
+* `skip = 0`
+
+Example (Convo):
+
+```ts
+const messages = await this.storage.listThreadMessages(threadId, skip, limit)
+```
+
+### 3.3 Avoid Expensive Query Patterns
+
+Do **not**:
+
+* Perform unbounded collection scans
+* Query across all protocols
+* Sort without an index
+* Use `$regex` or `$text` filters
+
+If complex queries are needed, break them into:
+
+* Indexed filters
+* Bounded pagination
+* Incremental lookups
+
+---
+
+## 4. Topic Manager Best Practices
+
+Topic Managers determine which outputs belong to your overlay. They must validate and parse data correctly.
+
+Best practices:
+
+1. **Validate admissible outputs** using PushDrop decoding.
+2. **Store only necessary fields** extracted from PushDrop.
+3. **Reject malformed or irrelevant outputs** early.
+4. **Handle duplicate admissions** cleanly.
+5. **Avoid heavy computation** inside the Topic Manager.
+
+Example (Convo):
+
+```ts
+const decoded = PushDrop.decode(output.lockingScript)
+const marker = Utils.toUTF8(fields[0])
+const protocol = Utils.toUTF8(fields[1])
+
+if (marker === 'convo' && protocol === 'tmconvo') {
+  admissibleOutputs.push(index)
+}
+```
+
+---
+
+## 5. Data Retention and Cleanup
+
+Overlays accumulate data continuously. Without retention logic, performance may degrade.
+
+Common retention approaches:
+
+* Delete old entries if your application doesn’t need them
+* Archive long-lived data to another collection
+* Rebuild indexes periodically
+* Limit history for ephemeral overlays
+
+Retention periods depend on the app:
+
+* **Convo Messenger:** often stores all messages indefinitely
+* **Task-based overlays:** may delete resolved items after 30–90 days
+
+---
+
+## 6. Summary of Recommendations
+
+| Area            | Recommendation                                                   |
+| --------------- | ---------------------------------------------------------------- |
+| Required Fields | Always store `txid`, `outputIndex`, and protocol ID.             |
+| Metadata        | Store only small, queryable fields (e.g., `threadId`, `sender`). |
+| Large Data      | Use UHRP references instead of storing raw data.                 |
+| Queries         | Use selective, indexed fields.                                   |
+| Pagination      | Always include `skip` and `limit` in list endpoints.             |
+| Topic Manager   | Validate and parse overlay-specific outputs only.                |
+| Retention       | Prune or archive data as needed.                                 |
+
+---
+
+Next file: [`02-query-performance.md`](./02-query-performance.md) — how to design and optimize overlay queries.