@bsv/overlay 2.2.0 → 2.2.7

package/docs/concepts/01-best-practices.md

@@ -1,202 +0,0 @@
-# Overlay Best Practices for Developers
-
-[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)
-
-**Navigation:** [Overview](./00-overview.md) | [Best Practices](./01-best-practices.md) | [Query Performance](./02-query-performance.md) | [Database Monitoring](./03-database-monitoring.md) | [Pagination Example](./04-pagination-example.md) | [Recommendations Summary](./05-recommendations-summary.md)
-
----
-
-## Overview
-
-This document provides **best practices for designing and maintaining overlay services** in the BSV ecosystem. It focuses on the practical concerns of developers who are building their **own overlays**—not on BSVA internals or modifications to Overlay Express.
-
-The goal is to help developers make consistent and scalable decisions when implementing **Topic Managers**, **Lookup Services**, and associated **MongoDB storage layers**. Examples reference **Convo Messenger**, but the principles apply to all overlay-based systems.
-
----
-
-## 1. Data Storage Principles
-
-Overlay Lookup Services should store **small, structured metadata**, not entire payloads. The goal is fast, predictable querying.
-
-### 1.1 Required Fields for Any Overlay
-
-Every Lookup Service must store enough information to uniquely reference an on-chain output. This includes:
-
-* **`txid`** — transaction ID containing the output
-* **`outputIndex`** — index of the admitted output
-* **`protocol`** — identifies which overlay protocol this entry belongs to
-
-These fields ensure that applications can always locate the exact on-chain output referenced by the overlay.
-
-### 1.2 Common Metadata Fields
-
-In addition to required fields, overlays typically store small metadata extracted from PushDrop fields or derived from application logic.
-
-Examples:
-
-* **`timestamp` / `createdAt`** — when the message or record was created
-* **`sender`** — sender's pubkey or DID
-* **`threadId`** — group/thread identifier
-* **`parentMessageId`** (optional) — for reply messages
-* **reaction fields** (when applicable)
-
-Example (Convo Messenger):
-
-```ts
-{
-  txid: "abc123...",
-  outputIndex: 0,
-  protocol: [2, "convo"],
-  threadId: "thread-xyz",
-  sender: "035a1b...",
-  createdAt: 1730000000000,
-  encryptedPayload: [...], // ciphertext array
-  header: [...],           // CurvePoint header
-  uniqueId: "optional-value",
-  parentMessageId: "optional-parent-txid"
-}
-```
-
-**Note:** Convo stores *encryptedPayload* and *header* because these remain small arrays needed by the client. Overlays must avoid storing anything large (e.g., full media files).
-
-### 1.3 When to Use UHRP
-
-If an overlay needs to handle large data (text bodies, images, files), Lookup Services should **only store a UHRP reference**:
-
-* Never store full file contents
-* Never store large plaintext or ciphertext blobs
-* Let the client fetch large data from UHRP when needed
-
-Lookup Services should remain lightweight.
-
----
-
-## 2. What Lookup Services Should *Not* Store
-
-To maintain performance and predictable scaling, Lookup Services must avoid:
-
-* Full PushDrop payloads or full scripts
-* Large encrypted or plaintext message bodies
-* Media files or binary attachments
-* Identity certificates
-* Redundant or duplicate on-chain data
-
-**Rule of thumb:**
-If it cannot be indexed efficiently, it does not belong in the Lookup DB.
-
----
-
-## 3. Structuring Lookup Queries
-
-### 3.1 Use Indexed, Selective Fields
-
-All queries should filter using indexed, selective fields such as:
-
-* `threadId`
-* `sender`
-* `parentMessageId`
-* `createdAt`
-
-Examples from Convo’s actual Mongo indexes:
-
-```js
-db.convoMessages.createIndex({ threadId: 1 })
-db.convoMessages.createIndex({ parentMessageId: 1 })
-db.convoMessages.createIndex({ threadId: 1, createdAt: -1 })
-db.convoReactions.createIndex({ threadId: 1 })
-```
-
-### 3.2 Use Pagination Everywhere
-
-List endpoints must support:
-
-* **`skip`** — starting offset
-* **`limit`** — number of items
-
-Typical defaults:
-
-* `limit = 50`
-* `skip = 0`
-
-Example (Convo):
-
-```ts
-const messages = await this.storage.listThreadMessages(threadId, skip, limit)
-```
-
-### 3.3 Avoid Expensive Query Patterns
-
-Do **not**:
-
-* Perform unbounded collection scans
-* Query across all protocols
-* Sort without an index
-* Use `$regex` or `$text` filters
-
-If complex queries are needed, break them into:
-
-* Indexed filters
-* Bounded pagination
-* Incremental lookups
-
----
-
-## 4. Topic Manager Best Practices
-
-Topic Managers determine which outputs belong to your overlay. They must validate and parse data correctly.
-
-Best practices:
-
-1. **Validate admissible outputs** using PushDrop decoding.
-2. **Store only necessary fields** extracted from PushDrop.
-3. **Reject malformed or irrelevant outputs** early.
-4. **Handle duplicate admissions** cleanly.
-5. **Avoid heavy computation** inside the Topic Manager.
-
-Example (Convo):
-
-```ts
-const decoded = PushDrop.decode(output.lockingScript)
-const marker = Utils.toUTF8(fields[0])
-const protocol = Utils.toUTF8(fields[1])
-
-if (marker === 'convo' && protocol === 'tmconvo') {
-  admissibleOutputs.push(index)
-}
-```
-
----
-
-## 5. Data Retention and Cleanup
-
-Overlays accumulate data continuously. Without retention logic, performance may degrade.
-
-Common retention approaches:
-
-* Delete old entries if your application doesn’t need them
-* Archive long-lived data to another collection
-* Rebuild indexes periodically
-* Limit history for ephemeral overlays
-
-Retention periods depend on the app:
-
-* **Convo Messenger:** often stores all messages indefinitely
-* **Task-based overlays:** may delete resolved items after 30–90 days
-
----
-
-## 6. Summary of Recommendations
-
-| Area            | Recommendation                                                   |
-| --------------- | ---------------------------------------------------------------- |
-| Required Fields | Always store `txid`, `outputIndex`, and protocol ID.             |
-| Metadata        | Store only small, queryable fields (e.g., `threadId`, `sender`). |
-| Large Data      | Use UHRP references instead of storing raw data.                 |
-| Queries         | Use selective, indexed fields.                                   |
-| Pagination      | Always include `skip` and `limit` in list endpoints.             |
-| Topic Manager   | Validate and parse overlay-specific outputs only.                |
-| Retention       | Prune or archive data as needed.                                 |
-
----
-
-Next file: [`02-query-performance.md`](./02-query-performance.md) — how to design and optimize overlay queries.

package/docs/concepts/02-query-performance.md

@@ -1,345 +0,0 @@
-# Query Performance and Indexing for Overlays
-
-[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)
-
-**Navigation:** [Overview](./00-overview.md) | [Best Practices](./01-best-practices.md) | [Query Performance](./02-query-performance.md) | [Database Monitoring](./03-database-monitoring.md) | [Pagination Example](./04-pagination-example.md) | [Recommendations Summary](./05-recommendations-summary.md)
-
----
-
-## Overview
-
-Efficient query design is critical for scalable overlay performance. Lookup Services often serve thousands of queries per day—fetching messages, reactions, and thread activity—so poor indexing or unbounded queries can quickly degrade performance.
-
-This document explains how to design efficient queries for overlay Lookup Services, using real examples from Convo Messenger.
-
----
-
-## 1. Query Design Guidelines
-
-### 1.1 Use Indexed Fields for All Queries
-
-Indexes are critical for overlay scalability. Without them, MongoDB must perform full collection scans — a major performance bottleneck.
-
-**Always index fields used in query filters**, such as:
-
-* `threadId`
-* `sender`
-* `recipient`
-* `createdAt`
-
-Example:
-
-```js
-db.convoMessages.createIndex({ threadId: 1 });
-db.convoMessages.createIndex({ parentMessageId: 1 });
-db.convoMessages.createIndex({ threadId: 1, createdAt: -1 });
-db.convoReactions.createIndex({ threadId: 1 });
-```
-
-These fields match the filters used by:
-
-* `listThreadMessages`
-* `listReplies`
-* `listThreadReactions`
-* `countThreadMessages`
-* `countReplies`
-
-Rule:
-Design indexes to match the exact fields your Lookup queries use.
-
-### 1.2 Use Range Queries for Time-Based Lookups
-
-If you need to filter messages by time, use
-
-```typescript
-{ createdAt: { $gte: start, $lte: end } }
-```
-
-Example pattern (recommended if your overlay uses time-range queries):
-
-```typescript
-db.messages.find({
-  threadId,
-  createdAt: { $gte: startTime }
-})
-.sort({ createdAt: -1 })
-.limit(50)
-```
-
-This ensures MongoDB uses timestamp indexes efficiently.
-
-### 1.3 Combine Pagination With Indexed Sorts
-
-When paginating through sorted data, combine `skip` / `limit` with an indexed sort key.
-
-```typescript
-async listThreadMessages(threadId, skip, limit) {
-  return await this.messages
-    .aggregate([
-      { $match: { threadId } },
-      { $sort: { createdAt: -1 } },
-      { $skip: skip },
-      { $limit: limit }
-    ])
-    .toArray();
-}
-```
-
-Ensure the sort key (`createdAt`) matches an existing index to avoid in-memory sorting.
-
----
-
-## 2. Common Anti-Patterns
-
-| Problematic Pattern          | Description                                             | Recommended Fix                                     |
-| ---------------------------- | ------------------------------------------------------- | --------------------------------------------------- |
-| **Unindexed Filters**        | Querying on non-indexed fields causes full scans.       | Add composite indexes on frequent filter fields.    |
-| **Regex or `$text` queries** | Regex searches prevent index usage.                     | Use prefix matches or precomputed lowercase fields. |
-| **Large `$in` filters**      | `$in` with many values increases memory and CPU load.   | Batch requests or use intermediate caching.         |
-| **Unbounded queries**        | Returning unbounded lists consumes memory.              | Always apply limit.                                 |
-| **Sorting without an index** | Forces MongoDB to sort in memory.                       | Add compound index with your sort key.                 |
-
-Examples to avoid:
-
-```typescript
-// ❌ DO NOT DO THIS
-db.messages.find({}).sort({ createdAt: -1 });
-
-// ❌ Breaks index usage
-db.messages.find({ sender: { $regex: '^03' } });
-```
-
----
-
-## 3. MongoDB Indexing Best Practices
-
-### 3.1 Use Compound Indexes
-
-For overlays, most queries filter by a primary grouping field (such as threadId) and a secondary field (such as createdAt). Compound indexes drastically improve performance:
-
-```js
-// convoMessages collection
-db.convoMessages.createIndex({ threadId: 1 });
-db.convoMessages.createIndex({ parentMessageId: 1 });
-db.convoMessages.createIndex({ threadId: 1, createdAt: -1 });
-
-// convoReactions collection
-db.convoReactions.createIndex({ threadId: 1 });
-
-// convoThreads collection
-db.convoThreads.createIndex({ threadId: 1 }, { unique: true });
-```
-
-These support efficient implmentations of:
-
-**Messages in a thread**
-```typescript
-this.messages
-  .find({ threadId })
-  .sort({ createdAt: 1 });
-```
-
-**Replies to a message**
-```typescript
-this.messages
-  .find({ parentMessageId })
-  .sort({ createdAt: 1 });
-```
-
-**Latest messages across all threads**
-Aggregation pipeline:
-```typescript
-[
-  { $sort: { createdAt: -1 } },
-  { $group: {
-      _id: "$threadId",
-      threadId: { $first: "$threadId" },
-      txid: { $first: "$txid" },
-      outputIndex: { $first: "$outputIndex" },
-      sender: { $first: "$sender" },
-      header: { $first: "$header" },
-      encryptedPayload: { $first: "$encryptedPayload" },
-      createdAt: { $first: "$createdAt" },
-      threadName: { $first: "$threadName" },
-      parentMessageId: { $first: "$parentMessageId" },
-      uniqueId: { $first: "$uniqueId" }
-  }},
-  { $sort: { createdAt: -1 } },
-  { $skip: skip },
-  { $limit: limit }
-]
-```
-
-All supported by the existing indexes.
-
----
-
-### 3.2 Monitor Index Usage
-
-Use MongoDB’s `explain()` command to verify index utilization:
-
-Example (threads’ messages lookup):
-```js
-db.convoMessages
-  .find({ threadId: "abc123" })
-  .sort({ createdAt: -1 })
-  .explain("executionStats");
-```
-
-Key metrics to monitor:
-
-* `executionTimeMillis` - how long the query took
-* `totalDocsExamined` - number of documents scanned
-* `totalKeysExamined` - number of index entries scanned
-
-**Goal:**
-`totalDocsExamined` should be low (ideally equal to or close to limit) for all Convo queries.
-
-### 3.3 Rebuild Indexes Periodically
-
-Indexes can fragment over time, especially with:
-
-* frequent inserts
-* deletes (like messages deleted when spent)
-* updates
-
-Schedule index rebuilds during low-traffic periods:
-```js
-db.convoMessages.reIndex();
-db.convoReactions.reIndex();
-db.convoThreads.reIndex();
-```
-
----
-
-## 4. Measuring Query Performance
-
-### 4.1 Use Built-In Mongo Metrics
-
-MongoDB includes a lightweight profiler that records slow queries.
-This is the easiest way for overlay authors to discover inefficient lookups.
-
-Enable profiling for any query taking longer than 100 ms:
-
-```js
-db.setProfilingLevel(1, { slowms: 100 });
-```
-
-Slow query entries appear in the system.profile collection and can be inspected manually or exported to monitoring tools such as:
-
-* Prometheus
-* Grafana
-* ELK / OpenSearch
-* Custom dashboards
-
-What to look for:
-
-* High executionTimeMillis
-* High totalDocsExamined (index not used)
-* High totalKeysExamined (index scan too large)
-
-Correctly indexed Convo queries should examine very small numbers of documents.
-
-### 4.2 Add Overlay-Level Timing Logs
-
-Your Lookup Service can log query timing directly at the overlay level.
-This measures actual latency experienced by clients, not just database timings.
-
-Example (Convo):
-
-```typescript
-const start = performance.now();
-const messages = await this.storage.listThreadMessages(
-  threadId,
-  skip,
-  limit
-);
-console.log(
-  `[ls_convo] listThreadMessages(${threadId}) took ${performance.now() - start} ms`
-);
-```
-
-You can add similar timing logs to:
-
-* listThreadReactions
-* listReplies
-* listLatestMessages
-*countThreadMessages
-* countReplies
-
-This makes it easy to identify:
-
-* Query patterns that degrade at scale
-* Missed indexes
-* Inefficient aggregation pipelines
-
-Overlay-level logs are invaluable for practical debugging because they show the exact request→response timing seen by apps like Convo Messenger.
-
-### 4.3 Performance Visualization Tooling (Proposed)
-
-Although not required for overlay authors today, you can add your own simple visualizations.
-
-Potential enhancements include:
-
-* **Query Time Dashboard**
-
-    A small local dashboard showing average duration per lookup type (e.g., listThreadMessages, listReplies).
-
-* **Slow Query Warnings**
-
-    Log a warning if any Lookup query exceeds a threshold (e.g., 200 ms).
-
-* **Index Usage Reports**
-
-    A small script that uses explain("executionStats") on common queries and prints index efficiency.
-
-These tools can help team-level debugging (e.g., Convo, Tempo, MetaMarket), but they are not a requirement for overlay authors and should not be confused with BSVA-level monitoring.
-
----
-
-## 5. Practical Example: Convo Messenger
-
-Convo Messenger provides a useful real-world example of how overlay query performance improves when Lookup queries are designed around indexed fields and predictable patterns.
-
-Convo optimized its Lookup queries by:
-
-Indexing fields used in queries, including:
-
-- `threadId`
-- `parentMessageId`
-- `createdAt`
-- compound index: `{ threadId: 1, createdAt: -1 }`
-
-Using pagination everywhere, with sensible limits such as:
-
-- `50` for message lists
-- `100` for reaction lists
-
-Ensuring query patterns always match existing indexes, including:
-
-- filtering by `threadId`
-- time ordering by `createdAt`
-- grouping aggregation only after an indexed sort
-- avoiding unbounded fetches (`find({})`)
-
-
-These design choices ensure Convo’s Lookup Service remains stable and scales predictably, even as message volume grows.
-
-This example illustrates how overlays can maintain efficient performance without requiring complex systems—just well-designed indexes, selective filters, and consistent pagination.
-
----
-
-## 6. Recommendations Summary
-
-| Area         | Recommendation                                              |
-| ------------ | ----------------------------------------------------------- |
-| Query Design | Always use indexed filters; avoid regex and `$in` scans.    |
-| Indexing     | Add indexes that match your overlay’s query filters (e.g., `{ threadId, createdAt }`).    |
-| Pagination   | Combine with indexed sorts to avoid in-memory sorting.      |
-| Measurement  | Use MongoDB profiler and `explain('executionStats')` to detect slow queries.                  |
-| Tooling      | Explore adding an Overlay Query Monitor to Overlay Express. |
-| Logging      | Add timing logs in Lookup Services to measure real client-facing latency |
-
----
-
-Next file: [`03-database-monitoring.md`](./03-database-monitoring.md) — focuses on database health, read-only access for debugging, and workflows for CARS admins.