@bsv/overlay 0.6.0 → 0.6.1

package/docs/concepts/04-pagination-example.md

@@ -0,0 +1,186 @@
+# Pagination in Overlay Queries: Convo Messenger Example
+
+[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)
+
+**Navigation:** [Overview](./00-overview.md) | [Best Practices](./01-best-practices.md) | [Query Performance](./02-query-performance.md) | [Database Monitoring](./03-database-monitoring.md) | [Pagination Example](./04-pagination-example.md) | [Recommendations Summary](./05-recommendations-summary.md)
+
+---
+
+## Overview
+
+Pagination is one of the simplest and most effective techniques for ensuring overlay lookup performance remains stable as datasets grow.
+Without pagination, queries that return lists—messages, reactions, or other records—can become increasingly slow and resource-intensive.
+
+This document illustrates why pagination matters and uses Convo Messenger as a real example of how it improves performance.
+It does not prescribe a required pattern or mandate changes; it simply shows how one overlay benefits from predictable, bounded queries.
+
+---
+
+## 1. Why Pagination Matters
+
+Overlay datasets grow continuously. Without pagination:
+
+* A lookup endpoint may return thousands of documents at once
+* Memory usage spikes on both the server and the client
+* Query latency increases as more documents accumulate
+* Sorting happens in memory instead of using indexes
+
+Pagination ensures each lookup returns only a small, fixed-size window of results.
+
+**Benefits**
+
+* Consistent performance regardless of dataset size
+* Lower memory pressure
+* Index-friendly access patterns
+* Improved UX in applications that display messages or lists
+
+---
+
+## 2. How Convo Messenger Uses Pagination (Example)
+
+Convo Messenger added pagination to all Lookup queries that return lists, including:
+
+* listThreadMessages
+* listLatestMessages
+* listReplies
+* listThreadReactions
+
+Below is one example from Convo’s Lookup Service showing how `skip` and `limit` values are interpreted when provided:
+
+```typescript
+if (query.type === 'listThreadMessages') {
+  const threadId = query.threadId ?? query.value?.threadId;
+  if (!threadId) throw new Error("threadId required");
+
+  const skip = query.skip ?? query.value?.skip ?? 0;
+  const limit = query.limit ?? query.value?.limit ?? 50;
+
+  const messages = await this.storage.listThreadMessages(threadId, skip, limit);
+  return this.formatAsLookupAnswers(messages);
+}
+```
+
+**Storage Layer Example (Convo)**
+
+Convo’s storage layer performs an indexed and paginated fetch:
+
+```typescript
+async listThreadMessages(threadId: string, skip = 0, limit = 50) {
+  const results = await this.messages
+    .aggregate([
+      { $match: { threadId } },
+      { $sort: { createdAt: -1 } },   // newest → oldest (indexed)
+      { $skip: skip },
+      { $limit: limit }
+    ])
+    .toArray();
+
+  return results.reverse(); // oldest → newest for UI ordering
+}
+```
+
+This matches Convo’s existing index:
+
+```typescript
+db.convoMessages.createIndex({ threadId: 1, createdAt: -1 });
+```
+
+**Client-Side Integration**
+
+In the frontend (React), pagination can be managed by maintaining a `page` or `offset` variable:
+
+```typescript
+const [messages, setMessages] = useState([]);
+const [page, setPage] = useState(0);
+const pageSize = 50;
+
+async function loadNextPage() {
+  const result = await lookup.query({
+    service: 'convo_lookup',
+    query: {
+      type: 'listThreadMessages',
+      threadId,
+      skip: page * pageSize,
+      limit: pageSize,
+    }
+  });
+
+  setMessages([...messages, ...result]);
+  setPage(page + 1);
+}
+```
+
+This simple approach allows seamless infinite scrolling or “Load More” functionality.
+
+Again, this is not a prescription—it simply demonstrates how Convo uses indexed pagination to maintain predictable performance.
+
+---
+
+## 3. Performance Impact (Convo Example)
+
+After adding pagination to its lookup queries, Convo observed substantial performance improvements.
+
+| Metric                     |  Before Pagination |       After Pagination |
+| -------------------------- | -----------------: | ---------------------: |
+| Average Query Time         |            ~420 ms |            **< 70 ms** |
+| Memory Usage (per request) |             ~75 MB |             **< 8 MB** |
+| CPU Load (peak)            |          Very High |         **Much Lower** |
+| UI Latency                 | Noticeable stutter | **Instantaneous load** |
+
+**Takeaways**
+* Avoiding unbounded queries dramatically stabilizes performance
+* Server resource usage becomes predictable
+* Larger threads no longer degrade lookup time
+* User experience improves without changing overlay protocols
+
+---
+
+## 4. Developer Notes (General Guidance)
+
+### 4.1 Recommended Defaults
+
+| Parameter | Recommended Value   | Notes                                                      |
+| --------- | ------------------- | ---------------------------------------------------------- |
+| `limit`   | 50                  | Reasonable for message lists; tweak based on dataset size. |
+| `skip`    | 0                   | Always provide, even if zero.                              |
+| sort    | `{ createdAt: -1 }` | Ensures chronological consistency.                         |
+
+### 4.2 Index Alignment
+
+For overlays that paginate ordered data, an index on the sort field is essential:
+
+```js
+{ threadId: 1, createdAt: -1 }
+```
+This avoids expensive in-memory sorts.
+
+### 4.3 Cursor-Based Pagination (Optional)
+
+Some overlays may use cursor-based pagination (e.g., createdAt > X) for large-scale workloads.
+---
+
+## 5. Visual Results (Convo Example)
+
+Pagination allowed Convo to:
+* Avoid returning entire threads at once
+* Use MongoDB indexes effectively
+* Keep lookup latency stable as message volume grew
+* Implement infinite scroll UX without heavy data loads
+
+This demonstrates how pagination helps overlays scale smoothly without requiring complicated architecture.
+
+---
+
+## 6. Recommendations Summary
+
+| Area               | Recommendation                                       |
+| ------------------ | ---------------------------------------------------- |
+| Implementation     | Always bound list queries with pagination parameters |
+| Client Integration | Incremental loading improves UX and stability        |
+| Indexing           | Align indexes with sort and filter fields            |
+| Optimization       | Cursor-based pagination can help for very large sets |
+| Benchmarking       | Measure latency and memory before/after deployment   |
+
+---
+
+Next file: [`05-recommendations-summary.md`](./05-recommendations-summary.md) — Summarizes all proposed overlay improvements for BSVA documentation and Overlay Express feature suggestions.

package/docs/concepts/05-recommendations-summary.md

@@ -0,0 +1,158 @@
+# Overlay Recommendations Summary
+
+[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)
+
+**Navigation:** [Overview](./00-overview.md) | [Best Practices](./01-best-practices.md) | [Query Performance](./02-query-performance.md) | [Database Monitoring](./03-database-monitoring.md) | [Pagination Example](./04-pagination-example.md) | [Recommendations Summary](./05-recommendations-summary.md)
+
+---
+
+## Overview
+
+This document summarizes the key recommendations from the overlay optimization series.
+These best practices help developers build overlays that remain efficient, stable, and easy to debug as datasets grow.
+The recommendations also highlight future opportunities for improvements to Overlay Express and related tooling.
+
+This summary is written for overlay authors and maintainers. It does not prescribe protocol-level changes.
+
+---
+
+## 1. General Design Recommendations
+
+| Area                         | Recommendation                 | Description                                                                                                                                      |
+| ---------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
+| **Lookup Data Structure**    | Store only indexable metadata  | Keep Lookup Services lightweight by storing only the fields needed for indexing and filtering. Store large data (media, payloads) externally.    |
+| **Topic Manager Validation** | Enforce schema and idempotency | Ensure each admitted transaction matches the expected structure and avoid duplicate writes.                                                      |
+| **Retention Policy**         | Implement periodic cleanup     | Schedule jobs to prune stale data and rebuild indexes to maintain predictable performance.                                                       |
+| **Security**                 | Restrict admin/debug access    | Provide safe read-only views only to trusted admins (e.g., CARS admins or overlay maintainers).                                                  |
+
+---
+
+## 2. Query and Indexing Recommendations
+
+### 2.1 Query Design
+
+* Always filter on indexed fields (e.g., `threadId`, `parentMessageId`, `createdAt`).
+* Avoid unbounded queries (`find({})`, no limit, no filter).
+* Do not use regex or `$text` queries—they prevent index usage.
+* Prefer range queries `($gte, $lte)` when filtering by time.
+
+### 2.2 Indexing Strategy
+
+* Add indexes that match your overlay’s most common query patterns:
+    * Example from Convo: `{ threadId: 1, createdAt: -1 }`
+    * Example for replies: `{ parentMessageId: 1, createdAt: -1 }`
+* Use MongoDB’s explain("executionStats") to confirm index usage.
+* Rebuild indexes periodically for overlays that experience frequent writes/deletes.
+
+### 2.3 Pagination
+
+* Implement skip/limit pagination on all list endpoints.
+* Align pagination with indexed sorting to avoid in-memory sorts.
+* For large-scale overlays, consider cursor-based pagination (e.g., createdAt > X).
+
+---
+
+## 3. Monitoring and Debugging
+
+### 3.1 Read-Only Admin Access
+
+Provide safe ways for overlay and CARS admins to inspect stored data:
+* Read-only database credentials
+* Or a protected REST endpoint for paginated inspection
+
+These allow debugging without risking data mutation.
+
+### 3.2 Health Metrics
+
+Track operational metrics such as:
+* Query latency (per lookup type)
+* Failed or rejected transactions
+* Topic Manager ingestion delay
+* Database and index size
+* Slow query counts
+
+Visualize these with Prometheus, Grafana, or MongoDB tools.
+
+### 3.3 Logging Standards
+
+* Log Topic Manager admissions and Lookup Service query timings.
+* Use consistent log tags (e.g., `[tm_convo]`, `[ls_market]`, `[overlay]`).
+* Avoid logging sensitive or encrypted data.
+
+---
+
+## 4. Performance Enhancements Verified in Convo Messenger
+
+| Optimization                      | Result                                        |
+| --------------------------------- | --------------------------------------------- |
+| Added pagination (`skip`/`limit`) | Reduced avg query time from 420ms → 70ms      |
+| Indexed `(threadId, createdAt)`   | Reduced CPU load and consistent scaling       |
+| Range Based queries               | More predictable performance                  |
+| MongoDB profiler + timing logs    | Identified slow or unindexed queries          |
+
+These tests confirm that even small backend changes yield **major end-user improvements** in message loading and overlay stability.
+
+---
+
+## 5. Proposed Overlay Express Enhancements
+
+### 5.1 Built-In Query Monitor (Future Proposal)
+
+A lightweight module that tracks and displays:
+* Average execution time per query type
+* Query counts
+* Slow query alerts
+* Index usage indicators
+
+### 5.2 Health Endpoint Standardization
+
+A recommended addition:
+
+```bash
+/health    → basic liveness  
+/metrics   → Prometheus-compatible performance stats  
+```
+
+### 5.3 Admin Console Integration
+
+A browser-based console for authorized users to:
+* Inspect Lookup data (read-only)
+* View recent admissions
+* Inspect slow queries
+* Monitor index statistics
+
+### 5.4 SDK Convenience Utilities
+
+Potential optional helpers:
+* `lookup.paginate()` wrapper
+* Standardized pagination response format
+* Typed client helpers for common query shapes
+
+These features would reduce boilerplate for overlay developers.
+---
+
+## 6. Next Steps for BSVA Integration
+
+1. Add these best practices to BSVA documentation and onboarding materials.
+2. Update example overlays (e.g., in Metanet Academy) to demonstrate pagination and indexing.
+3. Create an example overlay dashboard (Prometheus/Grafana) for developers.
+4. Review potential Overlay Express enhancements with maintainers.
+
+---
+
+## 7. Summary Table
+
+| Category        | Key Action                      | Impact                                  |
+| --------------- | ------------------------------- | --------------------------------------- |
+| Lookup Design   | Store metadata only             | Lower storage load and faster queries   |
+| Indexing        | Add indexes based on query usage| Lower latency and predictable behavior  |
+| Pagination      | Apply universally               | Stable performance as data grows        |
+| Monitoring      | Add metrics + profiling         | Early detection of issues               |
+| Admin Tools     | Provide safe, read-only access  | Easier debugging and transparency       |
+| Overlay Express | Consider Query Monitor tooling  | Unified visibility across overlays      |
+
+---
+
+**Conclusion:**
+These recommendations form a solid baseline for building fast, scalable, and maintainable overlays.
+They are intentionally lightweight and compatible with existing designs—no protocol changes required.

package/docs/concepts/README.md

@@ -1,4 +1,20 @@
 # Conceptual Topics
 
+[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)
+
+---
+
 These documents cover high-level conceptual information that will augment developers' understanding of the code-level Overlay Services Engine documentation:
 
+## Available Topics
+
+- [Overview](./00-overview.md) — Introduction to Overlay Use and Optimization in the BSV Ecosystem
+- [Best Practices](./01-best-practices.md) — Best practices for overlay design and implementation
+- [Query Performance](./02-query-performance.md) — Query optimization and performance considerations
+- [Database Monitoring](./03-database-monitoring.md) — Monitoring and debugging overlays
+- [Pagination Example](./04-pagination-example.md) — Pagination and practical examples
+- [Recommendations Summary](./05-recommendations-summary.md) — Quick reference of best practices
+
+---
+
+[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)

package/docs/examples/README.md

@@ -1,5 +1,15 @@
 # Examples
 
+[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](../concepts/README.md) | [📖 Examples](./README.md) | [⚙️ Internal](../internal/README.md)
+
+---
+
 Here, you will find documentation for common example usages of the Overlay Services Engine.
 
-- [Getting Started (WIP)](./gs-wip.md)
+## Available Examples
+
+- [Getting Started (WIP)](./gs-wip.md) — Introduction and getting started guide
+
+---
+
+[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](../concepts/README.md) | [📖 Examples](./README.md) | [⚙️ Internal](../internal/README.md)

package/docs/examples/gs-wip.md

@@ -1,4 +1,10 @@
-### Introduction to BSV Overlay Services Engine
+# Getting Started with BSV Overlay Services Engine
+
+[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](../concepts/README.md) | [📖 Examples](./README.md) | [⚙️ Internal](../internal/README.md)
+
+---
+
+## Introduction to BSV Overlay Services Engine
 
 The BSV Overlay Services Engine is designed to process transactions and manage data within a blockchain-based system, specifically targeting the Bitcoin SV (BSV) blockchain. It integrates various components such as Topic Managers, Lookup Services, Storage, and Chain Tracker to provide a robust environment for managing transaction data and overlay services.
 

package/docs/internal/README.md

@@ -1,4 +1,11 @@
 # Internals
 
-These documents cover the internal components of the Overlay Services engine. Generally, these are only useful for creating customized deployments:
+[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](../concepts/README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](./README.md)
 
+---
+
+These documents cover the internal components of the Overlay Services engine. Generally, these are only useful for creating customized deployments.
+
+---
+
+[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](../concepts/README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](./README.md)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bsv/overlay",
-  "version": "0.6.0",
+  "version": "0.6.1",
   "type": "module",
   "description": "BSV Blockchain Overlay Services Engine",
   "main": "dist/cjs/mod.js",
@@ -73,8 +73,8 @@
     "typescript": "^5.2.2"
   },
   "dependencies": {
-    "@bsv/gasp": "^1.2.0",
-    "@bsv/sdk": "^1.9.1",
+    "@bsv/gasp": "^1.2.2",
+    "@bsv/sdk": "^2.0.0",
     "knex": "^3.1.0"
   }
 }