@bsv/overlay 2.2.1 → 2.3.0

package/docs/concepts/03-database-monitoring.md

@@ -1,211 +0,0 @@
-# Overlay Database Monitoring and Debugging
-
-[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)
-
-**Navigation:** [Overview](./00-overview.md) | [Best Practices](./01-best-practices.md) | [Query Performance](./02-query-performance.md) | [Database Monitoring](./03-database-monitoring.md) | [Pagination Example](./04-pagination-example.md) | [Recommendations Summary](./05-recommendations-summary.md)
-
----
-
-## Overview
-
-Monitoring and debugging overlay databases helps ensure reliability, correctness, and predictable performance.
-As overlays grow, developers benefit from having ways to inspect what their Lookup Service is storing and validate that Topic Manager admission logic is working as intended.
-
-This document outlines optional strategies for:
-
-* Providing **safe, read-only** inspection of Lookup data
-* Building **debugging workflows** during overlay development
-* Tracking **basic performance and health metrics**
-* Ensuring correctness without exposing sensitive data or modifying overlay protocols
-
-These recommendations are intended for **overlay developers**, not system administrators of BSVA infrastructure.
-
----
-
-## 1. Goals
-
-Overlay authors may want to:
-
-* Verify what data is stored after Topic Manager admission
-* Inspect the parsed documents produced by PushDrop decoding
-* Measure lookup latency or query efficiency
-* Debug message ingestion, reactions, or thread activity during development
-* Provide trusted team members with read-only visibility in test or staging environments
-
-All monitoring discussed here is optional and applies only to the overlay developer’s own deployment.
-
----
-
-## 2. Read-Only Access for Debugging
-
-### 2.1 Purpose
-
-Read-only visibility is useful during development or internal debugging because it allows developers to:
-
-* Confirm that the Lookup Service stored the expected metadata
-* Inspect whether fields such as `threadId`, `createdAt`, or `parentMessageId` were parsed correctly
-* Understand why a transaction may not appear in query results
-
-### 2.2 Implementation Options
-
-Overlay authors can choose from several approaches depending on their internal workflow.
-
-#### Option A: MongoDB Read-Only Role
-
-MongoDB allows creation of roles that grant read-only access to specific collections:
-
-```js
-db.createUser({
-  user: "overlayReader",
-  pwd: "strongpassword",
-  roles: [
-    { role: "read", db: "overlaydb" }
-  ]
-});
-```
-
-This is useful for:
-
-* Internal developers
-* Debugging in staging environments
-* Letting trusted team members inspect lookup data
-
-#### Option B: Internal Read-Only Endpoint
-
-Some teams choose to expose a controlled debug route during development:
-
-```typescript
-router.get('/debug/messages', async (req, res) => {
-  const threadId = req.query.threadId;
-  const results = await storage.listThreadMessages(threadId, 0, 25);
-  res.json(results);
-});
-```
-
-**Important:**
-Such endpoints must not be exposed publicly. They should be protected with:
-
-* API keys
-* Admin tokens
-* IP allowlists
-* Or disabled entirely in production
-
-This is entirely optional and depends on each developer’s debugging workflow.
-
----
-
-## 3. Health and Performance Metrics
-
-These metrics are optional but useful during development or debugging.
-
-### 3.1 Basic Metrics
-
-Overlay authors may choose to track:
-
-* Collection size and document count
-* Index size and index efficiency
-* Average query execution time for common lookups
-* Latency from Topic Manager admission to Lookup write
-* Count of decode or parse failures
-
-These can be logged periodically or exported to monitoring tools like Prometheus or Grafana if desired.
-
-Example minimal metric:
-
-```typescript
-app.get('/metrics', async (req, res) => {
-  const stats = await db.collection('convoMessages').stats();
-  res.send(`convo_messages_bytes ${stats.size}\n`);
-});
-```
-
-### 3.2 Transaction Admission Tracking
-
-During development, Topic Managers can log admission decisions:
-
-```typescript
-console.log(`[tm_convo] admitted output ${txid}:${vout}`);
-```
-
-This helps correlate:
-
-* The BEEF transaction
-* Decoded PushDrop fields
-* The resulting Lookup Service document
-
-This logging is optional and purely for debugging the overlay’s own ingestion logic.
-
----
-
-## 4. Debugging Workflow Example
-
-A typical overlay debugging workflow might look like:
-
-1. Identify the txid of a message that should appear in the UI
-2. Query the Lookup data (using read-only DB access or internal endpoint)
-3. Verify that the Topic Manager correctly decoded the PushDrop fields
-4. Ensure Lookups return expected results (e.g., thread messages, replies, reactions)
-5. If performance issues appear, check index usage via `explain("executionStats")`
-6. If admission issues appear, review Topic Manager logs
-
-This workflow helps developers confirm correctness without modifying protocol-level logic.
-
----
-
-## 5. Optional Tools and Enhancements
-
-These tools are optional and useful only in development environments.
-
-### 5.1 Overlay Health Dashboard
-
-A lightweight web interface integrated into Overlay Express that would:
-
-* Display real-time query performance (e.g., avg lookup latency)
-* Show admission rates per Topic Manager
-* Track DB growth and index efficiency
-* Provide a query console for authorized users
-
-This is helpful for overlay teams during debugging or feature development.
-
-### 5.2 Integration With Existing Mongo Tooling
-
-Developers can use existing Mongo tools with read-only credentials:
-
-* Mongo Express
-* MongoDB Compass
-* Atlas Monitoring
-
-This allows safe browsing of Lookup documents without risk of modification.
-
----
-
-## 6. Security Considerations
-
-To avoid security risks:
-
-* Restrict read-only access to internal team members
-* Do not expose debugging endpoints publicly
-* Use authentication (API keys, tokens) for any admin routes
-* Avoid logging user data such as:
-  * CurvePoint headers
-  * Encrypted payloads
-  * Identity certificates
-  * Private metadata
-
-Logs should focus on metadata only, such as txid, threadId, and timestamps.
-
----
-
-## 7. Summary of Recommendations
-
-| Area        | Recommendation                                                     |
-| ----------- | ------------------------------------------------------------------ |
-| Read Access | Provide secure read-only access for overlay and CARS admins.       |
-| Metrics     | Track query latency, DB size, and failed transactions.             |
-| Logging     | Record admission flow from Topic Manager to Lookup.                |
-| Tools       | Use Mongo Express or Compass with read-only roles.                 |
-| Security    | Restrict debugging endpoints and sanitize data.                    |
-
----
-
-Next file: [`04-pagination-example.md`](./04-pagination-example.md) — Demonstrates performance improvements from pagination in Convo Messenger overlay queries.

package/docs/concepts/04-pagination-example.md

@@ -1,186 +0,0 @@
-# Pagination in Overlay Queries: Convo Messenger Example
-
-[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)
-
-**Navigation:** [Overview](./00-overview.md) | [Best Practices](./01-best-practices.md) | [Query Performance](./02-query-performance.md) | [Database Monitoring](./03-database-monitoring.md) | [Pagination Example](./04-pagination-example.md) | [Recommendations Summary](./05-recommendations-summary.md)
-
----
-
-## Overview
-
-Pagination is one of the simplest and most effective techniques for ensuring overlay lookup performance remains stable as datasets grow.
-Without pagination, queries that return lists—messages, reactions, or other records—can become increasingly slow and resource-intensive.
-
-This document illustrates why pagination matters and uses Convo Messenger as a real example of how it improves performance.
-It does not prescribe a required pattern or mandate changes; it simply shows how one overlay benefits from predictable, bounded queries.
-
----
-
-## 1. Why Pagination Matters
-
-Overlay datasets grow continuously. Without pagination:
-
-* A lookup endpoint may return thousands of documents at once
-* Memory usage spikes on both the server and the client
-* Query latency increases as more documents accumulate
-* Sorting happens in memory instead of using indexes
-
-Pagination ensures each lookup returns only a small, fixed-size window of results.
-
-**Benefits**
-
-* Consistent performance regardless of dataset size
-* Lower memory pressure
-* Index-friendly access patterns
-* Improved UX in applications that display messages or lists
-
----
-
-## 2. How Convo Messenger Uses Pagination (Example)
-
-Convo Messenger added pagination to all Lookup queries that return lists, including:
-
-* listThreadMessages
-* listLatestMessages
-* listReplies
-* listThreadReactions
-
-Below is one example from Convo’s Lookup Service showing how `skip` and `limit` values are interpreted when provided:
-
-```typescript
-if (query.type === 'listThreadMessages') {
-  const threadId = query.threadId ?? query.value?.threadId;
-  if (!threadId) throw new Error("threadId required");
-
-  const skip = query.skip ?? query.value?.skip ?? 0;
-  const limit = query.limit ?? query.value?.limit ?? 50;
-
-  const messages = await this.storage.listThreadMessages(threadId, skip, limit);
-  return this.formatAsLookupAnswers(messages);
-}
-```
-
-**Storage Layer Example (Convo)**
-
-Convo’s storage layer performs an indexed and paginated fetch:
-
-```typescript
-async listThreadMessages(threadId: string, skip = 0, limit = 50) {
-  const results = await this.messages
-    .aggregate([
-      { $match: { threadId } },
-      { $sort: { createdAt: -1 } },   // newest → oldest (indexed)
-      { $skip: skip },
-      { $limit: limit }
-    ])
-    .toArray();
-
-  return results.reverse(); // oldest → newest for UI ordering
-}
-```
-
-This matches Convo’s existing index:
-
-```typescript
-db.convoMessages.createIndex({ threadId: 1, createdAt: -1 });
-```
-
-**Client-Side Integration**
-
-In the frontend (React), pagination can be managed by maintaining a `page` or `offset` variable:
-
-```typescript
-const [messages, setMessages] = useState([]);
-const [page, setPage] = useState(0);
-const pageSize = 50;
-
-async function loadNextPage() {
-  const result = await lookup.query({
-    service: 'convo_lookup',
-    query: {
-      type: 'listThreadMessages',
-      threadId,
-      skip: page * pageSize,
-      limit: pageSize,
-    }
-  });
-
-  setMessages([...messages, ...result]);
-  setPage(page + 1);
-}
-```
-
-This simple approach allows seamless infinite scrolling or “Load More” functionality.
-
-Again, this is not a prescription—it simply demonstrates how Convo uses indexed pagination to maintain predictable performance.
-
----
-
-## 3. Performance Impact (Convo Example)
-
-After adding pagination to its lookup queries, Convo observed substantial performance improvements.
-
-| Metric                     |  Before Pagination |       After Pagination |
-| -------------------------- | -----------------: | ---------------------: |
-| Average Query Time         |            ~420 ms |            **< 70 ms** |
-| Memory Usage (per request) |             ~75 MB |             **< 8 MB** |
-| CPU Load (peak)            |          Very High |         **Much Lower** |
-| UI Latency                 | Noticeable stutter | **Instantaneous load** |
-
-**Takeaways**
-* Avoiding unbounded queries dramatically stabilizes performance
-* Server resource usage becomes predictable
-* Larger threads no longer degrade lookup time
-* User experience improves without changing overlay protocols
-
----
-
-## 4. Developer Notes (General Guidance)
-
-### 4.1 Recommended Defaults
-
-| Parameter | Recommended Value   | Notes                                                      |
-| --------- | ------------------- | ---------------------------------------------------------- |
-| `limit`   | 50                  | Reasonable for message lists; tweak based on dataset size. |
-| `skip`    | 0                   | Always provide, even if zero.                              |
-| sort    | `{ createdAt: -1 }` | Ensures chronological consistency.                         |
-
-### 4.2 Index Alignment
-
-For overlays that paginate ordered data, an index on the sort field is essential:
-
-```js
-{ threadId: 1, createdAt: -1 }
-```
-This avoids expensive in-memory sorts.
-
-### 4.3 Cursor-Based Pagination (Optional)
-
-Some overlays may use cursor-based pagination (e.g., createdAt > X) for large-scale workloads.
----
-
-## 5. Visual Results (Convo Example)
-
-Pagination allowed Convo to:
-* Avoid returning entire threads at once
-* Use MongoDB indexes effectively
-* Keep lookup latency stable as message volume grew
-* Implement infinite scroll UX without heavy data loads
-
-This demonstrates how pagination helps overlays scale smoothly without requiring complicated architecture.
-
----
-
-## 6. Recommendations Summary
-
-| Area               | Recommendation                                       |
-| ------------------ | ---------------------------------------------------- |
-| Implementation     | Always bound list queries with pagination parameters |
-| Client Integration | Incremental loading improves UX and stability        |
-| Indexing           | Align indexes with sort and filter fields            |
-| Optimization       | Cursor-based pagination can help for very large sets |
-| Benchmarking       | Measure latency and memory before/after deployment   |
-
----
-
-Next file: [`05-recommendations-summary.md`](./05-recommendations-summary.md) — Summarizes all proposed overlay improvements for BSVA documentation and Overlay Express feature suggestions.

package/docs/concepts/05-recommendations-summary.md

@@ -1,158 +0,0 @@
-# Overlay Recommendations Summary
-
-[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)
-
-**Navigation:** [Overview](./00-overview.md) | [Best Practices](./01-best-practices.md) | [Query Performance](./02-query-performance.md) | [Database Monitoring](./03-database-monitoring.md) | [Pagination Example](./04-pagination-example.md) | [Recommendations Summary](./05-recommendations-summary.md)
-
----
-
-## Overview
-
-This document summarizes the key recommendations from the overlay optimization series.
-These best practices help developers build overlays that remain efficient, stable, and easy to debug as datasets grow.
-The recommendations also highlight future opportunities for improvements to Overlay Express and related tooling.
-
-This summary is written for overlay authors and maintainers. It does not prescribe protocol-level changes.
-
----
-
-## 1. General Design Recommendations
-
-| Area                         | Recommendation                 | Description                                                                                                                                      |
-| ---------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------ |
-| **Lookup Data Structure**    | Store only indexable metadata  | Keep Lookup Services lightweight by storing only the fields needed for indexing and filtering. Store large data (media, payloads) externally.    |
-| **Topic Manager Validation** | Enforce schema and idempotency | Ensure each admitted transaction matches the expected structure and avoid duplicate writes.                                                      |
-| **Retention Policy**         | Implement periodic cleanup     | Schedule jobs to prune stale data and rebuild indexes to maintain predictable performance.                                                       |
-| **Security**                 | Restrict admin/debug access    | Provide safe read-only views only to trusted admins (e.g., CARS admins or overlay maintainers).                                                  |
-
----
-
-## 2. Query and Indexing Recommendations
-
-### 2.1 Query Design
-
-* Always filter on indexed fields (e.g., `threadId`, `parentMessageId`, `createdAt`).
-* Avoid unbounded queries (`find({})`, no limit, no filter).
-* Do not use regex or `$text` queries—they prevent index usage.
-* Prefer range queries `($gte, $lte)` when filtering by time.
-
-### 2.2 Indexing Strategy
-
-* Add indexes that match your overlay’s most common query patterns:
-    * Example from Convo: `{ threadId: 1, createdAt: -1 }`
-    * Example for replies: `{ parentMessageId: 1, createdAt: -1 }`
-* Use MongoDB’s explain("executionStats") to confirm index usage.
-* Rebuild indexes periodically for overlays that experience frequent writes/deletes.
-
-### 2.3 Pagination
-
-* Implement skip/limit pagination on all list endpoints.
-* Align pagination with indexed sorting to avoid in-memory sorts.
-* For large-scale overlays, consider cursor-based pagination (e.g., createdAt > X).
-
----
-
-## 3. Monitoring and Debugging
-
-### 3.1 Read-Only Admin Access
-
-Provide safe ways for overlay and CARS admins to inspect stored data:
-* Read-only database credentials
-* Or a protected REST endpoint for paginated inspection
-
-These allow debugging without risking data mutation.
-
-### 3.2 Health Metrics
-
-Track operational metrics such as:
-* Query latency (per lookup type)
-* Failed or rejected transactions
-* Topic Manager ingestion delay
-* Database and index size
-* Slow query counts
-
-Visualize these with Prometheus, Grafana, or MongoDB tools.
-
-### 3.3 Logging Standards
-
-* Log Topic Manager admissions and Lookup Service query timings.
-* Use consistent log tags (e.g., `[tm_convo]`, `[ls_market]`, `[overlay]`).
-* Avoid logging sensitive or encrypted data.
-
----
-
-## 4. Performance Enhancements Verified in Convo Messenger
-
-| Optimization                      | Result                                        |
-| --------------------------------- | --------------------------------------------- |
-| Added pagination (`skip`/`limit`) | Reduced avg query time from 420ms → 70ms      |
-| Indexed `(threadId, createdAt)`   | Reduced CPU load and consistent scaling       |
-| Range Based queries               | More predictable performance                  |
-| MongoDB profiler + timing logs    | Identified slow or unindexed queries          |
-
-These tests confirm that even small backend changes yield **major end-user improvements** in message loading and overlay stability.
-
----
-
-## 5. Proposed Overlay Express Enhancements
-
-### 5.1 Built-In Query Monitor (Future Proposal)
-
-A lightweight module that tracks and displays:
-* Average execution time per query type
-* Query counts
-* Slow query alerts
-* Index usage indicators
-
-### 5.2 Health Endpoint Standardization
-
-A recommended addition:
-
-```bash
-/health    → basic liveness  
-/metrics   → Prometheus-compatible performance stats  
-```
-
-### 5.3 Admin Console Integration
-
-A browser-based console for authorized users to:
-* Inspect Lookup data (read-only)
-* View recent admissions
-* Inspect slow queries
-* Monitor index statistics
-
-### 5.4 SDK Convenience Utilities
-
-Potential optional helpers:
-* `lookup.paginate()` wrapper
-* Standardized pagination response format
-* Typed client helpers for common query shapes
-
-These features would reduce boilerplate for overlay developers.
----
-
-## 6. Next Steps for BSVA Integration
-
-1. Add these best practices to BSVA documentation and onboarding materials.
-2. Update example overlays (e.g., in Metanet Academy) to demonstrate pagination and indexing.
-3. Create an example overlay dashboard (Prometheus/Grafana) for developers.
-4. Review potential Overlay Express enhancements with maintainers.
-
----
-
-## 7. Summary Table
-
-| Category        | Key Action                      | Impact                                  |
-| --------------- | ------------------------------- | --------------------------------------- |
-| Lookup Design   | Store metadata only             | Lower storage load and faster queries   |
-| Indexing        | Add indexes based on query usage| Lower latency and predictable behavior  |
-| Pagination      | Apply universally               | Stable performance as data grows        |
-| Monitoring      | Add metrics + profiling         | Early detection of issues               |
-| Admin Tools     | Provide safe, read-only access  | Easier debugging and transparency       |
-| Overlay Express | Consider Query Monitor tooling  | Unified visibility across overlays      |
-
----
-
-**Conclusion:**
-These recommendations form a solid baseline for building fast, scalable, and maintainable overlays.
-They are intentionally lightweight and compatible with existing designs—no protocol changes required.

package/docs/concepts/README.md

@@ -1,20 +0,0 @@
-# Conceptual Topics
-
-[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)
-
----
-
-These documents cover high-level conceptual information that will augment developers' understanding of the code-level Overlay Services Engine documentation:
-
-## Available Topics
-
-- [Overview](./00-overview.md) — Introduction to Overlay Use and Optimization in the BSV Ecosystem
-- [Best Practices](./01-best-practices.md) — Best practices for overlay design and implementation
-- [Query Performance](./02-query-performance.md) — Query optimization and performance considerations
-- [Database Monitoring](./03-database-monitoring.md) — Monitoring and debugging overlays
-- [Pagination Example](./04-pagination-example.md) — Pagination and practical examples
-- [Recommendations Summary](./05-recommendations-summary.md) — Quick reference of best practices
-
----
-
-[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](./README.md) | [📖 Examples](../examples/README.md) | [⚙️ Internal](../internal/README.md)

package/docs/examples/README.md

@@ -1,15 +0,0 @@
-# Examples
-
-[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](../concepts/README.md) | [📖 Examples](./README.md) | [⚙️ Internal](../internal/README.md)
-
----
-
-Here, you will find documentation for common example usages of the Overlay Services Engine.
-
-## Available Examples
-
-- [Getting Started (WIP)](./gs-wip.md) — Introduction and getting started guide
-
----
-
-[🏠 Home](../README.md) | [📚 API](../API.md) | [💡 Concepts](../concepts/README.md) | [📖 Examples](./README.md) | [⚙️ Internal](../internal/README.md)