soroban-events 2.0.2 → 3.0.0

package/README.md

@@ -1,46 +1,61 @@
 # soroban-events
 
-Lightweight Soroban RPC event ingestion, pagination, retries, deduplication, and ScVal decoding for Node.js.
+Resilient, windowed event streaming and XDR decoding for Soroban RPC.
 
-Build Soroban dashboards, activity feeds, lightweight indexers, analytics tools, monitoring services, bots, and event-driven backends without implementing RPC pagination and event decoding yourself.
+`soroban-events` provides production-oriented primitives for retrieving, decoding, processing, storing, replaying, and exporting Soroban contract events.
 
 ## Features
 
-- Ledger-range windowing
-- RPC cursor pagination
-- Event deduplication
-- Rate-limit and transient retries
-- Soroban `ScVal` decoding
-- Large integer precision preservation
-- Contract event filtering
-- Recent-event lookup with `tail()`
-- Async event streaming with `stream()`
-- AbortSignal support
-
-## Install
+- Safe windowed retrieval for ledger ranges larger than the Soroban RPC range limit.
+- Cursor-based pagination.
+- Event-ID deduplication.
+- Unlimited internal pagination with `limit: null`.
+- Reliable ledger checkpointing.
+- Durable file checkpoints.
+- Memory and SQLite event stores.
+- Backfill and historical replay.
+- Backfill/live handoff protection.
+- Reorg detection and recovery.
+- RPC health and ledger-retention inspection.
+- Configurable RPC failover.
+- Circuit-breaker protection.
+- Adaptive rate-limit handling.
+- Exponential retry with jitter.
+- Event filtering and custom predicates.
+- Deterministic event querying.
+- Event processing pipelines.
+- JSONL and CSV export.
+- Streaming export.
+- Optional dead-letter queue primitives.
+- CLI commands for status, backfill, replay, and health inspection.
+- TypeScript declarations.
+- Packed npm-consumer verification.
+
+## Requirements
+
+Node.js 22.12.0 or newer.
+
+The runtime requirement follows the current Stellar SDK dependency requirements.
+
+## Installation
 
 ```bash
 npm install soroban-events
 ```
 
-Requires Node.js 20+.
-
-## Quick start
+## Basic usage
 
 ```js
-import { SorobanEventStreamer } from 'soroban-events';
+import { SorobanEventStreamer } from "soroban-events";
 
 const streamer = new SorobanEventStreamer(
-  'https://soroban-testnet.stellar.org'
+  "https://soroban-testnet.stellar.org"
 );
 
-const latest = await streamer.getLatestLedger();
-
 const events = await streamer.getEventsWindowed({
-  startLedger: latest - 100,
-  endLedger: latest,
-  filters: [{ type: 'contract' }],
-  limit: 10
+  startLedger: 1000000,
+  endLedger: 1010000,
+  limit: null
 });
 
 for (const event of events) {
@@ -48,183 +63,455 @@ for (const event of events) {
 }
 ```
 
-## Filter by contract
+## Latest ledger
 
 ```js
-const events = await streamer.getEventsWindowed({
-  startLedger: latest - 1000,
-  endLedger: latest,
-  filters: [
-    {
-      type: 'contract',
-      contractIds: ['YOUR_CONTRACT_ID']
-    }
-  ],
-  limit: 100
+const latest = await streamer.getLatestLedger();
+
+console.log(latest);
+```
+
+Metadata can be requested when ledger information beyond the sequence is needed:
+
+```js
+const latest = await streamer.getLatestLedger({
+  metadata: true
 });
+
+console.log(latest.sequence);
+console.log(latest.hash);
 ```
 
-## Get recent events
+## Large event ranges
+
+Soroban RPC limits individual ledger-range queries. `soroban-events` automatically divides large ranges into safe windows.
+
+Internal consumers such as backfill and consume use unlimited pagination so events are not silently truncated by a default event limit.
+
+Public window retrieval accepts either a finite limit or `null` for unlimited retrieval:
 
 ```js
-const events = await streamer.tail({
-  contractId: 'YOUR_CONTRACT_ID',
-  limit: 20
+const events = await streamer.getEventsWindowed({
+  startLedger: 1000000,
+  endLedger: 1200000,
+  limit: null
 });
+```
+
+## Backfill
 
-console.log(events);
+```js
+import { backfill } from "soroban-events";
+
+await backfill(streamer, {
+  startLedger: 1000000,
+  endLedger: 1100000,
+  onEvent: async (event) => {
+    console.log(event);
+  }
+});
 ```
 
-## Stream events
+Backfill supports checkpointing, event stores, pipelines, progress reporting, and concurrent window processing.
+
+## Replay
 
 ```js
-for await (const event of streamer.stream({
-  startLedger: 4750000,
-  filters: [
-    {
-      type: 'contract',
-      contractIds: ['YOUR_CONTRACT_ID']
-    }
-  ]
-})) {
-  console.log('New event:', event);
-}
+import { EventReplay } from "soroban-events";
+
+const replay = new EventReplay(streamer);
+
+await replay.run({
+  startLedger: 1000000,
+  endLedger: 1010000,
+  onEvent: async (event) => {
+    console.log(event);
+  }
+});
 ```
 
-Stop a stream with `AbortController`:
+Replay supports checkpoint recovery, pipelines, persistence, and abort signals.
+
+## Streaming
 
 ```js
 const controller = new AbortController();
 
-setTimeout(() => controller.abort(), 30_000);
-
 for await (const event of streamer.stream({
-  startLedger: 4750000,
+  startLedger: 1000000,
   signal: controller.signal
 })) {
   console.log(event);
 }
+
+controller.abort();
 ```
 
-## Decode ScVal
+Stream checkpointing does not advance past a ledger until all events from that ledger have been successfully consumed.
 
-The decoder can also be used independently:
+## Checkpointing
+
+Checkpoint stores use the raw store contract:
 
 ```js
-import { unwrapScVal } from 'soroban-events';
+await store.save(key, ledger);
+const ledger = await store.load(key);
+await store.clear(key);
+```
 
-const value = unwrapScVal(scVal);
-console.log(value);
+For monotonic checkpoint management and explicit rewinds, use `CheckpointManager`:
+
+```js
+const manager = new CheckpointManager(store);
+
+await manager.save(ledger, "my-consumer");
+
+const resumeLedger = await manager.resumeFrom(
+  "my-consumer",
+  startLedger
+);
+
+await manager.rewind(
+  rewindLedger,
+  "my-consumer"
+);
 ```
 
-Common Soroban values supported include integers, symbols, strings, booleans, bytes, addresses, vectors, and maps.
+Normal checkpoint saves cannot move backwards. Explicit rewinds are used for recovery workflows.
 
-Large integer values are normalized without relying on JavaScript `Number` precision.
+## File checkpoints
 
-## Event format
+```js
+import {
+  FileCheckpointStore,
+  CheckpointManager
+} from "soroban-events";
 
-Events are normalized into a consistent object containing fields such as:
+const store = new FileCheckpointStore("./checkpoints");
+const checkpoints = new CheckpointManager(store);
+
+await checkpoints.save(
+  1000000,
+  "my-consumer"
+);
+
+const resumeLedger = await checkpoints.resumeFrom(
+  "my-consumer",
+  1
+);
+```
+
+File checkpoints use atomic replacement so a partially written checkpoint does not replace the last valid checkpoint.
+
+## Event filtering
 
 ```js
-{
-  id,
-  type,
-  ledger,
-  ledgerClosedAt,
-  contractId,
-  transactionIndex,
-  operationIndex,
-  txHash,
-  topics,
-  value,
-  inSuccessfulContractCall
-}
+import { createEventFilter } from "soroban-events";
+
+const filter = createEventFilter({
+  contractId: "C...",
+  type: "transfer"
+});
+
+const matching = events.filter(filter);
 ```
 
-## Configuration
+Filters can match:
+
+- contract ID
+- event type
+- topics
+- transaction hash
+- exact ledger
+- ledger ranges
+- multiple contracts
+- custom predicates
+
+## Event querying
 
 ```js
-const streamer = new SorobanEventStreamer(RPC_URL, {
-  pollInterval: 3000,
-  windowSize: 9500,
-  pageSize: 1000,
-  maxRetries: 3,
-  retryBaseMs: 500
+import { createEventQuery } from "soroban-events";
+
+const query = createEventQuery(store);
+
+const results = await query({
+  contractId: "C...",
+  startLedger: 1000000,
+  endLedger: 1010000,
+  limit: 100
 });
 ```
 
-| Option | Default | Description |
-|---|---:|---|
-| `pollInterval` | `3000` | Polling interval for `stream()` |
-| `windowSize` | `9500` | Maximum ledger window |
-| `pageSize` | `1000` | RPC page size |
-| `maxRetries` | `3` | Maximum retry attempts |
-| `retryBaseMs` | `500` | Base retry delay |
-
-## How it works
-
-```text
-Soroban RPC
-    |
-    v
-ledger windows
-    |
-    v
-cursor pagination
-    |
-    v
-retry handling
-    |
-    v
-deduplication
-    |
-    v
-ScVal decoding
-    |
-    v
-your application
-```
-
-No database is required.
-
-No full indexer stack is required.
-
-Use the normalized events in whatever application or storage layer you need.
+Queries provide deterministic ordering and pagination.
 
-## Testing
+## Event storage
+
+The package provides memory and SQLite event stores.
+
+Memory storage:
+
+```js
+import { MemoryEventStore } from "soroban-events";
+
+const store = new MemoryEventStore();
+
+await store.put(event);
+
+const events = await store.query({
+  startLedger: 1000000,
+  endLedger: 1010000
+});
+```
+
+SQLite storage:
+
+```js
+import { SqliteEventStore } from "soroban-events";
+
+const store = new SqliteEventStore("./events.db");
+
+await store.put(event);
+```
+
+SQLite uses Node's built-in SQLite support and therefore requires a supported Node.js runtime.
+
+## Event pipelines
+
+Pipelines can filter, transform, observe, batch, and handle event-processing failures.
+
+```js
+import { EventPipeline } from "soroban-events";
+
+const pipeline = new EventPipeline()
+  .filter((event) => event.type === "transfer")
+  .map((event) => ({
+    ...event,
+    processed: true
+  }))
+  .tap((event) => {
+    console.log("processed", event.id);
+  });
+```
+
+## RPC health
+
+```js
+const health = await streamer.getHealth();
+
+console.log(health.status);
+console.log(health.latestLedger);
+console.log(health.oldestLedger);
+console.log(health.ledgerRetentionWindow);
+```
+
+Retention can be checked directly:
 
-Run the unit tests:
+```js
+const result = await streamer.checkRetention(1000000);
+
+console.log(result.retained);
+```
+
+## RPC failover
+
+```js
+const streamer = new SorobanEventStreamer(
+  "https://primary-rpc.example",
+  {
+    failoverRpcUrls: [
+      "https://secondary-rpc.example",
+      "https://third-rpc.example"
+    ]
+  }
+);
+```
+
+The streamer can rotate through configured RPC endpoints when transient failures exhaust retry attempts.
+
+## Circuit breaker
+
+Circuit-breaker behavior can be configured:
+
+```js
+const streamer = new SorobanEventStreamer(
+  rpcUrl,
+  {
+    circuitBreakerThreshold: 3,
+    circuitBreakerCooldownMs: 30000
+  }
+);
+```
+
+The circuit protects the consumer from repeatedly hammering an unavailable RPC endpoint.
+
+## Adaptive retry
+
+Retry behavior supports exponential backoff, rate-limit adaptation, maximum delay, and jitter:
+
+```js
+const streamer = new SorobanEventStreamer(
+  rpcUrl,
+  {
+    maxRetries: 3,
+    retryBaseMs: 500,
+    retryMaxMs: 30000,
+    retryJitter: 0.2,
+    adaptiveRateLimit: true
+  }
+);
+```
+
+## Reorg recovery
+
+The package provides ledger consistency tracking and reorg recovery primitives.
+
+Recovery can:
+
+- detect ledger hash mismatches;
+- determine an affected rewind point;
+- rewind checkpoints;
+- invalidate affected stored events;
+- restore the backfill/live handoff boundary.
+
+## Event export
+
+JSONL:
+
+```js
+import { exportEvents } from "soroban-events";
+
+const jsonl = exportEvents(events, {
+  format: "jsonl"
+});
+```
+
+CSV:
+
+```js
+const csv = exportEvents(events, {
+  format: "csv"
+});
+```
+
+Streaming export is also supported for large event collections.
+
+## CLI
+
+Check RPC status:
 
 ```bash
-npm test
+npx soroban-events status --rpc <RPC_URL>
+```
+
+Backfill a ledger range:
+
+```bash
+npx soroban-events backfill --rpc <RPC_URL> --start <LEDGER> --end <LEDGER>
 ```
 
-Live Stellar Testnet verification:
+Replay a ledger range:
 
 ```bash
-npm run test:live
+npx soroban-events replay --rpc <RPC_URL> --start <LEDGER> --end <LEDGER>
 ```
 
-The project currently includes 22 automated tests covering windowing, pagination, deduplication, limits, retries, abort handling, filtering, ScVal decoding, large integers, and event normalization.
+Health information:
+
+```bash
+npx soroban-events health --rpc <RPC_URL>
+```
+
+The CLI supports JSONL and CSV export options where applicable.
+
+## TypeScript
+
+TypeScript declarations are included with the package.
+
+```ts
+import {
+  SorobanEventStreamer,
+  CheckpointManager,
+  MemoryCheckpointStore
+} from "soroban-events";
+
+const store = new MemoryCheckpointStore();
+const checkpoints = new CheckpointManager(store);
+```
+
+## Reliability guarantees
+
+The library is designed around several important ingestion properties.
+
+### Pagination safety
+
+Large RPC ranges are split into safe ledger windows and paginated using RPC cursors.
+
+### Event deduplication
+
+Event IDs are deduplicated across pagination and processing boundaries.
 
-The live test has also been verified against Stellar Testnet RPC.
+### Checkpoint safety
 
-## Status
+Consumer checkpoints are advanced only after the relevant ledger has been completely processed.
 
-**v0.1.0**
+### Restart recovery
 
-The current release focuses on the core RPC ingestion and ScVal decoding layer.
+A failed consumer can resume from its last durable checkpoint and reprocess the uncommitted ledger.
+
+### Backfill/live handoff
+
+The handoff layer protects the transition between historical backfill and live consumption from duplicate delivery.
+
+### Reorg handling
+
+Ledger consistency checks can detect hash mismatches and trigger controlled checkpoint rewind and storage invalidation.
+
+### RPC resilience
+
+Transient failures, rate limits, endpoint failures, and unavailable RPC services can be handled through retry, adaptive backoff, failover, and circuit-breaking mechanisms.
+
+## Testing
+
+Run the complete test suite:
+
+```bash
+npm test
+```
+
+Run the packed-consumer test:
+
+```bash
+npm run test:pack
+```
+
+Run the release verification suite:
+
+```bash
+npm run test:release
+```
 
-This is intentionally a lightweight library, not a database-backed blockchain indexer.
+The release verification includes:
 
-Potential future work includes durable cursor persistence, stronger recovery strategies, webhook delivery, richer filtering helpers, and production indexing integrations.
+- complete test suite;
+- packed npm consumer verification;
+- npm pack dry-run;
+- Git whitespace validation.
 
-## Contributing
+## v3.0.0
 
-Bug reports, edge cases, documentation improvements, test cases, and pull requests are welcome.
+Version 3 is a major compatibility boundary for the expanded public API.
 
-When reporting an RPC issue, include the endpoint/network, ledger range, relevant RPC error or response, expected behavior, and actual behavior.
+Before upgrading, review:
 
-## License
+- checkpoint store method signatures;
+- `CheckpointManager` usage;
+- streamer `limit` behavior;
+- event filtering and querying;
+- event storage APIs;
+- export APIs;
+- pipeline APIs;
+- RPC failover and retry configuration;
+- TypeScript declarations;
+- Node.js 22.12.0 or newer requirement.
 
-MIT
+The v3.0.0 release is currently unreleased.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "soroban-events",
-  "version": "2.0.2",
+  "version": "3.0.0",
   "description": "Resilient, windowed event streamer and XDR decoder for Soroban RPC",
   "type": "module",
   "main": "./src/index.js",
@@ -22,7 +22,9 @@
     "cli": "node src/cli.js",
     "test:unit": "node --test tests/*.test.js",
     "audit": "node scripts/audit.mjs",
-    "pack:audit": "node scripts/pack-audit.mjs"
+    "pack:audit": "node scripts/pack-audit.mjs",
+    "test:pack": "node --test tests/packed-consumer.test.js",
+    "test:release": "npm test && npm run test:pack && npm pack --dry-run && git diff --check"
   },
   "keywords": [
     "soroban",
@@ -39,7 +41,7 @@
   ],
   "license": "MIT",
   "engines": {
-    "node": ">=20"
+    "node": ">=22.12.0"
   },
   "dependencies": {
     "@stellar/stellar-sdk": "^17.1.0"

package/src/backfill.js

@@ -291,7 +291,7 @@ export class BackfillEngine {
           startLedger: window.startLedger,
           endLedger: window.endLedger,
           filters,
-          limit: 10000,
+          limit: null,
           signal
         });