effective-indexer 0.2.6 → 0.2.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (6) hide show
  1. package/README.md +202 -341
  2. package/dist/index.cjs +13 -6
  3. package/dist/index.cjs.map +1 -1
  4. package/dist/index.js +13 -6
  5. package/dist/index.js.map +1 -1
  6. package/package.json +1 -1
package/README.md CHANGED
@@ -2,15 +2,17 @@
2
2
 
3
3
  EVM event indexing without hosted lock-in.
4
4
 
5
- `effective-indexer` runs as your own worker, writes directly to your SQLite database, and gives you a typed query API.
5
+ One worker process, one config file, your own SQLite database. No subgraph deployment pipeline, no token staking, no PhD required.
6
6
 
7
- ## Why this approach
7
+ Works with any EVM chain: Ethereum, Rootstock, Polygon, Arbitrum, Base, you name it.
8
8
 
9
- - **Own your data**: events are stored in your DB, not in a third-party service.
10
- - **Simple operations**: one worker process, one config file, no subgraph deployment pipeline.
11
- - **Production-safe behavior**: checkpoint resume, reorg detection, retry/backoff, live polling.
12
- - **Fast backfill**: parallel `eth_getLogs` with deterministic chunk ordering.
13
- - **Typed DX**: TypeScript-first config and query surface.
9
+ ## Why
10
+
11
+ - **Own your data** — events land in your DB, not in someone else's cloud.
12
+ - **Fast backfill** — parallel `eth_getLogs` with deterministic chunk ordering.
13
+ - **Production-safe** — checkpoint resume, reorg detection, retry with backoff, live polling.
14
+ - **Self-healing** — automatic crash recovery with configurable alerting.
15
+ - **Typed DX** — TypeScript-first config and query API, Hardhat-style config files.
14
16
 
15
17
  ## Install
16
18
 
@@ -18,85 +20,80 @@ EVM event indexing without hosted lock-in.
18
20
  npm install effective-indexer effect
19
21
  ```
20
22
 
21
- `effect` is a peer dependency.
22
-
23
- ## License
23
+ `effect` is a peer dependency — the only runtime dependency besides `viem`.
24
24
 
25
- Free for noncommercial use under PolyForm Noncommercial 1.0.0.
26
- Commercial use requires a paid commercial license (see `LICENSE`).
27
- Contact: Aleksandr Shenshin <shenshin@me.com>.
25
+ ## Quick start
28
26
 
29
- ## 5-minute setup
27
+ ### 1. Config file
30
28
 
31
- ### 1) Create `indexer.config.ts`
29
+ Create `indexer.config.ts`:
32
30
 
33
31
  ```ts
34
32
  import { defineIndexerConfig } from "effective-indexer"
35
33
  import type { Abi } from "viem"
36
34
 
37
- const transferAbi: Abi = [
38
- {
39
- type: "event",
40
- name: "Transfer",
41
- inputs: [
42
- { indexed: true, name: "from", type: "address" },
43
- { indexed: true, name: "to", type: "address" },
44
- { indexed: false, name: "value", type: "uint256" },
45
- ],
46
- },
35
+ const abi: Abi = [
36
+ {
37
+ type: "event",
38
+ name: "Transfer",
39
+ inputs: [
40
+ { indexed: true, name: "from", type: "address" },
41
+ { indexed: true, name: "to", type: "address" },
42
+ { indexed: false, name: "value", type: "uint256" },
43
+ ],
44
+ },
47
45
  ]
48
46
 
49
47
  export default defineIndexerConfig({
50
- rpcUrl: "https://rpc.mainnet.rootstock.io/{{EVM_RPC_API_KEY}}",
51
- dbPath: "./data/events.db",
52
- contracts: [
53
- {
54
- name: "Token",
55
- address: "0xYourContractAddress",
56
- abi: transferAbi,
57
- events: ["Transfer"],
58
- startBlock: 0n,
59
- },
60
- ],
61
- network: {
62
- logs: {
63
- chunkSize: 2000,
64
- parallelRequests: 3,
65
- },
66
- },
48
+ rpcUrl: "https://rpc.mainnet.rootstock.io/{{EVM_RPC_API_KEY}}",
49
+ dbPath: "./data/events.db",
50
+ contracts: [
51
+ {
52
+ name: "Token",
53
+ address: "0xYourContractAddress",
54
+ abi,
55
+ events: ["Transfer"],
56
+ startBlock: 0n,
57
+ },
58
+ ],
67
59
  })
68
60
  ```
69
61
 
70
- ### 2) Create `scripts/indexer.ts`
62
+ `{{EVM_RPC_API_KEY}}` is resolved from env at runtime. Secrets stay in `.env`, config stays typed.
63
+
64
+ ### 2. Worker script
65
+
66
+ Create `scripts/indexer.ts`:
71
67
 
72
68
  ```ts
73
69
  import config from "../indexer.config"
74
70
  import { resolveIndexerConfigFromEnv, runIndexerWorker } from "effective-indexer"
75
71
 
76
- const resolvedConfig = resolveIndexerConfigFromEnv(config)
72
+ const resolved = resolveIndexerConfigFromEnv(config)
77
73
 
78
- runIndexerWorker(resolvedConfig).catch(error => {
79
- console.error("Indexer worker failed:", error)
80
- process.exit(1)
74
+ runIndexerWorker(resolved).catch(error => {
75
+ console.error("Indexer worker failed:", error)
76
+ process.exit(1)
81
77
  })
82
78
  ```
83
79
 
84
- ### 3) Add env and run
80
+ ### 3. Environment and run
85
81
 
86
82
  `.env`:
87
83
 
88
84
  ```bash
89
- EVM_RPC_API_KEY=your-rpc-api-key
90
- # Optional full URL override:
91
- # EVM_RPC_URL=https://rpc.mainnet.rootstock.io/<API_KEY>
85
+ EVM_RPC_API_KEY=your-api-key
86
+ # Full URL override (takes priority over template):
87
+ # EVM_RPC_URL=https://eth.llamarpc.com
92
88
  ```
93
89
 
94
- Run:
95
-
96
90
  ```bash
91
+ npm install -D tsx
97
92
  node --import tsx ./scripts/indexer.ts
98
93
  ```
99
94
 
95
+ That's it. The worker creates the DB directory, connects, backfills, and switches to live polling.
96
+
100
97
  ## Query data
101
98
 
102
99
  ```ts
@@ -106,307 +103,192 @@ import { Indexer, resolveIndexerConfigFromEnv } from "effective-indexer"
106
103
  const indexer = Indexer.create(resolveIndexerConfigFromEnv(config))
107
104
 
108
105
  const events = await indexer.query({
109
- contractName: "Token",
110
- eventName: "Transfer",
111
- order: "desc",
112
- limit: 50,
106
+ contractName: "Token",
107
+ eventName: "Transfer",
108
+ order: "desc",
109
+ limit: 50,
113
110
  })
114
111
 
115
- console.log(events.length)
112
+ console.log(events)
116
113
  await indexer.stop()
117
114
  ```
118
115
 
119
- ## Public API
120
-
121
- - `defineIndexerConfig(config)`
122
- Identity helper for typed config files (Hardhat-style).
123
- - `resolveIndexerConfigFromEnv(config, options?)`
124
- Resolves `{{ENV_VAR}}` placeholders and optional RPC URL override.
125
- - `runIndexerWorker(config, options?)`
126
- Runs long-lived worker with built-in DB directory creation and graceful shutdown.
127
- - `Indexer.create(config)`
128
- Returns handle: `start()`, `stop()`, `query()`, `count()`.
129
-
130
- ## Config essentials
131
-
132
- - `rpcUrl`: RPC endpoint URL (supports placeholders like `{{EVM_RPC_API_KEY}}`)
133
- - `dbPath`: SQLite path (default `./indexer.db`)
134
- - `contracts`: non-empty list of contracts and events to index
135
- - `network.polling`: block polling interval and confirmations
136
- - `network.logs`: chunk size, retries, parallel requests
137
- - `network.reorg.depth`: reorg buffer depth
138
- - `telemetry.progress`: CLI progress rendering
139
- - `logLevel`, `logFormat`, `enableTelemetry`
140
-
141
- ## Operational notes
142
-
143
- - Run a single writer process per SQLite file.
144
- - Keep DB on persistent storage.
145
- - Worker resumes from checkpoint after restart.
146
- - RPC must support `eth_getLogs`.
116
+ ## API
147
117
 
148
- ## Development
118
+ ### `Indexer.create(config): IndexerHandle`
149
119
 
150
- ```bash
151
- npm run build
152
- npm run typecheck
153
- npm run test
154
- npm run check
155
- ```
120
+ Creates an indexer instance. Returns:
156
121
 
157
- Repository: [github.com/cybervoid0/effective-indexer](https://github.com/cybervoid0/effective-indexer)
158
- # Effective Indexer
122
+ | Method | Description |
123
+ |--------|-------------|
124
+ | `start()` | Start indexing (non-blocking, runs in background) |
125
+ | `stop()` | Gracefully stop and dispose runtime (idempotent) |
126
+ | `waitForExit()` | Await the indexing loop (rejects on crash) |
127
+ | `query(q?)` | Query stored events → `Promise<ParsedEvent[]>` |
128
+ | `count(q?)` | Count stored events → `Promise<number>` |
159
129
 
160
- Lightweight EVM smart contract event indexer built with [Effect](https://effect.website).
130
+ ### `defineIndexerConfig(config)`
161
131
 
162
- Index EVM events to your own database in minutes — no hosted lock-in, no PhD required.
132
+ Identity function for typed config files. Zero runtime cost, pure DX.
163
133
 
164
- Repository: [github.com/cybervoid0/effective-indexer](https://github.com/cybervoid0/effective-indexer)
165
-
166
- Indexes smart contract events into SQLite with:
167
- - Historical backfill (`eth_getLogs` in chunks)
168
- - Live polling for new blocks
169
- - Checkpoint resume after restart
170
- - Reorg detection and rollback
134
+ ### `resolveIndexerConfigFromEnv(config, options?)`
171
135
 
172
- Works with any EVM-compatible chain (Ethereum, Rootstock, Polygon, Arbitrum, etc.).
136
+ Resolves `{{ENV_VAR}}` placeholders in `rpcUrl` from `process.env`. Supports:
173
137
 
174
- ## Requirements
138
+ - **Sensitive placeholders** — read via `Config.redacted` (default: `EVM_RPC_API_KEY`)
139
+ - **Full URL override** — `EVM_RPC_URL` env var takes priority over the template
140
+ - **Custom env source** — pass `{ env: myEnvMap }` for testing
175
141
 
176
- - Node.js `>=20`
177
- - RPC endpoint with `eth_getLogs` support
142
+ ### `runIndexerWorker(config, options?)`
178
143
 
179
- ## Install
144
+ Long-lived worker with batteries included:
180
145
 
181
- ```bash
182
- npm install effective-indexer effect
183
- ```
146
+ - Creates DB directory if missing
147
+ - Registers `SIGINT`/`SIGTERM` handlers for graceful shutdown
148
+ - Keeps the process alive during live polling
149
+ - Auto-restarts on crash with exponential backoff
150
+ - Calls `onRecoveryFailure` webhook when recovery window is exhausted
151
+ - Always re-throws the original error (notification failures are logged, never mask the cause)
184
152
 
185
- `effect` is a peer dependency.
153
+ ### `createWebhookNotifier(url, init?)`
186
154
 
187
- ## Quick Start
155
+ Helper that returns an `onRecoveryFailure` callback — POSTs a JSON payload to the given URL.
188
156
 
189
157
  ```ts
190
- import { Indexer } from "effective-indexer"
191
- import type { Abi } from "viem"
192
-
193
- const abi: Abi = [
194
- {
195
- type: "event",
196
- name: "Transfer",
197
- inputs: [
198
- { indexed: true, name: "from", type: "address" },
199
- { indexed: true, name: "to", type: "address" },
200
- { indexed: false, name: "value", type: "uint256" },
201
- ],
202
- },
203
- ]
204
-
205
- const indexer = Indexer.create({
206
- rpcUrl: "https://eth.llamarpc.com",
207
- dbPath: "./data/events.db",
208
- contracts: [
209
- {
210
- name: "USDT",
211
- address: "0xdAC17F958D2ee523a2206206994597C13D831ec7",
212
- abi,
213
- events: ["Transfer"],
214
- startBlock: 19000000n,
215
- },
216
- ],
217
- network: {
218
- polling: { intervalMs: 12000, confirmations: 2 },
219
- logs: { chunkSize: 2000 },
220
- reorg: { depth: 64 },
221
- },
222
- })
223
-
224
- await indexer.start() // non-blocking, runs in background
225
-
226
- const events = await indexer.query({
227
- contractName: "USDT",
228
- eventName: "Transfer",
229
- limit: 50,
230
- order: "desc",
231
- })
232
-
233
- console.log(events.length)
158
+ import { createWebhookNotifier } from "effective-indexer"
234
159
 
235
- // later
236
- await indexer.stop()
160
+ const notify = createWebhookNotifier("https://hooks.slack.com/...")
237
161
  ```
238
162
 
239
- ## API
240
-
241
- ### `Indexer.create(config)`
242
-
243
- Returns `IndexerHandle`:
244
- - `start(): Promise<void>` start indexing loop (non-blocking)
245
- - `stop(): Promise<void>` stop and dispose runtime
246
- - `query(q?: EventQuery): Promise<ParsedEvent[]>`
247
- - `count(q?: EventQuery): Promise<number>`
163
+ Or configure it in the config file directly (see `worker.alert.webhookUrl` below).
248
164
 
249
- ### `defineIndexerConfig(config)`
165
+ ### `EventQuery`
250
166
 
251
- Identity helper for a typed config file (Hardhat-style DX).
167
+ | Field | Type | Description |
168
+ |-------|------|-------------|
169
+ | `contractName` | `string?` | Filter by contract name |
170
+ | `eventName` | `string?` | Filter by event name |
171
+ | `fromBlock` | `bigint?` | Min block number |
172
+ | `toBlock` | `bigint?` | Max block number |
173
+ | `txHash` | `string?` | Filter by transaction hash |
174
+ | `limit` | `number?` | Max results |
175
+ | `offset` | `number?` | Skip first N results |
176
+ | `order` | `"asc" \| "desc"?` | Sort by block number |
252
177
 
253
- ### `resolveIndexerConfigFromEnv(config, options?)`
178
+ ### `ParsedEvent`
254
179
 
255
- Resolves `{{ENV_VAR}}` placeholders in `rpcUrl` and supports optional RPC override from env (`EVM_RPC_URL` by default).
180
+ | Field | Type |
181
+ |-------|------|
182
+ | `id` | `number` |
183
+ | `contractName` | `string` |
184
+ | `eventName` | `string` |
185
+ | `blockNumber` | `bigint` |
186
+ | `txHash` | `string` |
187
+ | `logIndex` | `number` |
188
+ | `timestamp` | `number \| null` |
189
+ | `args` | `Record<string, unknown>` |
256
190
 
257
- ### `runIndexerWorker(config, options?)`
191
+ ## Full config reference
258
192
 
259
- Runs a long-lived worker with built-in:
260
- - SQLite directory creation
261
- - graceful shutdown on `SIGINT` / `SIGTERM`
262
- - keep-alive process loop
193
+ All fields except `rpcUrl` and `contracts` are optional — sensible defaults are applied.
263
194
 
264
- ### `IndexerConfig`
195
+ ### Top-level
265
196
 
266
197
  | Field | Type | Default | Description |
267
198
  |-------|------|---------|-------------|
268
- | `rpcUrl` | `string` | — | RPC endpoint URL |
199
+ | `rpcUrl` | `string` | — | RPC endpoint (supports `{{ENV}}` placeholders) |
269
200
  | `dbPath` | `string` | `"./indexer.db"` | SQLite database path |
270
- | `contracts` | `ContractConfig[]` | — | Contracts to index |
271
- | `network` | `NetworkConfig` | see below | Network tuning |
272
- | `telemetry` | `TelemetryConfig` | see below | Backfill progress settings |
273
- | `logLevel` | `string` | `"info"` | Minimum log level |
274
- | `logFormat` | `string` | `"pretty"` | Log output format |
275
- | `enableTelemetry` | `boolean` | `true` | Set `false` for errors-only |
276
-
277
- ### `NetworkConfig`
201
+ | `contracts` | `ContractConfig[]` | — | At least one contract required |
202
+ | `network` | `NetworkConfig` | see below | RPC and chain tuning |
203
+ | `telemetry` | `TelemetryConfig` | see below | Progress rendering |
204
+ | `worker` | `WorkerConfig` | see below | Recovery and alerting |
205
+ | `logLevel` | `string` | `"info"` | `trace \| debug \| info \| warning \| error \| none` |
206
+ | `logFormat` | `string` | `"pretty"` | `pretty \| json \| structured` |
207
+ | `enableTelemetry` | `boolean` | `true` | `false` = errors only, no progress bar |
208
+
209
+ ### `contracts[]`
210
+
211
+ | Field | Type | Description |
212
+ |-------|------|-------------|
213
+ | `name` | `string` | Unique contract name (used in queries) |
214
+ | `address` | `string` | Contract address (hex) |
215
+ | `abi` | `Abi` | Viem-compatible ABI (only event entries needed) |
216
+ | `events` | `[string, ...string[]]` | Event names to index (non-empty) |
217
+ | `startBlock` | `bigint?` | Block to start indexing from (default: `0n`) |
218
+
219
+ ### `network`
278
220
 
279
221
  ```ts
280
- {
222
+ network: {
281
223
  polling: {
282
- intervalMs: 12000, // block polling interval
283
- confirmations: 1, // blocks behind head to consider confirmed
224
+ intervalMs: 12000, // block poll interval (ms)
225
+ confirmations: 1, // blocks behind tip = "confirmed"
284
226
  },
285
227
  logs: {
286
- chunkSize: 5000, // blocks per eth_getLogs request
287
- maxRetries: 5, // retry count on RPC failure
288
- parallelRequests: 1, // concurrent eth_getLogs requests during backfill
228
+ chunkSize: 5000, // blocks per eth_getLogs request
229
+ maxRetries: 5, // retries per failed RPC call
230
+ parallelRequests: 1, // concurrent eth_getLogs during backfill
289
231
  retry: {
290
- baseDelayMs: 1000, // initial retry delay
291
- maxDelayMs: 30000, // cap for exponential backoff
232
+ baseDelayMs: 1000, // initial retry delay
233
+ maxDelayMs: 30000, // backoff cap
292
234
  },
293
235
  },
294
236
  reorg: {
295
- depth: 20, // block hash buffer depth for reorg detection
237
+ depth: 20, // block hash buffer for reorg detection
296
238
  },
297
239
  }
298
240
  ```
299
241
 
300
- All fields are optional — defaults are shown above.
301
-
302
- ### `TelemetryConfig`
242
+ ### `telemetry`
303
243
 
304
244
  ```ts
305
- {
306
- telemetry: {
307
- progress: {
308
- enabled: true, // show backfill progress in terminal
309
- intervalMs: 3000, // progress update frequency (ms, minimum 500)
310
- },
245
+ telemetry: {
246
+ progress: {
247
+ enabled: true, // show live progress bar during backfill
248
+ intervalMs: 3000, // render frequency (min 500ms)
311
249
  },
312
250
  }
313
251
  ```
314
252
 
315
- `enableTelemetry: false` disables progress rendering and keeps error-level logs only.
316
-
317
- When enabled, the indexer displays a live progress line during backfill:
253
+ When enabled, the terminal shows a live progress line:
318
254
 
319
255
  ```
320
256
  [Backfill] Token 42.8% | 1,234,000/2,880,000 blocks | 3,450 blk/s | 12.4 ev/s | ETA 00:07:43 | p=3 | chunk=5000
321
257
  ```
322
258
 
323
- On non-TTY environments, periodic info logs are emitted instead. A final summary is logged when backfill completes:
259
+ On non-TTY (CI, logs), periodic info messages are emitted instead.
324
260
 
325
- ```
326
- [Backfill complete] Token: 2,880,000 blocks | 45,230 events | 312 chunks | 00:13:54 (3,453 blk/s, 54.2 ev/s) | p=3 | chunkSize=5000
327
- ```
328
-
329
- ## Worker Setup (Recommended)
330
-
331
- Run the indexer as a dedicated long-lived worker process (not in request handlers).
332
-
333
- Create `scripts/indexer.ts`:
261
+ ### `worker`
334
262
 
335
263
  ```ts
336
- import config from "../indexer.config"
337
- import { resolveIndexerConfigFromEnv, runIndexerWorker } from "effective-indexer"
338
-
339
- const resolvedConfig = resolveIndexerConfigFromEnv(config)
340
-
341
- runIndexerWorker(resolvedConfig).catch(error => {
342
- console.error("Indexer worker failed:", error)
343
- process.exit(1)
344
- })
345
- ```
346
-
347
- Create `indexer.config.ts`:
348
-
349
- ```ts
350
- import { defineIndexerConfig } from "effective-indexer"
351
- import type { Abi } from "viem"
352
-
353
- const transferAbi: Abi = [
354
- {
355
- type: "event",
356
- name: "Transfer",
357
- inputs: [
358
- { indexed: true, name: "from", type: "address" },
359
- { indexed: true, name: "to", type: "address" },
360
- { indexed: false, name: "value", type: "uint256" },
361
- ],
362
- },
363
- ]
364
-
365
- export default defineIndexerConfig({
366
- rpcUrl: "https://rpc.mainnet.rootstock.io/{{EVM_RPC_API_KEY}}",
367
- dbPath: "./data/events.db",
368
- contracts: [
369
- {
370
- name: "Token",
371
- address: "0xYourContractAddress",
372
- abi: transferAbi,
373
- events: ["Transfer"],
374
- startBlock: 0n,
375
- },
376
- ],
377
- })
378
- ```
379
-
380
- Create `.env`:
381
-
382
- ```bash
383
- EVM_RPC_API_KEY=your-rpc-api-key
384
- # Optional full RPC URL override:
385
- # EVM_RPC_URL=https://rpc.mainnet.rootstock.io/<API_KEY>
264
+ worker: {
265
+ recovery: {
266
+ enabled: true, // auto-restart on crash
267
+ maxRecoveryDurationMs: 900000, // give up after 15 min of failures
268
+ initialRetryDelayMs: 1000, // first retry delay
269
+ maxRetryDelayMs: 30000, // backoff cap
270
+ backoffFactor: 2, // exponential multiplier
271
+ },
272
+ alert: {
273
+ webhookUrl: "", // POST failure notification here
274
+ },
275
+ }
386
276
  ```
387
277
 
388
- Add scripts (with `tsx` installed):
278
+ When the worker crashes, it automatically restarts with exponential backoff. If it keeps failing beyond `maxRecoveryDurationMs`, it sends a JSON notification to `alert.webhookUrl` (if set) and exits with the original error.
389
279
 
390
- ```bash
391
- npm install -D tsx
392
- ```
280
+ The notification payload (`WorkerFailureNotification`):
393
281
 
394
- ```json
282
+ ```ts
395
283
  {
396
- "scripts": {
397
- "indexer:start": "node --import tsx ./scripts/indexer.ts",
398
- "indexer:debug": "INDEXER_LOG_LEVEL=debug node --import tsx ./scripts/indexer.ts"
399
- }
284
+ attempts: number // total restart attempts
285
+ recoveryDurationMs: number // time since first failure
286
+ error: unknown // the error that killed it
287
+ timestamp: string // ISO timestamp
400
288
  }
401
289
  ```
402
290
 
403
- Run:
404
-
405
- ```bash
406
- npm run indexer:start
407
- ```
408
-
409
- ### Network Tuning Profiles
291
+ ## Chain tuning profiles
410
292
 
411
293
  | Chain | `polling.intervalMs` | `polling.confirmations` | `logs.chunkSize` | `reorg.depth` |
412
294
  |-------|---------------------|------------------------|------------------|---------------|
@@ -415,71 +297,42 @@ npm run indexer:start
415
297
  | Polygon | 2000 | 32 | 2000 | 128 |
416
298
  | Arbitrum | 1000 | 0 | 5000 | 1 |
417
299
 
418
- ### `EventQuery`
419
-
420
- - `contractName?: string`
421
- - `eventName?: string`
422
- - `fromBlock?: bigint`
423
- - `toBlock?: bigint`
424
- - `txHash?: string`
425
- - `limit?: number`
426
- - `offset?: number`
427
- - `order?: "asc" | "desc"`
428
-
429
- ## Telemetry & Logging
430
-
431
- The indexer uses Effect's native logging system. All log output is controlled via config — no `console.log` calls in source.
432
-
433
- | Level | What's emitted |
434
- |-------|---------------|
435
- | `error` | Indexer errors (RPC failures, storage errors) |
436
- | `warning` | Reorg detection, parent hash mismatches |
437
- | `info` | Indexer start/stop, backfill start/complete, reorg handled |
438
- | `debug` | Chunk indexed, block indexed, storage init, query/count execution, reorg rollback, BlockCursor init |
439
- | `trace` | Individual log fetches, block emissions, no-new-blocks polls |
440
-
441
- ### Recommendations
442
-
443
- - **Production**: `logLevel: "info"` — lifecycle events and warnings
444
- - **Troubleshooting**: `logLevel: "debug"` — per-chunk/block detail
445
- - **Deep inspection**: `logLevel: "trace"` — every RPC call and poll
446
- - **Silent**: `enableTelemetry: false` — only errors
447
-
448
- ## Operational Notes
449
-
450
- - Use one writer process per SQLite database file.
451
- - Keep database file on persistent storage.
452
- - On restart, the indexer resumes from checkpoint and backfills missed blocks.
453
- - If RPC does not support `eth_getLogs`, indexing cannot work.
454
-
455
- ## Parallel Backfill
300
+ ## Parallel backfill
456
301
 
457
- Set `network.logs.parallelRequests` to speed up historical backfill by issuing multiple `eth_getLogs` requests concurrently. Chunk ordering is preserved regardless of concurrency.
302
+ Set `network.logs.parallelRequests` to speed up historical indexing. Chunk ordering is preserved regardless of concurrency.
458
303
 
459
304
  ```ts
460
- const indexer = Indexer.create({
461
- rpcUrl: "https://eth.llamarpc.com",
462
- contracts: [/* ... */],
463
- network: {
464
- logs: {
465
- chunkSize: 2000,
466
- parallelRequests: 4,
467
- },
305
+ network: {
306
+ logs: {
307
+ chunkSize: 2000,
308
+ parallelRequests: 4,
468
309
  },
469
- })
310
+ }
470
311
  ```
471
312
 
472
- **Recommended values**: Start with `parallelRequests: 3` and increase if the RPC allows. Public endpoints may rate-limit above 5-10 concurrent requests.
313
+ Start with `3` and increase if the RPC allows. Public endpoints may rate-limit above 5–10.
314
+
315
+ ## Logging
316
+
317
+ Uses Effect's native logging — no `console.log` in source code.
318
+
319
+ | Level | What you see |
320
+ |-------|-------------|
321
+ | `error` | RPC failures, storage errors |
322
+ | `warning` | Reorg detection, parent hash mismatches |
323
+ | `info` | Start/stop, backfill progress, reorg handled |
324
+ | `debug` | Per-chunk detail, queries, storage init |
325
+ | `trace` | Every RPC call and poll tick |
473
326
 
474
- ### Benchmarking
327
+ **Production**: `logLevel: "info"`. **Debugging**: `"debug"`. **Silent**: `enableTelemetry: false`.
475
328
 
476
- To measure the effect of parallelism:
329
+ ## Operational notes
477
330
 
478
- 1. Use a fixed RPC endpoint and contract/block range
479
- 2. Start with an empty database each run
480
- 3. Compare `parallelRequests` values 1, 2, 3, 4
481
- 4. Run 3 times each and take the median
482
- 5. Use the progress summary line for timing: `[Backfill complete] ... blk/s`
331
+ - One writer process per SQLite file. This is not a suggestion.
332
+ - Keep the DB on persistent storage.
333
+ - On restart, the indexer resumes from the last checkpoint.
334
+ - RPC must support `eth_getLogs` — if it doesn't, nothing will work.
335
+ - Graceful shutdown: `Ctrl+C` or `kill <pid>` — the worker finishes the current operation and writes the checkpoint.
483
336
 
484
337
  ## Development
485
338
 
@@ -489,3 +342,11 @@ npm run typecheck
489
342
  npm run test
490
343
  npm run check
491
344
  ```
345
+
346
+ ## License
347
+
348
+ Free for noncommercial use under PolyForm Noncommercial 1.0.0.
349
+ Commercial use requires a paid license — see `LICENSE`.
350
+ Contact: Aleksandr Shenshin <shenshin@me.com>.
351
+
352
+ Repository: [github.com/cybervoid0/effective-indexer](https://github.com/cybervoid0/effective-indexer)