@x12i/countex 1.8.3 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +866 -858
- package/dist/cli.js +6 -6
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
- package/dist/insights/advisors/resource-advisor.js +5 -1
- package/dist/insights/advisors/resource-advisor.js.map +1 -1
- package/dist/insights/types.d.ts +4 -0
- package/dist/insights/types.d.ts.map +1 -1
- package/dist/storage/postgres.d.ts +104 -0
- package/dist/storage/postgres.d.ts.map +1 -0
- package/dist/storage/postgres.js +1023 -0
- package/dist/storage/postgres.js.map +1 -0
- package/dist/types/index.d.ts +11 -5
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +399 -375
- package/docs/END_TO_END_EXAMPLE.md +316 -316
- package/docs/INSIGHTS.md +59 -59
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
- package/docs/guides/README.md +36 -34
- package/docs/guides/adapter-memory.md +60 -60
- package/docs/guides/adapter-mongo.md +64 -62
- package/docs/guides/adapter-postgres.md +90 -0
- package/docs/guides/adapter-redis.md +89 -89
- package/docs/guides/backup-export.md +135 -135
- package/docs/guides/cardinality.md +50 -50
- package/docs/guides/cli.md +53 -53
- package/docs/guides/client-events.md +65 -65
- package/docs/guides/counter-maps.md +160 -160
- package/docs/guides/idempotency.md +39 -39
- package/docs/guides/ingestion-and-flush.md +67 -67
- package/docs/guides/insights-advisors-and-optimization.md +39 -39
- package/docs/guides/insights-anomalies-and-rules.md +48 -48
- package/docs/guides/insights-comparison-and-trends.md +47 -47
- package/docs/guides/insights-forecasting-and-budget.md +61 -61
- package/docs/guides/insights-overview.md +82 -82
- package/docs/guides/insights-ranking.md +81 -81
- package/docs/guides/measures.md +71 -71
- package/docs/guides/migrating-mongo-to-postgres.md +127 -0
- package/docs/guides/preview.md +92 -92
- package/docs/guides/querying.md +178 -178
- package/docs/guides/simulation-profiles.md +33 -33
- package/docs/guides/simulation-resource-estimation.md +31 -30
- package/docs/guides/simulation.md +36 -36
- package/docs/guides/timeframes-and-bucketing.md +55 -55
- package/docs/internals/README.md +21 -21
- package/docs/internals/internal-architecture-overview.md +65 -65
- package/docs/internals/internal-backup-stream.md +23 -23
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
- package/docs/internals/internal-cardinality.md +30 -30
- package/docs/internals/internal-dimension-hash.md +34 -34
- package/docs/internals/internal-idempotency.md +36 -36
- package/docs/internals/internal-ingest-pipeline.md +52 -52
- package/docs/internals/internal-insights-layer.md +36 -36
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
- package/docs/internals/internal-measures-and-scales.md +35 -35
- package/docs/internals/internal-query-path.md +46 -46
- package/docs/internals/internal-storage-memory.md +13 -13
- package/docs/internals/internal-storage-mongo.md +44 -44
- package/docs/internals/internal-storage-postgres.md +32 -0
- package/docs/internals/internal-storage-redis.md +63 -63
- package/docs/internals/internal-timeframe-bucketing.md +33 -33
- package/docs/use-cases/README.md +56 -56
- package/docs/use-cases/audit-security-log-counts.md +550 -550
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
- package/docs/use-cases/multi-tenant-isolation.md +203 -203
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
- package/docs/use-cases/processing-workflow-activity.md +165 -165
- package/docs/use-cases/product-feature-usage.md +240 -240
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
- package/package.json +184 -169
package/README.md
CHANGED
|
@@ -1,858 +1,866 @@
|
|
|
1
|
-
# @x12i/countex
|
|
2
|
-
|
|
3
|
-
> Generic sparse counter matrices for JSON events.
|
|
4
|
-
> Dynamic counting, aggregation, and retrieval from mapped JSON inputs.
|
|
5
|
-
|
|
6
|
-
`@x12i/countex` is a **generic mapped-event counter**: a source-agnostic counting engine. You give it any JSON event, a declarative mapping config, and storage adapters; it projects events into sparse counter cells, batches deltas locally, and persists them atomically to a durable store (MongoDB) and a hot cache (Redis).
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
**
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
npm install
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
|
42
|
-
|
|
43
|
-
|
|
|
44
|
-
|
|
|
45
|
-
|
|
|
46
|
-
|
|
|
47
|
-
|
|
|
48
|
-
|
|
|
49
|
-
|
|
|
50
|
-
|
|
|
51
|
-
|
|
|
52
|
-
|
|
|
53
|
-
|
|
|
54
|
-
|
|
|
55
|
-
|
|
|
56
|
-
|
|
|
57
|
-
|
|
|
58
|
-
|
|
|
59
|
-
|
|
|
60
|
-
|
|
|
61
|
-
|
|
|
62
|
-
|
|
|
63
|
-
|
|
|
64
|
-
|
|
|
65
|
-
|
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
|
70
|
-
|
|
71
|
-
|
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
|
76
|
-
|
|
77
|
-
|
|
|
78
|
-
|
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
- **
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
**`
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
},
|
|
161
|
-
"
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
```
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
```
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
"
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
```
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
```
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
"
|
|
207
|
-
"
|
|
208
|
-
"
|
|
209
|
-
"
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
"
|
|
215
|
-
"
|
|
216
|
-
}
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
```
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
{
|
|
258
|
-
],
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
}
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
}
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
328
|
-
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
348
|
-
|
|
349
|
-
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
|
390
|
-
|
|
391
|
-
| `
|
|
392
|
-
| `
|
|
393
|
-
| `
|
|
394
|
-
| `
|
|
395
|
-
| `
|
|
396
|
-
|
|
397
|
-
|
|
398
|
-
|
|
399
|
-
|
|
400
|
-
|
|
401
|
-
|
|
|
402
|
-
|
|
403
|
-
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
429
|
-
|
|
430
|
-
|
|
431
|
-
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
|
|
440
|
-
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
`
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
"
|
|
475
|
-
"
|
|
476
|
-
"
|
|
477
|
-
"
|
|
478
|
-
|
|
479
|
-
|
|
480
|
-
|
|
481
|
-
|
|
482
|
-
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
```
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
```
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
564
|
-
|
|
565
|
-
|
|
566
|
-
format: "table",
|
|
567
|
-
|
|
568
|
-
|
|
569
|
-
|
|
570
|
-
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
|
|
574
|
-
|
|
575
|
-
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
|
|
599
|
-
|
|
600
|
-
|
|
601
|
-
|
|
602
|
-
|
|
603
|
-
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
|
|
609
|
-
|
|
610
|
-
|
|
611
|
-
|
|
612
|
-
|
|
613
|
-
|
|
614
|
-
|
|
615
|
-
|
|
616
|
-
|
|
617
|
-
|
|
618
|
-
|
|
619
|
-
|
|
620
|
-
|
|
621
|
-
|
|
622
|
-
|
|
623
|
-
|
|
624
|
-
|
|
625
|
-
|
|
626
|
-
|
|
627
|
-
|
|
628
|
-
|
|
629
|
-
|
|
630
|
-
|
|
631
|
-
|
|
632
|
-
|
|
633
|
-
|
|
634
|
-
|
|
635
|
-
|
|
636
|
-
|
|
637
|
-
|
|
638
|
-
```
|
|
639
|
-
|
|
640
|
-
|
|
641
|
-
|
|
642
|
-
|
|
643
|
-
|
|
644
|
-
|
|
645
|
-
|
|
646
|
-
|
|
647
|
-
|
|
648
|
-
|
|
649
|
-
|
|
650
|
-
|
|
651
|
-
|
|
652
|
-
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
|
|
657
|
-
|
|
658
|
-
|
|
659
|
-
|
|
660
|
-
|
|
661
|
-
|
|
662
|
-
|
|
663
|
-
|
|
664
|
-
|
|
665
|
-
|
|
666
|
-
|
|
667
|
-
|
|
668
|
-
|
|
669
|
-
Flush
|
|
670
|
-
|
|
671
|
-
|
|
672
|
-
|
|
673
|
-
|
|
674
|
-
|
|
675
|
-
|
|
676
|
-
|
|
677
|
-
-
|
|
678
|
-
|
|
679
|
-
|
|
680
|
-
|
|
681
|
-
|
|
682
|
-
|
|
683
|
-
|
|
684
|
-
|
|
685
|
-
|
|
686
|
-
|
|
687
|
-
|
|
688
|
-
|
|
689
|
-
|
|
690
|
-
|
|
691
|
-
|
|
692
|
-
|
|
693
|
-
|
|
694
|
-
|
|
695
|
-
|
|
696
|
-
|
|
697
|
-
|
|
698
|
-
|
|
699
|
-
|
|
700
|
-
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
|
|
704
|
-
|
|
705
|
-
|
|
706
|
-
|
|
707
|
-
|
|
708
|
-
|
|
709
|
-
|
|
710
|
-
|
|
711
|
-
|
|
712
|
-
|
|
713
|
-
|
|
714
|
-
|
|
715
|
-
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
|
|
720
|
-
|
|
721
|
-
|
|
722
|
-
|
|
723
|
-
|
|
724
|
-
|
|
725
|
-
|
|
726
|
-
|
|
727
|
-
|
|
728
|
-
|
|
729
|
-
|
|
730
|
-
|
|
731
|
-
|
|
732
|
-
|
|
733
|
-
|
|
734
|
-
|
|
735
|
-
|
|
736
|
-
|
|
737
|
-
|
|
738
|
-
|
|
739
|
-
|
|
740
|
-
|
|
741
|
-
|
|
742
|
-
|
|
743
|
-
|
|
744
|
-
//
|
|
745
|
-
const
|
|
746
|
-
|
|
747
|
-
|
|
748
|
-
|
|
749
|
-
|
|
750
|
-
|
|
751
|
-
|
|
752
|
-
|
|
753
|
-
|
|
754
|
-
|
|
755
|
-
|
|
756
|
-
|
|
757
|
-
|
|
758
|
-
|
|
759
|
-
|
|
760
|
-
|
|
761
|
-
|
|
762
|
-
|
|
763
|
-
|
|
764
|
-
|
|
765
|
-
|
|
766
|
-
|
|
767
|
-
|
|
768
|
-
**
|
|
769
|
-
|
|
770
|
-
|
|
771
|
-
|
|
772
|
-
|
|
773
|
-
|
|
774
|
-
|
|
775
|
-
|
|
776
|
-
|
|
777
|
-
|
|
778
|
-
|
|
779
|
-
|
|
780
|
-
|
|
781
|
-
|
|
782
|
-
|
|
783
|
-
|
|
784
|
-
|
|
785
|
-
|
|
786
|
-
|
|
787
|
-
|
|
788
|
-
|
|
789
|
-
|
|
790
|
-
|
|
791
|
-
|
|
792
|
-
|
|
793
|
-
|
|
794
|
-
|
|
795
|
-
|
|
796
|
-
|
|
797
|
-
|
|
798
|
-
|
|
799
|
-
|
|
800
|
-
|
|
801
|
-
|
|
802
|
-
|
|
803
|
-
|
|
804
|
-
|
|
805
|
-
|
|
806
|
-
|
|
807
|
-
|
|
808
|
-
|
|
809
|
-
`
|
|
810
|
-
|
|
811
|
-
|
|
812
|
-
|
|
813
|
-
|
|
814
|
-
|
|
815
|
-
|
|
816
|
-
|
|
817
|
-
|
|
818
|
-
|
|
819
|
-
|
|
820
|
-
|
|
821
|
-
|
|
822
|
-
|
|
823
|
-
|
|
824
|
-
|
|
825
|
-
|
|
826
|
-
|
|
827
|
-
|
|
|
828
|
-
|
|
829
|
-
| `src/
|
|
830
|
-
| `
|
|
831
|
-
| `
|
|
832
|
-
| `
|
|
833
|
-
| `
|
|
834
|
-
|
|
835
|
-
|
|
836
|
-
|
|
837
|
-
|
|
838
|
-
|
|
839
|
-
|
|
840
|
-
|
|
841
|
-
|
|
842
|
-
|
|
843
|
-
|
|
844
|
-
|
|
845
|
-
|
|
846
|
-
|
|
847
|
-
|
|
848
|
-
|
|
849
|
-
|
|
850
|
-
|
|
851
|
-
|
|
852
|
-
|
|
853
|
-
|
|
854
|
-
|
|
855
|
-
|
|
856
|
-
|
|
857
|
-
|
|
858
|
-
|
|
1
|
+
# @x12i/countex
|
|
2
|
+
|
|
3
|
+
> Generic sparse counter matrices for JSON events.
|
|
4
|
+
> Dynamic counting, aggregation, and retrieval from mapped JSON inputs.
|
|
5
|
+
|
|
6
|
+
`@x12i/countex` is a **generic mapped-event counter**: a source-agnostic counting engine. You give it any JSON event, a declarative mapping config, and storage adapters; it projects events into sparse counter cells, batches deltas locally, and persists them atomically to a durable store (MongoDB or PostgreSQL) and a hot cache (Redis).
|
|
7
|
+
|
|
8
|
+
**Package policy:** Countex is generic. It never references a specific host product, client name, schema, or storage layout. Host-specific behavior belongs in the host application, not in this package.
|
|
9
|
+
|
|
10
|
+
It does **not** know about users, organizations, documents, labels, or events. It only knows generic primitives: **scope, subject, metric, dimensions, timeframe, delta**. Whatever your domain calls those things — Countex doesn't care.
|
|
11
|
+
|
|
12
|
+
**Terminology:** `metric` is the generic Countex name for the thing being counted or categorized. In your domain that may be a label, status, event name, operation, outcome, class, model result, or business category — the `metrics` array in a map simply lists JSON paths that resolve to those categories. Storage and APIs refer to dictionary **`sourceKey`** (the string extracted from the event); avoid confusing that with application-level UUIDs unless your map uses UUIDs as keys.
|
|
13
|
+
|
|
14
|
+
**Process memory vs retained state:** Each worker keeps **unflushed deltas** in an in-memory buffer. After flush, authoritative aggregates live in **durable** storage (MongoDB or PostgreSQL); **Redis** holds a hot cache. Restarting a worker loses only buffered deltas that were not flushed yet — tune flush intervals accordingly.
|
|
15
|
+
|
|
16
|
+
```
|
|
17
|
+
event JSON ──► map ──► raw increments ──► buffer ──► flush ──► Mongo or Postgres (durable)
|
|
18
|
+
└───► Redis (hot cache)
|
|
19
|
+
|
|
20
|
+
◄── query ◄── Redis (full coverage) or ◄── durable fallback ──◄
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## Install
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
npm install @x12i/countex
|
|
29
|
+
# Bring your own clients — peer deps, both optional:
|
|
30
|
+
npm install mongodb ioredis
|
|
31
|
+
# or, for Postgres durable storage instead of Mongo:
|
|
32
|
+
npm install pg ioredis
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Node 18+. Zero runtime dependencies in the core package.
|
|
36
|
+
|
|
37
|
+
### Documentation shipped with the package
|
|
38
|
+
|
|
39
|
+
These guides are included in the published tarball under `docs/` and exposed as stable import subpaths (resolve to `.md` files for editors and tooling):
|
|
40
|
+
|
|
41
|
+
| Topic | Package subpath | File |
|
|
42
|
+
|-------|-----------------|------|
|
|
43
|
+
| Release & production deployment | `@x12i/countex/docs/deployment` | `docs/DEPLOYMENT.md` |
|
|
44
|
+
| Tutorial — install → map → ingest → query | `@x12i/countex/docs/end-to-end-example` | `docs/END_TO_END_EXAMPLE.md` |
|
|
45
|
+
| Feature guides index | `@x12i/countex/docs/guides` | `docs/guides/README.md` |
|
|
46
|
+
| Counter maps | `@x12i/countex/docs/guides/counter-maps` | `docs/guides/counter-maps.md` |
|
|
47
|
+
| Preview a map (dry-run) | `@x12i/countex/docs/guides/preview` | `docs/guides/preview.md` |
|
|
48
|
+
| Ingestion & flush | `@x12i/countex/docs/guides/ingestion-and-flush` | `docs/guides/ingestion-and-flush.md` |
|
|
49
|
+
| Timeframes | `@x12i/countex/docs/guides/timeframes-and-bucketing` | `docs/guides/timeframes-and-bucketing.md` |
|
|
50
|
+
| Measures | `@x12i/countex/docs/guides/measures` | `docs/guides/measures.md` |
|
|
51
|
+
| Querying | `@x12i/countex/docs/guides/querying` | `docs/guides/querying.md` |
|
|
52
|
+
| Memory adapters | `@x12i/countex/docs/guides/adapter-memory` | `docs/guides/adapter-memory.md` |
|
|
53
|
+
| Mongo durable | `@x12i/countex/docs/guides/adapter-mongo` | `docs/guides/adapter-mongo.md` |
|
|
54
|
+
| Postgres durable | `@x12i/countex/docs/guides/adapter-postgres` | `docs/guides/adapter-postgres.md` |
|
|
55
|
+
| Mongo to Postgres | `@x12i/countex/docs/guides/migrating-mongo-to-postgres` | `docs/guides/migrating-mongo-to-postgres.md` |
|
|
56
|
+
| Redis cache | `@x12i/countex/docs/guides/adapter-redis` | `docs/guides/adapter-redis.md` |
|
|
57
|
+
| Cardinality | `@x12i/countex/docs/guides/cardinality` | `docs/guides/cardinality.md` |
|
|
58
|
+
| Idempotency | `@x12i/countex/docs/guides/idempotency` | `docs/guides/idempotency.md` |
|
|
59
|
+
| Backup export | `@x12i/countex/docs/guides/backup-export` | `docs/guides/backup-export.md` |
|
|
60
|
+
| Client events | `@x12i/countex/docs/guides/client-events` | `docs/guides/client-events.md` |
|
|
61
|
+
| Insights overview | `@x12i/countex/docs/guides/insights-overview` | `docs/guides/insights-overview.md` |
|
|
62
|
+
| Insights ranking | `@x12i/countex/docs/guides/insights-ranking` | `docs/guides/insights-ranking.md` |
|
|
63
|
+
| Insights comparison & trends | `@x12i/countex/docs/guides/insights-comparison-and-trends` | `docs/guides/insights-comparison-and-trends.md` |
|
|
64
|
+
| Insights anomalies & rules | `@x12i/countex/docs/guides/insights-anomalies-and-rules` | `docs/guides/insights-anomalies-and-rules.md` |
|
|
65
|
+
| Insights forecasting & budget | `@x12i/countex/docs/guides/insights-forecasting-and-budget` | `docs/guides/insights-forecasting-and-budget.md` |
|
|
66
|
+
| Insights advisors | `@x12i/countex/docs/guides/insights-advisors-and-optimization` | `docs/guides/insights-advisors-and-optimization.md` |
|
|
67
|
+
| CLI | `@x12i/countex/docs/guides/cli` | `docs/guides/cli.md` |
|
|
68
|
+
| Simulation runtime (JS API) | `@x12i/countex/simulation` | `dist/simulation/` |
|
|
69
|
+
| Simulation overview | `@x12i/countex/docs/guides/simulation` | `docs/guides/simulation.md` |
|
|
70
|
+
| Simulation profiles | `@x12i/countex/docs/guides/simulation-profiles` | `docs/guides/simulation-profiles.md` |
|
|
71
|
+
| Simulation resource estimation | `@x12i/countex/docs/guides/simulation-resource-estimation` | `docs/guides/simulation-resource-estimation.md` |
|
|
72
|
+
|
|
73
|
+
**Use cases** (technical playbooks: contracts, fan-out math, exact `QuerySpec`s, production wiring, verification — see [`docs/use-cases/README.md`](docs/use-cases/README.md)):
|
|
74
|
+
|
|
75
|
+
| Topic | Package subpath | File |
|
|
76
|
+
|-------|-----------------|------|
|
|
77
|
+
| Use cases index | `@x12i/countex/docs/use-cases` | `docs/use-cases/README.md` |
|
|
78
|
+
| Audit & security logs (volume, situations, distinct entities, logical event totals) | `@x12i/countex/docs/use-cases/audit-security-log-counts` | `docs/use-cases/audit-security-log-counts.md` |
|
|
79
|
+
| Product & feature activity | `@x12i/countex/docs/use-cases/product-feature-usage` | `docs/use-cases/product-feature-usage.md` |
|
|
80
|
+
| LLM cost & token metering | `@x12i/countex/docs/use-cases/llm-cost-and-token-metering` | `docs/use-cases/llm-cost-and-token-metering.md` |
|
|
81
|
+
| Multi-tenant isolation | `@x12i/countex/docs/use-cases/multi-tenant-isolation` | `docs/use-cases/multi-tenant-isolation.md` |
|
|
82
|
+
| Idempotent webhooks & queues | `@x12i/countex/docs/use-cases/webhook-idempotent-ingestion` | `docs/use-cases/webhook-idempotent-ingestion.md` |
|
|
83
|
+
| Ops dashboards & rollups | `@x12i/countex/docs/use-cases/ops-dashboard-daily-rollups` | `docs/use-cases/ops-dashboard-daily-rollups.md` |
|
|
84
|
+
| Processing & workflow activity | `@x12i/countex/docs/use-cases/processing-workflow-activity` | `docs/use-cases/processing-workflow-activity.md` |
|
|
85
|
+
|
|
86
|
+
**Internals** (how the engine works — diagrams and traces) live under `docs/internals/` in the tarball; browse [`docs/internals/README.md`](docs/internals/README.md).
|
|
87
|
+
|
|
88
|
+
After install: `node_modules/@x12i/countex/docs/`.
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## The mental model
|
|
93
|
+
|
|
94
|
+
Countex turns mapped JSON events into sparse aggregate cells.
|
|
95
|
+
|
|
96
|
+
A counter cell is uniquely identified by:
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
(scope, counterId, timeframeType, timeframeKey, subjectType, subjectId, metricId, dimensionsHash)
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
Each event can produce one or many cell increments. A counter increment is defined by:
|
|
103
|
+
|
|
104
|
+
- **scope** — your tenant / org / account boundary
|
|
105
|
+
- **counterId** — which counter family this is (e.g. `access_by_subject_metric`)
|
|
106
|
+
- **timeframe** — `minute` / `hour` / `day` / `week` / `month` / `quarter` / `year` / `all`
|
|
107
|
+
- **subject** — the thing being counted against (e.g. an actor or a target)
|
|
108
|
+
- **metric** — the count category / label
|
|
109
|
+
- **dimensions** — arbitrary additional breakdowns
|
|
110
|
+
- **delta** — `+1`, `-1`, or any integer
|
|
111
|
+
|
|
112
|
+
Countex buffers increments locally, flushes atomic deltas to durable + cache, and serves queries from cache with durable fallback.
|
|
113
|
+
|
|
114
|
+
### Measures and rates
|
|
115
|
+
|
|
116
|
+
Maps may declare optional **`measures`**: named numeric fields per event (cost, tokens, latency, …) with aggregations `sum`, `count`, `min`, and `max`. **`avg`** is computed at query time when both `sum` and `count` exist on the measure. Use **`scale`** on a measure for fixed-point storage (e.g. cents). Extracted values attach to each fan-out increment the same way as **`delta`** (see RFC-style docs for fan-out semantics).
|
|
117
|
+
|
|
118
|
+
Queries accept **`measures`** (explicit list or `"*"`), **`orderBy`** on a measure aggregation, and optional **`rate`** (count or a measure’s sum divided by elapsed time in the queried window; for `format: "timeseries"`, per bucket). **`rate`** is invalid for timeframe **`all`**.
|
|
119
|
+
|
|
120
|
+
**Redis cache** stores **sum** and **count** per measure only; **min** / **max** are durable-backed. If a query needs min/max (including `orderBy` on them, or requesting measures that maintain min/max when using `"*"`), the planner reads from **durable** first so cache cannot drift.
|
|
121
|
+
|
|
122
|
+
**`bucketRange`** caps at **10,000** buckets; at **`minute`** resolution that is about **6.94 days** — keep queries within that bound or narrow the range.
|
|
123
|
+
|
|
124
|
+
### Percentiles & histograms
|
|
125
|
+
|
|
126
|
+
Core cells store **additive** measure aggregates (`sum / count / min / max`) per sparse cell. They do **not** store distributions, so percentile-style answers (p50/p90/p95/p99) can only come from min/max approximation or from histogram/sketch storage you add yourself. Insights advisors that mention percentile-shaped recommendations are heuristic; see [`docs/guides/insights-overview.md`](docs/guides/insights-overview.md) §"Distributions" and [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) "Architecture decisions".
|
|
127
|
+
|
|
128
|
+
---
|
|
129
|
+
|
|
130
|
+
## What kinds of events Countex is for
|
|
131
|
+
|
|
132
|
+
Countex is not tied to one event source or domain. It is a **generic mapped-event counter**: any JSON-like input stream where useful aggregates can be produced from paths in the event.
|
|
133
|
+
|
|
134
|
+
Typical sources include:
|
|
135
|
+
|
|
136
|
+
| Source family | Examples | Common Countex projection |
|
|
137
|
+
|---|---|---|
|
|
138
|
+
| Audit and security logs | Microsoft 365 audit logs, Google Workspace logs, Okta logs, cloud audit logs | actor/resource/action/situation/severity |
|
|
139
|
+
| Product and user activity | Woopra, Segment, PostHog-style events, application activity logs | user/account/feature/event/page/funnel |
|
|
140
|
+
| Processing and workflow activity | ETL runs, background jobs, agent graphs, workflow engines, queue processors | job/task/node/status/error/runtime |
|
|
141
|
+
| LLM and AI service usage | Model gateway logs, OpenAI/Anthropic calls, internal LLM services | app/user/model/provider/tokens/cost/latency/result |
|
|
142
|
+
| Business/domain events | CRM, support tickets, payments, commerce, lifecycle events | customer/object/state/category/outcome |
|
|
143
|
+
|
|
144
|
+
Countex only needs a map. The map tells Countex where to find:
|
|
145
|
+
|
|
146
|
+
- the scope boundary
|
|
147
|
+
- the event time
|
|
148
|
+
- the subjects
|
|
149
|
+
- the metrics
|
|
150
|
+
- the dimensions
|
|
151
|
+
- the delta/count
|
|
152
|
+
- optional numeric measures such as cost, tokens, latency, or duration
|
|
153
|
+
|
|
154
|
+
The original event can come from a log, webhook, queue message, analytics event, system activity, processing trace, or generated test fixture. Countex does not care about the event source as long as the map can project it into generic counter primitives.
|
|
155
|
+
|
|
156
|
+
### Audit-log style event
|
|
157
|
+
|
|
158
|
+
```json
|
|
159
|
+
{
|
|
160
|
+
"tenant": { "id": "org_123" },
|
|
161
|
+
"event": {
|
|
162
|
+
"id": "audit_001",
|
|
163
|
+
"createdAt": "2026-05-08T14:33:00Z",
|
|
164
|
+
"operation": "FileAccessed",
|
|
165
|
+
"situation": "external"
|
|
166
|
+
},
|
|
167
|
+
"actor": { "id": "user_42", "type": "user" },
|
|
168
|
+
"resource": { "id": "file_99", "type": "document" },
|
|
169
|
+
"labels": ["PII", "Financial"]
|
|
170
|
+
}
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
Counts:
|
|
174
|
+
|
|
175
|
+
```text
|
|
176
|
+
actor/resource × label × operation/situation/resourceType × timeframe
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
### Product-activity style event
|
|
180
|
+
|
|
181
|
+
```json
|
|
182
|
+
{
|
|
183
|
+
"account": { "id": "acct_123" },
|
|
184
|
+
"user": { "id": "user_42" },
|
|
185
|
+
"event": {
|
|
186
|
+
"name": "dashboard_viewed",
|
|
187
|
+
"createdAt": "2026-05-08T14:33:00Z"
|
|
188
|
+
},
|
|
189
|
+
"product": {
|
|
190
|
+
"feature": "risk_dashboard",
|
|
191
|
+
"plan": "enterprise"
|
|
192
|
+
}
|
|
193
|
+
}
|
|
194
|
+
```
|
|
195
|
+
|
|
196
|
+
Counts:
|
|
197
|
+
|
|
198
|
+
```text
|
|
199
|
+
user/account × activity metric × feature/plan × timeframe
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
### LLM-processing style event
|
|
203
|
+
|
|
204
|
+
```json
|
|
205
|
+
{
|
|
206
|
+
"scope": { "id": "org_123" },
|
|
207
|
+
"request": {
|
|
208
|
+
"id": "llm_req_001",
|
|
209
|
+
"createdAt": "2026-05-08T14:33:00Z",
|
|
210
|
+
"status": "success"
|
|
211
|
+
},
|
|
212
|
+
"app": { "id": "app-1" },
|
|
213
|
+
"model": {
|
|
214
|
+
"provider": "openai",
|
|
215
|
+
"name": "gpt-4.1"
|
|
216
|
+
},
|
|
217
|
+
"usage": {
|
|
218
|
+
"inputTokens": 1200,
|
|
219
|
+
"outputTokens": 350,
|
|
220
|
+
"costCents": 4,
|
|
221
|
+
"latencyMs": 1800
|
|
222
|
+
}
|
|
223
|
+
}
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
Counts/measures:
|
|
227
|
+
|
|
228
|
+
```text
|
|
229
|
+
app/model/provider/status × timeframe
|
|
230
|
+
sum inputTokens
|
|
231
|
+
sum outputTokens
|
|
232
|
+
sum costCents
|
|
233
|
+
avg latencyMs
|
|
234
|
+
count requests
|
|
235
|
+
```
|
|
236
|
+
|
|
237
|
+
---
|
|
238
|
+
|
|
239
|
+
## Quick start
|
|
240
|
+
|
|
241
|
+
```ts
|
|
242
|
+
import { createCountex } from "@x12i/countex";
|
|
243
|
+
import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory";
|
|
244
|
+
|
|
245
|
+
const accessMap = {
|
|
246
|
+
counterId: "access_by_subject_metric",
|
|
247
|
+
|
|
248
|
+
scope: { path: "$.tenant.id" },
|
|
249
|
+
eventTime: { path: "$.event.createdAt" },
|
|
250
|
+
|
|
251
|
+
subjects: [
|
|
252
|
+
{ type: "actor", path: "$.actor.id", required: true },
|
|
253
|
+
{ type: "target", path: "$.target.id", required: false },
|
|
254
|
+
],
|
|
255
|
+
|
|
256
|
+
metrics: [
|
|
257
|
+
{ path: "$.labels[*]", required: true },
|
|
258
|
+
],
|
|
259
|
+
|
|
260
|
+
dimensions: [
|
|
261
|
+
{ name: "eventType", path: "$.event.type" },
|
|
262
|
+
{ name: "situation", path: "$.event.situation" },
|
|
263
|
+
{ name: "targetType", path: "$.target.type" },
|
|
264
|
+
],
|
|
265
|
+
|
|
266
|
+
delta: { default: 1 },
|
|
267
|
+
timeframes: ["day", "week", "month", "quarter", "year"],
|
|
268
|
+
} as const;
|
|
269
|
+
|
|
270
|
+
const countex = createCountex({
|
|
271
|
+
maps: [accessMap],
|
|
272
|
+
durable: new MemoryDurable(),
|
|
273
|
+
cache: new MemoryCache(),
|
|
274
|
+
});
|
|
275
|
+
|
|
276
|
+
await countex.ingest({
|
|
277
|
+
tenant: { id: "org_123" },
|
|
278
|
+
event: { type: "share", situation: "external", createdAt: "2026-05-08T14:33:00Z" },
|
|
279
|
+
actor: { id: "user_42" },
|
|
280
|
+
target: { id: "doc_99", type: "document" },
|
|
281
|
+
labels: ["PCI", "PII"],
|
|
282
|
+
});
|
|
283
|
+
|
|
284
|
+
await countex.flush();
|
|
285
|
+
|
|
286
|
+
const result = await countex.query({
|
|
287
|
+
scope: "org_123",
|
|
288
|
+
counterId: "access_by_subject_metric",
|
|
289
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
|
|
290
|
+
groupBy: ["subject", "metric", "dim:eventType"],
|
|
291
|
+
format: "table",
|
|
292
|
+
orderBy: "-count",
|
|
293
|
+
limit: 100,
|
|
294
|
+
});
|
|
295
|
+
|
|
296
|
+
console.log(result.rows);
|
|
297
|
+
|
|
298
|
+
await countex.shutdown();
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
That single ingest produces:
|
|
302
|
+
|
|
303
|
+
- 1 actor + 1 target = **2 subjects**
|
|
304
|
+
- 2 labels = **2 metrics**
|
|
305
|
+
- 5 timeframes
|
|
306
|
+
- → **2 × 2 × 5 = 20 cell increments**
|
|
307
|
+
|
|
308
|
+
Each cell uses an atomic `$inc` on durable and `HINCRBY` on the cache, so 20 ingest workers can safely run in parallel.
|
|
309
|
+
|
|
310
|
+
### Preview a map (dry-run)
|
|
311
|
+
|
|
312
|
+
Use `countex.preview(event, { counterId? })` to inspect what one event would produce without any I/O — no dictionary allocation, no buffering, no writes. Returns one `MapPreviewResult` per matching map, each with `cells: PreviewIncrementCell[]` (subjects, metrics, dimensions, dimensionsHash, timeframe keys, deltas, measures), `extractionWarnings`, and `droppedAfterCardinality`. See [docs/guides/preview.md](docs/guides/preview.md).
|
|
313
|
+
|
|
314
|
+
---
|
|
315
|
+
|
|
316
|
+
## Production setup (Mongo + Redis)
|
|
317
|
+
|
|
318
|
+
Do **not** derive Mongo database names directly from raw tenant strings (length, `$`, `/`, and Unicode can break DB naming rules or leak structure). Hash or normalize first:
|
|
319
|
+
|
|
320
|
+
```ts
|
|
321
|
+
import { createHash } from "node:crypto";
|
|
322
|
+
import { MongoClient } from "mongodb";
|
|
323
|
+
import IORedis from "ioredis";
|
|
324
|
+
import { createCountex } from "@x12i/countex";
|
|
325
|
+
import { MongoDurable } from "@x12i/countex/adapters/mongo";
|
|
326
|
+
import { RedisCache } from "@x12i/countex/adapters/redis";
|
|
327
|
+
|
|
328
|
+
function hashScopeId(scope: string): string {
|
|
329
|
+
return createHash("sha256").update(scope, "utf8").digest("hex").slice(0, 16);
|
|
330
|
+
}
|
|
331
|
+
|
|
332
|
+
const mongo = new MongoClient(process.env.MONGO_URI!);
|
|
333
|
+
await mongo.connect();
|
|
334
|
+
const redis = new IORedis(process.env.REDIS_URL!);
|
|
335
|
+
|
|
336
|
+
const countex = createCountex({
|
|
337
|
+
maps: [accessMap],
|
|
338
|
+
durable: new MongoDurable({
|
|
339
|
+
client: mongo,
|
|
340
|
+
dbResolver: ({ scope }) => `countex_${hashScopeId(scope)}`,
|
|
341
|
+
}),
|
|
342
|
+
cache: new RedisCache({
|
|
343
|
+
client: redis,
|
|
344
|
+
keyPrefix: "countex",
|
|
345
|
+
// Optional adapter default TTLs; when `retention.redis` is set below, the
|
|
346
|
+
// engine resolves those strings to seconds and forwards them via
|
|
347
|
+
// `setRetentionPolicy`, overriding adapter defaults for matching timeframes.
|
|
348
|
+
}),
|
|
349
|
+
retention: {
|
|
350
|
+
redis: {
|
|
351
|
+
minute: "48h",
|
|
352
|
+
hour: "14d",
|
|
353
|
+
day: "92d",
|
|
354
|
+
week: "52w",
|
|
355
|
+
month: "24mo",
|
|
356
|
+
quarter: "24mo",
|
|
357
|
+
year: "forever",
|
|
358
|
+
all: "forever",
|
|
359
|
+
},
|
|
360
|
+
// Durable TTL / prune is not enforced by the engine. Operators apply Mongo
|
|
361
|
+
// TTL indexes, Postgres partition drops, or scheduled jobs.
|
|
362
|
+
// `retention.mongo` and `retention.postgres` are optional per-timeframe
|
|
363
|
+
// documentation (same duration grammar as redis); see types / DEPLOYMENT.
|
|
364
|
+
},
|
|
365
|
+
flush: {
|
|
366
|
+
everyMs: 60_000,
|
|
367
|
+
maxBufferedCells: 100_000,
|
|
368
|
+
maxBufferedEvents: 50_000,
|
|
369
|
+
flushOnShutdown: true,
|
|
370
|
+
},
|
|
371
|
+
cardinality: {
|
|
372
|
+
maxDimensionValuesPerName: 10_000,
|
|
373
|
+
onLimit: "overflow",
|
|
374
|
+
overflowValue: "__OTHER__",
|
|
375
|
+
},
|
|
376
|
+
});
|
|
377
|
+
|
|
378
|
+
await countex.init();
|
|
379
|
+
|
|
380
|
+
process.on("SIGTERM", async () => { await countex.shutdown(); });
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
---
|
|
384
|
+
|
|
385
|
+
## Mapping reference
|
|
386
|
+
|
|
387
|
+
A `CounterMap` is the heart of the package. Everything Countex knows about your data is in here.
|
|
388
|
+
|
|
389
|
+
| Field | Type | Notes |
|
|
390
|
+
|---------------|----------------------------|-------|
|
|
391
|
+
| `counterId` | `string` | Stable identifier for this counter family. |
|
|
392
|
+
| `scope` | `{ path }` | Path into the event for the tenant/scope id. Required. |
|
|
393
|
+
| `eventTime` | `{ path }` | Path to a Date / ISO string / epoch ms. Drives bucket selection. |
|
|
394
|
+
| `subjects` | `SubjectMapping[]` | One or many. Each has `type` and `path`; multi-value paths fan out. |
|
|
395
|
+
| `metrics` | `MetricMapping[]` | Same fanout semantics. `$.labels[*]` is common. |
|
|
396
|
+
| `measures` | `MeasureMapping[]` | Optional numeric fields per event (`sum` / `count` / `min` / `max`); see [Measures](./docs/guides/measures.md). |
|
|
397
|
+
| `dimensions` | `DimensionMapping[]` | Named extra breakdowns. Optional by default. |
|
|
398
|
+
| `delta` | `{ path?, default? }` | Numeric value at runtime. Default `1`. Negative deltas are fine. |
|
|
399
|
+
| `timeframes` | `TimeframeType[]` | Subset of `minute` / `hour` / `day` / `week` / `month` / `quarter` / `year` / `all`. |
|
|
400
|
+
| `idempotency` | `IdempotencyPolicy` | Optional; needs a cache adapter that implements `markEventSeen`. |
|
|
401
|
+
| `cardinality` | `CardinalityPolicy` | Optional; per-map override of engine-level defaults. |
|
|
402
|
+
| `options` | `MappingOptions` | `skipMissingOptionalDimensions` (default true), `normalizeValues` (default true). |
|
|
403
|
+
|
|
404
|
+
### Path syntax
|
|
405
|
+
|
|
406
|
+
A small JSONPath subset:
|
|
407
|
+
|
|
408
|
+
| Syntax | Meaning |
|
|
409
|
+
|-------------------|--------------------------------------------------|
|
|
410
|
+
| `$` | Root. |
|
|
411
|
+
| `.name` | Property. |
|
|
412
|
+
| `['name']` | Property (bracket form, allows arbitrary chars). |
|
|
413
|
+
| `[N]` | Numeric array index (negatives allowed). |
|
|
414
|
+
| `[*]` | Wildcard — fan out across array elems / object values. |
|
|
415
|
+
|
|
416
|
+
Examples:
|
|
417
|
+
|
|
418
|
+
```
|
|
419
|
+
$.tenant.id
|
|
420
|
+
$.actor.id
|
|
421
|
+
$.labels[*]
|
|
422
|
+
$.items[*].id
|
|
423
|
+
$['weird.key.with.dots']
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
### Fanout
|
|
427
|
+
|
|
428
|
+
A single event with 2 subjects and 3 labels (metrics) and 5 selected timeframes produces **2 × 3 × 5 = 30** atomic increments. This is intentional — read your mapping carefully. Countex emits a `dropped` event when cardinality protection bites, so you'll know.
|
|
429
|
+
|
|
430
|
+
---
|
|
431
|
+
|
|
432
|
+
## Storage adapters
|
|
433
|
+
|
|
434
|
+
Countex defines two contracts and ships three adapters.
|
|
435
|
+
|
|
436
|
+
```ts
|
|
437
|
+
interface DurableStorage {
|
|
438
|
+
applyDeltas(deltas: PendingDelta[]): Promise<void>;
|
|
439
|
+
readCells(spec: QuerySpec): Promise<CounterCell[]>;
|
|
440
|
+
resolveIds(lookups: DictionaryLookup[]): Promise<Map<string, number>>;
|
|
441
|
+
streamCells(filter): AsyncIterable<CounterCell>;
|
|
442
|
+
init?(): Promise<void>;
|
|
443
|
+
close?(): Promise<void>;
|
|
444
|
+
}
|
|
445
|
+
|
|
446
|
+
interface CacheStorage {
|
|
447
|
+
applyDeltas(deltas: PendingDelta[]): Promise<void>;
|
|
448
|
+
readCells(spec: QuerySpec): Promise<{ cells: CounterCell[]; coverage: "full" | "partial" | "miss" }>;
|
|
449
|
+
warm(cells: CounterCell[]): Promise<void>;
|
|
450
|
+
markEventSeen?(scopeId, eventId, ttlSeconds): Promise<boolean>;
|
|
451
|
+
init?(): Promise<void>;
|
|
452
|
+
close?(): Promise<void>;
|
|
453
|
+
}
|
|
454
|
+
```
|
|
455
|
+
|
|
456
|
+
### Built-in adapters
|
|
457
|
+
|
|
458
|
+
| Module | Purpose |
|
|
459
|
+
|-----------------------------------|--------------------------------------|
|
|
460
|
+
| `@x12i/countex/adapters/memory` | In-process durable + cache; tests / local dev. `MemoryCache` + Mongo is single-process only (`init` warns). |
|
|
461
|
+
| `@x12i/countex/adapters/mongo` | Durable. Requires `mongodb` peer. |
|
|
462
|
+
| `@x12i/countex/adapters/postgres` | Durable. Requires `pg` peer. PostgreSQL 15+. |
|
|
463
|
+
| `@x12i/countex/adapters/redis` | Hot cache. Requires `ioredis` peer. |
|
|
464
|
+
| `@x12i/countex/adapters/durable-cache` | No Redis: each query reads durable once. Low volume / one machine. |
|
|
465
|
+
|
|
466
|
+
Bring your own clients — Countex doesn't open or close them. That keeps lifecycle in the host app's hands.
|
|
467
|
+
|
|
468
|
+
### Mongo schema
|
|
469
|
+
|
|
470
|
+
`countex_cells`:
|
|
471
|
+
|
|
472
|
+
```json
|
|
473
|
+
{
|
|
474
|
+
"scopeId": "org_123",
|
|
475
|
+
"counterId": "access_by_subject_metric",
|
|
476
|
+
"timeframeType": "day",
|
|
477
|
+
"timeframeKey": "2026-05-08",
|
|
478
|
+
"subjectType": "actor",
|
|
479
|
+
"subjectId": 10001,
|
|
480
|
+
"metricId": 41,
|
|
481
|
+
"dimensionsHash": "a8f21c0b1d2e",
|
|
482
|
+
"dimensions": { "eventType": "share", "situation": "external" },
|
|
483
|
+
"count": 42,
|
|
484
|
+
"createdAt": "...",
|
|
485
|
+
"updatedAt": "..."
|
|
486
|
+
}
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
Indexes (created automatically on first write):
|
|
490
|
+
|
|
491
|
+
- `uniq_cell_identity` (unique, on the full identity tuple)
|
|
492
|
+
- `by_timeframe` — for range scans
|
|
493
|
+
- `by_subject` — for subject filters
|
|
494
|
+
- `by_metric` — for metric filters
|
|
495
|
+
|
|
496
|
+
`countex_dict` — the per-scope subject/metric dictionary, with monotonic ids allocated via `countex_seq`. Allocation is **concurrency-safe by construction**: read existing → atomic sequence `$inc` → `insertMany({ ordered: false })` → on duplicate-key, re-read. See [`docs/guides/adapter-mongo.md`](docs/guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency" and [`docs/internals/internal-storage-mongo.md`](docs/internals/internal-storage-mongo.md).
|
|
497
|
+
|
|
498
|
+
`dimensionsHash` is canonicalized so any two workers produce the same hash for the same logical dimension tuple: keys sorted lexicographically, values type-tagged (`n: / b: / i: / s:`), serialized, hashed (SHA-1, truncated to 12 hex). See [`docs/internals/internal-dimension-hash.md`](docs/internals/internal-dimension-hash.md).
|
|
499
|
+
|
|
500
|
+
### Redis layout
|
|
501
|
+
|
|
502
|
+
```
|
|
503
|
+
countex:{scope}:{counterId}:{timeframeType}:{timeframeKey} → HASH
|
|
504
|
+
field = "{subjectType}|{subjectId}|{metricId}|{dimensionsHash}"
|
|
505
|
+
value = count
|
|
506
|
+
|
|
507
|
+
countex:{scope}:{counterId}:{timeframeType}:{timeframeKey}:cov → HASH (coverage meta)
|
|
508
|
+
status = complete | partial | rebuilding (see BucketCoverageMeta; reads may infer missing)
|
|
509
|
+
lastWarmAt = epoch ms (string)
|
|
510
|
+
mapVersion = optional opaque integer (string)
|
|
511
|
+
schemaVersion = cache layout version (string)
|
|
512
|
+
cellCount = cells last warmed into this bucket (string, when known)
|
|
513
|
+
|
|
514
|
+
countex:{scope}:{counterId}:...:{timeframeKey}:meta:{field} → JSON of dimensions
|
|
515
|
+
|
|
516
|
+
countex:{scope}:dedupe:{eventId} → "1" with TTL
|
|
517
|
+
```
|
|
518
|
+
|
|
519
|
+
One hash per `(scope, counterId, bucket)` keeps Redis key counts small and per-bucket reads to a single `HGETALL`. **Do not** treat “cell hash is non-empty” as full cache coverage — use the **`:cov`** hash (`status === complete` after `warm`, `partial` after incremental `applyDeltas`, `rebuilding` during cache rebuild). See [`docs/guides/adapter-redis.md`](docs/guides/adapter-redis.md) and [`docs/internals/internal-storage-redis.md`](docs/internals/internal-storage-redis.md).
|
|
520
|
+
|
|
521
|
+
---
|
|
522
|
+
|
|
523
|
+
## Query API
|
|
524
|
+
|
|
525
|
+
```ts
|
|
526
|
+
const result = await countex.query({
|
|
527
|
+
scope: "org_123",
|
|
528
|
+
counterId: "access_by_subject_metric",
|
|
529
|
+
|
|
530
|
+
timeframe: {
|
|
531
|
+
type: "day",
|
|
532
|
+
from: "2026-05-01",
|
|
533
|
+
to: "2026-05-08",
|
|
534
|
+
},
|
|
535
|
+
|
|
536
|
+
// Filters — all optional (subject.ids / metrics: internal numbers and/or dictionary source strings)
|
|
537
|
+
subject: { type: "actor", ids: [10001, 10002, "user_uuid_a"] },
|
|
538
|
+
metrics: [41, 42, "PCI"],
|
|
539
|
+
dimensions: { eventType: ["share", "download"], situation: ["external"] },
|
|
540
|
+
|
|
541
|
+
// Shape
|
|
542
|
+
groupBy: ["subject", "metric", "dim:eventType"],
|
|
543
|
+
format: "table", // "cells" | "table" | "matrix" | "timeseries"
|
|
544
|
+
orderBy: "-count",
|
|
545
|
+
limit: 100,
|
|
546
|
+
|
|
547
|
+
// Optional: use dictionary source keys in grouped keys / cells instead of numeric ids only
|
|
548
|
+
resolveLabels: true,
|
|
549
|
+
|
|
550
|
+
// Diagnostics
|
|
551
|
+
bypassCache: false,
|
|
552
|
+
signal: AbortSignal.timeout(5_000),
|
|
553
|
+
});
|
|
554
|
+
```
|
|
555
|
+
|
|
556
|
+
**Filters:** `subject.ids` and `metrics` may mix **internal numeric ids** (from ingest) and **string source keys** (UUIDs, labels, and so on). String subject keys require `subject.type`. Set **`resolveLabels: true`** to return source keys in grouped `keys.subject` / `keys.metric`, and `subjectSourceKey` / `metricSourceKey` on **`format: "cells"`** rows. See [`docs/guides/querying.md`](docs/guides/querying.md).
|
|
557
|
+
|
|
558
|
+
`explainQuery` runs the planner **without** materializing cells. It returns a **`QueryPlan`**: expanded `bucketsRequested`, `cacheEligible` / `cacheReason`, predicted `durableFallbackPredicted`, `cacheBucketsExpectedHit` (when the cache adapter supports `probeBucketCoverage`), heuristic `plannedCellsEstimate`, and cost hints (`expensive`, `groupByCost`). Use it to preflight dashboards and expensive `groupBy` shapes.
|
|
559
|
+
|
|
560
|
+
```ts
|
|
561
|
+
const plan = await countex.explainQuery({
|
|
562
|
+
scope: "org_123",
|
|
563
|
+
counterId: "access_by_subject_metric",
|
|
564
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
|
|
565
|
+
groupBy: ["subject", "metric"],
|
|
566
|
+
format: "table",
|
|
567
|
+
});
|
|
568
|
+
```
|
|
569
|
+
|
|
570
|
+
Result:
|
|
571
|
+
|
|
572
|
+
```ts
|
|
573
|
+
{
|
|
574
|
+
format: "table",
|
|
575
|
+
rows: [
|
|
576
|
+
{ keys: { subject: 10001, metric: 41, "dim:eventType": "share" }, count: 142 },
|
|
577
|
+
// With resolveLabels: true (and known dictionary rows), subject/metric keys are strings:
|
|
578
|
+
// { keys: { subject: "user_42", metric: "PCI", "dim:eventType": "share" }, count: 142 },
|
|
579
|
+
...
|
|
580
|
+
],
|
|
581
|
+
total: 1234,
|
|
582
|
+
source: "cache" | "durable" | "merged",
|
|
583
|
+
}
|
|
584
|
+
```
|
|
585
|
+
|
|
586
|
+
String filters are resolved to internal ids before reads; unknown source keys are omitted from the filter. With **`resolveLabels: true`**, **`format: "cells"`** rows include **`subjectSourceKey`** and **`metricSourceKey`** in addition to numeric ids.
|
|
587
|
+
|
|
588
|
+
`groupBy` accepts:
|
|
589
|
+
- `"subject"`, `"subjectType"`, `"metric"`, `"timeframeKey"`
|
|
590
|
+
- `"dim:NAME"` — group by a dimension value
|
|
591
|
+
|
|
592
|
+
---
|
|
593
|
+
|
|
594
|
+
## Insights and advisors
|
|
595
|
+
|
|
596
|
+
Countex can optionally analyze stored counters and measures to produce higher-level insights. The core engine answers “what happened?” by maintaining sparse aggregate cells. The insights layer answers “what matters?” by ranking, comparing periods, detecting anomalies, forecasting usage, and emitting evidence-backed recommendations.
|
|
597
|
+
|
|
598
|
+
Typical uses:
|
|
599
|
+
|
|
600
|
+
- Top / bottom N subjects, metrics, or dimensions; top/bottom percent of groups; Pareto and contribution views
|
|
601
|
+
- Period-over-period comparison and simple trends on timeseries
|
|
602
|
+
- Explainable anomaly detection (robust z-score, thresholds, missing activity)
|
|
603
|
+
- Rule evaluation with normalized recommendations
|
|
604
|
+
- Cost / budget / max-token style advisors (generic measures only — no domain lock-in)
|
|
605
|
+
- Map and cardinality health when you pass the counter map into `createCountexInsights`
|
|
606
|
+
- Resource projections from observed cells via `estimateResourcesFromActuals` (requires the durable adapter reference)
|
|
607
|
+
|
|
608
|
+
**Safety:** Insights honor `maxBuckets` (aligned with the core 10,000 bucket cap) and `maxGroups`. Grouped queries load all matching cells for the timeframe before aggregation, so narrow filters when possible.
|
|
609
|
+
|
|
610
|
+
**Sparse counters:** Bottom-N style rankings omit zero rows by default; Countex only stores cells that received increments.
|
|
611
|
+
|
|
612
|
+
```ts
|
|
613
|
+
import { createCountexInsights } from "@x12i/countex/insights";
|
|
614
|
+
|
|
615
|
+
const insights = createCountexInsights({
|
|
616
|
+
countex,
|
|
617
|
+
durable, // optional; required for estimateResourcesFromActuals → streamCells
|
|
618
|
+
mapsByCounterId: new Map(maps.map((m) => [m.counterId, m])),
|
|
619
|
+
});
|
|
620
|
+
|
|
621
|
+
const topCostDrivers = await insights.topPercent({
|
|
622
|
+
scope: "org_123",
|
|
623
|
+
counterId: "llm_usage",
|
|
624
|
+
timeframe: { type: "month", from: "2026-05", to: "2026-05" },
|
|
625
|
+
groupBy: ["subject"],
|
|
626
|
+
measure: "costMicros.sum",
|
|
627
|
+
percent: 20,
|
|
628
|
+
});
|
|
629
|
+
```
|
|
630
|
+
|
|
631
|
+
Further reading: [`docs/guides/insights-overview.md`](docs/guides/insights-overview.md), [`docs/guides/insights-ranking.md`](docs/guides/insights-ranking.md), [`docs/guides/insights-comparison-and-trends.md`](docs/guides/insights-comparison-and-trends.md), [`docs/guides/insights-anomalies-and-rules.md`](docs/guides/insights-anomalies-and-rules.md), [`docs/guides/insights-forecasting-and-budget.md`](docs/guides/insights-forecasting-and-budget.md), [`docs/guides/insights-advisors-and-optimization.md`](docs/guides/insights-advisors-and-optimization.md). Deeper topic notes also live under `docs/INSIGHTS*.md` for maintainers.
|
|
632
|
+
|
|
633
|
+
CLI (after `npm run build` / install):
|
|
634
|
+
|
|
635
|
+
```bash
|
|
636
|
+
npx countex insights:top ./path/to/insight-context.mjs --scope org_123 --counter my_counter
|
|
637
|
+
# TypeScript configs: node --import=tsx node_modules/@x12i/countex/dist/cli.js insights:pareto ./ctx.ts
|
|
638
|
+
```
|
|
639
|
+
|
|
640
|
+
---
|
|
641
|
+
|
|
642
|
+
## Events
|
|
643
|
+
|
|
644
|
+
The Countex client is an `EventEmitter`. Subscribe to observe what's happening:
|
|
645
|
+
|
|
646
|
+
```ts
|
|
647
|
+
countex.on("flush", ({ cells, durationMs }) => {});
|
|
648
|
+
countex.on("flush_error", ({ error }) => {});
|
|
649
|
+
countex.on("cardinality_warning", ({ kind, count, limit })=> {});
|
|
650
|
+
countex.on("overflow", ({ kind, sourceKey }) => {});
|
|
651
|
+
countex.on("dropped", ({ reason }) => {});
|
|
652
|
+
countex.on("duplicate", ({ scopeId, eventId }) => {});
|
|
653
|
+
```
|
|
654
|
+
|
|
655
|
+
All event payloads are typed via `CountexEvents`.
|
|
656
|
+
|
|
657
|
+
---
|
|
658
|
+
|
|
659
|
+
## Operational notes
|
|
660
|
+
|
|
661
|
+
### Buffering and flushing
|
|
662
|
+
|
|
663
|
+
Each Countex instance keeps only a delta buffer in memory:
|
|
664
|
+
|
|
665
|
+
```
|
|
666
|
+
Map<cellKey, delta>
|
|
667
|
+
```
|
|
668
|
+
|
|
669
|
+
Flush triggers (any of):
|
|
670
|
+
|
|
671
|
+
- `flush.everyMs` timer (default 60s)
|
|
672
|
+
- `flush.maxBufferedCells` reached (default 100k)
|
|
673
|
+
- `flush.maxBufferedEvents` reached (default 50k)
|
|
674
|
+
- `shutdown()` if `flushOnShutdown` is true (default)
|
|
675
|
+
- explicit `flush()`
|
|
676
|
+
|
|
677
|
+
Flush is **two-phase**: durable first, cache second. If durable fails, the snapshot is merged back into the buffer — counts are not lost. If cache fails after durable succeeded, it's logged and continues; the next read warms cache from durable.
|
|
678
|
+
|
|
679
|
+
At most one flush runs at a time. Concurrent callers receive the same in-flight promise.
|
|
680
|
+
|
|
681
|
+
### Cardinality protection
|
|
682
|
+
|
|
683
|
+
A bad mapping (e.g. a free-form URL as a dimension) can balloon the dictionary. Countex tracks per-`(scope, kind, qualifier)` how many distinct keys it has minted, and applies a policy when limits hit:
|
|
684
|
+
|
|
685
|
+
- `overflow` (default) — rewrite the source key to `__OTHER__`. Emits `overflow`.
|
|
686
|
+
- `drop` — silently drop the increment. Emits `dropped`.
|
|
687
|
+
- `error` — throw `CardinalityLimitError`.
|
|
688
|
+
|
|
689
|
+
Defaults are conservative; tune them per workload. A warning fires at 80% of the limit (`cardinality_warning`).
|
|
690
|
+
|
|
691
|
+
### Idempotency
|
|
692
|
+
|
|
693
|
+
Optional and per-map. Requires a cache adapter that implements `markEventSeen` (the included `RedisCache` and `MemoryCache` both do).
|
|
694
|
+
|
|
695
|
+
```ts
|
|
696
|
+
idempotency: {
|
|
697
|
+
enabled: true,
|
|
698
|
+
eventIdPath: "$.event.id",
|
|
699
|
+
ttlSeconds: 86_400,
|
|
700
|
+
}
|
|
701
|
+
```
|
|
702
|
+
|
|
703
|
+
Dedupe keys are namespaced by `counterId`, so different maps with their own idempotency don't collide on shared event ids.
|
|
704
|
+
|
|
705
|
+
### Late events
|
|
706
|
+
|
|
707
|
+
Bucket selection always uses the **event time**, never the ingestion time. Late-arriving events go into the correct historical bucket — `$inc` doesn't care about ordering.
|
|
708
|
+
|
|
709
|
+
### Concurrency safety
|
|
710
|
+
|
|
711
|
+
All writes are atomic per cell:
|
|
712
|
+
|
|
713
|
+
- Mongo: `bulkWrite` of `updateOne` with `$inc` and `upsert: true`
|
|
714
|
+
- Redis: `HINCRBY`
|
|
715
|
+
|
|
716
|
+
20+ Countex workers can safely run in parallel against the same scope. Reads are eventually consistent across the two stores; if Redis is behind durable, the planner will detect partial coverage and read durable.
|
|
717
|
+
|
|
718
|
+
### Backup / verify / restore / rebuild
|
|
719
|
+
|
|
720
|
+
Top-level free functions and a `countex.backup` namespace bound to your engine:
|
|
721
|
+
|
|
722
|
+
```ts
|
|
723
|
+
import {
|
|
724
|
+
exportCells,
|
|
725
|
+
computeCellsFingerprint,
|
|
726
|
+
createBackupManifest,
|
|
727
|
+
verifyBackupAgainstManifest,
|
|
728
|
+
restoreSnapshotCells,
|
|
729
|
+
rebuildCache,
|
|
730
|
+
COUNTEX_CELL_SCHEMA_VERSION,
|
|
731
|
+
} from "@x12i/countex";
|
|
732
|
+
|
|
733
|
+
const filter = {
|
|
734
|
+
scope: "org_123",
|
|
735
|
+
counterId: "access_by_subject_metric",
|
|
736
|
+
timeframe: { type: "day", from: "2026-02-01", to: "2026-05-08" },
|
|
737
|
+
};
|
|
738
|
+
|
|
739
|
+
// Stream all cells in a window — bounded async iteration.
|
|
740
|
+
for await (const cell of exportCells(durable, filter)) {
|
|
741
|
+
// ndjson, S3, whatever you want
|
|
742
|
+
}
|
|
743
|
+
|
|
744
|
+
// Fingerprint + manifest for cold-storage verification.
|
|
745
|
+
const fp = await countex.backup.fingerprint(filter); // { cellCount, contentSha256, countSum }
|
|
746
|
+
const manifest = countex.backup.createManifest(filter, fp); // { manifestVersion, schemaVersion, filter, … }
|
|
747
|
+
|
|
748
|
+
// Verify a manifest against the live durable adapter.
|
|
749
|
+
const ok = await countex.backup.verify(manifest);
|
|
750
|
+
if (!ok.ok) throw new Error(ok.detail);
|
|
751
|
+
|
|
752
|
+
// Restore. Modes: "validate-only" | "replace-window" (default) | "merge-deltas" (reserved).
|
|
753
|
+
const { restored, mode } = await countex.backup.restore(
|
|
754
|
+
countex.backup.export(filter),
|
|
755
|
+
{ mode: "replace-window" },
|
|
756
|
+
);
|
|
757
|
+
|
|
758
|
+
// Re-warm cache from durable. Each warmed bucket's :cov flips to status=complete.
|
|
759
|
+
await countex.backup.rebuildCache(filter);
|
|
760
|
+
```
|
|
761
|
+
|
|
762
|
+
See [`docs/guides/backup-export.md`](docs/guides/backup-export.md) for the full recipe and manifest layout.
|
|
763
|
+
|
|
764
|
+
---
|
|
765
|
+
|
|
766
|
+
## Docker integration testing
|
|
767
|
+
|
|
768
|
+
Countex ships a Docker-based stack (MongoDB, Redis, MinIO) and integration tests under `tests/integration/docker/` that exercise real adapters: dictionary allocation under concurrency, Mongo **`lookupIds` / `reverseLookupIds`**, an end-to-end **`query()`** with **source-key filters** and **`resolveLabels`**, multi-worker ingestion, Redis cache coverage and durable fallback, `rebuildCache`, simulation fingerprints against a memory reference, and backup / manifest / verify / restore to S3-compatible storage (via the AWS SDK in devDependencies).
|
|
769
|
+
|
|
770
|
+
**One-shot local run** (starts services, runs tests with default localhost URLs, tears volumes down):
|
|
771
|
+
|
|
772
|
+
```bash
|
|
773
|
+
npm run test:e2e
|
|
774
|
+
```
|
|
775
|
+
|
|
776
|
+
**Host-runner** (services in Docker, tests on the host):
|
|
777
|
+
|
|
778
|
+
```bash
|
|
779
|
+
npm run docker:up
|
|
780
|
+
export COUNTEX_MONGO_URI='mongodb://countex:countex@127.0.0.1:27017/admin'
|
|
781
|
+
export COUNTEX_REDIS_URL='redis://127.0.0.1:6379'
|
|
782
|
+
export COUNTEX_S3_ENDPOINT='http://127.0.0.1:9000'
|
|
783
|
+
export COUNTEX_S3_REGION='us-east-1'
|
|
784
|
+
export COUNTEX_S3_ACCESS_KEY_ID='countex'
|
|
785
|
+
export COUNTEX_S3_SECRET_ACCESS_KEY='countex-password'
|
|
786
|
+
export COUNTEX_S3_BUCKET='countex-backups'
|
|
787
|
+
export COUNTEX_INTEGRATION=1
|
|
788
|
+
npm run test:integration
|
|
789
|
+
npm run docker:down
|
|
790
|
+
```
|
|
791
|
+
|
|
792
|
+
Optional tuning: `COUNTEX_TEST_PROFILE`, `COUNTEX_TEST_EVENTS`, `COUNTEX_TEST_WORKERS`. Stress-style full pipeline (larger simulation + S3 round-trip): set `COUNTEX_TEST_FULL=1` (requires S3 env as above).
|
|
793
|
+
|
|
794
|
+
**Container-runner** (CI parity): `npm run test:integration:docker` builds `docker/test-runner.Dockerfile` and runs `npm run test:integration:exec` with service hostnames injected by Compose.
|
|
795
|
+
|
|
796
|
+
Reports are written to `artifacts/integration/report.json` and `artifacts/integration/report.md` after `npm run test:integration`.
|
|
797
|
+
|
|
798
|
+
---
|
|
799
|
+
|
|
800
|
+
## Errors
|
|
801
|
+
|
|
802
|
+
All errors thrown by Countex extend `CountexError`:
|
|
803
|
+
|
|
804
|
+
| Class | When |
|
|
805
|
+
|--------------------------|------|
|
|
806
|
+
| `MappingError` | Invalid mapping config (caught at engine startup). |
|
|
807
|
+
| `ExtractionError` | Malformed path or value during extraction. |
|
|
808
|
+
| `CardinalityLimitError` | Cardinality limit hit when policy is `error`. |
|
|
809
|
+
| `StorageError` | Wrapping a storage-adapter failure. |
|
|
810
|
+
|
|
811
|
+
---
|
|
812
|
+
|
|
813
|
+
## Versioning & scope
|
|
814
|
+
|
|
815
|
+
> **Additive counts are not distinct counts.** If `user_42` appears in 100 events, `count = 100`, not 1 unique user. Counter cells store sums of deltas, so a query result of `count = 100` from a subject-grouped view means "100 increments hit this subject", not "100 distinct subjects". For exact distinct subject counts, group by `["subjectType", "subject"]` and use `rows.length` after retrieving every matching cell — see [`docs/guides/querying.md`](docs/guides/querying.md) §"Distinct subjects / metrics as row cardinality". Approximate distinct counts (HLL/sets) are a planned opt-in extension, not core.
|
|
816
|
+
|
|
817
|
+
`v0.x` is for stabilizing the public API. Once it's hardened:
|
|
818
|
+
|
|
819
|
+
- exact unique counts (HLL / sets) are an opt-in extension, not core
|
|
820
|
+
- complex expression deltas are not in core
|
|
821
|
+
- distributed locks / exactly-once: not Countex's job; durable storage is the source of truth and `$inc` makes worker-count irrelevant
|
|
822
|
+
|
|
823
|
+
---
|
|
824
|
+
|
|
825
|
+
## Repository layout
|
|
826
|
+
|
|
827
|
+
| Path | Role |
|
|
828
|
+
|------|------|
|
|
829
|
+
| `src/index.ts` | Public exports (`createCountex`, types, errors). |
|
|
830
|
+
| `src/core/` | Engine: ingest pipeline, buffer, extraction, mapping validation, time buckets, cardinality. |
|
|
831
|
+
| `src/query/` | Query planner (`planQuery` + `executeQuery`) and result formatting. |
|
|
832
|
+
| `src/storage/` | Adapters: `memory` (in-process), `mongo` (durable), `redis` (cache); shared coverage helpers. |
|
|
833
|
+
| `src/utils/` | JSONPath subset, hashing, logging. |
|
|
834
|
+
| `src/backup/` | Export, fingerprint, manifest, verify, restore, cache rebuild helpers. |
|
|
835
|
+
| `src/simulation/` | Simulation engine, profiles, expected aggregates, resource reports. |
|
|
836
|
+
| `src/insights/` | Ranking, comparison, anomaly detection, advisors, recommendations. |
|
|
837
|
+
| `src/cli.ts` | CLI commands for simulation, insights, export, and operational helpers. |
|
|
838
|
+
| `examples/` | Runnable demos (`npx tsx examples/basic.ts`). |
|
|
839
|
+
| `docs/` | Deployment guide, tutorial, guides, internals, and use cases (published in the npm package). |
|
|
840
|
+
| `tests/unit/` | Fast Node test runner tests (`npm test`). |
|
|
841
|
+
| `tests/integration/` | Opt-in live tests against real services (`npm run test:integration`). |
|
|
842
|
+
|
|
843
|
+
---
|
|
844
|
+
|
|
845
|
+
## Publishing (npm)
|
|
846
|
+
|
|
847
|
+
This package is scoped as **`@x12i/countex`** and is configured for **public** registry access (`publishConfig.access` in `package.json`).
|
|
848
|
+
|
|
849
|
+
1. Ensure you are logged in and allowed to publish under the **`@x12i`** scope on [npm](https://www.npmjs.com/).
|
|
850
|
+
2. Put auth tokens only in **`~/.npmrc`** — do not commit project `.npmrc` with credentials (it is listed in `.gitignore`).
|
|
851
|
+
3. From the repo root:
|
|
852
|
+
|
|
853
|
+
```bash
|
|
854
|
+
npm test
|
|
855
|
+
npm publish
|
|
856
|
+
```
|
|
857
|
+
|
|
858
|
+
`prepublishOnly` runs a clean build so the tarball always contains a fresh `dist/`.
|
|
859
|
+
|
|
860
|
+
The **`docs/`** directory (`DEPLOYMENT.md`, `END_TO_END_EXAMPLE.md`) is part of the published package (`files` in `package.json`).
|
|
861
|
+
|
|
862
|
+
---
|
|
863
|
+
|
|
864
|
+
## License
|
|
865
|
+
|
|
866
|
+
MIT — see `LICENSE`.
|