@x12i/countex 1.2.0 → 1.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (104) hide show
  1. package/README.md +23 -1
  2. package/dist/backup/manifest.d.ts +49 -0
  3. package/dist/backup/manifest.d.ts.map +1 -0
  4. package/dist/backup/manifest.js +103 -0
  5. package/dist/backup/manifest.js.map +1 -0
  6. package/dist/backup/snapshot-encoding.d.ts +8 -0
  7. package/dist/backup/snapshot-encoding.d.ts.map +1 -0
  8. package/dist/backup/snapshot-encoding.js +27 -0
  9. package/dist/backup/snapshot-encoding.js.map +1 -0
  10. package/dist/cli.js +16 -0
  11. package/dist/cli.js.map +1 -1
  12. package/dist/core/countex.d.ts +14 -1
  13. package/dist/core/countex.d.ts.map +1 -1
  14. package/dist/core/countex.js +141 -45
  15. package/dist/core/countex.js.map +1 -1
  16. package/dist/core/extraction.d.ts +1 -5
  17. package/dist/core/extraction.d.ts.map +1 -1
  18. package/dist/core/extraction.js.map +1 -1
  19. package/dist/core/mapping.d.ts +5 -0
  20. package/dist/core/mapping.d.ts.map +1 -1
  21. package/dist/core/mapping.js +27 -0
  22. package/dist/core/mapping.js.map +1 -1
  23. package/dist/index.d.ts +4 -1
  24. package/dist/index.d.ts.map +1 -1
  25. package/dist/index.js +13 -1
  26. package/dist/index.js.map +1 -1
  27. package/dist/query/formatter.d.ts +2 -2
  28. package/dist/query/formatter.d.ts.map +1 -1
  29. package/dist/query/formatter.js +9 -6
  30. package/dist/query/formatter.js.map +1 -1
  31. package/dist/query/planner.d.ts.map +1 -1
  32. package/dist/query/planner.js +16 -2
  33. package/dist/query/planner.js.map +1 -1
  34. package/dist/simulation/context.d.ts +5 -0
  35. package/dist/simulation/context.d.ts.map +1 -0
  36. package/dist/simulation/context.js +56 -0
  37. package/dist/simulation/context.js.map +1 -0
  38. package/dist/simulation/events.d.ts +3 -0
  39. package/dist/simulation/events.d.ts.map +1 -0
  40. package/dist/simulation/events.js +67 -0
  41. package/dist/simulation/events.js.map +1 -0
  42. package/dist/simulation/index.d.ts +7 -0
  43. package/dist/simulation/index.d.ts.map +1 -0
  44. package/dist/simulation/index.js +16 -0
  45. package/dist/simulation/index.js.map +1 -0
  46. package/dist/simulation/prng.d.ts +4 -0
  47. package/dist/simulation/prng.d.ts.map +1 -0
  48. package/dist/simulation/prng.js +18 -0
  49. package/dist/simulation/prng.js.map +1 -0
  50. package/dist/simulation/resource-estimate.d.ts +6 -0
  51. package/dist/simulation/resource-estimate.d.ts.map +1 -0
  52. package/dist/simulation/resource-estimate.js +11 -0
  53. package/dist/simulation/resource-estimate.js.map +1 -0
  54. package/dist/simulation/simulator.d.ts +15 -0
  55. package/dist/simulation/simulator.d.ts.map +1 -0
  56. package/dist/simulation/simulator.js +67 -0
  57. package/dist/simulation/simulator.js.map +1 -0
  58. package/dist/simulation/tracker.d.ts +8 -0
  59. package/dist/simulation/tracker.d.ts.map +1 -0
  60. package/dist/simulation/tracker.js +18 -0
  61. package/dist/simulation/tracker.js.map +1 -0
  62. package/dist/simulation/types.d.ts +49 -0
  63. package/dist/simulation/types.d.ts.map +1 -0
  64. package/dist/simulation/types.js +21 -0
  65. package/dist/simulation/types.js.map +1 -0
  66. package/dist/storage/memory.d.ts +3 -0
  67. package/dist/storage/memory.d.ts.map +1 -1
  68. package/dist/storage/memory.js +35 -0
  69. package/dist/storage/memory.js.map +1 -1
  70. package/dist/storage/mongo.d.ts +5 -0
  71. package/dist/storage/mongo.d.ts.map +1 -1
  72. package/dist/storage/mongo.js +68 -1
  73. package/dist/storage/mongo.js.map +1 -1
  74. package/dist/storage/redis.d.ts +6 -2
  75. package/dist/storage/redis.d.ts.map +1 -1
  76. package/dist/storage/redis.js +18 -8
  77. package/dist/storage/redis.js.map +1 -1
  78. package/dist/types/index.d.ts +51 -0
  79. package/dist/types/index.d.ts.map +1 -1
  80. package/docs/DEPLOYMENT.md +40 -2
  81. package/docs/countex_gap_analysis_complete_checklist.md +1340 -0
  82. package/docs/guides/README.md +5 -0
  83. package/docs/guides/adapter-mongo.md +12 -0
  84. package/docs/guides/adapter-redis.md +6 -0
  85. package/docs/guides/backup-export.md +46 -2
  86. package/docs/guides/cli.md +8 -0
  87. package/docs/guides/counter-maps.md +6 -0
  88. package/docs/guides/insights-overview.md +6 -0
  89. package/docs/guides/measures.md +4 -0
  90. package/docs/guides/querying.md +40 -0
  91. package/docs/guides/simulation-profiles.md +33 -0
  92. package/docs/guides/simulation-resource-estimation.md +30 -0
  93. package/docs/guides/simulation.md +36 -0
  94. package/docs/internals/README.md +1 -0
  95. package/docs/internals/internal-dimension-hash.md +34 -0
  96. package/docs/use-cases/README.md +56 -0
  97. package/docs/use-cases/audit-security-log-counts.md +550 -0
  98. package/docs/use-cases/llm-cost-and-token-metering.md +255 -0
  99. package/docs/use-cases/multi-tenant-isolation.md +203 -0
  100. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -0
  101. package/docs/use-cases/processing-workflow-activity.md +165 -0
  102. package/docs/use-cases/product-feature-usage.md +240 -0
  103. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -0
  104. package/package.json +18 -2
@@ -0,0 +1,1340 @@
1
+ # @x12i/countex — Gap Analysis: Actionable Version
2
+
3
+ ## Purpose
4
+
5
+ This document compares the current `@x12i/countex` README/spec direction against the original product intent and turns the gaps into concrete fixes.
6
+
7
+ This is not a generic checklist. Each item below has:
8
+
9
+ ```text
10
+ Status
11
+ Gap
12
+ Why it matters
13
+ Decision / fix
14
+ Priority
15
+ ```
16
+
17
+ ---
18
+
19
+ # 1. Bottom-line assessment
20
+
21
+ The current Countex direction is strong. It already covers the core idea well:
22
+
23
+ ```text
24
+ Generic JSON event → declarative map → sparse aggregate cells → buffered deltas → Mongo durable store + Redis hot cache → query + insights
25
+ ```
26
+
27
+ The package has grown into three natural layers:
28
+
29
+ ```text
30
+ Countex Core = count anything
31
+ Countex Simulation = test and size anything
32
+ Countex Insights = understand and optimize anything
33
+ ```
34
+
35
+ The biggest remaining gaps are not conceptual. They are implementation-precision gaps that can confuse code agents or create production risk.
36
+
37
+ The highest-priority fixes are:
38
+
39
+ ```text
40
+ 1. Make simulation visible in README/docs.
41
+ 2. Fix mapping reference mismatch: measures + minute/hour.
42
+ 3. Define dictionary allocation under concurrency.
43
+ 4. Define dimension canonicalization and hashing.
44
+ 5. Define cache coverage and query planner behavior.
45
+ 6. Define backup restore/verification, not only export.
46
+ 7. Define retention policy for Mongo and Redis.
47
+ 8. Add production/self-managed deployment guide and resource math guide.
48
+ 9. Clarify unique counts are not additive counts.
49
+ 10. Decide whether percentile/histogram support is part of core measures or an optional extension.
50
+ ```
51
+
52
+ ---
53
+
54
+ # 2. Original intent vs current state
55
+
56
+ ## 2.1 Pure generic mapped-event counter
57
+
58
+ ### Status
59
+
60
+ **Good / mostly complete.**
61
+
62
+ The README now clearly positions Countex as a generic mapped-event counter. It says the package does not know about users, organizations, documents, labels, or events, and only knows generic primitives:
63
+
64
+ ```text
65
+ scope
66
+ subject
67
+ metric
68
+ dimensions
69
+ timeframe
70
+ delta
71
+ ```
72
+
73
+ ### Remaining gap
74
+
75
+ The README still uses labels in examples, which is fine, but it should explicitly say:
76
+
77
+ ```text
78
+ A label is only one possible domain-specific metric.
79
+ ```
80
+
81
+ Otherwise users may think `metrics` means only labels/classifications.
82
+
83
+ ### Fix
84
+
85
+ Add this note under the mapping reference:
86
+
87
+ ```md
88
+ `metric` is the generic Countex term for the thing being counted or categorized. In some domains this may be a label, status, event name, operation, outcome, class, model result, or business category.
89
+ ```
90
+
91
+ ### Priority
92
+
93
+ **P0 — documentation clarity.**
94
+
95
+ ---
96
+
97
+ ## 2.2 Mapping from arbitrary JSON paths
98
+
99
+ ### Status
100
+
101
+ **Good foundation, but not complete enough for implementation.**
102
+
103
+ Covered now:
104
+
105
+ ```text
106
+ scope path
107
+ eventTime path
108
+ subject paths
109
+ metric paths
110
+ dimension paths
111
+ delta path/default
112
+ JSONPath subset
113
+ wildcard fanout
114
+ ```
115
+
116
+ ### Remaining gap
117
+
118
+ The mapping system needs stronger operational features:
119
+
120
+ ```text
121
+ fallback paths
122
+ value normalization
123
+ field-level default values
124
+ computed/transformed values
125
+ map preview/dry-run
126
+ extraction diagnostics
127
+ ```
128
+
129
+ These are important because real event sources are messy. Microsoft audit logs, product activity logs, workflow logs, and LLM gateway logs will not all have perfect stable paths.
130
+
131
+ ### Fix
132
+
133
+ Add these to the spec:
134
+
135
+ ```ts
136
+ subjects: [
137
+ {
138
+ type: "actor",
139
+ path: "$.actor.id",
140
+ fallbackPaths: ["$.user.id", "$.user.email"],
141
+ required: true,
142
+ normalize: "lowercase"
143
+ }
144
+ ]
145
+ ```
146
+
147
+ Add a preview API:
148
+
149
+ ```ts
150
+ const preview = await countex.preview(event, { counterId: "access_by_subject_metric" });
151
+ ```
152
+
153
+ Preview output should include:
154
+
155
+ ```text
156
+ extracted scope
157
+ extracted event time
158
+ subjects
159
+ metrics
160
+ dimensions
161
+ delta
162
+ timeframes
163
+ number of generated increments
164
+ warnings/errors
165
+ ```
166
+
167
+ ### Priority
168
+
169
+ **P0 — needed before code-agent implementation.**
170
+
171
+ ---
172
+
173
+ ## 2.3 Mapping reference mismatch: measures missing
174
+
175
+ ### Status
176
+
177
+ **Gap.**
178
+
179
+ The README explains measures and rates in the mental model, but the `Mapping reference` table does not include `measures` as a map field.
180
+
181
+ ### Why it matters
182
+
183
+ Measures are central for:
184
+
185
+ ```text
186
+ cost
187
+ tokens
188
+ latency
189
+ duration
190
+ items processed
191
+ retry counts
192
+ budget projections
193
+ max-token optimization
194
+ ```
195
+
196
+ If `measures` is not in the map reference, implementers may miss it or implement it inconsistently.
197
+
198
+ ### Fix
199
+
200
+ Add this row to the Mapping reference table:
201
+
202
+ | Field | Type | Notes |
203
+ |---|---|---|
204
+ | `measures` | `MeasureMapping[]` | Optional numeric values extracted from the event, such as cost, tokens, latency, or duration. Supports `sum`, `count`, `min`, `max`; `avg` is computed at query time. |
205
+
206
+ Add example:
207
+
208
+ ```ts
209
+ measures: [
210
+ {
211
+ name: "inputTokens",
212
+ path: "$.usage.inputTokens",
213
+ aggregations: ["sum", "count", "min", "max"]
214
+ },
215
+ {
216
+ name: "costMicros",
217
+ path: "$.usage.costMicros",
218
+ aggregations: ["sum", "count"],
219
+ scale: 1
220
+ }
221
+ ]
222
+ ```
223
+
224
+ ### Priority
225
+
226
+ **P0 — README/spec correctness.**
227
+
228
+ ---
229
+
230
+ ## 2.4 Mapping reference mismatch: minute/hour timeframes
231
+
232
+ ### Status
233
+
234
+ **Gap.**
235
+
236
+ The mental model says timeframes include:
237
+
238
+ ```text
239
+ minute / hour / day / week / month / quarter / year / all
240
+ ```
241
+
242
+ But the mapping reference says the subset is:
243
+
244
+ ```text
245
+ day / week / month / quarter / year / all
246
+ ```
247
+
248
+ ### Why it matters
249
+
250
+ This creates ambiguity for processing activity, LLM usage, latency/rate dashboards, and operations monitoring where minute/hour buckets are important.
251
+
252
+ ### Fix
253
+
254
+ Update the mapping reference row:
255
+
256
+ ```md
257
+ `timeframes` — subset of `minute` / `hour` / `day` / `week` / `month` / `quarter` / `year` / `all`.
258
+ ```
259
+
260
+ ### Priority
261
+
262
+ **P0 — direct spec mismatch.**
263
+
264
+ ---
265
+
266
+ ## 2.5 Dynamic dimensions
267
+
268
+ ### Status
269
+
270
+ **Concept covered, but hashing/canonicalization is missing.**
271
+
272
+ The README supports arbitrary dynamic dimensions and query grouping via `dim:NAME`.
273
+
274
+ ### Remaining gap
275
+
276
+ The package needs a precise rule for how dimension tuples become `dimensionsHash`.
277
+
278
+ Without this, two workers can generate different hashes for the same logical dimension tuple.
279
+
280
+ Example risk:
281
+
282
+ ```ts
283
+ { eventType: "share", situation: "external" }
284
+ ```
285
+
286
+ and:
287
+
288
+ ```ts
289
+ { situation: "external", eventType: "share" }
290
+ ```
291
+
292
+ must produce the same hash.
293
+
294
+ ### Fix
295
+
296
+ Define canonical dimension hashing:
297
+
298
+ ```text
299
+ 1. Remove undefined dimensions.
300
+ 2. Normalize values according to map options.
301
+ 3. Sort dimension keys lexicographically.
302
+ 4. Serialize as stable JSON.
303
+ 5. Hash with stable algorithm.
304
+ ```
305
+
306
+ Example:
307
+
308
+ ```ts
309
+ canonicalDimensions = [
310
+ ["eventType", "share"],
311
+ ["situation", "external"]
312
+ ]
313
+ ```
314
+
315
+ Then:
316
+
317
+ ```ts
318
+ dimensionsHash = hashStable(canonicalDimensions)
319
+ ```
320
+
321
+ ### Priority
322
+
323
+ **P0 — correctness across workers.**
324
+
325
+ ---
326
+
327
+ ## 2.6 Source key / UID dictionary
328
+
329
+ ### Status
330
+
331
+ **Concept present, but implementation-critical details are missing.**
332
+
333
+ We agreed that raw source identifiers should be mapped to compact internal IDs:
334
+
335
+ ```text
336
+ sourceKey -> internalId
337
+ ```
338
+
339
+ The README mentions `countex_dict` and `countex_seq`.
340
+
341
+ ### Remaining gap
342
+
343
+ The spec does not yet define how dictionary allocation is concurrency-safe across many workers.
344
+
345
+ This is a major implementation risk.
346
+
347
+ Counters are safe because `$inc` is atomic. But dictionary allocation is a separate problem.
348
+
349
+ ### Why it matters
350
+
351
+ If two workers see the same new source key at the same time, they must not create two different internal IDs.
352
+
353
+ Bad outcome:
354
+
355
+ ```text
356
+ worker A: user_42 -> 10001
357
+ worker B: user_42 -> 10002
358
+ ```
359
+
360
+ Then counts fragment across two IDs.
361
+
362
+ ### Fix
363
+
364
+ Define dictionary allocation as an atomic operation.
365
+
366
+ Recommended Mongo-backed flow:
367
+
368
+ ```text
369
+ 1. Try find existing dictionary record by unique identity:
370
+ scopeId + kind + qualifier + sourceKey
371
+
372
+ 2. If missing, allocate next sequence ID atomically.
373
+
374
+ 3. Insert dictionary record with unique index.
375
+
376
+ 4. If duplicate insert error occurs, re-read existing record.
377
+
378
+ 5. Cache result in Redis/local LRU.
379
+ ```
380
+
381
+ Unique index:
382
+
383
+ ```js
384
+ db.countex_dict.createIndex(
385
+ { scopeId: 1, kind: 1, qualifier: 1, sourceKeyHash: 1 },
386
+ { unique: true, name: "uniq_dict_key" }
387
+ )
388
+ ```
389
+
390
+ Dictionary record:
391
+
392
+ ```json
393
+ {
394
+ "scopeId": "org_123",
395
+ "kind": "subject",
396
+ "qualifier": "actor",
397
+ "sourceKeyHash": "...",
398
+ "sourceKeyPreview": "user_42",
399
+ "internalId": 10001,
400
+ "createdAt": "..."
401
+ }
402
+ ```
403
+
404
+ ### Priority
405
+
406
+ **P0 — production correctness.**
407
+
408
+ ---
409
+
410
+ ## 2.7 Local memory vs 24-hour state
411
+
412
+ ### Status
413
+
414
+ **Architecture improved, but original expectation should be explicitly reconciled.**
415
+
416
+ Original idea mentioned 24 hours in memory. The better architecture is:
417
+
418
+ ```text
419
+ worker memory = only unflushed delta buffer
420
+ Redis = hot retained state
421
+ Mongo = durable retained state
422
+ ```
423
+
424
+ ### Gap
425
+
426
+ README says each instance keeps only a delta buffer, but it does not explicitly explain that workers do **not** keep 24 hours of state.
427
+
428
+ ### Fix
429
+
430
+ Add an operational note:
431
+
432
+ ```md
433
+ Countex workers are not the retained-state store. Each worker keeps only a bounded unflushed delta buffer. Hot retained aggregates live in Redis; durable retained aggregates live in Mongo. This allows many workers to run without multiplying the 24h/92d memory footprint.
434
+ ```
435
+
436
+ ### Priority
437
+
438
+ **P0 — prevents wrong architecture implementation.**
439
+
440
+ ---
441
+
442
+ ## 2.8 Flush/save strategy
443
+
444
+ ### Status
445
+
446
+ **Good foundation.**
447
+
448
+ Covered:
449
+
450
+ ```text
451
+ time-based flush
452
+ maxBufferedCells
453
+ maxBufferedEvents
454
+ manual flush
455
+ shutdown flush
456
+ durable-first
457
+ merge buffer back if durable fails
458
+ cache failure does not lose durable data
459
+ ```
460
+
461
+ ### Remaining gap
462
+
463
+ The spec should define:
464
+
465
+ ```text
466
+ retry/backoff
467
+ max batch size
468
+ partial Mongo bulkWrite failure handling
469
+ Redis pipeline failure handling
470
+ buffer overflow behavior
471
+ optional local WAL
472
+ ```
473
+
474
+ ### Fix
475
+
476
+ Add flush config:
477
+
478
+ ```ts
479
+ flush: {
480
+ everyMs: 60_000,
481
+ maxBufferedCells: 100_000,
482
+ maxBufferedEvents: 50_000,
483
+ maxDeltasPerBatch: 10_000,
484
+ retry: {
485
+ attempts: 3,
486
+ backoffMs: 500,
487
+ backoffMultiplier: 2
488
+ },
489
+ onBufferOverflow: "block", // "block" | "drop" | "error" | "spillToDisk"
490
+ localWal: {
491
+ enabled: false,
492
+ path: "./.countex-wal"
493
+ }
494
+ }
495
+ ```
496
+
497
+ ### Priority
498
+
499
+ **P1 — production readiness.**
500
+
501
+ ---
502
+
503
+ ## 2.9 Multi-worker safety
504
+
505
+ ### Status
506
+
507
+ **Partially covered.**
508
+
509
+ Atomic counter increments are covered:
510
+
511
+ ```text
512
+ Mongo $inc
513
+ Redis HINCRBY
514
+ ```
515
+
516
+ ### Gap
517
+
518
+ The README says 20+ workers can write safely, but that is only fully true if these are also safe:
519
+
520
+ ```text
521
+ dictionary allocation
522
+ dimension hashing
523
+ map version consistency
524
+ idempotency keys
525
+ ```
526
+
527
+ ### Fix
528
+
529
+ Add this note:
530
+
531
+ ```md
532
+ Multi-worker safety depends on three stable identities: dictionary IDs, dimensionsHash, and map version. Counter increments are atomic, but all workers must resolve source keys and dimension tuples consistently.
533
+ ```
534
+
535
+ Add integration test:
536
+
537
+ ```text
538
+ 20 workers ingest same new subjects/metrics/dimensions concurrently.
539
+ Expected: one internal ID per source key, one dimensionsHash per tuple, correct final count.
540
+ ```
541
+
542
+ ### Priority
543
+
544
+ **P0/P1 — core correctness.**
545
+
546
+ ---
547
+
548
+ ## 2.10 Mongo per org / self-managed Mongo
549
+
550
+ ### Status
551
+
552
+ **Good concept, needs safety details.**
553
+
554
+ README supports:
555
+
556
+ ```ts
557
+ dbResolver: ({ scope }) => `countex_${scope}`
558
+ ```
559
+
560
+ ### Gap
561
+
562
+ Raw scope values should not be blindly used as DB names.
563
+
564
+ ### Why it matters
565
+
566
+ Scope values may contain unsafe characters, long strings, user-provided data, or collisions.
567
+
568
+ ### Fix
569
+
570
+ Add safe dbResolver guidance:
571
+
572
+ ```ts
573
+ dbResolver: ({ scope }) => `countex_${sanitizeScope(scope)}`
574
+ ```
575
+
576
+ Or:
577
+
578
+ ```ts
579
+ dbResolver: ({ scope }) => `countex_${hashScope(scope)}`
580
+ ```
581
+
582
+ Add docs:
583
+
584
+ ```text
585
+ For tens/hundreds of tenants, DB-per-scope is fine.
586
+ For thousands of tenants, prefer shared DB + indexed scopeId.
587
+ ```
588
+
589
+ ### Priority
590
+
591
+ **P1 — production/tenant safety.**
592
+
593
+ ---
594
+
595
+ ## 2.11 Redis 92-day hot cache
596
+
597
+ ### Status
598
+
599
+ **Concept covered, retention model underdefined.**
600
+
601
+ README shows:
602
+
603
+ ```ts
604
+ ttlSeconds: 92 * 24 * 3600
605
+ ```
606
+
607
+ ### Gap
608
+
609
+ A single TTL is too simple for the full model.
610
+
611
+ We need retention per timeframe:
612
+
613
+ ```text
614
+ minute: maybe 2 days
615
+ hour: maybe 30 days
616
+ day: 92 days
617
+ week: 52 weeks
618
+ month: 24 months
619
+ quarter: 12 quarters
620
+ year: 7 years
621
+ all: forever
622
+ ```
623
+
624
+ ### Fix
625
+
626
+ Add cache retention config:
627
+
628
+ ```ts
629
+ cache: {
630
+ hotTimeframes: {
631
+ minute: { retain: "2d" },
632
+ hour: { retain: "30d" },
633
+ day: { retain: "92d" },
634
+ week: { retain: "52w" },
635
+ month: { retain: "24mo" },
636
+ quarter: { retain: "12q" },
637
+ year: { retain: "7y" },
638
+ all: { retain: "forever" }
639
+ }
640
+ }
641
+ ```
642
+
643
+ Also document Redis production setting:
644
+
645
+ ```text
646
+ maxmemory-policy should usually be noeviction.
647
+ ```
648
+
649
+ ### Priority
650
+
651
+ **P1 — production operations.**
652
+
653
+ ---
654
+
655
+ ## 2.12 Query planner and cache coverage
656
+
657
+ ### Status
658
+
659
+ **High-level concept covered, exact behavior missing.**
660
+
661
+ README says query can return source:
662
+
663
+ ```text
664
+ cache | durable | merged
665
+ ```
666
+
667
+ ### Gap
668
+
669
+ The mechanism for deciding cache coverage is not defined.
670
+
671
+ ### Why it matters
672
+
673
+ Without coverage metadata, Countex may think Redis is complete when it is partial.
674
+
675
+ Examples:
676
+
677
+ ```text
678
+ Redis key expired for one day in a 20-day query.
679
+ Redis write failed after Mongo succeeded.
680
+ Redis rebuild is only halfway done.
681
+ ```
682
+
683
+ ### Fix
684
+
685
+ Define per-bucket cache metadata:
686
+
687
+ ```text
688
+ countex:{scope}:{counterId}:{timeframeType}:{timeframeKey}:coverage
689
+ ```
690
+
691
+ Fields:
692
+
693
+ ```json
694
+ {
695
+ "status": "complete",
696
+ "lastWarmAt": "...",
697
+ "lastDurableSyncAt": "...",
698
+ "schemaVersion": 1
699
+ }
700
+ ```
701
+
702
+ Query planner behavior:
703
+
704
+ ```text
705
+ 1. Determine required buckets.
706
+ 2. Check Redis coverage for each bucket.
707
+ 3. If all complete, read Redis.
708
+ 4. If some missing/partial, read missing from Mongo.
709
+ 5. Merge results.
710
+ 6. Optionally warm Redis for missing buckets.
711
+ ```
712
+
713
+ ### Priority
714
+
715
+ **P0 — correctness of Redis-first fetch.**
716
+
717
+ ---
718
+
719
+ ## 2.13 Measures and max-token optimization
720
+
721
+ ### Status
722
+
723
+ **Measures foundation is good, but max-token recommendations need percentiles.**
724
+
725
+ Current measures support:
726
+
727
+ ```text
728
+ sum
729
+ count
730
+ min
731
+ max
732
+ avg at query time
733
+ rate
734
+ ```
735
+
736
+ ### Gap
737
+
738
+ Max-token optimization needs distribution information:
739
+
740
+ ```text
741
+ p50
742
+ p75
743
+ p90
744
+ p95
745
+ p99
746
+ ```
747
+
748
+ Without histograms/sketches, recommendations will be weaker.
749
+
750
+ ### Fix
751
+
752
+ Decision needed:
753
+
754
+ ```text
755
+ Option A: Add histogram/distribution support to core measures.
756
+ Option B: Keep histograms as optional insight extension.
757
+ Option C: Defer percentiles and make max-token advisor low-confidence using avg/max only.
758
+ ```
759
+
760
+ Recommended:
761
+
762
+ ```text
763
+ Add optional histogram support, off by default.
764
+ ```
765
+
766
+ Example:
767
+
768
+ ```ts
769
+ measures: [
770
+ {
771
+ name: "outputTokens",
772
+ path: "$.usage.outputTokens",
773
+ aggregations: ["sum", "count", "min", "max"],
774
+ distribution: {
775
+ enabled: true,
776
+ type: "histogram",
777
+ buckets: "auto-log"
778
+ }
779
+ }
780
+ ]
781
+ ```
782
+
783
+ ### Priority
784
+
785
+ **P2 for core, P0 if max-token advisor is part of first release.**
786
+
787
+ ---
788
+
789
+ ## 2.14 Unique counts
790
+
791
+ ### Status
792
+
793
+ **Correctly excluded from core, but needs clearer user-facing warning.**
794
+
795
+ ### Gap
796
+
797
+ Users may assume:
798
+
799
+ ```text
800
+ count of events involving users = unique users
801
+ ```
802
+
803
+ That is wrong.
804
+
805
+ ### Fix
806
+
807
+ Add README warning:
808
+
809
+ ```md
810
+ Countex additive counts are not distinct counts. If the same subject appears in 100 events, the count is 100, not 1. Exact unique counts require raw data or explicit unique-count extensions. Approximate unique counts may be added later via HyperLogLog or similar structures.
811
+ ```
812
+
813
+ Add future API boundary:
814
+
815
+ ```ts
816
+ await countex.unique.queryExact({ ... }); // future/on-demand, not core additive counter
817
+ await countex.unique.queryApprox({ ... }); // future HLL extension
818
+ ```
819
+
820
+ ### Priority
821
+
822
+ **P1 — avoids misuse.**
823
+
824
+ ---
825
+
826
+ ## 2.15 Backup/export/rebuild
827
+
828
+ ### Status
829
+
830
+ **Export and rebuild are covered; restore and verification are missing.**
831
+
832
+ Current README has:
833
+
834
+ ```text
835
+ exportCells
836
+ rebuildCache
837
+ ```
838
+
839
+ ### Gap
840
+
841
+ A backup story is incomplete without:
842
+
843
+ ```text
844
+ verify
845
+ restore
846
+ backup manifest
847
+ restore mode
848
+ retention policy
849
+ ```
850
+
851
+ ### Fix
852
+
853
+ Add backup APIs:
854
+
855
+ ```ts
856
+ await countex.backup.export({
857
+ scope,
858
+ counterId,
859
+ timeframe,
860
+ format: "ndjson.gz",
861
+ target: "s3://bucket/path/",
862
+ verify: true
863
+ });
864
+ ```
865
+
866
+ ```ts
867
+ await countex.backup.verify({
868
+ source: "s3://bucket/path/manifest.json",
869
+ checks: ["rowCount", "countSum", "hash"]
870
+ });
871
+ ```
872
+
873
+ ```ts
874
+ await countex.backup.restore({
875
+ source: "s3://bucket/path/manifest.json",
876
+ mode: "replace-window" // "replace-window" | "merge-deltas" | "validate-only"
877
+ });
878
+ ```
879
+
880
+ ### Priority
881
+
882
+ **P1 — production durability.**
883
+
884
+ ---
885
+
886
+ ## 2.16 Simulation
887
+
888
+ ### Status
889
+
890
+ **Specified separately, missing from current README.**
891
+
892
+ ### Gap
893
+
894
+ The README lists many docs and insights, but does not yet include a Simulation section.
895
+
896
+ ### Why it matters
897
+
898
+ Simulation is central to Countex’s value because users need to know before production:
899
+
900
+ ```text
901
+ how many cells a map creates
902
+ how much Redis/Mongo it needs
903
+ whether cardinality explodes
904
+ whether queries match expected counts
905
+ ```
906
+
907
+ ### Fix
908
+
909
+ Add README section:
910
+
911
+ ```md
912
+ ## Simulation
913
+
914
+ Countex includes a simulation toolkit for generating deterministic event streams, validating maps, estimating resource usage, and stress-testing cardinality behavior before production data exists.
915
+ ```
916
+
917
+ Add docs table rows:
918
+
919
+ ```text
920
+ Simulation overview | @x12i/countex/docs/guides/simulation | docs/guides/simulation.md
921
+ Simulation profiles | @x12i/countex/docs/guides/simulation-profiles | docs/guides/simulation-profiles.md
922
+ Simulation resource estimation | @x12i/countex/docs/guides/simulation-resource-estimation | docs/guides/simulation-resource-estimation.md
923
+ ```
924
+
925
+ Add entry point:
926
+
927
+ ```ts
928
+ import { createCountexSimulator } from "@x12i/countex/simulation";
929
+ ```
930
+
931
+ ### Priority
932
+
933
+ **P0 — we already designed it; it should be visible.**
934
+
935
+ ---
936
+
937
+ ## 2.17 Insights/advisors
938
+
939
+ ### Status
940
+
941
+ **README now has a good overview. Implementation/API still needs full contracts.**
942
+
943
+ Covered:
944
+
945
+ ```text
946
+ top/bottom
947
+ percent slices
948
+ Pareto
949
+ period comparison
950
+ anomalies
951
+ rules
952
+ budget/max-token advisors
953
+ map/cardinality/resource advisors
954
+ ```
955
+
956
+ ### Remaining gap
957
+
958
+ The README gives examples but not enough API specificity for code agents.
959
+
960
+ ### Fix
961
+
962
+ Create formal docs:
963
+
964
+ ```text
965
+ docs/guides/insights-ranking.md
966
+ docs/guides/insights-comparison-and-trends.md
967
+ docs/guides/insights-anomalies-and-rules.md
968
+ docs/guides/insights-forecasting-and-budget.md
969
+ docs/guides/insights-advisors-and-optimization.md
970
+ ```
971
+
972
+ Each should include:
973
+
974
+ ```text
975
+ input type
976
+ output type
977
+ edge cases
978
+ examples
979
+ test cases
980
+ query cost warnings
981
+ ```
982
+
983
+ ### Priority
984
+
985
+ **P1/P2 — valuable, but core must be stable first.**
986
+
987
+ ---
988
+
989
+ ## 2.18 Processing/workflow use case
990
+
991
+ ### Status
992
+
993
+ **Mentioned, but missing as listed use-case doc.**
994
+
995
+ The README source-family table includes processing/workflow activity, but the use-cases table does not include a dedicated doc.
996
+
997
+ ### Fix
998
+
999
+ Add:
1000
+
1001
+ ```text
1002
+ Processing & workflow activity | @x12i/countex/docs/use-cases/processing-workflow-activity | docs/use-cases/processing-workflow-activity.md
1003
+ ```
1004
+
1005
+ ### Priority
1006
+
1007
+ **P1 — documentation completeness.**
1008
+
1009
+ ---
1010
+
1011
+ # 3. Concrete missing features by layer
1012
+
1013
+ ## 3.1 Core — missing or underdefined
1014
+
1015
+ | Area | Status | Concrete fix | Priority |
1016
+ |---|---|---|---|
1017
+ | `measures` in mapping table | Missing | Add table row + example | P0 |
1018
+ | `minute/hour` in mapping table | Mismatch | Update timeframe list | P0 |
1019
+ | Map preview | Missing | Add `countex.preview(event)` | P0 |
1020
+ | Map validation | Missing | Add `validateMap(map, samples)` | P0 |
1021
+ | Dimension hash canonicalization | Missing | Define stable sorting + serialization + hash | P0 |
1022
+ | Dictionary concurrency | Missing | Atomic upsert/sequence allocation design | P0 |
1023
+ | Cache coverage | Missing | Per-bucket coverage metadata | P0 |
1024
+ | Query explain | Missing | Add `explainQuery(query)` | P1 |
1025
+ | Retry/backoff | Partial | Add flush retry policy | P1 |
1026
+ | Local WAL | Missing optional | Add optional crash-loss reduction | P2 |
1027
+ | Retention cleanup | Missing | Add `retention.apply()` | P1 |
1028
+ | Schema versioning | Missing | Add `schemaVersion`, `mapVersion` | P1 |
1029
+
1030
+ ---
1031
+
1032
+ ## 3.2 Storage — missing or underdefined
1033
+
1034
+ | Area | Status | Concrete fix | Priority |
1035
+ |---|---|---|---|
1036
+ | Mongo DB-per-scope safety | Partial | Sanitize/hash scope IDs | P1 |
1037
+ | Mongo indexes | Partial | Add full required/optional index list | P0 |
1038
+ | Mongo migrations | Missing | Versioned migrations or `init()` migration mode | P1 |
1039
+ | Redis retention | Partial | Per-timeframe retention config | P1 |
1040
+ | Redis coverage metadata | Missing | Coverage keys/status per bucket | P0 |
1041
+ | Redis maxmemory guidance | Missing | Recommend `noeviction` | P1 |
1042
+ | Dictionary cache | Partial | Define Redis/local LRU behavior | P1 |
1043
+ | Backup restore | Missing | `restore`, `verify`, manifest | P1 |
1044
+
1045
+ ---
1046
+
1047
+ ## 3.3 Simulation — missing from package surface
1048
+
1049
+ | Area | Status | Concrete fix | Priority |
1050
+ |---|---|---|---|
1051
+ | README section | Missing | Add Simulation section | P0 |
1052
+ | Docs table rows | Missing | Add simulation docs rows | P0 |
1053
+ | Entry point | Missing | `@x12i/countex/simulation` | P1 |
1054
+ | Seeded PRNG | Specified, not implemented | Add deterministic generator | P1 |
1055
+ | Event profiles | Specified | Implement audit/product/processing/LLM/custom | P1 |
1056
+ | Expected aggregates | Specified | Implement validation oracle | P1 |
1057
+ | Resource reports | Specified | Implement actual/estimated reports | P1 |
1058
+ | CLI | Specified | `countex simulate:*` commands | P2 |
1059
+
1060
+ ---
1061
+
1062
+ ## 3.4 Insights — needs API hardening
1063
+
1064
+ | Area | Status | Concrete fix | Priority |
1065
+ |---|---|---|---|
1066
+ | Ranking | Concept | Implement topN/bottomN/topPercent/Pareto | P1 |
1067
+ | Period compare | Concept | Implement comparePeriods/trend | P1 |
1068
+ | Anomaly detection | Concept | Implement robust-zscore + threshold + missing | P2 |
1069
+ | Rules | Concept | Implement evaluateRules | P2 |
1070
+ | Resource advisor | Concept | Implement estimateResourcesFromActuals | P2 |
1071
+ | Map health | Concept | Implement analyzeMapHealth | P2 |
1072
+ | Max-token advisor | Concept | Needs histograms or fallback | P2/P3 |
1073
+ | Recommendation object | Specified | Add typed contract | P1 |
1074
+ | Safety limits | Mentioned | Enforce maxGroups/maxBuckets/timeouts | P1 |
1075
+
1076
+ ---
1077
+
1078
+ # 4. Decisions still needed
1079
+
1080
+ ## Decision 1 — One package or multiple packages?
1081
+
1082
+ Recommended decision:
1083
+
1084
+ ```text
1085
+ One package with subpath exports.
1086
+ ```
1087
+
1088
+ Why:
1089
+
1090
+ ```text
1091
+ @x12i/countex
1092
+ @x12i/countex/adapters/mongo
1093
+ @x12i/countex/adapters/redis
1094
+ @x12i/countex/simulation
1095
+ @x12i/countex/insights
1096
+ ```
1097
+
1098
+ This keeps adoption simple while preserving clean boundaries.
1099
+
1100
+ ---
1101
+
1102
+ ## Decision 2 — Dictionary IDs: Number or BigInt?
1103
+
1104
+ Recommended decision:
1105
+
1106
+ ```text
1107
+ Use JS number internally while under Number.MAX_SAFE_INTEGER.
1108
+ Store Mongo values as Long-compatible where needed.
1109
+ Expose IDs as string | number in APIs.
1110
+ Avoid BigInt in hot JS paths unless required.
1111
+ ```
1112
+
1113
+ Why:
1114
+
1115
+ ```text
1116
+ BigInt complicates JSON, Redis, and user APIs.
1117
+ ```
1118
+
1119
+ ---
1120
+
1121
+ ## Decision 3 — Percentiles/histograms now or later?
1122
+
1123
+ Recommended decision:
1124
+
1125
+ ```text
1126
+ Core v1 supports sum/count/min/max.
1127
+ Histogram distribution is optional v1.1 or feature-flagged v1.
1128
+ Max-token advisor can ship with low-confidence fallback, then improve with histograms.
1129
+ ```
1130
+
1131
+ Why:
1132
+
1133
+ ```text
1134
+ Do not block core on percentile storage, but do not pretend max-token recommendations are strong without percentiles.
1135
+ ```
1136
+
1137
+ ---
1138
+
1139
+ ## Decision 4 — Restore semantics
1140
+
1141
+ Recommended decision:
1142
+
1143
+ Support three modes:
1144
+
1145
+ ```text
1146
+ replace-window
1147
+ merge-deltas
1148
+ validate-only
1149
+ ```
1150
+
1151
+ Default:
1152
+
1153
+ ```text
1154
+ validate-only for dry run
1155
+ replace-window for disaster restore
1156
+ merge-deltas only for advanced use
1157
+ ```
1158
+
1159
+ ---
1160
+
1161
+ ## Decision 5 — Timezone behavior
1162
+
1163
+ Recommended decision:
1164
+
1165
+ ```text
1166
+ Default UTC buckets.
1167
+ ISO week.
1168
+ Optional map-level timezone override.
1169
+ ```
1170
+
1171
+ Why:
1172
+
1173
+ ```text
1174
+ UTC keeps distributed systems predictable.
1175
+ ```
1176
+
1177
+ ---
1178
+
1179
+ # 5. Implementation-ready priority plan
1180
+
1181
+ ## P0 — Fix before core implementation continues
1182
+
1183
+ These directly affect correctness or code-agent interpretation.
1184
+
1185
+ 1. Add `measures` to Mapping reference.
1186
+ 2. Add `minute/hour` to Mapping reference timeframe list.
1187
+ 3. Define dimension canonicalization/hash algorithm.
1188
+ 4. Define dictionary allocation under concurrency.
1189
+ 5. Define cache coverage metadata and query planner behavior.
1190
+ 6. Add map preview/dry-run API.
1191
+ 7. Add Simulation section to README/docs table.
1192
+ 8. Normalize insight docs paths in README.
1193
+ 9. Clarify local worker memory vs Redis retained state.
1194
+ 10. Clarify metric vs label and sourceKey vs uid terminology.
1195
+
1196
+ ---
1197
+
1198
+ ## P1 — Production-readiness next
1199
+
1200
+ 1. Add backup verify/restore APIs.
1201
+ 2. Add retention policy for Mongo and Redis.
1202
+ 3. Add self-managed deployment guide.
1203
+ 4. Add resource math/sizing guide.
1204
+ 5. Add Mongo/Redis operational recommendations.
1205
+ 6. Add schemaVersion/mapVersion.
1206
+ 7. Add query explain and query metadata.
1207
+ 8. Add multi-worker dictionary integration test.
1208
+ 9. Add failure-mode tests.
1209
+ 10. Add security/tenant isolation guidance.
1210
+
1211
+ ---
1212
+
1213
+ ## P2 — Product-value layer
1214
+
1215
+ 1. Implement simulation engine.
1216
+ 2. Implement simulation profiles.
1217
+ 3. Implement expected aggregate validation.
1218
+ 4. Implement insights ranking/slicing/Pareto.
1219
+ 5. Implement comparePeriods/trend.
1220
+ 6. Implement anomaly detection/rules.
1221
+ 7. Implement resource/map advisors.
1222
+ 8. Implement LLM/cost/token advisors with fallback.
1223
+
1224
+ ---
1225
+
1226
+ ## P3 — Advanced/later
1227
+
1228
+ 1. Histogram/percentile storage if not done earlier.
1229
+ 2. Approx unique counts via HLL.
1230
+ 3. Parquet export.
1231
+ 4. OpenTelemetry integration.
1232
+ 5. Sharding guidance.
1233
+ 6. Web demo/dashboard.
1234
+ 7. Distributed simulation cluster.
1235
+
1236
+ ---
1237
+
1238
+ # 6. Final complete action list
1239
+
1240
+ This is the direct list I would give to a code agent or implementation team.
1241
+
1242
+ ## README/spec patch actions
1243
+
1244
+ 1. Patch Mapping reference: add `measures`.
1245
+ 2. Patch Mapping reference: add `minute/hour` to timeframe list.
1246
+ 3. Add note: `metric` is generic; `label` is one example.
1247
+ 4. Add note: use `sourceKey`, not `uid`, in generic docs.
1248
+ 5. Add note: workers keep only unflushed deltas; retained state is Mongo/Redis.
1249
+ 6. Add Simulation section.
1250
+ 7. Add simulation docs rows to docs table.
1251
+ 8. Add processing/workflow use-case doc row.
1252
+ 9. Normalize Insights doc links so paths match actual package files.
1253
+ 10. Add Unique Counts warning section.
1254
+
1255
+ ## Core implementation actions
1256
+
1257
+ 1. Implement deterministic dimension canonicalization and hashing.
1258
+ 2. Implement concurrency-safe dictionary allocation.
1259
+ 3. Implement `preview(event)`.
1260
+ 4. Implement `validateMap(map, sampleEvents)`.
1261
+ 5. Implement cache coverage metadata.
1262
+ 6. Implement query planner full/partial/miss logic.
1263
+ 7. Implement query result metadata: buckets scanned, cells scanned, source.
1264
+ 8. Add flush retry/backoff and batch limits.
1265
+ 9. Add mapVersion/schemaVersion fields.
1266
+ 10. Add retention policy engine.
1267
+
1268
+ ## Storage actions
1269
+
1270
+ 1. Add Mongo unique index for cell identity.
1271
+ 2. Add Mongo unique index for dictionary identity.
1272
+ 3. Add safe `dbResolver` examples.
1273
+ 4. Add Redis per-timeframe TTL/retention.
1274
+ 5. Add Redis coverage keys.
1275
+ 6. Add Redis rebuild from Mongo by bucket range.
1276
+ 7. Add backup export manifest.
1277
+ 8. Add backup verify.
1278
+ 9. Add backup restore modes.
1279
+ 10. Add backup/rebuild tests.
1280
+
1281
+ ## Simulation actions
1282
+
1283
+ 1. Add `@x12i/countex/simulation` entry point.
1284
+ 2. Implement seeded PRNG.
1285
+ 3. Implement pools and relations.
1286
+ 4. Implement audit-log profile.
1287
+ 5. Implement product-activity profile.
1288
+ 6. Implement processing-activity profile.
1289
+ 7. Implement llm-usage profile.
1290
+ 8. Implement custom profile.
1291
+ 9. Implement expected aggregate tracker.
1292
+ 10. Implement simulation report.
1293
+ 11. Implement resource estimator.
1294
+ 12. Implement simulation CLI.
1295
+
1296
+ ## Insights actions
1297
+
1298
+ 1. Add `@x12i/countex/insights` entry point.
1299
+ 2. Implement topN/bottomN.
1300
+ 3. Implement topPercent/bottomPercent.
1301
+ 4. Implement Pareto/contribution.
1302
+ 5. Implement comparePeriods.
1303
+ 6. Implement trend.
1304
+ 7. Implement detectAnomalies basic.
1305
+ 8. Implement evaluateRules.
1306
+ 9. Implement estimateResourcesFromActuals.
1307
+ 10. Implement analyzeMapHealth.
1308
+ 11. Implement costDrivers.
1309
+ 12. Implement budgetProjection.
1310
+ 13. Implement recommendMaxTokens fallback.
1311
+ 14. Add histogram support or mark percentile recommendations as limited.
1312
+
1313
+ ---
1314
+
1315
+ # 7. Final conclusion
1316
+
1317
+ The current Countex design is directionally complete but not yet implementation-complete.
1318
+
1319
+ The main risk is not that we missed the product idea. The product idea is now strong:
1320
+
1321
+ ```text
1322
+ Generic mapped counters + simulation + insights
1323
+ ```
1324
+
1325
+ The real risk is ambiguity in the mechanics:
1326
+
1327
+ ```text
1328
+ dictionary IDs
1329
+ dimension hashing
1330
+ cache coverage
1331
+ retention
1332
+ backup restore
1333
+ measure distributions
1334
+ query safety
1335
+ ```
1336
+
1337
+ Those are exactly the areas to tighten before handing this to code agents.
1338
+
1339
+ Once the P0 items are patched, the package spec becomes much safer to implement.
1340
+