@x12i/countex 1.2.0 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (112) hide show
  1. package/README.md +80 -18
  2. package/dist/backup/manifest.d.ts +70 -0
  3. package/dist/backup/manifest.d.ts.map +1 -0
  4. package/dist/backup/manifest.js +131 -0
  5. package/dist/backup/manifest.js.map +1 -0
  6. package/dist/backup/snapshot-encoding.d.ts +8 -0
  7. package/dist/backup/snapshot-encoding.d.ts.map +1 -0
  8. package/dist/backup/snapshot-encoding.js +27 -0
  9. package/dist/backup/snapshot-encoding.js.map +1 -0
  10. package/dist/cli.js +16 -0
  11. package/dist/cli.js.map +1 -1
  12. package/dist/core/countex.d.ts +79 -1
  13. package/dist/core/countex.d.ts.map +1 -1
  14. package/dist/core/countex.js +254 -49
  15. package/dist/core/countex.js.map +1 -1
  16. package/dist/core/extraction.d.ts +1 -5
  17. package/dist/core/extraction.d.ts.map +1 -1
  18. package/dist/core/extraction.js.map +1 -1
  19. package/dist/core/mapping.d.ts +5 -0
  20. package/dist/core/mapping.d.ts.map +1 -1
  21. package/dist/core/mapping.js +27 -0
  22. package/dist/core/mapping.js.map +1 -1
  23. package/dist/index.d.ts +5 -2
  24. package/dist/index.d.ts.map +1 -1
  25. package/dist/index.js +15 -1
  26. package/dist/index.js.map +1 -1
  27. package/dist/query/formatter.d.ts +2 -2
  28. package/dist/query/formatter.d.ts.map +1 -1
  29. package/dist/query/formatter.js +9 -6
  30. package/dist/query/formatter.js.map +1 -1
  31. package/dist/query/planner.d.ts +15 -1
  32. package/dist/query/planner.d.ts.map +1 -1
  33. package/dist/query/planner.js +152 -3
  34. package/dist/query/planner.js.map +1 -1
  35. package/dist/simulation/context.d.ts +5 -0
  36. package/dist/simulation/context.d.ts.map +1 -0
  37. package/dist/simulation/context.js +56 -0
  38. package/dist/simulation/context.js.map +1 -0
  39. package/dist/simulation/events.d.ts +3 -0
  40. package/dist/simulation/events.d.ts.map +1 -0
  41. package/dist/simulation/events.js +67 -0
  42. package/dist/simulation/events.js.map +1 -0
  43. package/dist/simulation/index.d.ts +7 -0
  44. package/dist/simulation/index.d.ts.map +1 -0
  45. package/dist/simulation/index.js +16 -0
  46. package/dist/simulation/index.js.map +1 -0
  47. package/dist/simulation/prng.d.ts +4 -0
  48. package/dist/simulation/prng.d.ts.map +1 -0
  49. package/dist/simulation/prng.js +18 -0
  50. package/dist/simulation/prng.js.map +1 -0
  51. package/dist/simulation/resource-estimate.d.ts +6 -0
  52. package/dist/simulation/resource-estimate.d.ts.map +1 -0
  53. package/dist/simulation/resource-estimate.js +11 -0
  54. package/dist/simulation/resource-estimate.js.map +1 -0
  55. package/dist/simulation/simulator.d.ts +15 -0
  56. package/dist/simulation/simulator.d.ts.map +1 -0
  57. package/dist/simulation/simulator.js +67 -0
  58. package/dist/simulation/simulator.js.map +1 -0
  59. package/dist/simulation/tracker.d.ts +8 -0
  60. package/dist/simulation/tracker.d.ts.map +1 -0
  61. package/dist/simulation/tracker.js +18 -0
  62. package/dist/simulation/tracker.js.map +1 -0
  63. package/dist/simulation/types.d.ts +49 -0
  64. package/dist/simulation/types.d.ts.map +1 -0
  65. package/dist/simulation/types.js +21 -0
  66. package/dist/simulation/types.js.map +1 -0
  67. package/dist/storage/coverage.d.ts +13 -0
  68. package/dist/storage/coverage.d.ts.map +1 -0
  69. package/dist/storage/coverage.js +37 -0
  70. package/dist/storage/coverage.js.map +1 -0
  71. package/dist/storage/memory.d.ts +23 -5
  72. package/dist/storage/memory.d.ts.map +1 -1
  73. package/dist/storage/memory.js +162 -3
  74. package/dist/storage/memory.js.map +1 -1
  75. package/dist/storage/mongo.d.ts +5 -0
  76. package/dist/storage/mongo.d.ts.map +1 -1
  77. package/dist/storage/mongo.js +68 -1
  78. package/dist/storage/mongo.js.map +1 -1
  79. package/dist/storage/redis.d.ts +40 -11
  80. package/dist/storage/redis.d.ts.map +1 -1
  81. package/dist/storage/redis.js +202 -44
  82. package/dist/storage/redis.js.map +1 -1
  83. package/dist/types/index.d.ts +165 -7
  84. package/dist/types/index.d.ts.map +1 -1
  85. package/docs/DEPLOYMENT.md +125 -3
  86. package/docs/countex_gap_analysis_complete_checklist.md +1340 -0
  87. package/docs/guides/README.md +5 -0
  88. package/docs/guides/adapter-mongo.md +12 -0
  89. package/docs/guides/adapter-redis.md +46 -4
  90. package/docs/guides/backup-export.md +106 -10
  91. package/docs/guides/cli.md +8 -0
  92. package/docs/guides/counter-maps.md +14 -0
  93. package/docs/guides/insights-overview.md +6 -0
  94. package/docs/guides/measures.md +4 -0
  95. package/docs/guides/preview.md +92 -0
  96. package/docs/guides/querying.md +70 -0
  97. package/docs/guides/simulation-profiles.md +33 -0
  98. package/docs/guides/simulation-resource-estimation.md +30 -0
  99. package/docs/guides/simulation.md +36 -0
  100. package/docs/internals/README.md +1 -0
  101. package/docs/internals/internal-dimension-hash.md +34 -0
  102. package/docs/internals/internal-storage-mongo.md +24 -4
  103. package/docs/internals/internal-storage-redis.md +41 -9
  104. package/docs/use-cases/README.md +56 -0
  105. package/docs/use-cases/audit-security-log-counts.md +550 -0
  106. package/docs/use-cases/llm-cost-and-token-metering.md +255 -0
  107. package/docs/use-cases/multi-tenant-isolation.md +203 -0
  108. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -0
  109. package/docs/use-cases/processing-workflow-activity.md +165 -0
  110. package/docs/use-cases/product-feature-usage.md +240 -0
  111. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -0
  112. package/package.json +18 -2
@@ -4,21 +4,41 @@
4
4
 
5
5
  - `applyDeltas` — upserts with `$inc` style operations for counts and compatible measure fields.
6
6
  - `readCells` — satisfies `QuerySpec` filters using compound indexes.
7
- - `resolveIds` — transactional or sequential allocation of dictionary ids (`countex_dict`, `countex_seq`).
7
+ - `resolveIds` — concurrency-safe allocation of dictionary ids (`countex_dict`, `countex_seq`).
8
8
  - `streamCells` — bounded iteration for backup/export.
9
+ - `importSnapshotCells` — replace-window restore target (called by `restoreSnapshotCells`).
9
10
 
10
11
  ## Write path interaction
11
12
 
12
- Flush snapshots arrive as arrays of `PendingDelta`; Mongo applies them idempotently per cell identity tuple.
13
+ Flush snapshots arrive as arrays of `PendingDelta`; Mongo applies them idempotently per cell identity tuple via `bulkWrite` of `updateOne` operations with `$inc` and `upsert: true`.
13
14
 
14
15
  ## Indexes
15
16
 
16
- Created lazily on first write paths (see README schema excerpt): uniqueness on full identity, secondary indexes for timeframe/subject/metric scans.
17
+ Created lazily on first write paths:
18
+
19
+ - `countex_cells` — `uniq_cell_identity` (unique on the full identity tuple), `by_timeframe`, `by_subject`, `by_metric`.
20
+ - `countex_dict` — `uniq_dict_key` (unique on `scopeId, kind, qualifier, sourceKey`).
21
+
22
+ ## Dictionary allocation under concurrency
23
+
24
+ `resolveIds` follows a strict five-step recipe so 20+ workers seeing the same new `sourceKey` at once still converge on a single internal id:
25
+
26
+ 1. **Read** existing dict rows for the requested `(scopeId, kind, qualifier, sourceKey)` batch in one round-trip.
27
+ 2. **Allocate** a contiguous block of ids from the per-scope sequence document via `findOneAndUpdate({ _id }, { $inc: { seq: count } })` — atomic per sequence document.
28
+ 3. **Insert** the new dict rows with `insertMany({ ordered: false })`. The `uniq_dict_key` index forbids duplicates.
29
+ 4. On **duplicate-key error** (another worker won the race), **re-read** the dict for the same identities — the winning row is now visible.
30
+ 5. **Cache** the resolved id in the local LRU and (optionally) Redis.
31
+
32
+ Bad outcome avoided: two workers each minting `user_42 → 10001` and `user_42 → 10002`. With this protocol one insert wins, the loser re-reads the winner's id, and counts never fragment across two ids.
33
+
34
+ The user-facing recipe lives in [`docs/guides/adapter-mongo.md`](../guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency".
17
35
 
18
36
  ## Dictionary vs dimensions
19
37
 
20
- Subjects/metrics map to integers; dimensions remain embedded documents/hashes for cells — matching core engine assumptions.
38
+ Subjects/metrics map to integers via `countex_dict`. Dimension VALUES are stored as raw values on cells (not via the dictionary), keeping cells self-describing and avoiding dictionary churn for low-cardinality dimensions.
21
39
 
22
40
  ## See also
23
41
 
24
42
  - [Backup streaming](./internal-backup-stream.md)
43
+ - [Dimension hash](./internal-dimension-hash.md) — how the `dimensionsHash` part of the cell identity is computed.
44
+ - [Mongo adapter guide](../guides/adapter-mongo.md)
@@ -1,18 +1,49 @@
1
1
  # Internal: Redis cache adapter
2
2
 
3
- `RedisCache` (`src/storage/redis.ts`) maps each `(scope, counterId, timeframeType, timeframeKey)` bucket to a **HASH**:
3
+ `RedisCache` (`src/storage/redis.ts`) maps each `(scope, counterId, timeframeType, timeframeKey)` bucket to a **HASH** of cell entries plus a sibling **`:cov` HASH** of coverage metadata:
4
4
 
5
- - Field encodes `subjectType|subjectId|metricId|dimensionsHash`
6
- - Value stores encoded counts / measure sums suitable for fast `HGETALL`
5
+ ```
6
+ countex:{scope}:{counterId}:{tf}:{bucket} → HASH (cells)
7
+ field = "{subjectType}|{subjectId}|{metricId}|{dimensionsHash}"
8
+ value = count (integer)
9
+ field = "{base}:m:{measureName}:sum" → measure sum
10
+ field = "{base}:m:{measureName}:count" → measure event count
7
11
 
8
- Metadata hashes may store dimension JSON for reconstruction.
12
+ countex:{scope}:{counterId}:{tf}:{bucket}:cov → HASH (coverage meta)
13
+ status = "complete" | "partial" | "rebuilding"
14
+ lastWarmAt = epoch ms
15
+ cellCount = number of cells last warmed (set on warm)
16
+ schemaVersion = REDIS_CACHE_SCHEMA_VERSION
17
+ mapVersion = optional opaque integer
18
+
19
+ countex:{scope}:{counterId}:{tf}:{bucket}:meta:{baseField} → JSON of dimensions
20
+ ```
21
+
22
+ Per-bucket coverage gives the planner authoritative information about which buckets the cache can answer with confidence, instead of guessing from "did `HGETALL` return anything?".
23
+
24
+ ## Coverage state machine
25
+
26
+ | Triggered by | Effect on `:cov.status` |
27
+ |---|---|
28
+ | First `applyDeltas` to a previously-empty bucket | `partial` (via `HSETNX`) — never downgrades a `complete` or `rebuilding` bucket |
29
+ | Subsequent `applyDeltas` | unchanged status; `lastWarmAt` updated |
30
+ | `warm()` | `complete`, with `cellCount`, `schemaVersion`, `mapVersion`, `lastWarmAt` written atomically |
31
+ | `markBucketRebuilding()` | `rebuilding` (called by `rebuildCache` before refilling) |
32
+ | TTL expiry | key disappears → `missing` on next read |
9
33
 
10
34
  ## Read coverage
11
35
 
12
- `readCells` returns:
36
+ `readCells` pipelines both `HGETALL bucket` and `HGETALL :cov`, then derives per-bucket coverage from the meta:
37
+
38
+ - All buckets `complete` → aggregate `full`.
39
+ - Any `complete | partial | rebuilding` mixed with the rest → aggregate `partial`.
40
+ - All `missing` → aggregate `miss`.
41
+
42
+ When the planner supplies `expectedMapVersion` or `expectedSchemaVersion` and the bucket's stored version differs, `complete` is demoted to `partial` (cache is treated as stale for that bucket). The planner then falls back to durable.
43
+
44
+ ## Probe-only path
13
45
 
14
- - `coverage: "full"` when all requested buckets present
15
- - `"partial"` / `"miss"` otherwise — planner falls back to durable
46
+ `probeBucketCoverage(scope, counterId, tf, [keys])` returns just the `:cov` hashes without touching cell data. The `Countex.explainQuery(spec)` API uses this to predict the cache hit rate without materializing rows.
16
47
 
17
48
  ## Dedupe keys
18
49
 
@@ -20,12 +51,13 @@ Metadata hashes may store dimension JSON for reconstruction.
20
51
 
21
52
  ## Warm path
22
53
 
23
- `warm` hydrates hashes from `CounterCell[]` after durable reads.
54
+ `warm` hydrates hashes from `CounterCell[]` after durable reads, then sets `:cov.status=complete`. Used by `rebuildCache` and by the query planner's best-effort warm-after-fallback.
24
55
 
25
56
  ## Error stance
26
57
 
27
- Apply failures during flush are swallowed at Countex level after durable success — subsequent reads rebuild cache.
58
+ Apply failures during flush are swallowed at Countex level after durable success — subsequent reads rebuild cache. Coverage meta is best-effort: a stale `:cov.status=complete` only causes a temporary divergence (caller sees stale-but-consistent reads until TTL expires).
28
59
 
29
60
  ## See also
30
61
 
31
62
  - [Query path](./internal-query-path.md)
63
+ - [Storage memory](./internal-storage-memory.md) — mirrors the same coverage contract for tests.
@@ -0,0 +1,56 @@
1
+ # Countex use cases — technical playbooks
2
+
3
+ These documents are **end-to-end operational specifications**: they start from a concrete delivery problem, define **exact** input contracts and Countex primitives, then walk through **numbered steps** you can implement without improvising.
4
+
5
+ They are published with `@x12i/countex` under `docs/use-cases/` and exposed as stable `package.json` exports (see root `README.md`).
6
+
7
+ ## How each playbook is structured
8
+
9
+ Every use-case file follows the same skeleton:
10
+
11
+ 1. **Problem & constraints** — what you are counting, latency/durability expectations, and what Countex does *not* solve.
12
+ 2. **Primitives mapping** — explicit table: JSON field → `scope` / `subject` / `metric` / `dimension` / `eventTime` / `delta`.
13
+ 3. **Canonical payload** — JSON example plus a **required-field matrix** (what must be present or ingest skips).
14
+ 4. **`CounterMap`** — copy-paste literal with a **field-by-field rationale**.
15
+ 5. **Fan-out arithmetic** — formula and a worked example for capacity planning.
16
+ 6. **Implementation steps** — repository layout, dependencies, `createCountex` options, `init` / `ingest` / `flush` / `shutdown`, event subscriptions.
17
+ 7. **Query specifications** — literal `QuerySpec` objects; how to read **`QueryRow.keys`** (dictionary ids vs literal dimension strings).
18
+ 8. **Production hardening** — Mongo / Redis wiring patterns, cardinality, idempotency TTL math, observability.
19
+ 9. **Verification** — checks you can run after deploy.
20
+ 10. **Failure modes** — symptom → cause → exact mitigation.
21
+
22
+ ## Stable import paths (npm)
23
+
24
+ | Document | Subpath |
25
+ |----------|---------|
26
+ | Index (this list) | `@x12i/countex/docs/use-cases` |
27
+ | [Audit & security logs](./audit-security-log-counts.md) (includes **volume, situations, distinct entities, logical event counts**) | `@x12i/countex/docs/use-cases/audit-security-log-counts` |
28
+ | [Product & feature activity](./product-feature-usage.md) | `@x12i/countex/docs/use-cases/product-feature-usage` |
29
+ | [LLM metering](./llm-cost-and-token-metering.md) | `@x12i/countex/docs/use-cases/llm-cost-and-token-metering` |
30
+ | [Multi-tenant isolation](./multi-tenant-isolation.md) | `@x12i/countex/docs/use-cases/multi-tenant-isolation` |
31
+ | [Webhooks & queues](./webhook-idempotent-ingestion.md) | `@x12i/countex/docs/use-cases/webhook-idempotent-ingestion` |
32
+ | [Ops dashboards](./ops-dashboard-daily-rollups.md) | `@x12i/countex/docs/use-cases/ops-dashboard-daily-rollups` |
33
+ | [Processing & workflow activity](./processing-workflow-activity.md) | `@x12i/countex/docs/use-cases/processing-workflow-activity` |
34
+
35
+ ## Prerequisites (global)
36
+
37
+ - **Node.js** ≥ 18 (see `package.json` `engines`).
38
+ - Core package: `@x12i/countex`.
39
+ - Optional peers for production paths: `mongodb`, `ioredis` (your process owns client lifecycle).
40
+
41
+ ## Terminology (read this once)
42
+
43
+ | Term | Meaning |
44
+ |------|---------|
45
+ | **Scope** | Tenant boundary string (`scopeId` in storage). Every query **must** pin `scope`. |
46
+ | **Subject** | `(subjectType, subjectId)` after dictionary resolution; `subjectId` is an **integer**. |
47
+ | **Metric** | `(metricId)` after dictionary resolution; labels/strings become stable numeric ids per scope. |
48
+ | **Dimension** | Named breakdown; values stored **literally** on cells (not dictionary-encoded). |
49
+ | **Timeframe key** | UTC bucket string, e.g. `2026-05-08` for `day` (see `bucketKey` in `src/core/timeframe.ts`). |
50
+ | **Fan-out** | Cartesian product of subject extractions × metric extractions × number of `timeframes` entries. |
51
+
52
+ ## Related documentation
53
+
54
+ - Feature reference: [`docs/guides/`](../guides/README.md)
55
+ - Narrative tutorial: [`END_TO_END_EXAMPLE.md`](../END_TO_END_EXAMPLE.md)
56
+ - Deployment: [`DEPLOYMENT.md`](../DEPLOYMENT.md)