@x12i/countex 1.5.0 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. package/README.md +57 -17
  2. package/dist/backup/manifest.d.ts +22 -1
  3. package/dist/backup/manifest.d.ts.map +1 -1
  4. package/dist/backup/manifest.js +34 -6
  5. package/dist/backup/manifest.js.map +1 -1
  6. package/dist/core/countex.d.ts +66 -1
  7. package/dist/core/countex.d.ts.map +1 -1
  8. package/dist/core/countex.js +113 -4
  9. package/dist/core/countex.js.map +1 -1
  10. package/dist/index.d.ts +4 -4
  11. package/dist/index.d.ts.map +1 -1
  12. package/dist/index.js +3 -1
  13. package/dist/index.js.map +1 -1
  14. package/dist/query/planner.d.ts +15 -1
  15. package/dist/query/planner.d.ts.map +1 -1
  16. package/dist/query/planner.js +136 -1
  17. package/dist/query/planner.js.map +1 -1
  18. package/dist/storage/coverage.d.ts +13 -0
  19. package/dist/storage/coverage.d.ts.map +1 -0
  20. package/dist/storage/coverage.js +37 -0
  21. package/dist/storage/coverage.js.map +1 -0
  22. package/dist/storage/memory.d.ts +20 -5
  23. package/dist/storage/memory.d.ts.map +1 -1
  24. package/dist/storage/memory.js +127 -3
  25. package/dist/storage/memory.js.map +1 -1
  26. package/dist/storage/redis.d.ts +36 -11
  27. package/dist/storage/redis.d.ts.map +1 -1
  28. package/dist/storage/redis.js +189 -41
  29. package/dist/storage/redis.js.map +1 -1
  30. package/dist/types/index.d.ts +114 -7
  31. package/dist/types/index.d.ts.map +1 -1
  32. package/docs/DEPLOYMENT.md +87 -3
  33. package/docs/guides/adapter-redis.md +40 -4
  34. package/docs/guides/backup-export.md +84 -32
  35. package/docs/guides/counter-maps.md +8 -0
  36. package/docs/guides/preview.md +92 -0
  37. package/docs/guides/querying.md +30 -0
  38. package/docs/internals/internal-storage-mongo.md +24 -4
  39. package/docs/internals/internal-storage-redis.md +41 -9
  40. package/package.json +1 -1
@@ -114,9 +114,17 @@ Total raw increments per event ≈
114
114
 
115
115
  Plan capacity accordingly.
116
116
 
117
+ ## Preview before ingesting
118
+
119
+ Run `countex.preview(event, { counterId })` on a sample event to see exactly what cells your map would generate — without any dictionary writes or buffering. See [Preview a map](./preview.md) for the full result shape (`subjectSourceKey`, `metricSourceKey`, `dimensionsHash`, `timeframes`, `delta`, `measures`, plus `extractionWarnings` and `droppedAfterCardinality`).
120
+
121
+ Dimension keys/values feeding into `dimensionsHash` are canonicalized: keys sorted lexicographically, values type-tagged, hashed (SHA-1 truncated to 12 hex). Any two workers always produce the same hash for the same logical dimension tuple. See [internal-dimension-hash.md](../internals/internal-dimension-hash.md) for the exact recipe.
122
+
117
123
  ## See also
118
124
 
119
125
  - [Timeframes](./timeframes-and-bucketing.md)
120
126
  - [Measures](./measures.md)
121
127
  - [Cardinality](./cardinality.md)
128
+ - [Preview a map](./preview.md)
122
129
  - Internal: [Mapping validation & extraction](../internals/internal-mapping-validation-and-extraction.md)
130
+ - Internal: [Dimension hash](../internals/internal-dimension-hash.md)
@@ -0,0 +1,92 @@
1
+ # Guide: Preview a map (dry-run)
2
+
3
+ `countex.preview(event, options?)` runs **extraction + cardinality** for one event against your maps and returns the increments it _would_ produce — without dictionary I/O, without buffering, without any storage writes. Use it to debug a counter map before ingesting millions of events.
4
+
5
+ ## Step 1: Construct a Countex instance
6
+
7
+ ```ts
8
+ import { createCountex } from "@x12i/countex";
9
+ import { MemoryDurable } from "@x12i/countex/adapters/memory";
10
+
11
+ const countex = createCountex({
12
+ maps: [accessMap],
13
+ durable: new MemoryDurable(),
14
+ flush: { everyMs: 0 },
15
+ });
16
+ ```
17
+
18
+ A real durable adapter is fine too — preview never calls it.
19
+
20
+ ## Step 2: Run preview
21
+
22
+ ```ts
23
+ const previews = countex.preview(event, {
24
+ counterId: "access_by_subject_metric", // optional; defaults to all maps
25
+ });
26
+ ```
27
+
28
+ `previews` is an array of `MapPreviewResult`, one per map.
29
+
30
+ ## Result shape
31
+
32
+ ```ts
33
+ interface MapPreviewResult {
34
+ counterId: string;
35
+ skipped: boolean;
36
+ extractionWarnings: ExtractionWarning[];
37
+ cells: PreviewIncrementCell[];
38
+ /** Increments dropped by cardinality guard. Preview runs silently — no `dropped` events. */
39
+ droppedAfterCardinality: number;
40
+ }
41
+
42
+ interface PreviewIncrementCell {
43
+ scopeId: string;
44
+ counterId: string;
45
+ timeframeType: TimeframeType;
46
+ timeframeKey: string;
47
+ subjectType: string;
48
+ subjectSourceKey: string;
49
+ metricSourceKey: string;
50
+ dimensions: Dimensions;
51
+ dimensionsHash: string;
52
+ delta: number;
53
+ measures: Record<string, { value: number; aggs: MeasureAgg[] }>;
54
+ }
55
+ ```
56
+
57
+ `subjectSourceKey` and `metricSourceKey` are the raw strings extracted from the event — preview does not allocate dictionary ids. `dimensionsHash` matches the canonical form computed during real ingestion (see [internal-dimension-hash.md](../internals/internal-dimension-hash.md)).
58
+
59
+ ## Counting fan-out
60
+
61
+ ```ts
62
+ const total = previews.reduce((s, p) => s + p.cells.length, 0);
63
+ console.log(`This event would generate ${total} cell increments.`);
64
+ ```
65
+
66
+ For the canonical example (1 actor × 2 metrics × 5 timeframes), `total === 10`.
67
+
68
+ ## Diagnostics: warnings and skips
69
+
70
+ `extractionWarnings` includes things like `"measure_invalid"`, `"measure_missing"`, missing required dimensions, and bad path resolutions. Use them to spot a misconfigured map without having to ingest and watch event listeners.
71
+
72
+ ```ts
73
+ for (const p of previews) {
74
+ if (p.skipped) console.warn(`map "${p.counterId}" skipped this event`);
75
+ for (const w of p.extractionWarnings) {
76
+ console.warn(`extraction warning: ${w.reason}`, w.meta);
77
+ }
78
+ }
79
+ ```
80
+
81
+ ## Caveats
82
+
83
+ - **Cardinality guard state is shared with the live engine.** Preview runs the same `CardinalityGuard.check`, so calling preview repeatedly counts toward the per-scope cardinality limits. For isolated dry-runs, construct a separate `Countex` instance just for previewing.
84
+ - **No idempotency check.** Preview ignores `idempotency` — same event can be previewed any number of times.
85
+ - **No measure scaling on output.** `measures[name].value` is the raw extracted number; on the live ingest path it would be multiplied by `measure.scale` before persistence.
86
+ - **No dropped/overflow events emitted.** Use `droppedAfterCardinality` and `extractionWarnings` instead.
87
+
88
+ ## See also
89
+
90
+ - [Counter maps guide](./counter-maps.md)
91
+ - [Cardinality guide](./cardinality.md)
92
+ - Internal: [Mapping validation & extraction](../internals/internal-mapping-validation-and-extraction.md)
@@ -81,9 +81,39 @@ Result includes `source`: `"cache"`, `"durable"`, or `"merged"`.
81
81
  | `cacheAttempted` | `true` if the engine consulted the cache adapter (cache configured, not `bypassCache`, and the planner allowed cache for this query shape). |
82
82
  | `cacheCoverage` | When `cacheAttempted` is true: `"full"` if the cache answered completely; `"partial"` or `"miss"` if the planner fell through to durable for authoritative cells. |
83
83
  | `cellsMaterialized` | Number of counter cells loaded from storage before grouping/formatting. |
84
+ | `bucketsRequested` | Bucket keys the query expanded to (after `bucketRange`). |
85
+ | `cacheBucketsHit` | Number of buckets the cache reported as `complete`. |
86
+ | `durableFallback` | `true` when the planner had to read durable for some/all buckets. |
87
+ | `plannedCellsEstimate` | Heuristic estimate of cells before grouping (matches `cellsMaterialized` after execution). |
88
+ | `groupByCost` | `"trivial"` (no groupBy), `"low"`, or `"high"` (3+ keys or 2+ dimensions). |
89
+ | `expensive` | Heuristic flag — `true` for big windows, high groupByCost, or `format: "matrix"` with many cells. |
84
90
 
85
91
  Use this for tests, dashboards, and verifying Redis vs Mongo paths.
86
92
 
93
+ ### Explain a query without running it
94
+
95
+ `countex.explainQuery(spec)` returns a {@link QueryPlan} without touching durable storage and only probing per-bucket cache coverage:
96
+
97
+ ```ts
98
+ const plan = await countex.explainQuery({
99
+ scope: "org_1",
100
+ counterId: "events",
101
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-31" },
102
+ groupBy: ["dim:operation", "dim:situation", "subject"],
103
+ });
104
+ // {
105
+ // bucketsRequested: ["2026-05-01", …, "2026-05-31"],
106
+ // cacheEligible: true,
107
+ // cacheBucketsExpectedHit: 18,
108
+ // durableFallbackPredicted: true,
109
+ // plannedCellsEstimate: 1_240,
110
+ // groupByCost: "high",
111
+ // expensive: true,
112
+ // }
113
+ ```
114
+
115
+ `cacheBucketsExpectedHit` and `plannedCellsEstimate` are populated when the cache adapter implements `probeBucketCoverage` (the bundled `RedisCache` and `MemoryCache` both do).
116
+
87
117
  ## Step 6: Totals, event volume, and “distinct entities”
88
118
 
89
119
  ### Grand total (sum all matching cells)
@@ -4,21 +4,41 @@
4
4
 
5
5
  - `applyDeltas` — upserts with `$inc` style operations for counts and compatible measure fields.
6
6
  - `readCells` — satisfies `QuerySpec` filters using compound indexes.
7
- - `resolveIds` — transactional or sequential allocation of dictionary ids (`countex_dict`, `countex_seq`).
7
+ - `resolveIds` — concurrency-safe allocation of dictionary ids (`countex_dict`, `countex_seq`).
8
8
  - `streamCells` — bounded iteration for backup/export.
9
+ - `importSnapshotCells` — replace-window restore target (called by `restoreSnapshotCells`).
9
10
 
10
11
  ## Write path interaction
11
12
 
12
- Flush snapshots arrive as arrays of `PendingDelta`; Mongo applies them idempotently per cell identity tuple.
13
+ Flush snapshots arrive as arrays of `PendingDelta`; Mongo applies them idempotently per cell identity tuple via `bulkWrite` of `updateOne` operations with `$inc` and `upsert: true`.
13
14
 
14
15
  ## Indexes
15
16
 
16
- Created lazily on first write paths (see README schema excerpt): uniqueness on full identity, secondary indexes for timeframe/subject/metric scans.
17
+ Created lazily on first write paths:
18
+
19
+ - `countex_cells` — `uniq_cell_identity` (unique on the full identity tuple), `by_timeframe`, `by_subject`, `by_metric`.
20
+ - `countex_dict` — `uniq_dict_key` (unique on `scopeId, kind, qualifier, sourceKey`).
21
+
22
+ ## Dictionary allocation under concurrency
23
+
24
+ `resolveIds` follows a strict five-step recipe so 20+ workers seeing the same new `sourceKey` at once still converge on a single internal id:
25
+
26
+ 1. **Read** existing dict rows for the requested `(scopeId, kind, qualifier, sourceKey)` batch in one round-trip.
27
+ 2. **Allocate** a contiguous block of ids from the per-scope sequence document via `findOneAndUpdate({ _id }, { $inc: { seq: count } })` — atomic per sequence document.
28
+ 3. **Insert** the new dict rows with `insertMany({ ordered: false })`. The `uniq_dict_key` index forbids duplicates.
29
+ 4. On **duplicate-key error** (another worker won the race), **re-read** the dict for the same identities — the winning row is now visible.
30
+ 5. **Cache** the resolved id in the local LRU and (optionally) Redis.
31
+
32
+ Bad outcome avoided: two workers each minting `user_42 → 10001` and `user_42 → 10002`. With this protocol one insert wins, the loser re-reads the winner's id, and counts never fragment across two ids.
33
+
34
+ The user-facing recipe lives in [`docs/guides/adapter-mongo.md`](../guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency".
17
35
 
18
36
  ## Dictionary vs dimensions
19
37
 
20
- Subjects/metrics map to integers; dimensions remain embedded documents/hashes for cells — matching core engine assumptions.
38
+ Subjects/metrics map to integers via `countex_dict`. Dimension VALUES are stored as raw values on cells (not via the dictionary), keeping cells self-describing and avoiding dictionary churn for low-cardinality dimensions.
21
39
 
22
40
  ## See also
23
41
 
24
42
  - [Backup streaming](./internal-backup-stream.md)
43
+ - [Dimension hash](./internal-dimension-hash.md) — how the `dimensionsHash` part of the cell identity is computed.
44
+ - [Mongo adapter guide](../guides/adapter-mongo.md)
@@ -1,18 +1,49 @@
1
1
  # Internal: Redis cache adapter
2
2
 
3
- `RedisCache` (`src/storage/redis.ts`) maps each `(scope, counterId, timeframeType, timeframeKey)` bucket to a **HASH**:
3
+ `RedisCache` (`src/storage/redis.ts`) maps each `(scope, counterId, timeframeType, timeframeKey)` bucket to a **HASH** of cell entries plus a sibling **`:cov` HASH** of coverage metadata:
4
4
 
5
- - Field encodes `subjectType|subjectId|metricId|dimensionsHash`
6
- - Value stores encoded counts / measure sums suitable for fast `HGETALL`
5
+ ```
6
+ countex:{scope}:{counterId}:{tf}:{bucket} → HASH (cells)
7
+ field = "{subjectType}|{subjectId}|{metricId}|{dimensionsHash}"
8
+ value = count (integer)
9
+ field = "{base}:m:{measureName}:sum" → measure sum
10
+ field = "{base}:m:{measureName}:count" → measure event count
7
11
 
8
- Metadata hashes may store dimension JSON for reconstruction.
12
+ countex:{scope}:{counterId}:{tf}:{bucket}:cov → HASH (coverage meta)
13
+ status = "complete" | "partial" | "rebuilding"
14
+ lastWarmAt = epoch ms
15
+ cellCount = number of cells last warmed (set on warm)
16
+ schemaVersion = REDIS_CACHE_SCHEMA_VERSION
17
+ mapVersion = optional opaque integer
18
+
19
+ countex:{scope}:{counterId}:{tf}:{bucket}:meta:{baseField} → JSON of dimensions
20
+ ```
21
+
22
+ Per-bucket coverage gives the planner authoritative information about which buckets the cache can answer with confidence, instead of guessing from "did `HGETALL` return anything?".
23
+
24
+ ## Coverage state machine
25
+
26
+ | Triggered by | Effect on `:cov.status` |
27
+ |---|---|
28
+ | First `applyDeltas` to a previously-empty bucket | `partial` (via `HSETNX`) — never downgrades a `complete` or `rebuilding` bucket |
29
+ | Subsequent `applyDeltas` | unchanged status; `lastWarmAt` updated |
30
+ | `warm()` | `complete`, with `cellCount`, `schemaVersion`, `mapVersion`, `lastWarmAt` written atomically |
31
+ | `markBucketRebuilding()` | `rebuilding` (called by `rebuildCache` before refilling) |
32
+ | TTL expiry | key disappears → `missing` on next read |
9
33
 
10
34
  ## Read coverage
11
35
 
12
- `readCells` returns:
36
+ `readCells` pipelines both `HGETALL bucket` and `HGETALL :cov`, then derives per-bucket coverage from the meta:
37
+
38
+ - All buckets `complete` → aggregate `full`.
39
+ - Any `complete | partial | rebuilding` mixed with the rest → aggregate `partial`.
40
+ - All `missing` → aggregate `miss`.
41
+
42
+ When the planner supplies `expectedMapVersion` or `expectedSchemaVersion` and the bucket's stored version differs, `complete` is demoted to `partial` (cache is treated as stale for that bucket). The planner then falls back to durable.
43
+
44
+ ## Probe-only path
13
45
 
14
- - `coverage: "full"` when all requested buckets present
15
- - `"partial"` / `"miss"` otherwise — planner falls back to durable
46
+ `probeBucketCoverage(scope, counterId, tf, [keys])` returns just the `:cov` hashes without touching cell data. The `Countex.explainQuery(spec)` API uses this to predict the cache hit rate without materializing rows.
16
47
 
17
48
  ## Dedupe keys
18
49
 
@@ -20,12 +51,13 @@ Metadata hashes may store dimension JSON for reconstruction.
20
51
 
21
52
  ## Warm path
22
53
 
23
- `warm` hydrates hashes from `CounterCell[]` after durable reads.
54
+ `warm` hydrates hashes from `CounterCell[]` after durable reads, then sets `:cov.status=complete`. Used by `rebuildCache` and by the query planner's best-effort warm-after-fallback.
24
55
 
25
56
  ## Error stance
26
57
 
27
- Apply failures during flush are swallowed at Countex level after durable success — subsequent reads rebuild cache.
58
+ Apply failures during flush are swallowed at Countex level after durable success — subsequent reads rebuild cache. Coverage meta is best-effort: a stale `:cov.status=complete` only causes a temporary divergence (caller sees stale-but-consistent reads until TTL expires).
28
59
 
29
60
  ## See also
30
61
 
31
62
  - [Query path](./internal-query-path.md)
63
+ - [Storage memory](./internal-storage-memory.md) — mirrors the same coverage contract for tests.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@x12i/countex",
3
- "version": "1.5.0",
3
+ "version": "1.6.0",
4
4
  "description": "Generic sparse counter matrices for JSON events. Dynamic counting, aggregation, and retrieval from mapped JSON inputs.",
5
5
  "repository": {
6
6
  "type": "git",