@x12i/countex 1.2.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +80 -18
- package/dist/backup/manifest.d.ts +70 -0
- package/dist/backup/manifest.d.ts.map +1 -0
- package/dist/backup/manifest.js +131 -0
- package/dist/backup/manifest.js.map +1 -0
- package/dist/backup/snapshot-encoding.d.ts +8 -0
- package/dist/backup/snapshot-encoding.d.ts.map +1 -0
- package/dist/backup/snapshot-encoding.js +27 -0
- package/dist/backup/snapshot-encoding.js.map +1 -0
- package/dist/cli.js +16 -0
- package/dist/cli.js.map +1 -1
- package/dist/core/countex.d.ts +79 -1
- package/dist/core/countex.d.ts.map +1 -1
- package/dist/core/countex.js +254 -49
- package/dist/core/countex.js.map +1 -1
- package/dist/core/extraction.d.ts +1 -5
- package/dist/core/extraction.d.ts.map +1 -1
- package/dist/core/extraction.js.map +1 -1
- package/dist/core/mapping.d.ts +5 -0
- package/dist/core/mapping.d.ts.map +1 -1
- package/dist/core/mapping.js +27 -0
- package/dist/core/mapping.js.map +1 -1
- package/dist/index.d.ts +5 -2
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +15 -1
- package/dist/index.js.map +1 -1
- package/dist/query/formatter.d.ts +2 -2
- package/dist/query/formatter.d.ts.map +1 -1
- package/dist/query/formatter.js +9 -6
- package/dist/query/formatter.js.map +1 -1
- package/dist/query/planner.d.ts +15 -1
- package/dist/query/planner.d.ts.map +1 -1
- package/dist/query/planner.js +152 -3
- package/dist/query/planner.js.map +1 -1
- package/dist/simulation/context.d.ts +5 -0
- package/dist/simulation/context.d.ts.map +1 -0
- package/dist/simulation/context.js +56 -0
- package/dist/simulation/context.js.map +1 -0
- package/dist/simulation/events.d.ts +3 -0
- package/dist/simulation/events.d.ts.map +1 -0
- package/dist/simulation/events.js +67 -0
- package/dist/simulation/events.js.map +1 -0
- package/dist/simulation/index.d.ts +7 -0
- package/dist/simulation/index.d.ts.map +1 -0
- package/dist/simulation/index.js +16 -0
- package/dist/simulation/index.js.map +1 -0
- package/dist/simulation/prng.d.ts +4 -0
- package/dist/simulation/prng.d.ts.map +1 -0
- package/dist/simulation/prng.js +18 -0
- package/dist/simulation/prng.js.map +1 -0
- package/dist/simulation/resource-estimate.d.ts +6 -0
- package/dist/simulation/resource-estimate.d.ts.map +1 -0
- package/dist/simulation/resource-estimate.js +11 -0
- package/dist/simulation/resource-estimate.js.map +1 -0
- package/dist/simulation/simulator.d.ts +15 -0
- package/dist/simulation/simulator.d.ts.map +1 -0
- package/dist/simulation/simulator.js +67 -0
- package/dist/simulation/simulator.js.map +1 -0
- package/dist/simulation/tracker.d.ts +8 -0
- package/dist/simulation/tracker.d.ts.map +1 -0
- package/dist/simulation/tracker.js +18 -0
- package/dist/simulation/tracker.js.map +1 -0
- package/dist/simulation/types.d.ts +49 -0
- package/dist/simulation/types.d.ts.map +1 -0
- package/dist/simulation/types.js +21 -0
- package/dist/simulation/types.js.map +1 -0
- package/dist/storage/coverage.d.ts +13 -0
- package/dist/storage/coverage.d.ts.map +1 -0
- package/dist/storage/coverage.js +37 -0
- package/dist/storage/coverage.js.map +1 -0
- package/dist/storage/memory.d.ts +23 -5
- package/dist/storage/memory.d.ts.map +1 -1
- package/dist/storage/memory.js +162 -3
- package/dist/storage/memory.js.map +1 -1
- package/dist/storage/mongo.d.ts +5 -0
- package/dist/storage/mongo.d.ts.map +1 -1
- package/dist/storage/mongo.js +68 -1
- package/dist/storage/mongo.js.map +1 -1
- package/dist/storage/redis.d.ts +40 -11
- package/dist/storage/redis.d.ts.map +1 -1
- package/dist/storage/redis.js +202 -44
- package/dist/storage/redis.js.map +1 -1
- package/dist/types/index.d.ts +165 -7
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +125 -3
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -0
- package/docs/guides/README.md +5 -0
- package/docs/guides/adapter-mongo.md +12 -0
- package/docs/guides/adapter-redis.md +46 -4
- package/docs/guides/backup-export.md +106 -10
- package/docs/guides/cli.md +8 -0
- package/docs/guides/counter-maps.md +14 -0
- package/docs/guides/insights-overview.md +6 -0
- package/docs/guides/measures.md +4 -0
- package/docs/guides/preview.md +92 -0
- package/docs/guides/querying.md +70 -0
- package/docs/guides/simulation-profiles.md +33 -0
- package/docs/guides/simulation-resource-estimation.md +30 -0
- package/docs/guides/simulation.md +36 -0
- package/docs/internals/README.md +1 -0
- package/docs/internals/internal-dimension-hash.md +34 -0
- package/docs/internals/internal-storage-mongo.md +24 -4
- package/docs/internals/internal-storage-redis.md +41 -9
- package/docs/use-cases/README.md +56 -0
- package/docs/use-cases/audit-security-log-counts.md +550 -0
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -0
- package/docs/use-cases/multi-tenant-isolation.md +203 -0
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -0
- package/docs/use-cases/processing-workflow-activity.md +165 -0
- package/docs/use-cases/product-feature-usage.md +240 -0
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -0
- package/package.json +18 -2
package/docs/guides/README.md
CHANGED
|
@@ -25,5 +25,10 @@ These files are published with the npm package and exposed as stable import subp
|
|
|
25
25
|
| [Insights forecasting & budget](./insights-forecasting-and-budget.md) | `@x12i/countex/docs/guides/insights-forecasting-and-budget` |
|
|
26
26
|
| [Insights advisors](./insights-advisors-and-optimization.md) | `@x12i/countex/docs/guides/insights-advisors-and-optimization` |
|
|
27
27
|
| [CLI](./cli.md) | `@x12i/countex/docs/guides/cli` |
|
|
28
|
+
| [Simulation overview](./simulation.md) | `@x12i/countex/docs/guides/simulation` |
|
|
29
|
+
| [Simulation profiles](./simulation-profiles.md) | `@x12i/countex/docs/guides/simulation-profiles` |
|
|
30
|
+
| [Simulation resource estimation](./simulation-resource-estimation.md) | `@x12i/countex/docs/guides/simulation-resource-estimation` |
|
|
31
|
+
|
|
32
|
+
Scenario walkthroughs: [`docs/use-cases/`](../use-cases/README.md) (also exposed as `@x12i/countex/docs/use-cases/*`).
|
|
28
33
|
|
|
29
34
|
Additional narrative docs (also under `docs/`): `DEPLOYMENT.md`, `END_TO_END_EXAMPLE.md`, `INSIGHTS*.md`.
|
|
@@ -40,6 +40,18 @@ On first write the adapter ensures collections such as **`countex_cells`** and *
|
|
|
40
40
|
|
|
41
41
|
Flushes call `applyDeltas` with atomic increment operations so parallel writers can safely aggregate.
|
|
42
42
|
|
|
43
|
+
## Dictionary allocation (`resolveIds`) and concurrency
|
|
44
|
+
|
|
45
|
+
Dictionary rows map **`(scopeId, kind, qualifier, sourceKey)` → `internalId`** (integer). The adapter:
|
|
46
|
+
|
|
47
|
+
1. **Reads** existing rows for the batch.
|
|
48
|
+
2. **Allocates** a contiguous block of ids from a per-scope sequence document (`countex_seq`).
|
|
49
|
+
3. **Inserts** new rows with `insertMany({ ordered: false })`.
|
|
50
|
+
|
|
51
|
+
A **unique index** (`uniq_dict_key` on `scopeId`, `kind`, `qualifier`, `sourceKey`) prevents duplicate keys. If two workers race on the same new key, one insert wins and the other hits a duplicate-key error; the adapter **re-reads** and continues. Always create indexes via `init()` / first write (unless `skipIndexInit`).
|
|
52
|
+
|
|
53
|
+
Sequence allocation uses MongoDB **`findOneAndUpdate` `$inc`**, which is atomic per sequence document. Very large parallel bursts should still be safe; id blocks may interleave across workers but remain unique.
|
|
54
|
+
|
|
43
55
|
## Step 5: Shutdown
|
|
44
56
|
|
|
45
57
|
Close Countex, then close the Mongo client when your process exits.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Guide: Redis cache adapter
|
|
2
2
|
|
|
3
|
-
`RedisCache` stores hot counter cells for fast reads. It pairs with a durable store (typically Mongo).
|
|
3
|
+
`RedisCache` stores hot counter cells for fast reads. It pairs with a durable store (typically Mongo) and tracks per-bucket coverage so the query planner only treats fully-warmed buckets as cache-authoritative.
|
|
4
4
|
|
|
5
5
|
## Step 1: Install peer dependency
|
|
6
6
|
|
|
@@ -20,6 +20,14 @@ const cache = new RedisCache({
|
|
|
20
20
|
client: redis,
|
|
21
21
|
keyPrefix: "countex",
|
|
22
22
|
ttlSeconds: 92 * 24 * 3600,
|
|
23
|
+
// Optional: shorter hot buckets, longer rollups
|
|
24
|
+
ttlSecondsByTimeframe: {
|
|
25
|
+
minute: 48 * 3600,
|
|
26
|
+
hour: 14 * 24 * 3600,
|
|
27
|
+
day: 92 * 24 * 3600,
|
|
28
|
+
},
|
|
29
|
+
// Optional opaque map version stamped onto bucket coverage on warm()
|
|
30
|
+
mapVersion: 1,
|
|
23
31
|
});
|
|
24
32
|
|
|
25
33
|
const countex = createCountex({
|
|
@@ -31,15 +39,49 @@ const countex = createCountex({
|
|
|
31
39
|
await countex.init();
|
|
32
40
|
```
|
|
33
41
|
|
|
34
|
-
## Step 3:
|
|
42
|
+
## Step 3: Unified retention via `CountexConfig.retention`
|
|
35
43
|
|
|
36
|
-
|
|
44
|
+
Prefer the engine-level `retention` config so the same policy is enforced regardless of which cache adapter is used:
|
|
37
45
|
|
|
38
|
-
|
|
46
|
+
```ts
|
|
47
|
+
const countex = createCountex({
|
|
48
|
+
maps: [myMap],
|
|
49
|
+
durable,
|
|
50
|
+
cache,
|
|
51
|
+
retention: {
|
|
52
|
+
redis: {
|
|
53
|
+
minute: "2d",
|
|
54
|
+
hour: "30d",
|
|
55
|
+
day: "92d",
|
|
56
|
+
week: "52w",
|
|
57
|
+
month: "24mo",
|
|
58
|
+
quarter: "12q",
|
|
59
|
+
year: "7y",
|
|
60
|
+
all: "forever",
|
|
61
|
+
},
|
|
62
|
+
// Mongo retention is operator-managed today (TTL index or scheduled prune).
|
|
63
|
+
},
|
|
64
|
+
});
|
|
65
|
+
```
|
|
39
66
|
|
|
67
|
+
The engine forwards the resolved per-timeframe TTLs to `cache.setRetentionPolicy(...)` at construction. Engine policy overrides the adapter's own `ttlSecondsByTimeframe`.
|
|
68
|
+
|
|
69
|
+
Accepted duration strings: `<n><unit>` where unit is `s|m|h|d|w|mo|q|y`, or `"forever"` for no expiry. Plain numbers are seconds.
|
|
70
|
+
|
|
71
|
+
## Step 4: Operations
|
|
72
|
+
|
|
73
|
+
- **Production Redis must use `maxmemory-policy noeviction`.** Silent eviction breaks the per-bucket "complete" coverage assumption — a half-evicted bucket can no longer be served from cache, and silent eviction means the `:cov` key may report `complete` while the cells hash is gone.
|
|
40
74
|
- Cache misses fall back to durable reads; results may warm the cache afterward.
|
|
41
75
|
- Cache flush failures during write path are **non-fatal** (durable is source of truth).
|
|
42
76
|
|
|
77
|
+
## Step 5: Idempotency hook
|
|
78
|
+
|
|
79
|
+
When maps declare **idempotency**, the cache adapter implements **`markEventSeen`** so duplicates are skipped before buffering.
|
|
80
|
+
|
|
81
|
+
## Step 6: Per-bucket coverage
|
|
82
|
+
|
|
83
|
+
`RedisCache` maintains a `:cov` HASH per bucket with `status` (`complete | partial | rebuilding`), `lastWarmAt`, `cellCount`, `schemaVersion`, and optional `mapVersion`. The query planner reads it to know which buckets the cache can answer with confidence; details in [internals/internal-storage-redis.md](../internals/internal-storage-redis.md).
|
|
84
|
+
|
|
43
85
|
## See also
|
|
44
86
|
|
|
45
87
|
- [Idempotency](./idempotency.md)
|
|
@@ -1,8 +1,10 @@
|
|
|
1
|
-
# Guide: Backup
|
|
1
|
+
# Guide: Backup, manifest, verify, restore, and cache rebuild
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
`@x12i/countex` ships export, fingerprint, manifest, verify, restore, and cache-rebuild helpers. Use them as free functions or via the `countex.backup` namespace bound to your engine.
|
|
4
4
|
|
|
5
|
-
## Step 1: Export cells
|
|
5
|
+
## Step 1: Export cells (stream)
|
|
6
|
+
|
|
7
|
+
Free function:
|
|
6
8
|
|
|
7
9
|
```ts
|
|
8
10
|
import { exportCells } from "@x12i/countex";
|
|
@@ -12,28 +14,122 @@ for await (const cell of exportCells(durable, {
|
|
|
12
14
|
counterId: "events",
|
|
13
15
|
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
14
16
|
})) {
|
|
15
|
-
// serialize
|
|
17
|
+
// serialize to NDJSON / S3 / cold storage…
|
|
18
|
+
}
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Namespace:
|
|
22
|
+
|
|
23
|
+
```ts
|
|
24
|
+
for await (const cell of countex.backup.export({
|
|
25
|
+
scope: "org_1",
|
|
26
|
+
counterId: "events",
|
|
27
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
28
|
+
})) {
|
|
29
|
+
// …
|
|
16
30
|
}
|
|
17
31
|
```
|
|
18
32
|
|
|
19
33
|
Streaming is bounded by the `(scope, counterId, timeframe)` filter you provide.
|
|
20
34
|
|
|
21
|
-
## Step 2:
|
|
35
|
+
## Step 2: Fingerprint + manifest
|
|
36
|
+
|
|
37
|
+
The fingerprint sorts canonical JSON lines in memory and computes:
|
|
38
|
+
|
|
39
|
+
- `cellCount` — number of cells in the stream.
|
|
40
|
+
- `contentSha256` — SHA-256 of the sorted canonical lines.
|
|
41
|
+
- `countSum` — sum of `count` across all cells (sanity-check field).
|
|
22
42
|
|
|
23
43
|
```ts
|
|
24
|
-
import {
|
|
44
|
+
import { computeCellsFingerprint, createBackupManifest, COUNTEX_CELL_SCHEMA_VERSION } from "@x12i/countex";
|
|
25
45
|
|
|
26
|
-
const
|
|
46
|
+
const filter = {
|
|
27
47
|
scope: "org_1",
|
|
28
48
|
counterId: "events",
|
|
29
49
|
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
30
|
-
}
|
|
50
|
+
};
|
|
51
|
+
|
|
52
|
+
const fp = await computeCellsFingerprint(exportCells(durable, filter));
|
|
53
|
+
const manifest = createBackupManifest(filter, fp, COUNTEX_CELL_SCHEMA_VERSION);
|
|
54
|
+
// persist manifest JSON next to the NDJSON dump…
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Resulting manifest:
|
|
58
|
+
|
|
59
|
+
```json
|
|
60
|
+
{
|
|
61
|
+
"manifestVersion": 1,
|
|
62
|
+
"schemaVersion": 1,
|
|
63
|
+
"createdAtIso": "2026-05-09T08:00:00.000Z",
|
|
64
|
+
"filter": { "scope": "org_1", "counterId": "events", "timeframe": { "type": "day", "from": "2026-05-01", "to": "2026-05-07" } },
|
|
65
|
+
"cellCount": 1234,
|
|
66
|
+
"contentSha256": "…",
|
|
67
|
+
"countSum": 98765
|
|
68
|
+
}
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
## Step 3: Verify
|
|
72
|
+
|
|
73
|
+
```ts
|
|
74
|
+
import { verifyBackupAgainstManifest } from "@x12i/countex";
|
|
75
|
+
|
|
76
|
+
const check = await verifyBackupAgainstManifest(manifest, exportCells(durable, manifest.filter));
|
|
77
|
+
if (!check.ok) throw new Error(check.detail);
|
|
31
78
|
```
|
|
32
79
|
|
|
33
|
-
|
|
80
|
+
Or via the namespace, which uses the engine's durable adapter:
|
|
81
|
+
|
|
82
|
+
```ts
|
|
83
|
+
const check = await countex.backup.verify(manifest);
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
`verify` checks `cellCount`, `contentSha256`, and (when present) `countSum`. Any mismatch returns `{ ok: false, detail }`.
|
|
87
|
+
|
|
88
|
+
## Step 4: Restore (modes)
|
|
89
|
+
|
|
90
|
+
```ts
|
|
91
|
+
import { restoreSnapshotCells } from "@x12i/countex";
|
|
92
|
+
|
|
93
|
+
const { restored, mode } = await restoreSnapshotCells(
|
|
94
|
+
targetDurable,
|
|
95
|
+
exportCells(sourceDurable, filter),
|
|
96
|
+
{ mode: "replace-window", batchSize: 500 }
|
|
97
|
+
);
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Or via the namespace:
|
|
101
|
+
|
|
102
|
+
```ts
|
|
103
|
+
await countex.backup.restore(countex.backup.export(filter), { mode: "replace-window" });
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Modes:
|
|
107
|
+
|
|
108
|
+
| Mode | Behavior |
|
|
109
|
+
|---|---|
|
|
110
|
+
| `validate-only` | Count cells without writing. Equivalent to legacy `dryRun: true`. |
|
|
111
|
+
| `replace-window` | Replace each cell identity in the stream via `importSnapshotCells`. Default. |
|
|
112
|
+
| `merge-deltas` | **Reserved.** Throws `NotImplementedError` today. |
|
|
113
|
+
|
|
114
|
+
After a `replace-window` restore, **rebuild Redis** for the affected windows so coverage flips back to `complete`.
|
|
115
|
+
|
|
116
|
+
## Step 5: Cache rebuild
|
|
117
|
+
|
|
118
|
+
```ts
|
|
119
|
+
const { cellsWarmed } = await countex.backup.rebuildCache({
|
|
120
|
+
scope: "org_1",
|
|
121
|
+
counterId: "events",
|
|
122
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
123
|
+
});
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
`rebuildCache` streams `streamCells` from durable in batches and calls `cache.warm(batch)`. Each warmed bucket's `:cov` hash flips to `status=complete`.
|
|
127
|
+
|
|
128
|
+
## Cancellation
|
|
34
129
|
|
|
35
|
-
Pass `AbortSignal` where supported (`streamCells` / `warm` options propagate through adapters).
|
|
130
|
+
Pass `AbortSignal` where supported (`streamCells` / `warm` / `restoreSnapshotCells` options propagate through adapters).
|
|
36
131
|
|
|
37
132
|
## See also
|
|
38
133
|
|
|
39
134
|
- Internal: [Backup streaming](../internals/internal-backup-stream.md)
|
|
135
|
+
- Internal: [Mongo durable](../internals/internal-storage-mongo.md) — `importSnapshotCells` semantics.
|
package/docs/guides/cli.md
CHANGED
|
@@ -33,6 +33,14 @@ countex insights:resources ./ctx.mjs
|
|
|
33
33
|
|
|
34
34
|
Use `--format json|markdown` for output shaping.
|
|
35
35
|
|
|
36
|
+
## Simulation demo (no config file)
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
countex simulate:demo --events 200 --seed 1
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Emits JSON **`SimulationReport`** from a dry-run against the bundled demo map (`audit-log` profile). No Mongo/Redis required.
|
|
43
|
+
|
|
36
44
|
## Step 4: TypeScript configs
|
|
37
45
|
|
|
38
46
|
```bash
|
|
@@ -2,6 +2,12 @@
|
|
|
2
2
|
|
|
3
3
|
A **counter map** (`CounterMap`) tells Countex how to read one JSON event and turn it into sparse counter updates. This guide walks through defining a map from scratch.
|
|
4
4
|
|
|
5
|
+
## Metrics vs domain “labels”
|
|
6
|
+
|
|
7
|
+
Countex uses **`metrics`** in the map config: paths that resolve to the category or label you want one counter split per value (for example `$.labels[*]` or `$.event.operation`). That field name is generic — your payload might call them labels, statuses, tags, or event types. They all map to Countex **metrics**.
|
|
8
|
+
|
|
9
|
+
Documentation refers to **`sourceKey`** for the string form of a subject or metric before dictionary compaction; that is not necessarily a UUID unless you choose UUID strings as keys.
|
|
10
|
+
|
|
5
11
|
## What you will configure
|
|
6
12
|
|
|
7
13
|
- **scope** — tenant or isolation boundary (string id).
|
|
@@ -108,9 +114,17 @@ Total raw increments per event ≈
|
|
|
108
114
|
|
|
109
115
|
Plan capacity accordingly.
|
|
110
116
|
|
|
117
|
+
## Preview before ingesting
|
|
118
|
+
|
|
119
|
+
Run `countex.preview(event, { counterId })` on a sample event to see exactly what cells your map would generate — without any dictionary writes or buffering. See [Preview a map](./preview.md) for the full result shape (`subjectSourceKey`, `metricSourceKey`, `dimensionsHash`, `timeframes`, `delta`, `measures`, plus `extractionWarnings` and `droppedAfterCardinality`).
|
|
120
|
+
|
|
121
|
+
Dimension keys/values feeding into `dimensionsHash` are canonicalized: keys sorted lexicographically, values type-tagged, hashed (SHA-1 truncated to 12 hex). Any two workers always produce the same hash for the same logical dimension tuple. See [internal-dimension-hash.md](../internals/internal-dimension-hash.md) for the exact recipe.
|
|
122
|
+
|
|
111
123
|
## See also
|
|
112
124
|
|
|
113
125
|
- [Timeframes](./timeframes-and-bucketing.md)
|
|
114
126
|
- [Measures](./measures.md)
|
|
115
127
|
- [Cardinality](./cardinality.md)
|
|
128
|
+
- [Preview a map](./preview.md)
|
|
116
129
|
- Internal: [Mapping validation & extraction](../internals/internal-mapping-validation-and-extraction.md)
|
|
130
|
+
- Internal: [Dimension hash](../internals/internal-dimension-hash.md)
|
|
@@ -59,6 +59,12 @@ insights.on("insight_query", (e) => console.log(e));
|
|
|
59
59
|
|
|
60
60
|
Insights honor **`maxBuckets`** (aligned with core query expansion) and **`maxGroups`**. Prefer narrower filters when grouping heavily.
|
|
61
61
|
|
|
62
|
+
## Distributions, percentiles, and histograms
|
|
63
|
+
|
|
64
|
+
Core Countex cells store **additive** measure aggregates (`sum`, `count`, `min`, `max`) per sparse cell — not full distributions. Insights therefore reason about **rankings, deltas, thresholds, and derived averages**, not arbitrary percentile curves across arbitrary slices unless you add **explicit** histogram or sketch storage elsewhere.
|
|
65
|
+
|
|
66
|
+
Token and latency **advisors** use those summaries (and sometimes robust z-scores on bucketed series); treat percentile-style language in recommendations as **heuristic** unless you extend storage.
|
|
67
|
+
|
|
62
68
|
## Feature map
|
|
63
69
|
|
|
64
70
|
| Area | Methods |
|
package/docs/guides/measures.md
CHANGED
|
@@ -34,6 +34,10 @@ const map = {
|
|
|
34
34
|
|
|
35
35
|
Use `onMissing: "skip"` (default) or `"drop"` / `"zero"` per measure.
|
|
36
36
|
|
|
37
|
+
### Unique / distinct counts are not additive sums
|
|
38
|
+
|
|
39
|
+
Countex **counts** and **measure aggregations** (`sum`, `count`, `min`, `max`) behave like standard additive counters over cells. They do **not** answer “how many **distinct** users appeared in this bucket?” unless you model that explicitly (for example one subject id per event and interpret cell counts accordingly). Do **not** sum counts across overlapping populations and treat the result as a unique count. For large-cardinality uniques, plan on a dedicated approach (for example optional future HLL-style storage — see deployment notes); vanilla cells are the wrong tool for deduplicated set cardinality across arbitrary queries.
|
|
40
|
+
|
|
37
41
|
## Step 2: Ingest as usual
|
|
38
42
|
|
|
39
43
|
Each fan-out increment carries the **same** measure snapshot for that event.
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
# Guide: Preview a map (dry-run)
|
|
2
|
+
|
|
3
|
+
`countex.preview(event, options?)` runs **extraction + cardinality** for one event against your maps and returns the increments it _would_ produce — without dictionary I/O, without buffering, without any storage writes. Use it to debug a counter map before ingesting millions of events.
|
|
4
|
+
|
|
5
|
+
## Step 1: Construct a Countex instance
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
import { createCountex } from "@x12i/countex";
|
|
9
|
+
import { MemoryDurable } from "@x12i/countex/adapters/memory";
|
|
10
|
+
|
|
11
|
+
const countex = createCountex({
|
|
12
|
+
maps: [accessMap],
|
|
13
|
+
durable: new MemoryDurable(),
|
|
14
|
+
flush: { everyMs: 0 },
|
|
15
|
+
});
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
A real durable adapter is fine too — preview never calls it.
|
|
19
|
+
|
|
20
|
+
## Step 2: Run preview
|
|
21
|
+
|
|
22
|
+
```ts
|
|
23
|
+
const previews = countex.preview(event, {
|
|
24
|
+
counterId: "access_by_subject_metric", // optional; defaults to all maps
|
|
25
|
+
});
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
`previews` is an array of `MapPreviewResult`, one per map.
|
|
29
|
+
|
|
30
|
+
## Result shape
|
|
31
|
+
|
|
32
|
+
```ts
|
|
33
|
+
interface MapPreviewResult {
|
|
34
|
+
counterId: string;
|
|
35
|
+
skipped: boolean;
|
|
36
|
+
extractionWarnings: ExtractionWarning[];
|
|
37
|
+
cells: PreviewIncrementCell[];
|
|
38
|
+
/** Increments dropped by cardinality guard. Preview runs silently — no `dropped` events. */
|
|
39
|
+
droppedAfterCardinality: number;
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
interface PreviewIncrementCell {
|
|
43
|
+
scopeId: string;
|
|
44
|
+
counterId: string;
|
|
45
|
+
timeframeType: TimeframeType;
|
|
46
|
+
timeframeKey: string;
|
|
47
|
+
subjectType: string;
|
|
48
|
+
subjectSourceKey: string;
|
|
49
|
+
metricSourceKey: string;
|
|
50
|
+
dimensions: Dimensions;
|
|
51
|
+
dimensionsHash: string;
|
|
52
|
+
delta: number;
|
|
53
|
+
measures: Record<string, { value: number; aggs: MeasureAgg[] }>;
|
|
54
|
+
}
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
`subjectSourceKey` and `metricSourceKey` are the raw strings extracted from the event — preview does not allocate dictionary ids. `dimensionsHash` matches the canonical form computed during real ingestion (see [internal-dimension-hash.md](../internals/internal-dimension-hash.md)).
|
|
58
|
+
|
|
59
|
+
## Counting fan-out
|
|
60
|
+
|
|
61
|
+
```ts
|
|
62
|
+
const total = previews.reduce((s, p) => s + p.cells.length, 0);
|
|
63
|
+
console.log(`This event would generate ${total} cell increments.`);
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
For the canonical example (1 actor × 2 metrics × 5 timeframes), `total === 10`.
|
|
67
|
+
|
|
68
|
+
## Diagnostics: warnings and skips
|
|
69
|
+
|
|
70
|
+
`extractionWarnings` includes things like `"measure_invalid"`, `"measure_missing"`, missing required dimensions, and bad path resolutions. Use them to spot a misconfigured map without having to ingest and watch event listeners.
|
|
71
|
+
|
|
72
|
+
```ts
|
|
73
|
+
for (const p of previews) {
|
|
74
|
+
if (p.skipped) console.warn(`map "${p.counterId}" skipped this event`);
|
|
75
|
+
for (const w of p.extractionWarnings) {
|
|
76
|
+
console.warn(`extraction warning: ${w.reason}`, w.meta);
|
|
77
|
+
}
|
|
78
|
+
}
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
## Caveats
|
|
82
|
+
|
|
83
|
+
- **Cardinality guard state is shared with the live engine.** Preview runs the same `CardinalityGuard.check`, so calling preview repeatedly counts toward the per-scope cardinality limits. For isolated dry-runs, construct a separate `Countex` instance just for previewing.
|
|
84
|
+
- **No idempotency check.** Preview ignores `idempotency` — same event can be previewed any number of times.
|
|
85
|
+
- **No measure scaling on output.** `measures[name].value` is the raw extracted number; on the live ingest path it would be multiplied by `measure.scale` before persistence.
|
|
86
|
+
- **No dropped/overflow events emitted.** Use `droppedAfterCardinality` and `extractionWarnings` instead.
|
|
87
|
+
|
|
88
|
+
## See also
|
|
89
|
+
|
|
90
|
+
- [Counter maps guide](./counter-maps.md)
|
|
91
|
+
- [Cardinality guide](./cardinality.md)
|
|
92
|
+
- Internal: [Mapping validation & extraction](../internals/internal-mapping-validation-and-extraction.md)
|
package/docs/guides/querying.md
CHANGED
|
@@ -72,8 +72,78 @@ Do not use `rate` with timeframe **`all`**.
|
|
|
72
72
|
|
|
73
73
|
Result includes `source`: `"cache"`, `"durable"`, or `"merged"`.
|
|
74
74
|
|
|
75
|
+
### Query diagnostics
|
|
76
|
+
|
|
77
|
+
`result.diagnostics` (when present) adds planner detail:
|
|
78
|
+
|
|
79
|
+
| Field | Meaning |
|
|
80
|
+
|-------|---------|
|
|
81
|
+
| `cacheAttempted` | `true` if the engine consulted the cache adapter (cache configured, not `bypassCache`, and the planner allowed cache for this query shape). |
|
|
82
|
+
| `cacheCoverage` | When `cacheAttempted` is true: `"full"` if the cache answered completely; `"partial"` or `"miss"` if the planner fell through to durable for authoritative cells. |
|
|
83
|
+
| `cellsMaterialized` | Number of counter cells loaded from storage before grouping/formatting. |
|
|
84
|
+
| `bucketsRequested` | Bucket keys the query expanded to (after `bucketRange`). |
|
|
85
|
+
| `cacheBucketsHit` | Number of buckets the cache reported as `complete`. |
|
|
86
|
+
| `durableFallback` | `true` when the planner had to read durable for some/all buckets. |
|
|
87
|
+
| `plannedCellsEstimate` | Heuristic estimate of cells before grouping (matches `cellsMaterialized` after execution). |
|
|
88
|
+
| `groupByCost` | `"trivial"` (no groupBy), `"low"`, or `"high"` (3+ keys or 2+ dimensions). |
|
|
89
|
+
| `expensive` | Heuristic flag — `true` for big windows, high groupByCost, or `format: "matrix"` with many cells. |
|
|
90
|
+
|
|
91
|
+
Use this for tests, dashboards, and verifying Redis vs Mongo paths.
|
|
92
|
+
|
|
93
|
+
### Explain a query without running it
|
|
94
|
+
|
|
95
|
+
`countex.explainQuery(spec)` returns a {@link QueryPlan} without touching durable storage and only probing per-bucket cache coverage:
|
|
96
|
+
|
|
97
|
+
```ts
|
|
98
|
+
const plan = await countex.explainQuery({
|
|
99
|
+
scope: "org_1",
|
|
100
|
+
counterId: "events",
|
|
101
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-31" },
|
|
102
|
+
groupBy: ["dim:operation", "dim:situation", "subject"],
|
|
103
|
+
});
|
|
104
|
+
// {
|
|
105
|
+
// bucketsRequested: ["2026-05-01", …, "2026-05-31"],
|
|
106
|
+
// cacheEligible: true,
|
|
107
|
+
// cacheBucketsExpectedHit: 18,
|
|
108
|
+
// durableFallbackPredicted: true,
|
|
109
|
+
// plannedCellsEstimate: 1_240,
|
|
110
|
+
// groupByCost: "high",
|
|
111
|
+
// expensive: true,
|
|
112
|
+
// }
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
`cacheBucketsExpectedHit` and `plannedCellsEstimate` are populated when the cache adapter implements `probeBucketCoverage` (the bundled `RedisCache` and `MemoryCache` both do).
|
|
116
|
+
|
|
117
|
+
## Step 6: Totals, event volume, and “distinct entities”
|
|
118
|
+
|
|
119
|
+
### Grand total (sum all matching cells)
|
|
120
|
+
|
|
121
|
+
Omit **`groupBy`** (or use an empty list). For **`format: "table"`**, the engine merges every matching cell into **one row**; **`rows[0].count`** is the sum of **`count`** across those cells. **`result.total`** matches that sum for the returned rows (`sumRowCounts` in `src/query/formatter.ts`).
|
|
122
|
+
|
|
123
|
+
Use this for **total increment volume** in the query window. If one source event fans out into many cells (multiple subjects × metrics × timeframes), this total is **not** the same as “number of source events.”
|
|
124
|
+
|
|
125
|
+
### Distinct subjects / metrics as row cardinality
|
|
126
|
+
|
|
127
|
+
Countex does **not** store arbitrary **`COUNT(DISTINCT …)`** queries. **Exactly what to do** when you need “how many unique actors had activity?”:
|
|
128
|
+
|
|
129
|
+
1. **`subject: { type: "actor" }`** (optional **`ids`**) to restrict the subject family.
|
|
130
|
+
2. **`dimensions: { situation: ["external"] }`** (example) to restrict literals stored on cells.
|
|
131
|
+
3. **`groupBy: ["subjectType", "subject"]`** so **each row aggregates one dictionary subject id**.
|
|
132
|
+
4. **`rows.length`** after the query = number of distinct actors **with non-zero merged activity** in that slice (sparse — actors who never appeared produce no row).
|
|
133
|
+
|
|
134
|
+
**Critical:** if you set **`limit`**, **`rows.length ≤ limit`** and **`result.total`** only sum **returned** rows — you **cannot** infer global distinct counts from a capped query.
|
|
135
|
+
|
|
136
|
+
### Logical “one row per source event” totals
|
|
137
|
+
|
|
138
|
+
Maintain a **separate counter map** engineered for **one increment per deduped message** (single timeframe granularity, single synthetic metric field, minimal subjects). See [`docs/use-cases/audit-security-log-counts.md`](../use-cases/audit-security-log-counts.md) §10.6.
|
|
139
|
+
|
|
140
|
+
### Metric filters at rest
|
|
141
|
+
|
|
142
|
+
`MongoDurable` / `MemoryDurable` apply **`metrics`** filters using **numeric metric ids** parsed from `QuerySpec.metrics`. Pure string labels that are not finite numbers do **not** constrain the database filter — resolve ids from `countex_dict` or filter after read.
|
|
143
|
+
|
|
75
144
|
## See also
|
|
76
145
|
|
|
77
146
|
- [Measures](./measures.md)
|
|
78
147
|
- [Timeframes](./timeframes-and-bucketing.md)
|
|
148
|
+
- [Audit use case — analytics recipes](../use-cases/audit-security-log-counts.md) §10
|
|
79
149
|
- Internal: [Query path](../internals/internal-query-path.md)
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Guide: Simulation profiles
|
|
2
|
+
|
|
3
|
+
Profiles configure **what kind of events** the simulator emits and **how much fan-out** they produce.
|
|
4
|
+
|
|
5
|
+
## Built-in profile ids
|
|
6
|
+
|
|
7
|
+
| Profile id | Intent |
|
|
8
|
+
|------------|--------|
|
|
9
|
+
| `audit-log` | High-volume audit-style records: actors, resources, operations, optional label fan-out. |
|
|
10
|
+
| `product-activity` | Product analytics–style events: account scope, user subject, named event metric, feature/plan dimensions. |
|
|
11
|
+
| `processing-activity` | Jobs / pipelines / workflows: job id subject, task/step metrics, status dimensions, duration measures. |
|
|
12
|
+
| `llm-usage` | Token and cost measures with model/provider dimensions. |
|
|
13
|
+
| `custom` | Supply your own event factory; simulator handles scheduling and pools only. |
|
|
14
|
+
|
|
15
|
+
## Profile options (conceptual)
|
|
16
|
+
|
|
17
|
+
- **`durationMs`** — simulated wall-clock span of the run (logical time advancement between events).
|
|
18
|
+
- **`eventsPerTick`** — throughput knob for batch generators.
|
|
19
|
+
- **`poolSizes`** — caps on distinct subjects/metrics/dimension values (cardinality stress).
|
|
20
|
+
- **`weights`** — skew toward hot keys (Pareto-style) vs uniform draws.
|
|
21
|
+
|
|
22
|
+
Exact fields match `SimulationProfile` in `@x12i/countex/simulation` TypeScript exports.
|
|
23
|
+
|
|
24
|
+
## Choosing a profile
|
|
25
|
+
|
|
26
|
+
- Mirror the **same map** you plan for production; mismatched profiles hide JSONPath bugs.
|
|
27
|
+
- Start **uniform** pools, then add **skew** to reproduce hotspot behavior.
|
|
28
|
+
- Combine with **`validateMap(map, samples)`** from core so paths resolve on representative payloads.
|
|
29
|
+
|
|
30
|
+
## See also
|
|
31
|
+
|
|
32
|
+
- [Simulation overview](./simulation.md)
|
|
33
|
+
- [Simulation resource estimation](./simulation-resource-estimation.md)
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Guide: Simulation resource estimation
|
|
2
|
+
|
|
3
|
+
Use simulation outputs to reason about **Mongo**, **Redis**, and **process memory** before cutover.
|
|
4
|
+
|
|
5
|
+
## Signals from a run
|
|
6
|
+
|
|
7
|
+
After a simulator run (or CLI report):
|
|
8
|
+
|
|
9
|
+
- **Raw events** and **raw increments** (before cardinality drops) — multiply by your peak events/sec for ingress sizing.
|
|
10
|
+
- **Distinct dictionary keys** per kind (`subject`, `metric`, `dimensionValue`) — drives Mongo dictionary collection growth.
|
|
11
|
+
- **Distinct cells touched** — upper bound on sparse matrix cardinality for the window.
|
|
12
|
+
- **Flush batch sizes** — depends on buffer settings (`maxBufferedCells`, flush interval).
|
|
13
|
+
|
|
14
|
+
## Mapping to infrastructure
|
|
15
|
+
|
|
16
|
+
| Concern | What to measure |
|
|
17
|
+
|---------|-----------------|
|
|
18
|
+
| Mongo durable | Cell documents + dictionary docs; index sizes (see deployment guide). |
|
|
19
|
+
| Redis | Working set for hot timeframes; TTL/retention policy per timeframe type. |
|
|
20
|
+
| Application RAM | In-flight buffer + Node overhead; simulate worst-case burst with `onOverflow: "block"`. |
|
|
21
|
+
|
|
22
|
+
## Limits
|
|
23
|
+
|
|
24
|
+
Simulation **does not** replace load testing of your MongoDB cluster or Redis failover behavior; it estimates **logical** counter volume from maps and profiles.
|
|
25
|
+
|
|
26
|
+
## See also
|
|
27
|
+
|
|
28
|
+
- [Simulation overview](./simulation.md)
|
|
29
|
+
- [Ingestion & flush](./ingestion-and-flush.md)
|
|
30
|
+
- [`docs/DEPLOYMENT.md`](../DEPLOYMENT.md)
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Guide: Simulation overview
|
|
2
|
+
|
|
3
|
+
**Simulation** helps before production traffic exists: generate deterministic synthetic events, stress-test maps and cardinality, estimate storage and flush load, and validate that aggregates match expectations.
|
|
4
|
+
|
|
5
|
+
## Runtime entry point
|
|
6
|
+
|
|
7
|
+
When using the published package:
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
import { createCountexSimulator } from "@x12i/countex/simulation";
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The simulator is optional tooling on top of the core engine; it does not change ingest or query semantics.
|
|
14
|
+
|
|
15
|
+
## What simulation covers
|
|
16
|
+
|
|
17
|
+
- **Seeded randomness** — reproducible runs for CI and regression tests.
|
|
18
|
+
- **Pools & relations** — weighted entities (users, documents, models) with stable ids across events.
|
|
19
|
+
- **Profiles** — named scenarios (audit-style logs, product activity, LLM metering, processing pipelines); see [Simulation profiles](./simulation-profiles.md).
|
|
20
|
+
- **Expected aggregates** — optional tracker that compares simulated totals to counter reads for a window.
|
|
21
|
+
- **Reports** — structured summary (events emitted, fan-out estimates, errors) for sizing discussions.
|
|
22
|
+
|
|
23
|
+
## Relationship to core Countex
|
|
24
|
+
|
|
25
|
+
1. Build or load a `CounterMap` (same as production).
|
|
26
|
+
2. Instantiate `createCountexSimulator({ map, profile, seed, ... })`.
|
|
27
|
+
3. Drive **`generateEvents(n)`** or **`run()`** into `countex.ingest` / `ingestBatch` in tests or benchmarks.
|
|
28
|
+
4. Optionally attach **`ExpectedAggregateTracker`** and assert after flush + query.
|
|
29
|
+
|
|
30
|
+
See [Simulation resource estimation](./simulation-resource-estimation.md) for capacity hooks.
|
|
31
|
+
|
|
32
|
+
## See also
|
|
33
|
+
|
|
34
|
+
- [CLI](./cli.md) — `countex simulate:demo` for scripted dry-runs without a config module
|
|
35
|
+
- [Counter maps](./counter-maps.md)
|
|
36
|
+
- [Cardinality](./cardinality.md)
|
package/docs/internals/README.md
CHANGED
|
@@ -12,6 +12,7 @@ Start here:
|
|
|
12
12
|
| In-memory merge | [Buffer merge & flush](./internal-buffer-merge-and-flush.md) |
|
|
13
13
|
| Reads | [Query path](./internal-query-path.md) |
|
|
14
14
|
| Bucket keys | [Timeframe bucketing](./internal-timeframe-bucketing.md) |
|
|
15
|
+
| Dimension hashing | [Dimension canonicalization & hash](./internal-dimension-hash.md) |
|
|
15
16
|
| Measures | [Measures & scales](./internal-measures-and-scales.md) |
|
|
16
17
|
| Cardinality | [Cardinality guard](./internal-cardinality.md) |
|
|
17
18
|
| Dedupe | [Idempotency](./internal-idempotency.md) |
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Internal: Dimension canonicalization and hashing
|
|
2
|
+
|
|
3
|
+
Each counter cell identity includes a **`dimensionsHash`** derived from the dimension key/value object for that increment.
|
|
4
|
+
|
|
5
|
+
## Normalization
|
|
6
|
+
|
|
7
|
+
Before hashing, dimension values pass through `normalizeDimensionValue` in `src/core/normalization.ts`:
|
|
8
|
+
|
|
9
|
+
- `null` / missing optional dimensions → `null`.
|
|
10
|
+
- Booleans and finite numbers preserved as typed values.
|
|
11
|
+
- Strings: optionally trimmed only (`normalizeValues` on the map); types are **not** coerced (`"1"` and `1` stay distinct).
|
|
12
|
+
|
|
13
|
+
Dictionary keys for subjects/metrics use **`normalizeKey`** (trim + lowercase strings) — that is separate from dimension hashing.
|
|
14
|
+
|
|
15
|
+
## Canonical string form
|
|
16
|
+
|
|
17
|
+
`canonicalizeDimensions` in `src/utils/hash.ts`:
|
|
18
|
+
|
|
19
|
+
1. Sort dimension **names** lexicographically.
|
|
20
|
+
2. Serialize as `key=value` pairs joined with `&`.
|
|
21
|
+
3. Escape `\`, `=`, and `&` in keys and string values with backslash.
|
|
22
|
+
4. Encode values with type prefixes: `n:` (null), `b:0|1`, `i:<number>`, `s:<escaped>`.
|
|
23
|
+
|
|
24
|
+
## Hash
|
|
25
|
+
|
|
26
|
+
`dimensionsHash` is **SHA-1** of the canonical string, truncated to **12 hex characters** (48 bits). Empty dimensions → literal `"0"`.
|
|
27
|
+
|
|
28
|
+
Collisions are statistically negligible at intended cardinalities; cells are further scoped by `(scopeId, counterId, timeframe, subject, metric)`.
|
|
29
|
+
|
|
30
|
+
## See also
|
|
31
|
+
|
|
32
|
+
- `src/utils/hash.ts`
|
|
33
|
+
- `src/core/normalization.ts`
|
|
34
|
+
- [Cardinality](../guides/cardinality.md)
|