@x12i/countex 1.2.0 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +23 -1
- package/dist/backup/manifest.d.ts +49 -0
- package/dist/backup/manifest.d.ts.map +1 -0
- package/dist/backup/manifest.js +103 -0
- package/dist/backup/manifest.js.map +1 -0
- package/dist/backup/snapshot-encoding.d.ts +8 -0
- package/dist/backup/snapshot-encoding.d.ts.map +1 -0
- package/dist/backup/snapshot-encoding.js +27 -0
- package/dist/backup/snapshot-encoding.js.map +1 -0
- package/dist/cli.js +16 -0
- package/dist/cli.js.map +1 -1
- package/dist/core/countex.d.ts +14 -1
- package/dist/core/countex.d.ts.map +1 -1
- package/dist/core/countex.js +141 -45
- package/dist/core/countex.js.map +1 -1
- package/dist/core/extraction.d.ts +1 -5
- package/dist/core/extraction.d.ts.map +1 -1
- package/dist/core/extraction.js.map +1 -1
- package/dist/core/mapping.d.ts +5 -0
- package/dist/core/mapping.d.ts.map +1 -1
- package/dist/core/mapping.js +27 -0
- package/dist/core/mapping.js.map +1 -1
- package/dist/index.d.ts +4 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +13 -1
- package/dist/index.js.map +1 -1
- package/dist/query/formatter.d.ts +2 -2
- package/dist/query/formatter.d.ts.map +1 -1
- package/dist/query/formatter.js +9 -6
- package/dist/query/formatter.js.map +1 -1
- package/dist/query/planner.d.ts.map +1 -1
- package/dist/query/planner.js +16 -2
- package/dist/query/planner.js.map +1 -1
- package/dist/simulation/context.d.ts +5 -0
- package/dist/simulation/context.d.ts.map +1 -0
- package/dist/simulation/context.js +56 -0
- package/dist/simulation/context.js.map +1 -0
- package/dist/simulation/events.d.ts +3 -0
- package/dist/simulation/events.d.ts.map +1 -0
- package/dist/simulation/events.js +67 -0
- package/dist/simulation/events.js.map +1 -0
- package/dist/simulation/index.d.ts +7 -0
- package/dist/simulation/index.d.ts.map +1 -0
- package/dist/simulation/index.js +16 -0
- package/dist/simulation/index.js.map +1 -0
- package/dist/simulation/prng.d.ts +4 -0
- package/dist/simulation/prng.d.ts.map +1 -0
- package/dist/simulation/prng.js +18 -0
- package/dist/simulation/prng.js.map +1 -0
- package/dist/simulation/resource-estimate.d.ts +6 -0
- package/dist/simulation/resource-estimate.d.ts.map +1 -0
- package/dist/simulation/resource-estimate.js +11 -0
- package/dist/simulation/resource-estimate.js.map +1 -0
- package/dist/simulation/simulator.d.ts +15 -0
- package/dist/simulation/simulator.d.ts.map +1 -0
- package/dist/simulation/simulator.js +67 -0
- package/dist/simulation/simulator.js.map +1 -0
- package/dist/simulation/tracker.d.ts +8 -0
- package/dist/simulation/tracker.d.ts.map +1 -0
- package/dist/simulation/tracker.js +18 -0
- package/dist/simulation/tracker.js.map +1 -0
- package/dist/simulation/types.d.ts +49 -0
- package/dist/simulation/types.d.ts.map +1 -0
- package/dist/simulation/types.js +21 -0
- package/dist/simulation/types.js.map +1 -0
- package/dist/storage/memory.d.ts +3 -0
- package/dist/storage/memory.d.ts.map +1 -1
- package/dist/storage/memory.js +35 -0
- package/dist/storage/memory.js.map +1 -1
- package/dist/storage/mongo.d.ts +5 -0
- package/dist/storage/mongo.d.ts.map +1 -1
- package/dist/storage/mongo.js +68 -1
- package/dist/storage/mongo.js.map +1 -1
- package/dist/storage/redis.d.ts +6 -2
- package/dist/storage/redis.d.ts.map +1 -1
- package/dist/storage/redis.js +18 -8
- package/dist/storage/redis.js.map +1 -1
- package/dist/types/index.d.ts +51 -0
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +40 -2
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -0
- package/docs/guides/README.md +5 -0
- package/docs/guides/adapter-mongo.md +12 -0
- package/docs/guides/adapter-redis.md +6 -0
- package/docs/guides/backup-export.md +46 -2
- package/docs/guides/cli.md +8 -0
- package/docs/guides/counter-maps.md +6 -0
- package/docs/guides/insights-overview.md +6 -0
- package/docs/guides/measures.md +4 -0
- package/docs/guides/querying.md +40 -0
- package/docs/guides/simulation-profiles.md +33 -0
- package/docs/guides/simulation-resource-estimation.md +30 -0
- package/docs/guides/simulation.md +36 -0
- package/docs/internals/README.md +1 -0
- package/docs/internals/internal-dimension-hash.md +34 -0
- package/docs/use-cases/README.md +56 -0
- package/docs/use-cases/audit-security-log-counts.md +550 -0
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -0
- package/docs/use-cases/multi-tenant-isolation.md +203 -0
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -0
- package/docs/use-cases/processing-workflow-activity.md +165 -0
- package/docs/use-cases/product-feature-usage.md +240 -0
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -0
- package/package.json +18 -2
package/docs/guides/README.md
CHANGED
|
@@ -25,5 +25,10 @@ These files are published with the npm package and exposed as stable import subp
|
|
|
25
25
|
| [Insights forecasting & budget](./insights-forecasting-and-budget.md) | `@x12i/countex/docs/guides/insights-forecasting-and-budget` |
|
|
26
26
|
| [Insights advisors](./insights-advisors-and-optimization.md) | `@x12i/countex/docs/guides/insights-advisors-and-optimization` |
|
|
27
27
|
| [CLI](./cli.md) | `@x12i/countex/docs/guides/cli` |
|
|
28
|
+
| [Simulation overview](./simulation.md) | `@x12i/countex/docs/guides/simulation` |
|
|
29
|
+
| [Simulation profiles](./simulation-profiles.md) | `@x12i/countex/docs/guides/simulation-profiles` |
|
|
30
|
+
| [Simulation resource estimation](./simulation-resource-estimation.md) | `@x12i/countex/docs/guides/simulation-resource-estimation` |
|
|
31
|
+
|
|
32
|
+
Scenario walkthroughs: [`docs/use-cases/`](../use-cases/README.md) (also exposed as `@x12i/countex/docs/use-cases/*`).
|
|
28
33
|
|
|
29
34
|
Additional narrative docs (also under `docs/`): `DEPLOYMENT.md`, `END_TO_END_EXAMPLE.md`, `INSIGHTS*.md`.
|
|
@@ -40,6 +40,18 @@ On first write the adapter ensures collections such as **`countex_cells`** and *
|
|
|
40
40
|
|
|
41
41
|
Flushes call `applyDeltas` with atomic increment operations so parallel writers can safely aggregate.
|
|
42
42
|
|
|
43
|
+
## Dictionary allocation (`resolveIds`) and concurrency
|
|
44
|
+
|
|
45
|
+
Dictionary rows map **`(scopeId, kind, qualifier, sourceKey)` → `internalId`** (integer). The adapter:
|
|
46
|
+
|
|
47
|
+
1. **Reads** existing rows for the batch.
|
|
48
|
+
2. **Allocates** a contiguous block of ids from a per-scope sequence document (`countex_seq`).
|
|
49
|
+
3. **Inserts** new rows with `insertMany({ ordered: false })`.
|
|
50
|
+
|
|
51
|
+
A **unique index** (`uniq_dict_key` on `scopeId`, `kind`, `qualifier`, `sourceKey`) prevents duplicate keys. If two workers race on the same new key, one insert wins and the other hits a duplicate-key error; the adapter **re-reads** and continues. Always create indexes via `init()` / first write (unless `skipIndexInit`).
|
|
52
|
+
|
|
53
|
+
Sequence allocation uses MongoDB **`findOneAndUpdate` `$inc`**, which is atomic per sequence document. Very large parallel bursts should still be safe; id blocks may interleave across workers but remain unique.
|
|
54
|
+
|
|
43
55
|
## Step 5: Shutdown
|
|
44
56
|
|
|
45
57
|
Close Countex, then close the Mongo client when your process exits.
|
|
@@ -20,6 +20,12 @@ const cache = new RedisCache({
|
|
|
20
20
|
client: redis,
|
|
21
21
|
keyPrefix: "countex",
|
|
22
22
|
ttlSeconds: 92 * 24 * 3600,
|
|
23
|
+
// Optional: shorter hot buckets, longer rollups
|
|
24
|
+
ttlSecondsByTimeframe: {
|
|
25
|
+
minute: 48 * 3600,
|
|
26
|
+
hour: 14 * 24 * 3600,
|
|
27
|
+
day: 92 * 24 * 3600,
|
|
28
|
+
},
|
|
23
29
|
});
|
|
24
30
|
|
|
25
31
|
const countex = createCountex({
|
|
@@ -30,9 +30,53 @@ const { cellsWarmed } = await rebuildCache(durable, cache, {
|
|
|
30
30
|
}, { batchSize: 1000 });
|
|
31
31
|
```
|
|
32
32
|
|
|
33
|
-
## Step 3:
|
|
33
|
+
## Step 3: Manifest and verification
|
|
34
34
|
|
|
35
|
-
|
|
35
|
+
After streaming an export, compute a stable fingerprint and save a small JSON manifest (e.g. next to your NDJSON dump):
|
|
36
|
+
|
|
37
|
+
```ts
|
|
38
|
+
import {
|
|
39
|
+
computeCellsFingerprint,
|
|
40
|
+
createBackupManifest,
|
|
41
|
+
verifyBackupAgainstManifest,
|
|
42
|
+
exportCells,
|
|
43
|
+
COUNTEX_CELL_SCHEMA_VERSION,
|
|
44
|
+
} from "@x12i/countex";
|
|
45
|
+
|
|
46
|
+
const filter = {
|
|
47
|
+
scope: "org_1",
|
|
48
|
+
counterId: "events",
|
|
49
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
50
|
+
};
|
|
51
|
+
|
|
52
|
+
const fp = await computeCellsFingerprint(exportCells(durable, filter));
|
|
53
|
+
const manifest = createBackupManifest(filter, fp, COUNTEX_CELL_SCHEMA_VERSION);
|
|
54
|
+
// persist manifest JSON + cell lines...
|
|
55
|
+
|
|
56
|
+
const check = await verifyBackupAgainstManifest(manifest, exportCells(durable, filter));
|
|
57
|
+
if (!check.ok) throw new Error(check.detail);
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Checksums sort canonical JSON lines in memory — use **timeframe-bounded** exports for very large datasets.
|
|
61
|
+
|
|
62
|
+
## Step 4: Restore snapshot
|
|
63
|
+
|
|
64
|
+
**Target:** durable storage only (`importSnapshotCells` on `MongoDurable` / `MemoryDurable`). After restore, **rebuild Redis** with `rebuildCache` for affected windows.
|
|
65
|
+
|
|
66
|
+
```ts
|
|
67
|
+
import { restoreSnapshotCells, exportCells } from "@x12i/countex";
|
|
68
|
+
|
|
69
|
+
const { restored } = await restoreSnapshotCells(targetDurable, exportCells(sourceDurable, filter), {
|
|
70
|
+
batchSize: 500,
|
|
71
|
+
});
|
|
72
|
+
// dryRun: true counts cells without writing
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Restore replaces absolute cell state for each identity in the stream — it is not an incremental delta merge.
|
|
76
|
+
|
|
77
|
+
## Step 5: Cancellation
|
|
78
|
+
|
|
79
|
+
Pass `AbortSignal` where supported (`streamCells` / `warm` / restore options propagate through adapters).
|
|
36
80
|
|
|
37
81
|
## See also
|
|
38
82
|
|
package/docs/guides/cli.md
CHANGED
|
@@ -33,6 +33,14 @@ countex insights:resources ./ctx.mjs
|
|
|
33
33
|
|
|
34
34
|
Use `--format json|markdown` for output shaping.
|
|
35
35
|
|
|
36
|
+
## Simulation demo (no config file)
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
countex simulate:demo --events 200 --seed 1
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Emits JSON **`SimulationReport`** from a dry-run against the bundled demo map (`audit-log` profile). No Mongo/Redis required.
|
|
43
|
+
|
|
36
44
|
## Step 4: TypeScript configs
|
|
37
45
|
|
|
38
46
|
```bash
|
|
@@ -2,6 +2,12 @@
|
|
|
2
2
|
|
|
3
3
|
A **counter map** (`CounterMap`) tells Countex how to read one JSON event and turn it into sparse counter updates. This guide walks through defining a map from scratch.
|
|
4
4
|
|
|
5
|
+
## Metrics vs domain “labels”
|
|
6
|
+
|
|
7
|
+
Countex uses **`metrics`** in the map config: paths that resolve to the category or label you want one counter split per value (for example `$.labels[*]` or `$.event.operation`). That field name is generic — your payload might call them labels, statuses, tags, or event types. They all map to Countex **metrics**.
|
|
8
|
+
|
|
9
|
+
Documentation refers to **`sourceKey`** for the string form of a subject or metric before dictionary compaction; that is not necessarily a UUID unless you choose UUID strings as keys.
|
|
10
|
+
|
|
5
11
|
## What you will configure
|
|
6
12
|
|
|
7
13
|
- **scope** — tenant or isolation boundary (string id).
|
|
@@ -59,6 +59,12 @@ insights.on("insight_query", (e) => console.log(e));
|
|
|
59
59
|
|
|
60
60
|
Insights honor **`maxBuckets`** (aligned with core query expansion) and **`maxGroups`**. Prefer narrower filters when grouping heavily.
|
|
61
61
|
|
|
62
|
+
## Distributions, percentiles, and histograms
|
|
63
|
+
|
|
64
|
+
Core Countex cells store **additive** measure aggregates (`sum`, `count`, `min`, `max`) per sparse cell — not full distributions. Insights therefore reason about **rankings, deltas, thresholds, and derived averages**, not arbitrary percentile curves across arbitrary slices unless you add **explicit** histogram or sketch storage elsewhere.
|
|
65
|
+
|
|
66
|
+
Token and latency **advisors** use those summaries (and sometimes robust z-scores on bucketed series); treat percentile-style language in recommendations as **heuristic** unless you extend storage.
|
|
67
|
+
|
|
62
68
|
## Feature map
|
|
63
69
|
|
|
64
70
|
| Area | Methods |
|
package/docs/guides/measures.md
CHANGED
|
@@ -34,6 +34,10 @@ const map = {
|
|
|
34
34
|
|
|
35
35
|
Use `onMissing: "skip"` (default) or `"drop"` / `"zero"` per measure.
|
|
36
36
|
|
|
37
|
+
### Unique / distinct counts are not additive sums
|
|
38
|
+
|
|
39
|
+
Countex **counts** and **measure aggregations** (`sum`, `count`, `min`, `max`) behave like standard additive counters over cells. They do **not** answer “how many **distinct** users appeared in this bucket?” unless you model that explicitly (for example one subject id per event and interpret cell counts accordingly). Do **not** sum counts across overlapping populations and treat the result as a unique count. For large-cardinality uniques, plan on a dedicated approach (for example optional future HLL-style storage — see deployment notes); vanilla cells are the wrong tool for deduplicated set cardinality across arbitrary queries.
|
|
40
|
+
|
|
37
41
|
## Step 2: Ingest as usual
|
|
38
42
|
|
|
39
43
|
Each fan-out increment carries the **same** measure snapshot for that event.
|
package/docs/guides/querying.md
CHANGED
|
@@ -72,8 +72,48 @@ Do not use `rate` with timeframe **`all`**.
|
|
|
72
72
|
|
|
73
73
|
Result includes `source`: `"cache"`, `"durable"`, or `"merged"`.
|
|
74
74
|
|
|
75
|
+
### Query diagnostics
|
|
76
|
+
|
|
77
|
+
`result.diagnostics` (when present) adds planner detail:
|
|
78
|
+
|
|
79
|
+
| Field | Meaning |
|
|
80
|
+
|-------|---------|
|
|
81
|
+
| `cacheAttempted` | `true` if the engine consulted the cache adapter (cache configured, not `bypassCache`, and the planner allowed cache for this query shape). |
|
|
82
|
+
| `cacheCoverage` | When `cacheAttempted` is true: `"full"` if the cache answered completely; `"partial"` or `"miss"` if the planner fell through to durable for authoritative cells. |
|
|
83
|
+
| `cellsMaterialized` | Number of counter cells loaded from storage before grouping/formatting. |
|
|
84
|
+
|
|
85
|
+
Use this for tests, dashboards, and verifying Redis vs Mongo paths.
|
|
86
|
+
|
|
87
|
+
## Step 6: Totals, event volume, and “distinct entities”
|
|
88
|
+
|
|
89
|
+
### Grand total (sum all matching cells)
|
|
90
|
+
|
|
91
|
+
Omit **`groupBy`** (or use an empty list). For **`format: "table"`**, the engine merges every matching cell into **one row**; **`rows[0].count`** is the sum of **`count`** across those cells. **`result.total`** matches that sum for the returned rows (`sumRowCounts` in `src/query/formatter.ts`).
|
|
92
|
+
|
|
93
|
+
Use this for **total increment volume** in the query window. If one source event fans out into many cells (multiple subjects × metrics × timeframes), this total is **not** the same as “number of source events.”
|
|
94
|
+
|
|
95
|
+
### Distinct subjects / metrics as row cardinality
|
|
96
|
+
|
|
97
|
+
Countex does **not** store arbitrary **`COUNT(DISTINCT …)`** queries. **Exactly what to do** when you need “how many unique actors had activity?”:
|
|
98
|
+
|
|
99
|
+
1. **`subject: { type: "actor" }`** (optional **`ids`**) to restrict the subject family.
|
|
100
|
+
2. **`dimensions: { situation: ["external"] }`** (example) to restrict literals stored on cells.
|
|
101
|
+
3. **`groupBy: ["subjectType", "subject"]`** so **each row aggregates one dictionary subject id**.
|
|
102
|
+
4. **`rows.length`** after the query = number of distinct actors **with non-zero merged activity** in that slice (sparse — actors who never appeared produce no row).
|
|
103
|
+
|
|
104
|
+
**Critical:** if you set **`limit`**, **`rows.length ≤ limit`** and **`result.total`** only sum **returned** rows — you **cannot** infer global distinct counts from a capped query.
|
|
105
|
+
|
|
106
|
+
### Logical “one row per source event” totals
|
|
107
|
+
|
|
108
|
+
Maintain a **separate counter map** engineered for **one increment per deduped message** (single timeframe granularity, single synthetic metric field, minimal subjects). See [`docs/use-cases/audit-security-log-counts.md`](../use-cases/audit-security-log-counts.md) §10.6.
|
|
109
|
+
|
|
110
|
+
### Metric filters at rest
|
|
111
|
+
|
|
112
|
+
`MongoDurable` / `MemoryDurable` apply **`metrics`** filters using **numeric metric ids** parsed from `QuerySpec.metrics`. Pure string labels that are not finite numbers do **not** constrain the database filter — resolve ids from `countex_dict` or filter after read.
|
|
113
|
+
|
|
75
114
|
## See also
|
|
76
115
|
|
|
77
116
|
- [Measures](./measures.md)
|
|
78
117
|
- [Timeframes](./timeframes-and-bucketing.md)
|
|
118
|
+
- [Audit use case — analytics recipes](../use-cases/audit-security-log-counts.md) §10
|
|
79
119
|
- Internal: [Query path](../internals/internal-query-path.md)
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Guide: Simulation profiles
|
|
2
|
+
|
|
3
|
+
Profiles configure **what kind of events** the simulator emits and **how much fan-out** they produce.
|
|
4
|
+
|
|
5
|
+
## Built-in profile ids
|
|
6
|
+
|
|
7
|
+
| Profile id | Intent |
|
|
8
|
+
|------------|--------|
|
|
9
|
+
| `audit-log` | High-volume audit-style records: actors, resources, operations, optional label fan-out. |
|
|
10
|
+
| `product-activity` | Product analytics–style events: account scope, user subject, named event metric, feature/plan dimensions. |
|
|
11
|
+
| `processing-activity` | Jobs / pipelines / workflows: job id subject, task/step metrics, status dimensions, duration measures. |
|
|
12
|
+
| `llm-usage` | Token and cost measures with model/provider dimensions. |
|
|
13
|
+
| `custom` | Supply your own event factory; simulator handles scheduling and pools only. |
|
|
14
|
+
|
|
15
|
+
## Profile options (conceptual)
|
|
16
|
+
|
|
17
|
+
- **`durationMs`** — simulated wall-clock span of the run (logical time advancement between events).
|
|
18
|
+
- **`eventsPerTick`** — throughput knob for batch generators.
|
|
19
|
+
- **`poolSizes`** — caps on distinct subjects/metrics/dimension values (cardinality stress).
|
|
20
|
+
- **`weights`** — skew toward hot keys (Pareto-style) vs uniform draws.
|
|
21
|
+
|
|
22
|
+
Exact fields match `SimulationProfile` in `@x12i/countex/simulation` TypeScript exports.
|
|
23
|
+
|
|
24
|
+
## Choosing a profile
|
|
25
|
+
|
|
26
|
+
- Mirror the **same map** you plan for production; mismatched profiles hide JSONPath bugs.
|
|
27
|
+
- Start **uniform** pools, then add **skew** to reproduce hotspot behavior.
|
|
28
|
+
- Combine with **`validateMap(map, samples)`** from core so paths resolve on representative payloads.
|
|
29
|
+
|
|
30
|
+
## See also
|
|
31
|
+
|
|
32
|
+
- [Simulation overview](./simulation.md)
|
|
33
|
+
- [Simulation resource estimation](./simulation-resource-estimation.md)
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Guide: Simulation resource estimation
|
|
2
|
+
|
|
3
|
+
Use simulation outputs to reason about **Mongo**, **Redis**, and **process memory** before cutover.
|
|
4
|
+
|
|
5
|
+
## Signals from a run
|
|
6
|
+
|
|
7
|
+
After a simulator run (or CLI report):
|
|
8
|
+
|
|
9
|
+
- **Raw events** and **raw increments** (before cardinality drops) — multiply by your peak events/sec for ingress sizing.
|
|
10
|
+
- **Distinct dictionary keys** per kind (`subject`, `metric`, `dimensionValue`) — drives Mongo dictionary collection growth.
|
|
11
|
+
- **Distinct cells touched** — upper bound on sparse matrix cardinality for the window.
|
|
12
|
+
- **Flush batch sizes** — depends on buffer settings (`maxBufferedCells`, flush interval).
|
|
13
|
+
|
|
14
|
+
## Mapping to infrastructure
|
|
15
|
+
|
|
16
|
+
| Concern | What to measure |
|
|
17
|
+
|---------|-----------------|
|
|
18
|
+
| Mongo durable | Cell documents + dictionary docs; index sizes (see deployment guide). |
|
|
19
|
+
| Redis | Working set for hot timeframes; TTL/retention policy per timeframe type. |
|
|
20
|
+
| Application RAM | In-flight buffer + Node overhead; simulate worst-case burst with `onOverflow: "block"`. |
|
|
21
|
+
|
|
22
|
+
## Limits
|
|
23
|
+
|
|
24
|
+
Simulation **does not** replace load testing of your MongoDB cluster or Redis failover behavior; it estimates **logical** counter volume from maps and profiles.
|
|
25
|
+
|
|
26
|
+
## See also
|
|
27
|
+
|
|
28
|
+
- [Simulation overview](./simulation.md)
|
|
29
|
+
- [Ingestion & flush](./ingestion-and-flush.md)
|
|
30
|
+
- [`docs/DEPLOYMENT.md`](../DEPLOYMENT.md)
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Guide: Simulation overview
|
|
2
|
+
|
|
3
|
+
**Simulation** helps before production traffic exists: generate deterministic synthetic events, stress-test maps and cardinality, estimate storage and flush load, and validate that aggregates match expectations.
|
|
4
|
+
|
|
5
|
+
## Runtime entry point
|
|
6
|
+
|
|
7
|
+
When using the published package:
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
import { createCountexSimulator } from "@x12i/countex/simulation";
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The simulator is optional tooling on top of the core engine; it does not change ingest or query semantics.
|
|
14
|
+
|
|
15
|
+
## What simulation covers
|
|
16
|
+
|
|
17
|
+
- **Seeded randomness** — reproducible runs for CI and regression tests.
|
|
18
|
+
- **Pools & relations** — weighted entities (users, documents, models) with stable ids across events.
|
|
19
|
+
- **Profiles** — named scenarios (audit-style logs, product activity, LLM metering, processing pipelines); see [Simulation profiles](./simulation-profiles.md).
|
|
20
|
+
- **Expected aggregates** — optional tracker that compares simulated totals to counter reads for a window.
|
|
21
|
+
- **Reports** — structured summary (events emitted, fan-out estimates, errors) for sizing discussions.
|
|
22
|
+
|
|
23
|
+
## Relationship to core Countex
|
|
24
|
+
|
|
25
|
+
1. Build or load a `CounterMap` (same as production).
|
|
26
|
+
2. Instantiate `createCountexSimulator({ map, profile, seed, ... })`.
|
|
27
|
+
3. Drive **`generateEvents(n)`** or **`run()`** into `countex.ingest` / `ingestBatch` in tests or benchmarks.
|
|
28
|
+
4. Optionally attach **`ExpectedAggregateTracker`** and assert after flush + query.
|
|
29
|
+
|
|
30
|
+
See [Simulation resource estimation](./simulation-resource-estimation.md) for capacity hooks.
|
|
31
|
+
|
|
32
|
+
## See also
|
|
33
|
+
|
|
34
|
+
- [CLI](./cli.md) — `countex simulate:demo` for scripted dry-runs without a config module
|
|
35
|
+
- [Counter maps](./counter-maps.md)
|
|
36
|
+
- [Cardinality](./cardinality.md)
|
package/docs/internals/README.md
CHANGED
|
@@ -12,6 +12,7 @@ Start here:
|
|
|
12
12
|
| In-memory merge | [Buffer merge & flush](./internal-buffer-merge-and-flush.md) |
|
|
13
13
|
| Reads | [Query path](./internal-query-path.md) |
|
|
14
14
|
| Bucket keys | [Timeframe bucketing](./internal-timeframe-bucketing.md) |
|
|
15
|
+
| Dimension hashing | [Dimension canonicalization & hash](./internal-dimension-hash.md) |
|
|
15
16
|
| Measures | [Measures & scales](./internal-measures-and-scales.md) |
|
|
16
17
|
| Cardinality | [Cardinality guard](./internal-cardinality.md) |
|
|
17
18
|
| Dedupe | [Idempotency](./internal-idempotency.md) |
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Internal: Dimension canonicalization and hashing
|
|
2
|
+
|
|
3
|
+
Each counter cell identity includes a **`dimensionsHash`** derived from the dimension key/value object for that increment.
|
|
4
|
+
|
|
5
|
+
## Normalization
|
|
6
|
+
|
|
7
|
+
Before hashing, dimension values pass through `normalizeDimensionValue` in `src/core/normalization.ts`:
|
|
8
|
+
|
|
9
|
+
- `null` / missing optional dimensions → `null`.
|
|
10
|
+
- Booleans and finite numbers preserved as typed values.
|
|
11
|
+
- Strings: optionally trimmed only (`normalizeValues` on the map); types are **not** coerced (`"1"` and `1` stay distinct).
|
|
12
|
+
|
|
13
|
+
Dictionary keys for subjects/metrics use **`normalizeKey`** (trim + lowercase strings) — that is separate from dimension hashing.
|
|
14
|
+
|
|
15
|
+
## Canonical string form
|
|
16
|
+
|
|
17
|
+
`canonicalizeDimensions` in `src/utils/hash.ts`:
|
|
18
|
+
|
|
19
|
+
1. Sort dimension **names** lexicographically.
|
|
20
|
+
2. Serialize as `key=value` pairs joined with `&`.
|
|
21
|
+
3. Escape `\`, `=`, and `&` in keys and string values with backslash.
|
|
22
|
+
4. Encode values with type prefixes: `n:` (null), `b:0|1`, `i:<number>`, `s:<escaped>`.
|
|
23
|
+
|
|
24
|
+
## Hash
|
|
25
|
+
|
|
26
|
+
`dimensionsHash` is **SHA-1** of the canonical string, truncated to **12 hex characters** (48 bits). Empty dimensions → literal `"0"`.
|
|
27
|
+
|
|
28
|
+
Collisions are statistically negligible at intended cardinalities; cells are further scoped by `(scopeId, counterId, timeframe, subject, metric)`.
|
|
29
|
+
|
|
30
|
+
## See also
|
|
31
|
+
|
|
32
|
+
- `src/utils/hash.ts`
|
|
33
|
+
- `src/core/normalization.ts`
|
|
34
|
+
- [Cardinality](../guides/cardinality.md)
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# Countex use cases — technical playbooks
|
|
2
|
+
|
|
3
|
+
These documents are **end-to-end operational specifications**: they start from a concrete delivery problem, define **exact** input contracts and Countex primitives, then walk through **numbered steps** you can implement without improvising.
|
|
4
|
+
|
|
5
|
+
They are published with `@x12i/countex` under `docs/use-cases/` and exposed as stable `package.json` exports (see root `README.md`).
|
|
6
|
+
|
|
7
|
+
## How each playbook is structured
|
|
8
|
+
|
|
9
|
+
Every use-case file follows the same skeleton:
|
|
10
|
+
|
|
11
|
+
1. **Problem & constraints** — what you are counting, latency/durability expectations, and what Countex does *not* solve.
|
|
12
|
+
2. **Primitives mapping** — explicit table: JSON field → `scope` / `subject` / `metric` / `dimension` / `eventTime` / `delta`.
|
|
13
|
+
3. **Canonical payload** — JSON example plus a **required-field matrix** (what must be present or ingest skips).
|
|
14
|
+
4. **`CounterMap`** — copy-paste literal with a **field-by-field rationale**.
|
|
15
|
+
5. **Fan-out arithmetic** — formula and a worked example for capacity planning.
|
|
16
|
+
6. **Implementation steps** — repository layout, dependencies, `createCountex` options, `init` / `ingest` / `flush` / `shutdown`, event subscriptions.
|
|
17
|
+
7. **Query specifications** — literal `QuerySpec` objects; how to read **`QueryRow.keys`** (dictionary ids vs literal dimension strings).
|
|
18
|
+
8. **Production hardening** — Mongo / Redis wiring patterns, cardinality, idempotency TTL math, observability.
|
|
19
|
+
9. **Verification** — checks you can run after deploy.
|
|
20
|
+
10. **Failure modes** — symptom → cause → exact mitigation.
|
|
21
|
+
|
|
22
|
+
## Stable import paths (npm)
|
|
23
|
+
|
|
24
|
+
| Document | Subpath |
|
|
25
|
+
|----------|---------|
|
|
26
|
+
| Index (this list) | `@x12i/countex/docs/use-cases` |
|
|
27
|
+
| [Audit & security logs](./audit-security-log-counts.md) (includes **volume, situations, distinct entities, logical event counts**) | `@x12i/countex/docs/use-cases/audit-security-log-counts` |
|
|
28
|
+
| [Product & feature activity](./product-feature-usage.md) | `@x12i/countex/docs/use-cases/product-feature-usage` |
|
|
29
|
+
| [LLM metering](./llm-cost-and-token-metering.md) | `@x12i/countex/docs/use-cases/llm-cost-and-token-metering` |
|
|
30
|
+
| [Multi-tenant isolation](./multi-tenant-isolation.md) | `@x12i/countex/docs/use-cases/multi-tenant-isolation` |
|
|
31
|
+
| [Webhooks & queues](./webhook-idempotent-ingestion.md) | `@x12i/countex/docs/use-cases/webhook-idempotent-ingestion` |
|
|
32
|
+
| [Ops dashboards](./ops-dashboard-daily-rollups.md) | `@x12i/countex/docs/use-cases/ops-dashboard-daily-rollups` |
|
|
33
|
+
| [Processing & workflow activity](./processing-workflow-activity.md) | `@x12i/countex/docs/use-cases/processing-workflow-activity` |
|
|
34
|
+
|
|
35
|
+
## Prerequisites (global)
|
|
36
|
+
|
|
37
|
+
- **Node.js** ≥ 18 (see `package.json` `engines`).
|
|
38
|
+
- Core package: `@x12i/countex`.
|
|
39
|
+
- Optional peers for production paths: `mongodb`, `ioredis` (your process owns client lifecycle).
|
|
40
|
+
|
|
41
|
+
## Terminology (read this once)
|
|
42
|
+
|
|
43
|
+
| Term | Meaning |
|
|
44
|
+
|------|---------|
|
|
45
|
+
| **Scope** | Tenant boundary string (`scopeId` in storage). Every query **must** pin `scope`. |
|
|
46
|
+
| **Subject** | `(subjectType, subjectId)` after dictionary resolution; `subjectId` is an **integer**. |
|
|
47
|
+
| **Metric** | `(metricId)` after dictionary resolution; labels/strings become stable numeric ids per scope. |
|
|
48
|
+
| **Dimension** | Named breakdown; values stored **literally** on cells (not dictionary-encoded). |
|
|
49
|
+
| **Timeframe key** | UTC bucket string, e.g. `2026-05-08` for `day` (see `bucketKey` in `src/core/timeframe.ts`). |
|
|
50
|
+
| **Fan-out** | Cartesian product of subject extractions × metric extractions × number of `timeframes` entries. |
|
|
51
|
+
|
|
52
|
+
## Related documentation
|
|
53
|
+
|
|
54
|
+
- Feature reference: [`docs/guides/`](../guides/README.md)
|
|
55
|
+
- Narrative tutorial: [`END_TO_END_EXAMPLE.md`](../END_TO_END_EXAMPLE.md)
|
|
56
|
+
- Deployment: [`DEPLOYMENT.md`](../DEPLOYMENT.md)
|