@x12i/countex 1.8.3 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +866 -858
  3. package/dist/cli.js +6 -6
  4. package/dist/index.d.ts +1 -0
  5. package/dist/index.d.ts.map +1 -1
  6. package/dist/index.js +1 -0
  7. package/dist/index.js.map +1 -1
  8. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  9. package/dist/insights/advisors/resource-advisor.js +5 -1
  10. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  11. package/dist/insights/types.d.ts +4 -0
  12. package/dist/insights/types.d.ts.map +1 -1
  13. package/dist/storage/postgres.d.ts +104 -0
  14. package/dist/storage/postgres.d.ts.map +1 -0
  15. package/dist/storage/postgres.js +1023 -0
  16. package/dist/storage/postgres.js.map +1 -0
  17. package/dist/types/index.d.ts +11 -5
  18. package/dist/types/index.d.ts.map +1 -1
  19. package/docs/DEPLOYMENT.md +399 -375
  20. package/docs/END_TO_END_EXAMPLE.md +316 -316
  21. package/docs/INSIGHTS.md +59 -59
  22. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  23. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  24. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  25. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  26. package/docs/guides/README.md +36 -34
  27. package/docs/guides/adapter-memory.md +60 -60
  28. package/docs/guides/adapter-mongo.md +64 -62
  29. package/docs/guides/adapter-postgres.md +90 -0
  30. package/docs/guides/adapter-redis.md +89 -89
  31. package/docs/guides/backup-export.md +135 -135
  32. package/docs/guides/cardinality.md +50 -50
  33. package/docs/guides/cli.md +53 -53
  34. package/docs/guides/client-events.md +65 -65
  35. package/docs/guides/counter-maps.md +160 -160
  36. package/docs/guides/idempotency.md +39 -39
  37. package/docs/guides/ingestion-and-flush.md +67 -67
  38. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  39. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  40. package/docs/guides/insights-comparison-and-trends.md +47 -47
  41. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  42. package/docs/guides/insights-overview.md +82 -82
  43. package/docs/guides/insights-ranking.md +81 -81
  44. package/docs/guides/measures.md +71 -71
  45. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  46. package/docs/guides/preview.md +92 -92
  47. package/docs/guides/querying.md +178 -178
  48. package/docs/guides/simulation-profiles.md +33 -33
  49. package/docs/guides/simulation-resource-estimation.md +31 -30
  50. package/docs/guides/simulation.md +36 -36
  51. package/docs/guides/timeframes-and-bucketing.md +55 -55
  52. package/docs/internals/README.md +21 -21
  53. package/docs/internals/internal-architecture-overview.md +65 -65
  54. package/docs/internals/internal-backup-stream.md +23 -23
  55. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  56. package/docs/internals/internal-cardinality.md +30 -30
  57. package/docs/internals/internal-dimension-hash.md +34 -34
  58. package/docs/internals/internal-idempotency.md +36 -36
  59. package/docs/internals/internal-ingest-pipeline.md +52 -52
  60. package/docs/internals/internal-insights-layer.md +36 -36
  61. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  62. package/docs/internals/internal-measures-and-scales.md +35 -35
  63. package/docs/internals/internal-query-path.md +46 -46
  64. package/docs/internals/internal-storage-memory.md +13 -13
  65. package/docs/internals/internal-storage-mongo.md +44 -44
  66. package/docs/internals/internal-storage-postgres.md +32 -0
  67. package/docs/internals/internal-storage-redis.md +63 -63
  68. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  69. package/docs/use-cases/README.md +56 -56
  70. package/docs/use-cases/audit-security-log-counts.md +550 -550
  71. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  72. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  73. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  74. package/docs/use-cases/processing-workflow-activity.md +165 -165
  75. package/docs/use-cases/product-feature-usage.md +240 -240
  76. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  77. package/package.json +184 -169
@@ -1,178 +1,178 @@
1
- # Guide: Querying
2
-
3
- `countex.query(spec)` reads aggregated cells for a scope and counter, with optional filters and output shaping.
4
-
5
- ## Step 1: Minimal query
6
-
7
- ```ts
8
- const result = await countex.query({
9
- scope: "org_1",
10
- counterId: "access_by_actor_label",
11
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
12
- });
13
- ```
14
-
15
- ## Step 2: Filter
16
-
17
- ```ts
18
- const result = await countex.query({
19
- scope: "org_1",
20
- counterId: "access_by_actor_label",
21
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
22
- subject: { type: "actor", ids: [10001, 10002] },
23
- metrics: [41, 42],
24
- dimensions: { operation: ["read", "write"] },
25
- });
26
- ```
27
-
28
- **Subject and metric filters** accept **internal numeric ids** (as stored on cells) **and/or dictionary source keys** (strings: UUIDs, opaque ids, label text — whatever your map extracted at ingest). Strings are resolved read-only via `durable.lookupIds` before storage reads; **unknown keys are dropped** from the filter (they do not allocate dictionary rows). When any `subject.ids` entry is a string, **`subject.type` is required** so the lookup is scoped to that subject family. In TypeScript / JSON, **`number` means an internal id** and **`string` means a source key** — the literal `"10001"` is never treated as numeric id `10001`. If `subject.ids` or `metrics` is present as an **empty array**, the query matches **no cells** for that filter.
29
-
30
- ```ts
31
- // Same slice using source keys from your domain (no need to remember internal ids)
32
- const bySourceKeys = await countex.query({
33
- scope: "org_1",
34
- counterId: "access_by_actor_label",
35
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
36
- subject: { type: "actor", ids: ["aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", "user_42"] },
37
- metrics: ["PCI", "PII"],
38
- dimensions: { operation: ["read", "write"] },
39
- });
40
- ```
41
-
42
- ## Human-readable labels (`resolveLabels`)
43
-
44
- Cells always store **numeric** `subjectId` / `metricId`. For grouped results and dashboards, set **`resolveLabels: true`** so `groupBy` keys for **`subject`** and **`metric`** use **source strings** where the dictionary has a reverse mapping; missing entries fall back to the numeric id as a string. With **`format: "cells"`**, each row also includes **`subjectSourceKey`** and **`metricSourceKey`**. This uses **`durable.reverseLookupIds`** on the ids present in the result set (extra durable reads; omit when you only need ids).
45
-
46
- ```ts
47
- const labeled = await countex.query({
48
- scope: "org_1",
49
- counterId: "access_by_actor_label",
50
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
51
- subject: { type: "actor", ids: ["aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"] },
52
- metrics: ["read"],
53
- groupBy: ["subject", "metric", "dim:operation"],
54
- format: "table",
55
- resolveLabels: true,
56
- });
57
- ```
58
-
59
- ## Step 3: Group and format
60
-
61
- `groupBy` accepts:
62
-
63
- - `subject`, `subjectType`, `metric`, `timeframeKey`
64
- - `dim:NAME` for a dimension
65
-
66
- `format`:
67
-
68
- - `cells` — raw-ish cell list
69
- - `table` / `matrix` — aggregated rows
70
- - `timeseries` — points per bucket (and optional extra grouping)
71
-
72
- ```ts
73
- const result = await countex.query({
74
- scope: "org_1",
75
- counterId: "access_by_actor_label",
76
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
77
- groupBy: ["subject", "dim:operation"],
78
- format: "table",
79
- orderBy: "-count",
80
- limit: 50,
81
- });
82
- ```
83
-
84
- ## Step 4: Measures and rates
85
-
86
- ```ts
87
- await countex.query({
88
- // ...
89
- measures: ["cost"],
90
- orderBy: { measure: "cost", agg: "sum", dir: "desc" },
91
- rate: { measure: "cost", agg: "sum" },
92
- });
93
- ```
94
-
95
- Do not use `rate` with timeframe **`all`**.
96
-
97
- ## Step 5: Cache control
98
-
99
- - `bypassCache: true` forces durable reads.
100
- - `signal` accepts `AbortSignal` for timeouts.
101
-
102
- Result includes `source`: `"cache"`, `"durable"`, or `"merged"`.
103
-
104
- ### Query diagnostics
105
-
106
- `result.diagnostics` (when present) adds planner detail:
107
-
108
- | Field | Meaning |
109
- |-------|---------|
110
- | `cacheAttempted` | `true` if the engine consulted the cache adapter (cache configured, not `bypassCache`, and the planner allowed cache for this query shape). |
111
- | `cacheCoverage` | When `cacheAttempted` is true: `"full"` if the cache answered completely; `"partial"` or `"miss"` if the planner fell through to durable for authoritative cells. |
112
- | `cellsMaterialized` | Number of counter cells loaded from storage before grouping/formatting. |
113
- | `bucketsRequested` | Bucket keys the query expanded to (after `bucketRange`). |
114
- | `cacheBucketsHit` | Number of buckets the cache reported as `complete`. |
115
- | `durableFallback` | `true` when the planner had to read durable for some/all buckets. |
116
- | `plannedCellsEstimate` | Heuristic estimate of cells before grouping (matches `cellsMaterialized` after execution). |
117
- | `groupByCost` | `"trivial"` (no groupBy), `"low"`, or `"high"` (3+ keys or 2+ dimensions). |
118
- | `expensive` | Heuristic flag — `true` for big windows, high groupByCost, or `format: "matrix"` with many cells. |
119
-
120
- Use this for tests, dashboards, and verifying Redis vs Mongo paths.
121
-
122
- ### Explain a query without running it
123
-
124
- `countex.explainQuery(spec)` returns a {@link QueryPlan} without touching durable storage and only probing per-bucket cache coverage:
125
-
126
- ```ts
127
- const plan = await countex.explainQuery({
128
- scope: "org_1",
129
- counterId: "events",
130
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-31" },
131
- groupBy: ["dim:operation", "dim:situation", "subject"],
132
- });
133
- // {
134
- // bucketsRequested: ["2026-05-01", …, "2026-05-31"],
135
- // cacheEligible: true,
136
- // cacheBucketsExpectedHit: 18,
137
- // durableFallbackPredicted: true,
138
- // plannedCellsEstimate: 1_240,
139
- // groupByCost: "high",
140
- // expensive: true,
141
- // }
142
- ```
143
-
144
- `cacheBucketsExpectedHit` and `plannedCellsEstimate` are populated when the cache adapter implements `probeBucketCoverage` (the bundled `RedisCache` and `MemoryCache` both do).
145
-
146
- ## Step 6: Totals, event volume, and “distinct entities”
147
-
148
- ### Grand total (sum all matching cells)
149
-
150
- Omit **`groupBy`** (or use an empty list). For **`format: "table"`**, the engine merges every matching cell into **one row**; **`rows[0].count`** is the sum of **`count`** across those cells. **`result.total`** matches that sum for the returned rows (`sumRowCounts` in `src/query/formatter.ts`).
151
-
152
- Use this for **total increment volume** in the query window. If one source event fans out into many cells (multiple subjects × metrics × timeframes), this total is **not** the same as “number of source events.”
153
-
154
- ### Distinct subjects / metrics as row cardinality
155
-
156
- Countex does **not** store arbitrary **`COUNT(DISTINCT …)`** queries. **Exactly what to do** when you need “how many unique actors had activity?”:
157
-
158
- 1. **`subject: { type: "actor" }`** (optional **`ids`**) to restrict the subject family.
159
- 2. **`dimensions: { situation: ["external"] }`** (example) to restrict literals stored on cells.
160
- 3. **`groupBy: ["subjectType", "subject"]`** so **each row aggregates one dictionary subject id**.
161
- 4. **`rows.length`** after the query = number of distinct actors **with non-zero merged activity** in that slice (sparse — actors who never appeared produce no row).
162
-
163
- **Critical:** if you set **`limit`**, **`rows.length ≤ limit`** and **`result.total`** only sum **returned** rows — you **cannot** infer global distinct counts from a capped query.
164
-
165
- ### Logical “one row per source event” totals
166
-
167
- Maintain a **separate counter map** engineered for **one increment per deduped message** (single timeframe granularity, single synthetic metric field, minimal subjects). See [`docs/use-cases/audit-security-log-counts.md`](../use-cases/audit-security-log-counts.md) §10.6.
168
-
169
- ### Metric and subject filters at rest
170
-
171
- `executeQuery` runs **`normalizeQueryDictionaryFilters`** first: any **string** entries in **`subject.ids`** / **`metrics`** are translated to internal ids via **`lookupIds`** (read-only). The spec passed to **`readCells`** therefore uses **numeric ids only**, so Mongo / memory index filters apply correctly. If every string key was unknown, **`ids` / `metrics` may become empty arrays**, which match **no** cells (same as an impossible id list).
172
-
173
- ## See also
174
-
175
- - [Measures](./measures.md)
176
- - [Timeframes](./timeframes-and-bucketing.md)
177
- - [Audit use case — analytics recipes](../use-cases/audit-security-log-counts.md) §10
178
- - Internal: [Query path](../internals/internal-query-path.md)
1
+ # Guide: Querying
2
+
3
+ `countex.query(spec)` reads aggregated cells for a scope and counter, with optional filters and output shaping.
4
+
5
+ ## Step 1: Minimal query
6
+
7
+ ```ts
8
+ const result = await countex.query({
9
+ scope: "org_1",
10
+ counterId: "access_by_actor_label",
11
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
12
+ });
13
+ ```
14
+
15
+ ## Step 2: Filter
16
+
17
+ ```ts
18
+ const result = await countex.query({
19
+ scope: "org_1",
20
+ counterId: "access_by_actor_label",
21
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
22
+ subject: { type: "actor", ids: [10001, 10002] },
23
+ metrics: [41, 42],
24
+ dimensions: { operation: ["read", "write"] },
25
+ });
26
+ ```
27
+
28
+ **Subject and metric filters** accept **internal numeric ids** (as stored on cells) **and/or dictionary source keys** (strings: UUIDs, opaque ids, label text — whatever your map extracted at ingest). Strings are resolved read-only via `durable.lookupIds` before storage reads; **unknown keys are dropped** from the filter (they do not allocate dictionary rows). When any `subject.ids` entry is a string, **`subject.type` is required** so the lookup is scoped to that subject family. In TypeScript / JSON, **`number` means an internal id** and **`string` means a source key** — the literal `"10001"` is never treated as numeric id `10001`. If `subject.ids` or `metrics` is present as an **empty array**, the query matches **no cells** for that filter.
29
+
30
+ ```ts
31
+ // Same slice using source keys from your domain (no need to remember internal ids)
32
+ const bySourceKeys = await countex.query({
33
+ scope: "org_1",
34
+ counterId: "access_by_actor_label",
35
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
36
+ subject: { type: "actor", ids: ["aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", "user_42"] },
37
+ metrics: ["PCI", "PII"],
38
+ dimensions: { operation: ["read", "write"] },
39
+ });
40
+ ```
41
+
42
+ ## Human-readable labels (`resolveLabels`)
43
+
44
+ Cells always store **numeric** `subjectId` / `metricId`. For grouped results and dashboards, set **`resolveLabels: true`** so `groupBy` keys for **`subject`** and **`metric`** use **source strings** where the dictionary has a reverse mapping; missing entries fall back to the numeric id as a string. With **`format: "cells"`**, each row also includes **`subjectSourceKey`** and **`metricSourceKey`**. This uses **`durable.reverseLookupIds`** on the ids present in the result set (extra durable reads; omit when you only need ids).
45
+
46
+ ```ts
47
+ const labeled = await countex.query({
48
+ scope: "org_1",
49
+ counterId: "access_by_actor_label",
50
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
51
+ subject: { type: "actor", ids: ["aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"] },
52
+ metrics: ["read"],
53
+ groupBy: ["subject", "metric", "dim:operation"],
54
+ format: "table",
55
+ resolveLabels: true,
56
+ });
57
+ ```
58
+
59
+ ## Step 3: Group and format
60
+
61
+ `groupBy` accepts:
62
+
63
+ - `subject`, `subjectType`, `metric`, `timeframeKey`
64
+ - `dim:NAME` for a dimension
65
+
66
+ `format`:
67
+
68
+ - `cells` — raw-ish cell list
69
+ - `table` / `matrix` — aggregated rows
70
+ - `timeseries` — points per bucket (and optional extra grouping)
71
+
72
+ ```ts
73
+ const result = await countex.query({
74
+ scope: "org_1",
75
+ counterId: "access_by_actor_label",
76
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
77
+ groupBy: ["subject", "dim:operation"],
78
+ format: "table",
79
+ orderBy: "-count",
80
+ limit: 50,
81
+ });
82
+ ```
83
+
84
+ ## Step 4: Measures and rates
85
+
86
+ ```ts
87
+ await countex.query({
88
+ // ...
89
+ measures: ["cost"],
90
+ orderBy: { measure: "cost", agg: "sum", dir: "desc" },
91
+ rate: { measure: "cost", agg: "sum" },
92
+ });
93
+ ```
94
+
95
+ Do not use `rate` with timeframe **`all`**.
96
+
97
+ ## Step 5: Cache control
98
+
99
+ - `bypassCache: true` forces durable reads.
100
+ - `signal` accepts `AbortSignal` for timeouts.
101
+
102
+ Result includes `source`: `"cache"`, `"durable"`, or `"merged"`.
103
+
104
+ ### Query diagnostics
105
+
106
+ `result.diagnostics` (when present) adds planner detail:
107
+
108
+ | Field | Meaning |
109
+ |-------|---------|
110
+ | `cacheAttempted` | `true` if the engine consulted the cache adapter (cache configured, not `bypassCache`, and the planner allowed cache for this query shape). |
111
+ | `cacheCoverage` | When `cacheAttempted` is true: `"full"` if the cache answered completely; `"partial"` or `"miss"` if the planner fell through to durable for authoritative cells. |
112
+ | `cellsMaterialized` | Number of counter cells loaded from storage before grouping/formatting. |
113
+ | `bucketsRequested` | Bucket keys the query expanded to (after `bucketRange`). |
114
+ | `cacheBucketsHit` | Number of buckets the cache reported as `complete`. |
115
+ | `durableFallback` | `true` when the planner had to read durable for some/all buckets. |
116
+ | `plannedCellsEstimate` | Heuristic estimate of cells before grouping (matches `cellsMaterialized` after execution). |
117
+ | `groupByCost` | `"trivial"` (no groupBy), `"low"`, or `"high"` (3+ keys or 2+ dimensions). |
118
+ | `expensive` | Heuristic flag — `true` for big windows, high groupByCost, or `format: "matrix"` with many cells. |
119
+
120
+ Use this for tests, dashboards, and verifying Redis vs Mongo paths.
121
+
122
+ ### Explain a query without running it
123
+
124
+ `countex.explainQuery(spec)` returns a {@link QueryPlan} without touching durable storage and only probing per-bucket cache coverage:
125
+
126
+ ```ts
127
+ const plan = await countex.explainQuery({
128
+ scope: "org_1",
129
+ counterId: "events",
130
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-31" },
131
+ groupBy: ["dim:operation", "dim:situation", "subject"],
132
+ });
133
+ // {
134
+ // bucketsRequested: ["2026-05-01", …, "2026-05-31"],
135
+ // cacheEligible: true,
136
+ // cacheBucketsExpectedHit: 18,
137
+ // durableFallbackPredicted: true,
138
+ // plannedCellsEstimate: 1_240,
139
+ // groupByCost: "high",
140
+ // expensive: true,
141
+ // }
142
+ ```
143
+
144
+ `cacheBucketsExpectedHit` and `plannedCellsEstimate` are populated when the cache adapter implements `probeBucketCoverage` (the bundled `RedisCache` and `MemoryCache` both do).
145
+
146
+ ## Step 6: Totals, event volume, and “distinct entities”
147
+
148
+ ### Grand total (sum all matching cells)
149
+
150
+ Omit **`groupBy`** (or use an empty list). For **`format: "table"`**, the engine merges every matching cell into **one row**; **`rows[0].count`** is the sum of **`count`** across those cells. **`result.total`** matches that sum for the returned rows (`sumRowCounts` in `src/query/formatter.ts`).
151
+
152
+ Use this for **total increment volume** in the query window. If one source event fans out into many cells (multiple subjects × metrics × timeframes), this total is **not** the same as “number of source events.”
153
+
154
+ ### Distinct subjects / metrics as row cardinality
155
+
156
+ Countex does **not** store arbitrary **`COUNT(DISTINCT …)`** queries. **Exactly what to do** when you need “how many unique actors had activity?”:
157
+
158
+ 1. **`subject: { type: "actor" }`** (optional **`ids`**) to restrict the subject family.
159
+ 2. **`dimensions: { situation: ["external"] }`** (example) to restrict literals stored on cells.
160
+ 3. **`groupBy: ["subjectType", "subject"]`** so **each row aggregates one dictionary subject id**.
161
+ 4. **`rows.length`** after the query = number of distinct actors **with non-zero merged activity** in that slice (sparse — actors who never appeared produce no row).
162
+
163
+ **Critical:** if you set **`limit`**, **`rows.length ≤ limit`** and **`result.total`** only sum **returned** rows — you **cannot** infer global distinct counts from a capped query.
164
+
165
+ ### Logical “one row per source event” totals
166
+
167
+ Maintain a **separate counter map** engineered for **one increment per deduped message** (single timeframe granularity, single synthetic metric field, minimal subjects). See [`docs/use-cases/audit-security-log-counts.md`](../use-cases/audit-security-log-counts.md) §10.6.
168
+
169
+ ### Metric and subject filters at rest
170
+
171
+ `executeQuery` runs **`normalizeQueryDictionaryFilters`** first: any **string** entries in **`subject.ids`** / **`metrics`** are translated to internal ids via **`lookupIds`** (read-only). The spec passed to **`readCells`** therefore uses **numeric ids only**, so Mongo / memory index filters apply correctly. If every string key was unknown, **`ids` / `metrics` may become empty arrays**, which match **no** cells (same as an impossible id list).
172
+
173
+ ## See also
174
+
175
+ - [Measures](./measures.md)
176
+ - [Timeframes](./timeframes-and-bucketing.md)
177
+ - [Audit use case — analytics recipes](../use-cases/audit-security-log-counts.md) §10
178
+ - Internal: [Query path](../internals/internal-query-path.md)
@@ -1,33 +1,33 @@
1
- # Guide: Simulation profiles
2
-
3
- Profiles configure **what kind of events** the simulator emits and **how much fan-out** they produce.
4
-
5
- ## Built-in profile ids
6
-
7
- | Profile id | Intent |
8
- |------------|--------|
9
- | `audit-log` | High-volume audit-style records: actors, resources, operations, optional label fan-out. |
10
- | `product-activity` | Product analytics–style events: account scope, user subject, named event metric, feature/plan dimensions. |
11
- | `processing-activity` | Jobs / pipelines / workflows: job id subject, task/step metrics, status dimensions, duration measures. |
12
- | `llm-usage` | Token and cost measures with model/provider dimensions. |
13
- | `custom` | Supply your own event factory; simulator handles scheduling and pools only. |
14
-
15
- ## Profile options (conceptual)
16
-
17
- - **`durationMs`** — simulated wall-clock span of the run (logical time advancement between events).
18
- - **`eventsPerTick`** — throughput knob for batch generators.
19
- - **`poolSizes`** — caps on distinct subjects/metrics/dimension values (cardinality stress).
20
- - **`weights`** — skew toward hot keys (Pareto-style) vs uniform draws.
21
-
22
- Exact fields match `SimulationProfile` in `@x12i/countex/simulation` TypeScript exports.
23
-
24
- ## Choosing a profile
25
-
26
- - Mirror the **same map** you plan for production; mismatched profiles hide JSONPath bugs.
27
- - Start **uniform** pools, then add **skew** to reproduce hotspot behavior.
28
- - Combine with **`validateMap(map, samples)`** from core so paths resolve on representative payloads.
29
-
30
- ## See also
31
-
32
- - [Simulation overview](./simulation.md)
33
- - [Simulation resource estimation](./simulation-resource-estimation.md)
1
+ # Guide: Simulation profiles
2
+
3
+ Profiles configure **what kind of events** the simulator emits and **how much fan-out** they produce.
4
+
5
+ ## Built-in profile ids
6
+
7
+ | Profile id | Intent |
8
+ |------------|--------|
9
+ | `audit-log` | High-volume audit-style records: actors, resources, operations, optional label fan-out. |
10
+ | `product-activity` | Product analytics–style events: account scope, user subject, named event metric, feature/plan dimensions. |
11
+ | `processing-activity` | Jobs / pipelines / workflows: job id subject, task/step metrics, status dimensions, duration measures. |
12
+ | `llm-usage` | Token and cost measures with model/provider dimensions. |
13
+ | `custom` | Supply your own event factory; simulator handles scheduling and pools only. |
14
+
15
+ ## Profile options (conceptual)
16
+
17
+ - **`durationMs`** — simulated wall-clock span of the run (logical time advancement between events).
18
+ - **`eventsPerTick`** — throughput knob for batch generators.
19
+ - **`poolSizes`** — caps on distinct subjects/metrics/dimension values (cardinality stress).
20
+ - **`weights`** — skew toward hot keys (Pareto-style) vs uniform draws.
21
+
22
+ Exact fields match `SimulationProfile` in `@x12i/countex/simulation` TypeScript exports.
23
+
24
+ ## Choosing a profile
25
+
26
+ - Mirror the **same map** you plan for production; mismatched profiles hide JSONPath bugs.
27
+ - Start **uniform** pools, then add **skew** to reproduce hotspot behavior.
28
+ - Combine with **`validateMap(map, samples)`** from core so paths resolve on representative payloads.
29
+
30
+ ## See also
31
+
32
+ - [Simulation overview](./simulation.md)
33
+ - [Simulation resource estimation](./simulation-resource-estimation.md)
@@ -1,30 +1,31 @@
1
- # Guide: Simulation resource estimation
2
-
3
- Use simulation outputs to reason about **Mongo**, **Redis**, and **process memory** before cutover.
4
-
5
- ## Signals from a run
6
-
7
- After a simulator run (or CLI report):
8
-
9
- - **Raw events** and **raw increments** (before cardinality drops) — multiply by your peak events/sec for ingress sizing.
10
- - **Distinct dictionary keys** per kind (`subject`, `metric`, `dimensionValue`) — drives Mongo dictionary collection growth.
11
- - **Distinct cells touched** — upper bound on sparse matrix cardinality for the window.
12
- - **Flush batch sizes** — depends on buffer settings (`maxBufferedCells`, flush interval).
13
-
14
- ## Mapping to infrastructure
15
-
16
- | Concern | What to measure |
17
- |---------|-----------------|
18
- | Mongo durable | Cell documents + dictionary docs; index sizes (see deployment guide). |
19
- | Redis | Working set for hot timeframes; TTL/retention policy per timeframe type. |
20
- | Application RAM | In-flight buffer + Node overhead; simulate worst-case burst with `onOverflow: "block"`. |
21
-
22
- ## Limits
23
-
24
- Simulation **does not** replace load testing of your MongoDB cluster or Redis failover behavior; it estimates **logical** counter volume from maps and profiles.
25
-
26
- ## See also
27
-
28
- - [Simulation overview](./simulation.md)
29
- - [Ingestion & flush](./ingestion-and-flush.md)
30
- - [`docs/DEPLOYMENT.md`](../DEPLOYMENT.md)
1
+ # Guide: Simulation resource estimation
2
+
3
+ Use simulation outputs to reason about **Mongo**, **Redis**, and **process memory** before cutover.
4
+
5
+ ## Signals from a run
6
+
7
+ After a simulator run (or CLI report):
8
+
9
+ - **Raw events** and **raw increments** (before cardinality drops) — multiply by your peak events/sec for ingress sizing.
10
+ - **Distinct dictionary keys** per kind (`subject`, `metric`, `dimensionValue`) — drives Mongo dictionary collection growth.
11
+ - **Distinct cells touched** — upper bound on sparse matrix cardinality for the window.
12
+ - **Flush batch sizes** — depends on buffer settings (`maxBufferedCells`, flush interval).
13
+
14
+ ## Mapping to infrastructure
15
+
16
+ | Concern | What to measure |
17
+ |---------|-----------------|
18
+ | Mongo durable | Cell documents + dictionary docs; index sizes (see deployment guide). Default planning constant: 750 bytes/cell. |
19
+ | Postgres durable | Packed cell rows + dict + indexes. Default planning constant: 250 bytes/cell (`postgresBytesPerCell`). Partition minute/hour and keep autovacuum aggressive. |
20
+ | Redis | Working set for hot timeframes; TTL/retention policy per timeframe type. |
21
+ | Application RAM | In-flight buffer + Node overhead; simulate worst-case burst with `onOverflow: "block"`. |
22
+
23
+ ## Limits
24
+
25
+ Simulation **does not** replace load testing of your MongoDB cluster or Redis failover behavior; it estimates **logical** counter volume from maps and profiles.
26
+
27
+ ## See also
28
+
29
+ - [Simulation overview](./simulation.md)
30
+ - [Ingestion & flush](./ingestion-and-flush.md)
31
+ - [`docs/DEPLOYMENT.md`](../DEPLOYMENT.md)
@@ -1,36 +1,36 @@
1
- # Guide: Simulation overview
2
-
3
- **Simulation** helps before production traffic exists: generate deterministic synthetic events, stress-test maps and cardinality, estimate storage and flush load, and validate that aggregates match expectations.
4
-
5
- ## Runtime entry point
6
-
7
- When using the published package:
8
-
9
- ```ts
10
- import { createCountexSimulator } from "@x12i/countex/simulation";
11
- ```
12
-
13
- The simulator is optional tooling on top of the core engine; it does not change ingest or query semantics.
14
-
15
- ## What simulation covers
16
-
17
- - **Seeded randomness** — reproducible runs for CI and regression tests.
18
- - **Pools & relations** — weighted entities (users, documents, models) with stable ids across events.
19
- - **Profiles** — named scenarios (audit-style logs, product activity, LLM metering, processing pipelines); see [Simulation profiles](./simulation-profiles.md).
20
- - **Expected aggregates** — optional tracker that compares simulated totals to counter reads for a window.
21
- - **Reports** — structured summary (events emitted, fan-out estimates, errors) for sizing discussions.
22
-
23
- ## Relationship to core Countex
24
-
25
- 1. Build or load a `CounterMap` (same as production).
26
- 2. Instantiate `createCountexSimulator({ map, profile, seed, ... })`.
27
- 3. Drive **`generateEvents(n)`** or **`run()`** into `countex.ingest` / `ingestBatch` in tests or benchmarks.
28
- 4. Optionally attach **`ExpectedAggregateTracker`** and assert after flush + query.
29
-
30
- See [Simulation resource estimation](./simulation-resource-estimation.md) for capacity hooks.
31
-
32
- ## See also
33
-
34
- - [CLI](./cli.md) — `countex simulate:demo` for scripted dry-runs without a config module
35
- - [Counter maps](./counter-maps.md)
36
- - [Cardinality](./cardinality.md)
1
+ # Guide: Simulation overview
2
+
3
+ **Simulation** helps before production traffic exists: generate deterministic synthetic events, stress-test maps and cardinality, estimate storage and flush load, and validate that aggregates match expectations.
4
+
5
+ ## Runtime entry point
6
+
7
+ When using the published package:
8
+
9
+ ```ts
10
+ import { createCountexSimulator } from "@x12i/countex/simulation";
11
+ ```
12
+
13
+ The simulator is optional tooling on top of the core engine; it does not change ingest or query semantics.
14
+
15
+ ## What simulation covers
16
+
17
+ - **Seeded randomness** — reproducible runs for CI and regression tests.
18
+ - **Pools & relations** — weighted entities (users, documents, models) with stable ids across events.
19
+ - **Profiles** — named scenarios (audit-style logs, product activity, LLM metering, processing pipelines); see [Simulation profiles](./simulation-profiles.md).
20
+ - **Expected aggregates** — optional tracker that compares simulated totals to counter reads for a window.
21
+ - **Reports** — structured summary (events emitted, fan-out estimates, errors) for sizing discussions.
22
+
23
+ ## Relationship to core Countex
24
+
25
+ 1. Build or load a `CounterMap` (same as production).
26
+ 2. Instantiate `createCountexSimulator({ map, profile, seed, ... })`.
27
+ 3. Drive **`generateEvents(n)`** or **`run()`** into `countex.ingest` / `ingestBatch` in tests or benchmarks.
28
+ 4. Optionally attach **`ExpectedAggregateTracker`** and assert after flush + query.
29
+
30
+ See [Simulation resource estimation](./simulation-resource-estimation.md) for capacity hooks.
31
+
32
+ ## See also
33
+
34
+ - [CLI](./cli.md) — `countex simulate:demo` for scripted dry-runs without a config module
35
+ - [Counter maps](./counter-maps.md)
36
+ - [Cardinality](./cardinality.md)