@x12i/countex 1.8.3 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +866 -858
- package/dist/cli.js +6 -6
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
- package/dist/insights/advisors/resource-advisor.js +5 -1
- package/dist/insights/advisors/resource-advisor.js.map +1 -1
- package/dist/insights/types.d.ts +4 -0
- package/dist/insights/types.d.ts.map +1 -1
- package/dist/storage/postgres.d.ts +104 -0
- package/dist/storage/postgres.d.ts.map +1 -0
- package/dist/storage/postgres.js +1023 -0
- package/dist/storage/postgres.js.map +1 -0
- package/dist/types/index.d.ts +11 -5
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +399 -375
- package/docs/END_TO_END_EXAMPLE.md +316 -316
- package/docs/INSIGHTS.md +59 -59
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
- package/docs/guides/README.md +36 -34
- package/docs/guides/adapter-memory.md +60 -60
- package/docs/guides/adapter-mongo.md +64 -62
- package/docs/guides/adapter-postgres.md +90 -0
- package/docs/guides/adapter-redis.md +89 -89
- package/docs/guides/backup-export.md +135 -135
- package/docs/guides/cardinality.md +50 -50
- package/docs/guides/cli.md +53 -53
- package/docs/guides/client-events.md +65 -65
- package/docs/guides/counter-maps.md +160 -160
- package/docs/guides/idempotency.md +39 -39
- package/docs/guides/ingestion-and-flush.md +67 -67
- package/docs/guides/insights-advisors-and-optimization.md +39 -39
- package/docs/guides/insights-anomalies-and-rules.md +48 -48
- package/docs/guides/insights-comparison-and-trends.md +47 -47
- package/docs/guides/insights-forecasting-and-budget.md +61 -61
- package/docs/guides/insights-overview.md +82 -82
- package/docs/guides/insights-ranking.md +81 -81
- package/docs/guides/measures.md +71 -71
- package/docs/guides/migrating-mongo-to-postgres.md +127 -0
- package/docs/guides/preview.md +92 -92
- package/docs/guides/querying.md +178 -178
- package/docs/guides/simulation-profiles.md +33 -33
- package/docs/guides/simulation-resource-estimation.md +31 -30
- package/docs/guides/simulation.md +36 -36
- package/docs/guides/timeframes-and-bucketing.md +55 -55
- package/docs/internals/README.md +21 -21
- package/docs/internals/internal-architecture-overview.md +65 -65
- package/docs/internals/internal-backup-stream.md +23 -23
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
- package/docs/internals/internal-cardinality.md +30 -30
- package/docs/internals/internal-dimension-hash.md +34 -34
- package/docs/internals/internal-idempotency.md +36 -36
- package/docs/internals/internal-ingest-pipeline.md +52 -52
- package/docs/internals/internal-insights-layer.md +36 -36
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
- package/docs/internals/internal-measures-and-scales.md +35 -35
- package/docs/internals/internal-query-path.md +46 -46
- package/docs/internals/internal-storage-memory.md +13 -13
- package/docs/internals/internal-storage-mongo.md +44 -44
- package/docs/internals/internal-storage-postgres.md +32 -0
- package/docs/internals/internal-storage-redis.md +63 -63
- package/docs/internals/internal-timeframe-bucketing.md +33 -33
- package/docs/use-cases/README.md +56 -56
- package/docs/use-cases/audit-security-log-counts.md +550 -550
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
- package/docs/use-cases/multi-tenant-isolation.md +203 -203
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
- package/docs/use-cases/processing-workflow-activity.md +165 -165
- package/docs/use-cases/product-feature-usage.md +240 -240
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
- package/package.json +184 -169
package/docs/guides/querying.md
CHANGED
|
@@ -1,178 +1,178 @@
|
|
|
1
|
-
# Guide: Querying
|
|
2
|
-
|
|
3
|
-
`countex.query(spec)` reads aggregated cells for a scope and counter, with optional filters and output shaping.
|
|
4
|
-
|
|
5
|
-
## Step 1: Minimal query
|
|
6
|
-
|
|
7
|
-
```ts
|
|
8
|
-
const result = await countex.query({
|
|
9
|
-
scope: "org_1",
|
|
10
|
-
counterId: "access_by_actor_label",
|
|
11
|
-
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
12
|
-
});
|
|
13
|
-
```
|
|
14
|
-
|
|
15
|
-
## Step 2: Filter
|
|
16
|
-
|
|
17
|
-
```ts
|
|
18
|
-
const result = await countex.query({
|
|
19
|
-
scope: "org_1",
|
|
20
|
-
counterId: "access_by_actor_label",
|
|
21
|
-
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
22
|
-
subject: { type: "actor", ids: [10001, 10002] },
|
|
23
|
-
metrics: [41, 42],
|
|
24
|
-
dimensions: { operation: ["read", "write"] },
|
|
25
|
-
});
|
|
26
|
-
```
|
|
27
|
-
|
|
28
|
-
**Subject and metric filters** accept **internal numeric ids** (as stored on cells) **and/or dictionary source keys** (strings: UUIDs, opaque ids, label text — whatever your map extracted at ingest). Strings are resolved read-only via `durable.lookupIds` before storage reads; **unknown keys are dropped** from the filter (they do not allocate dictionary rows). When any `subject.ids` entry is a string, **`subject.type` is required** so the lookup is scoped to that subject family. In TypeScript / JSON, **`number` means an internal id** and **`string` means a source key** — the literal `"10001"` is never treated as numeric id `10001`. If `subject.ids` or `metrics` is present as an **empty array**, the query matches **no cells** for that filter.
|
|
29
|
-
|
|
30
|
-
```ts
|
|
31
|
-
// Same slice using source keys from your domain (no need to remember internal ids)
|
|
32
|
-
const bySourceKeys = await countex.query({
|
|
33
|
-
scope: "org_1",
|
|
34
|
-
counterId: "access_by_actor_label",
|
|
35
|
-
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
36
|
-
subject: { type: "actor", ids: ["aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", "user_42"] },
|
|
37
|
-
metrics: ["PCI", "PII"],
|
|
38
|
-
dimensions: { operation: ["read", "write"] },
|
|
39
|
-
});
|
|
40
|
-
```
|
|
41
|
-
|
|
42
|
-
## Human-readable labels (`resolveLabels`)
|
|
43
|
-
|
|
44
|
-
Cells always store **numeric** `subjectId` / `metricId`. For grouped results and dashboards, set **`resolveLabels: true`** so `groupBy` keys for **`subject`** and **`metric`** use **source strings** where the dictionary has a reverse mapping; missing entries fall back to the numeric id as a string. With **`format: "cells"`**, each row also includes **`subjectSourceKey`** and **`metricSourceKey`**. This uses **`durable.reverseLookupIds`** on the ids present in the result set (extra durable reads; omit when you only need ids).
|
|
45
|
-
|
|
46
|
-
```ts
|
|
47
|
-
const labeled = await countex.query({
|
|
48
|
-
scope: "org_1",
|
|
49
|
-
counterId: "access_by_actor_label",
|
|
50
|
-
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
51
|
-
subject: { type: "actor", ids: ["aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"] },
|
|
52
|
-
metrics: ["read"],
|
|
53
|
-
groupBy: ["subject", "metric", "dim:operation"],
|
|
54
|
-
format: "table",
|
|
55
|
-
resolveLabels: true,
|
|
56
|
-
});
|
|
57
|
-
```
|
|
58
|
-
|
|
59
|
-
## Step 3: Group and format
|
|
60
|
-
|
|
61
|
-
`groupBy` accepts:
|
|
62
|
-
|
|
63
|
-
- `subject`, `subjectType`, `metric`, `timeframeKey`
|
|
64
|
-
- `dim:NAME` for a dimension
|
|
65
|
-
|
|
66
|
-
`format`:
|
|
67
|
-
|
|
68
|
-
- `cells` — raw-ish cell list
|
|
69
|
-
- `table` / `matrix` — aggregated rows
|
|
70
|
-
- `timeseries` — points per bucket (and optional extra grouping)
|
|
71
|
-
|
|
72
|
-
```ts
|
|
73
|
-
const result = await countex.query({
|
|
74
|
-
scope: "org_1",
|
|
75
|
-
counterId: "access_by_actor_label",
|
|
76
|
-
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
77
|
-
groupBy: ["subject", "dim:operation"],
|
|
78
|
-
format: "table",
|
|
79
|
-
orderBy: "-count",
|
|
80
|
-
limit: 50,
|
|
81
|
-
});
|
|
82
|
-
```
|
|
83
|
-
|
|
84
|
-
## Step 4: Measures and rates
|
|
85
|
-
|
|
86
|
-
```ts
|
|
87
|
-
await countex.query({
|
|
88
|
-
// ...
|
|
89
|
-
measures: ["cost"],
|
|
90
|
-
orderBy: { measure: "cost", agg: "sum", dir: "desc" },
|
|
91
|
-
rate: { measure: "cost", agg: "sum" },
|
|
92
|
-
});
|
|
93
|
-
```
|
|
94
|
-
|
|
95
|
-
Do not use `rate` with timeframe **`all`**.
|
|
96
|
-
|
|
97
|
-
## Step 5: Cache control
|
|
98
|
-
|
|
99
|
-
- `bypassCache: true` forces durable reads.
|
|
100
|
-
- `signal` accepts `AbortSignal` for timeouts.
|
|
101
|
-
|
|
102
|
-
Result includes `source`: `"cache"`, `"durable"`, or `"merged"`.
|
|
103
|
-
|
|
104
|
-
### Query diagnostics
|
|
105
|
-
|
|
106
|
-
`result.diagnostics` (when present) adds planner detail:
|
|
107
|
-
|
|
108
|
-
| Field | Meaning |
|
|
109
|
-
|-------|---------|
|
|
110
|
-
| `cacheAttempted` | `true` if the engine consulted the cache adapter (cache configured, not `bypassCache`, and the planner allowed cache for this query shape). |
|
|
111
|
-
| `cacheCoverage` | When `cacheAttempted` is true: `"full"` if the cache answered completely; `"partial"` or `"miss"` if the planner fell through to durable for authoritative cells. |
|
|
112
|
-
| `cellsMaterialized` | Number of counter cells loaded from storage before grouping/formatting. |
|
|
113
|
-
| `bucketsRequested` | Bucket keys the query expanded to (after `bucketRange`). |
|
|
114
|
-
| `cacheBucketsHit` | Number of buckets the cache reported as `complete`. |
|
|
115
|
-
| `durableFallback` | `true` when the planner had to read durable for some/all buckets. |
|
|
116
|
-
| `plannedCellsEstimate` | Heuristic estimate of cells before grouping (matches `cellsMaterialized` after execution). |
|
|
117
|
-
| `groupByCost` | `"trivial"` (no groupBy), `"low"`, or `"high"` (3+ keys or 2+ dimensions). |
|
|
118
|
-
| `expensive` | Heuristic flag — `true` for big windows, high groupByCost, or `format: "matrix"` with many cells. |
|
|
119
|
-
|
|
120
|
-
Use this for tests, dashboards, and verifying Redis vs Mongo paths.
|
|
121
|
-
|
|
122
|
-
### Explain a query without running it
|
|
123
|
-
|
|
124
|
-
`countex.explainQuery(spec)` returns a {@link QueryPlan} without touching durable storage and only probing per-bucket cache coverage:
|
|
125
|
-
|
|
126
|
-
```ts
|
|
127
|
-
const plan = await countex.explainQuery({
|
|
128
|
-
scope: "org_1",
|
|
129
|
-
counterId: "events",
|
|
130
|
-
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-31" },
|
|
131
|
-
groupBy: ["dim:operation", "dim:situation", "subject"],
|
|
132
|
-
});
|
|
133
|
-
// {
|
|
134
|
-
// bucketsRequested: ["2026-05-01", …, "2026-05-31"],
|
|
135
|
-
// cacheEligible: true,
|
|
136
|
-
// cacheBucketsExpectedHit: 18,
|
|
137
|
-
// durableFallbackPredicted: true,
|
|
138
|
-
// plannedCellsEstimate: 1_240,
|
|
139
|
-
// groupByCost: "high",
|
|
140
|
-
// expensive: true,
|
|
141
|
-
// }
|
|
142
|
-
```
|
|
143
|
-
|
|
144
|
-
`cacheBucketsExpectedHit` and `plannedCellsEstimate` are populated when the cache adapter implements `probeBucketCoverage` (the bundled `RedisCache` and `MemoryCache` both do).
|
|
145
|
-
|
|
146
|
-
## Step 6: Totals, event volume, and “distinct entities”
|
|
147
|
-
|
|
148
|
-
### Grand total (sum all matching cells)
|
|
149
|
-
|
|
150
|
-
Omit **`groupBy`** (or use an empty list). For **`format: "table"`**, the engine merges every matching cell into **one row**; **`rows[0].count`** is the sum of **`count`** across those cells. **`result.total`** matches that sum for the returned rows (`sumRowCounts` in `src/query/formatter.ts`).
|
|
151
|
-
|
|
152
|
-
Use this for **total increment volume** in the query window. If one source event fans out into many cells (multiple subjects × metrics × timeframes), this total is **not** the same as “number of source events.”
|
|
153
|
-
|
|
154
|
-
### Distinct subjects / metrics as row cardinality
|
|
155
|
-
|
|
156
|
-
Countex does **not** store arbitrary **`COUNT(DISTINCT …)`** queries. **Exactly what to do** when you need “how many unique actors had activity?”:
|
|
157
|
-
|
|
158
|
-
1. **`subject: { type: "actor" }`** (optional **`ids`**) to restrict the subject family.
|
|
159
|
-
2. **`dimensions: { situation: ["external"] }`** (example) to restrict literals stored on cells.
|
|
160
|
-
3. **`groupBy: ["subjectType", "subject"]`** so **each row aggregates one dictionary subject id**.
|
|
161
|
-
4. **`rows.length`** after the query = number of distinct actors **with non-zero merged activity** in that slice (sparse — actors who never appeared produce no row).
|
|
162
|
-
|
|
163
|
-
**Critical:** if you set **`limit`**, **`rows.length ≤ limit`** and **`result.total`** only sum **returned** rows — you **cannot** infer global distinct counts from a capped query.
|
|
164
|
-
|
|
165
|
-
### Logical “one row per source event” totals
|
|
166
|
-
|
|
167
|
-
Maintain a **separate counter map** engineered for **one increment per deduped message** (single timeframe granularity, single synthetic metric field, minimal subjects). See [`docs/use-cases/audit-security-log-counts.md`](../use-cases/audit-security-log-counts.md) §10.6.
|
|
168
|
-
|
|
169
|
-
### Metric and subject filters at rest
|
|
170
|
-
|
|
171
|
-
`executeQuery` runs **`normalizeQueryDictionaryFilters`** first: any **string** entries in **`subject.ids`** / **`metrics`** are translated to internal ids via **`lookupIds`** (read-only). The spec passed to **`readCells`** therefore uses **numeric ids only**, so Mongo / memory index filters apply correctly. If every string key was unknown, **`ids` / `metrics` may become empty arrays**, which match **no** cells (same as an impossible id list).
|
|
172
|
-
|
|
173
|
-
## See also
|
|
174
|
-
|
|
175
|
-
- [Measures](./measures.md)
|
|
176
|
-
- [Timeframes](./timeframes-and-bucketing.md)
|
|
177
|
-
- [Audit use case — analytics recipes](../use-cases/audit-security-log-counts.md) §10
|
|
178
|
-
- Internal: [Query path](../internals/internal-query-path.md)
|
|
1
|
+
# Guide: Querying
|
|
2
|
+
|
|
3
|
+
`countex.query(spec)` reads aggregated cells for a scope and counter, with optional filters and output shaping.
|
|
4
|
+
|
|
5
|
+
## Step 1: Minimal query
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const result = await countex.query({
|
|
9
|
+
scope: "org_1",
|
|
10
|
+
counterId: "access_by_actor_label",
|
|
11
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
12
|
+
});
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Step 2: Filter
|
|
16
|
+
|
|
17
|
+
```ts
|
|
18
|
+
const result = await countex.query({
|
|
19
|
+
scope: "org_1",
|
|
20
|
+
counterId: "access_by_actor_label",
|
|
21
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
22
|
+
subject: { type: "actor", ids: [10001, 10002] },
|
|
23
|
+
metrics: [41, 42],
|
|
24
|
+
dimensions: { operation: ["read", "write"] },
|
|
25
|
+
});
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
**Subject and metric filters** accept **internal numeric ids** (as stored on cells) **and/or dictionary source keys** (strings: UUIDs, opaque ids, label text — whatever your map extracted at ingest). Strings are resolved read-only via `durable.lookupIds` before storage reads; **unknown keys are dropped** from the filter (they do not allocate dictionary rows). When any `subject.ids` entry is a string, **`subject.type` is required** so the lookup is scoped to that subject family. In TypeScript / JSON, **`number` means an internal id** and **`string` means a source key** — the literal `"10001"` is never treated as numeric id `10001`. If `subject.ids` or `metrics` is present as an **empty array**, the query matches **no cells** for that filter.
|
|
29
|
+
|
|
30
|
+
```ts
|
|
31
|
+
// Same slice using source keys from your domain (no need to remember internal ids)
|
|
32
|
+
const bySourceKeys = await countex.query({
|
|
33
|
+
scope: "org_1",
|
|
34
|
+
counterId: "access_by_actor_label",
|
|
35
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
36
|
+
subject: { type: "actor", ids: ["aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee", "user_42"] },
|
|
37
|
+
metrics: ["PCI", "PII"],
|
|
38
|
+
dimensions: { operation: ["read", "write"] },
|
|
39
|
+
});
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Human-readable labels (`resolveLabels`)
|
|
43
|
+
|
|
44
|
+
Cells always store **numeric** `subjectId` / `metricId`. For grouped results and dashboards, set **`resolveLabels: true`** so `groupBy` keys for **`subject`** and **`metric`** use **source strings** where the dictionary has a reverse mapping; missing entries fall back to the numeric id as a string. With **`format: "cells"`**, each row also includes **`subjectSourceKey`** and **`metricSourceKey`**. This uses **`durable.reverseLookupIds`** on the ids present in the result set (extra durable reads; omit when you only need ids).
|
|
45
|
+
|
|
46
|
+
```ts
|
|
47
|
+
const labeled = await countex.query({
|
|
48
|
+
scope: "org_1",
|
|
49
|
+
counterId: "access_by_actor_label",
|
|
50
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
51
|
+
subject: { type: "actor", ids: ["aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee"] },
|
|
52
|
+
metrics: ["read"],
|
|
53
|
+
groupBy: ["subject", "metric", "dim:operation"],
|
|
54
|
+
format: "table",
|
|
55
|
+
resolveLabels: true,
|
|
56
|
+
});
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
## Step 3: Group and format
|
|
60
|
+
|
|
61
|
+
`groupBy` accepts:
|
|
62
|
+
|
|
63
|
+
- `subject`, `subjectType`, `metric`, `timeframeKey`
|
|
64
|
+
- `dim:NAME` for a dimension
|
|
65
|
+
|
|
66
|
+
`format`:
|
|
67
|
+
|
|
68
|
+
- `cells` — raw-ish cell list
|
|
69
|
+
- `table` / `matrix` — aggregated rows
|
|
70
|
+
- `timeseries` — points per bucket (and optional extra grouping)
|
|
71
|
+
|
|
72
|
+
```ts
|
|
73
|
+
const result = await countex.query({
|
|
74
|
+
scope: "org_1",
|
|
75
|
+
counterId: "access_by_actor_label",
|
|
76
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
77
|
+
groupBy: ["subject", "dim:operation"],
|
|
78
|
+
format: "table",
|
|
79
|
+
orderBy: "-count",
|
|
80
|
+
limit: 50,
|
|
81
|
+
});
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
## Step 4: Measures and rates
|
|
85
|
+
|
|
86
|
+
```ts
|
|
87
|
+
await countex.query({
|
|
88
|
+
// ...
|
|
89
|
+
measures: ["cost"],
|
|
90
|
+
orderBy: { measure: "cost", agg: "sum", dir: "desc" },
|
|
91
|
+
rate: { measure: "cost", agg: "sum" },
|
|
92
|
+
});
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
Do not use `rate` with timeframe **`all`**.
|
|
96
|
+
|
|
97
|
+
## Step 5: Cache control
|
|
98
|
+
|
|
99
|
+
- `bypassCache: true` forces durable reads.
|
|
100
|
+
- `signal` accepts `AbortSignal` for timeouts.
|
|
101
|
+
|
|
102
|
+
Result includes `source`: `"cache"`, `"durable"`, or `"merged"`.
|
|
103
|
+
|
|
104
|
+
### Query diagnostics
|
|
105
|
+
|
|
106
|
+
`result.diagnostics` (when present) adds planner detail:
|
|
107
|
+
|
|
108
|
+
| Field | Meaning |
|
|
109
|
+
|-------|---------|
|
|
110
|
+
| `cacheAttempted` | `true` if the engine consulted the cache adapter (cache configured, not `bypassCache`, and the planner allowed cache for this query shape). |
|
|
111
|
+
| `cacheCoverage` | When `cacheAttempted` is true: `"full"` if the cache answered completely; `"partial"` or `"miss"` if the planner fell through to durable for authoritative cells. |
|
|
112
|
+
| `cellsMaterialized` | Number of counter cells loaded from storage before grouping/formatting. |
|
|
113
|
+
| `bucketsRequested` | Bucket keys the query expanded to (after `bucketRange`). |
|
|
114
|
+
| `cacheBucketsHit` | Number of buckets the cache reported as `complete`. |
|
|
115
|
+
| `durableFallback` | `true` when the planner had to read durable for some/all buckets. |
|
|
116
|
+
| `plannedCellsEstimate` | Heuristic estimate of cells before grouping (matches `cellsMaterialized` after execution). |
|
|
117
|
+
| `groupByCost` | `"trivial"` (no groupBy), `"low"`, or `"high"` (3+ keys or 2+ dimensions). |
|
|
118
|
+
| `expensive` | Heuristic flag — `true` for big windows, high groupByCost, or `format: "matrix"` with many cells. |
|
|
119
|
+
|
|
120
|
+
Use this for tests, dashboards, and verifying Redis vs Mongo paths.
|
|
121
|
+
|
|
122
|
+
### Explain a query without running it
|
|
123
|
+
|
|
124
|
+
`countex.explainQuery(spec)` returns a {@link QueryPlan} without touching durable storage and only probing per-bucket cache coverage:
|
|
125
|
+
|
|
126
|
+
```ts
|
|
127
|
+
const plan = await countex.explainQuery({
|
|
128
|
+
scope: "org_1",
|
|
129
|
+
counterId: "events",
|
|
130
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-31" },
|
|
131
|
+
groupBy: ["dim:operation", "dim:situation", "subject"],
|
|
132
|
+
});
|
|
133
|
+
// {
|
|
134
|
+
// bucketsRequested: ["2026-05-01", …, "2026-05-31"],
|
|
135
|
+
// cacheEligible: true,
|
|
136
|
+
// cacheBucketsExpectedHit: 18,
|
|
137
|
+
// durableFallbackPredicted: true,
|
|
138
|
+
// plannedCellsEstimate: 1_240,
|
|
139
|
+
// groupByCost: "high",
|
|
140
|
+
// expensive: true,
|
|
141
|
+
// }
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
`cacheBucketsExpectedHit` and `plannedCellsEstimate` are populated when the cache adapter implements `probeBucketCoverage` (the bundled `RedisCache` and `MemoryCache` both do).
|
|
145
|
+
|
|
146
|
+
## Step 6: Totals, event volume, and “distinct entities”
|
|
147
|
+
|
|
148
|
+
### Grand total (sum all matching cells)
|
|
149
|
+
|
|
150
|
+
Omit **`groupBy`** (or use an empty list). For **`format: "table"`**, the engine merges every matching cell into **one row**; **`rows[0].count`** is the sum of **`count`** across those cells. **`result.total`** matches that sum for the returned rows (`sumRowCounts` in `src/query/formatter.ts`).
|
|
151
|
+
|
|
152
|
+
Use this for **total increment volume** in the query window. If one source event fans out into many cells (multiple subjects × metrics × timeframes), this total is **not** the same as “number of source events.”
|
|
153
|
+
|
|
154
|
+
### Distinct subjects / metrics as row cardinality
|
|
155
|
+
|
|
156
|
+
Countex does **not** store arbitrary **`COUNT(DISTINCT …)`** queries. **Exactly what to do** when you need “how many unique actors had activity?”:
|
|
157
|
+
|
|
158
|
+
1. **`subject: { type: "actor" }`** (optional **`ids`**) to restrict the subject family.
|
|
159
|
+
2. **`dimensions: { situation: ["external"] }`** (example) to restrict literals stored on cells.
|
|
160
|
+
3. **`groupBy: ["subjectType", "subject"]`** so **each row aggregates one dictionary subject id**.
|
|
161
|
+
4. **`rows.length`** after the query = number of distinct actors **with non-zero merged activity** in that slice (sparse — actors who never appeared produce no row).
|
|
162
|
+
|
|
163
|
+
**Critical:** if you set **`limit`**, **`rows.length ≤ limit`** and **`result.total`** only sum **returned** rows — you **cannot** infer global distinct counts from a capped query.
|
|
164
|
+
|
|
165
|
+
### Logical “one row per source event” totals
|
|
166
|
+
|
|
167
|
+
Maintain a **separate counter map** engineered for **one increment per deduped message** (single timeframe granularity, single synthetic metric field, minimal subjects). See [`docs/use-cases/audit-security-log-counts.md`](../use-cases/audit-security-log-counts.md) §10.6.
|
|
168
|
+
|
|
169
|
+
### Metric and subject filters at rest
|
|
170
|
+
|
|
171
|
+
`executeQuery` runs **`normalizeQueryDictionaryFilters`** first: any **string** entries in **`subject.ids`** / **`metrics`** are translated to internal ids via **`lookupIds`** (read-only). The spec passed to **`readCells`** therefore uses **numeric ids only**, so Mongo / memory index filters apply correctly. If every string key was unknown, **`ids` / `metrics` may become empty arrays**, which match **no** cells (same as an impossible id list).
|
|
172
|
+
|
|
173
|
+
## See also
|
|
174
|
+
|
|
175
|
+
- [Measures](./measures.md)
|
|
176
|
+
- [Timeframes](./timeframes-and-bucketing.md)
|
|
177
|
+
- [Audit use case — analytics recipes](../use-cases/audit-security-log-counts.md) §10
|
|
178
|
+
- Internal: [Query path](../internals/internal-query-path.md)
|
|
@@ -1,33 +1,33 @@
|
|
|
1
|
-
# Guide: Simulation profiles
|
|
2
|
-
|
|
3
|
-
Profiles configure **what kind of events** the simulator emits and **how much fan-out** they produce.
|
|
4
|
-
|
|
5
|
-
## Built-in profile ids
|
|
6
|
-
|
|
7
|
-
| Profile id | Intent |
|
|
8
|
-
|------------|--------|
|
|
9
|
-
| `audit-log` | High-volume audit-style records: actors, resources, operations, optional label fan-out. |
|
|
10
|
-
| `product-activity` | Product analytics–style events: account scope, user subject, named event metric, feature/plan dimensions. |
|
|
11
|
-
| `processing-activity` | Jobs / pipelines / workflows: job id subject, task/step metrics, status dimensions, duration measures. |
|
|
12
|
-
| `llm-usage` | Token and cost measures with model/provider dimensions. |
|
|
13
|
-
| `custom` | Supply your own event factory; simulator handles scheduling and pools only. |
|
|
14
|
-
|
|
15
|
-
## Profile options (conceptual)
|
|
16
|
-
|
|
17
|
-
- **`durationMs`** — simulated wall-clock span of the run (logical time advancement between events).
|
|
18
|
-
- **`eventsPerTick`** — throughput knob for batch generators.
|
|
19
|
-
- **`poolSizes`** — caps on distinct subjects/metrics/dimension values (cardinality stress).
|
|
20
|
-
- **`weights`** — skew toward hot keys (Pareto-style) vs uniform draws.
|
|
21
|
-
|
|
22
|
-
Exact fields match `SimulationProfile` in `@x12i/countex/simulation` TypeScript exports.
|
|
23
|
-
|
|
24
|
-
## Choosing a profile
|
|
25
|
-
|
|
26
|
-
- Mirror the **same map** you plan for production; mismatched profiles hide JSONPath bugs.
|
|
27
|
-
- Start **uniform** pools, then add **skew** to reproduce hotspot behavior.
|
|
28
|
-
- Combine with **`validateMap(map, samples)`** from core so paths resolve on representative payloads.
|
|
29
|
-
|
|
30
|
-
## See also
|
|
31
|
-
|
|
32
|
-
- [Simulation overview](./simulation.md)
|
|
33
|
-
- [Simulation resource estimation](./simulation-resource-estimation.md)
|
|
1
|
+
# Guide: Simulation profiles
|
|
2
|
+
|
|
3
|
+
Profiles configure **what kind of events** the simulator emits and **how much fan-out** they produce.
|
|
4
|
+
|
|
5
|
+
## Built-in profile ids
|
|
6
|
+
|
|
7
|
+
| Profile id | Intent |
|
|
8
|
+
|------------|--------|
|
|
9
|
+
| `audit-log` | High-volume audit-style records: actors, resources, operations, optional label fan-out. |
|
|
10
|
+
| `product-activity` | Product analytics–style events: account scope, user subject, named event metric, feature/plan dimensions. |
|
|
11
|
+
| `processing-activity` | Jobs / pipelines / workflows: job id subject, task/step metrics, status dimensions, duration measures. |
|
|
12
|
+
| `llm-usage` | Token and cost measures with model/provider dimensions. |
|
|
13
|
+
| `custom` | Supply your own event factory; simulator handles scheduling and pools only. |
|
|
14
|
+
|
|
15
|
+
## Profile options (conceptual)
|
|
16
|
+
|
|
17
|
+
- **`durationMs`** — simulated wall-clock span of the run (logical time advancement between events).
|
|
18
|
+
- **`eventsPerTick`** — throughput knob for batch generators.
|
|
19
|
+
- **`poolSizes`** — caps on distinct subjects/metrics/dimension values (cardinality stress).
|
|
20
|
+
- **`weights`** — skew toward hot keys (Pareto-style) vs uniform draws.
|
|
21
|
+
|
|
22
|
+
Exact fields match `SimulationProfile` in `@x12i/countex/simulation` TypeScript exports.
|
|
23
|
+
|
|
24
|
+
## Choosing a profile
|
|
25
|
+
|
|
26
|
+
- Mirror the **same map** you plan for production; mismatched profiles hide JSONPath bugs.
|
|
27
|
+
- Start **uniform** pools, then add **skew** to reproduce hotspot behavior.
|
|
28
|
+
- Combine with **`validateMap(map, samples)`** from core so paths resolve on representative payloads.
|
|
29
|
+
|
|
30
|
+
## See also
|
|
31
|
+
|
|
32
|
+
- [Simulation overview](./simulation.md)
|
|
33
|
+
- [Simulation resource estimation](./simulation-resource-estimation.md)
|
|
@@ -1,30 +1,31 @@
|
|
|
1
|
-
# Guide: Simulation resource estimation
|
|
2
|
-
|
|
3
|
-
Use simulation outputs to reason about **Mongo**, **Redis**, and **process memory** before cutover.
|
|
4
|
-
|
|
5
|
-
## Signals from a run
|
|
6
|
-
|
|
7
|
-
After a simulator run (or CLI report):
|
|
8
|
-
|
|
9
|
-
- **Raw events** and **raw increments** (before cardinality drops) — multiply by your peak events/sec for ingress sizing.
|
|
10
|
-
- **Distinct dictionary keys** per kind (`subject`, `metric`, `dimensionValue`) — drives Mongo dictionary collection growth.
|
|
11
|
-
- **Distinct cells touched** — upper bound on sparse matrix cardinality for the window.
|
|
12
|
-
- **Flush batch sizes** — depends on buffer settings (`maxBufferedCells`, flush interval).
|
|
13
|
-
|
|
14
|
-
## Mapping to infrastructure
|
|
15
|
-
|
|
16
|
-
| Concern | What to measure |
|
|
17
|
-
|---------|-----------------|
|
|
18
|
-
| Mongo durable | Cell documents + dictionary docs; index sizes (see deployment guide). |
|
|
19
|
-
|
|
|
20
|
-
|
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
- [
|
|
30
|
-
- [
|
|
1
|
+
# Guide: Simulation resource estimation
|
|
2
|
+
|
|
3
|
+
Use simulation outputs to reason about **Mongo**, **Redis**, and **process memory** before cutover.
|
|
4
|
+
|
|
5
|
+
## Signals from a run
|
|
6
|
+
|
|
7
|
+
After a simulator run (or CLI report):
|
|
8
|
+
|
|
9
|
+
- **Raw events** and **raw increments** (before cardinality drops) — multiply by your peak events/sec for ingress sizing.
|
|
10
|
+
- **Distinct dictionary keys** per kind (`subject`, `metric`, `dimensionValue`) — drives Mongo dictionary collection growth.
|
|
11
|
+
- **Distinct cells touched** — upper bound on sparse matrix cardinality for the window.
|
|
12
|
+
- **Flush batch sizes** — depends on buffer settings (`maxBufferedCells`, flush interval).
|
|
13
|
+
|
|
14
|
+
## Mapping to infrastructure
|
|
15
|
+
|
|
16
|
+
| Concern | What to measure |
|
|
17
|
+
|---------|-----------------|
|
|
18
|
+
| Mongo durable | Cell documents + dictionary docs; index sizes (see deployment guide). Default planning constant: 750 bytes/cell. |
|
|
19
|
+
| Postgres durable | Packed cell rows + dict + indexes. Default planning constant: 250 bytes/cell (`postgresBytesPerCell`). Partition minute/hour and keep autovacuum aggressive. |
|
|
20
|
+
| Redis | Working set for hot timeframes; TTL/retention policy per timeframe type. |
|
|
21
|
+
| Application RAM | In-flight buffer + Node overhead; simulate worst-case burst with `onOverflow: "block"`. |
|
|
22
|
+
|
|
23
|
+
## Limits
|
|
24
|
+
|
|
25
|
+
Simulation **does not** replace load testing of your MongoDB cluster or Redis failover behavior; it estimates **logical** counter volume from maps and profiles.
|
|
26
|
+
|
|
27
|
+
## See also
|
|
28
|
+
|
|
29
|
+
- [Simulation overview](./simulation.md)
|
|
30
|
+
- [Ingestion & flush](./ingestion-and-flush.md)
|
|
31
|
+
- [`docs/DEPLOYMENT.md`](../DEPLOYMENT.md)
|
|
@@ -1,36 +1,36 @@
|
|
|
1
|
-
# Guide: Simulation overview
|
|
2
|
-
|
|
3
|
-
**Simulation** helps before production traffic exists: generate deterministic synthetic events, stress-test maps and cardinality, estimate storage and flush load, and validate that aggregates match expectations.
|
|
4
|
-
|
|
5
|
-
## Runtime entry point
|
|
6
|
-
|
|
7
|
-
When using the published package:
|
|
8
|
-
|
|
9
|
-
```ts
|
|
10
|
-
import { createCountexSimulator } from "@x12i/countex/simulation";
|
|
11
|
-
```
|
|
12
|
-
|
|
13
|
-
The simulator is optional tooling on top of the core engine; it does not change ingest or query semantics.
|
|
14
|
-
|
|
15
|
-
## What simulation covers
|
|
16
|
-
|
|
17
|
-
- **Seeded randomness** — reproducible runs for CI and regression tests.
|
|
18
|
-
- **Pools & relations** — weighted entities (users, documents, models) with stable ids across events.
|
|
19
|
-
- **Profiles** — named scenarios (audit-style logs, product activity, LLM metering, processing pipelines); see [Simulation profiles](./simulation-profiles.md).
|
|
20
|
-
- **Expected aggregates** — optional tracker that compares simulated totals to counter reads for a window.
|
|
21
|
-
- **Reports** — structured summary (events emitted, fan-out estimates, errors) for sizing discussions.
|
|
22
|
-
|
|
23
|
-
## Relationship to core Countex
|
|
24
|
-
|
|
25
|
-
1. Build or load a `CounterMap` (same as production).
|
|
26
|
-
2. Instantiate `createCountexSimulator({ map, profile, seed, ... })`.
|
|
27
|
-
3. Drive **`generateEvents(n)`** or **`run()`** into `countex.ingest` / `ingestBatch` in tests or benchmarks.
|
|
28
|
-
4. Optionally attach **`ExpectedAggregateTracker`** and assert after flush + query.
|
|
29
|
-
|
|
30
|
-
See [Simulation resource estimation](./simulation-resource-estimation.md) for capacity hooks.
|
|
31
|
-
|
|
32
|
-
## See also
|
|
33
|
-
|
|
34
|
-
- [CLI](./cli.md) — `countex simulate:demo` for scripted dry-runs without a config module
|
|
35
|
-
- [Counter maps](./counter-maps.md)
|
|
36
|
-
- [Cardinality](./cardinality.md)
|
|
1
|
+
# Guide: Simulation overview
|
|
2
|
+
|
|
3
|
+
**Simulation** helps before production traffic exists: generate deterministic synthetic events, stress-test maps and cardinality, estimate storage and flush load, and validate that aggregates match expectations.
|
|
4
|
+
|
|
5
|
+
## Runtime entry point
|
|
6
|
+
|
|
7
|
+
When using the published package:
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
import { createCountexSimulator } from "@x12i/countex/simulation";
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The simulator is optional tooling on top of the core engine; it does not change ingest or query semantics.
|
|
14
|
+
|
|
15
|
+
## What simulation covers
|
|
16
|
+
|
|
17
|
+
- **Seeded randomness** — reproducible runs for CI and regression tests.
|
|
18
|
+
- **Pools & relations** — weighted entities (users, documents, models) with stable ids across events.
|
|
19
|
+
- **Profiles** — named scenarios (audit-style logs, product activity, LLM metering, processing pipelines); see [Simulation profiles](./simulation-profiles.md).
|
|
20
|
+
- **Expected aggregates** — optional tracker that compares simulated totals to counter reads for a window.
|
|
21
|
+
- **Reports** — structured summary (events emitted, fan-out estimates, errors) for sizing discussions.
|
|
22
|
+
|
|
23
|
+
## Relationship to core Countex
|
|
24
|
+
|
|
25
|
+
1. Build or load a `CounterMap` (same as production).
|
|
26
|
+
2. Instantiate `createCountexSimulator({ map, profile, seed, ... })`.
|
|
27
|
+
3. Drive **`generateEvents(n)`** or **`run()`** into `countex.ingest` / `ingestBatch` in tests or benchmarks.
|
|
28
|
+
4. Optionally attach **`ExpectedAggregateTracker`** and assert after flush + query.
|
|
29
|
+
|
|
30
|
+
See [Simulation resource estimation](./simulation-resource-estimation.md) for capacity hooks.
|
|
31
|
+
|
|
32
|
+
## See also
|
|
33
|
+
|
|
34
|
+
- [CLI](./cli.md) — `countex simulate:demo` for scripted dry-runs without a config module
|
|
35
|
+
- [Counter maps](./counter-maps.md)
|
|
36
|
+
- [Cardinality](./cardinality.md)
|