@x12i/countex 1.8.5 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +866 -860
- package/dist/cli.js +6 -6
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
- package/dist/insights/advisors/resource-advisor.js +5 -1
- package/dist/insights/advisors/resource-advisor.js.map +1 -1
- package/dist/insights/types.d.ts +4 -0
- package/dist/insights/types.d.ts.map +1 -1
- package/dist/storage/postgres.d.ts +104 -0
- package/dist/storage/postgres.d.ts.map +1 -0
- package/dist/storage/postgres.js +1023 -0
- package/dist/storage/postgres.js.map +1 -0
- package/dist/types/index.d.ts +11 -5
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +399 -375
- package/docs/END_TO_END_EXAMPLE.md +316 -316
- package/docs/INSIGHTS.md +59 -59
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
- package/docs/guides/README.md +36 -34
- package/docs/guides/adapter-memory.md +60 -60
- package/docs/guides/adapter-mongo.md +64 -62
- package/docs/guides/adapter-postgres.md +90 -0
- package/docs/guides/adapter-redis.md +89 -89
- package/docs/guides/backup-export.md +135 -135
- package/docs/guides/cardinality.md +50 -50
- package/docs/guides/cli.md +53 -53
- package/docs/guides/client-events.md +65 -65
- package/docs/guides/counter-maps.md +160 -160
- package/docs/guides/idempotency.md +39 -39
- package/docs/guides/ingestion-and-flush.md +67 -67
- package/docs/guides/insights-advisors-and-optimization.md +39 -39
- package/docs/guides/insights-anomalies-and-rules.md +48 -48
- package/docs/guides/insights-comparison-and-trends.md +47 -47
- package/docs/guides/insights-forecasting-and-budget.md +61 -61
- package/docs/guides/insights-overview.md +82 -82
- package/docs/guides/insights-ranking.md +81 -81
- package/docs/guides/measures.md +71 -71
- package/docs/guides/migrating-mongo-to-postgres.md +127 -0
- package/docs/guides/preview.md +92 -92
- package/docs/guides/querying.md +178 -178
- package/docs/guides/simulation-profiles.md +33 -33
- package/docs/guides/simulation-resource-estimation.md +31 -30
- package/docs/guides/simulation.md +36 -36
- package/docs/guides/timeframes-and-bucketing.md +55 -55
- package/docs/internals/README.md +21 -21
- package/docs/internals/internal-architecture-overview.md +65 -65
- package/docs/internals/internal-backup-stream.md +23 -23
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
- package/docs/internals/internal-cardinality.md +30 -30
- package/docs/internals/internal-dimension-hash.md +34 -34
- package/docs/internals/internal-idempotency.md +36 -36
- package/docs/internals/internal-ingest-pipeline.md +52 -52
- package/docs/internals/internal-insights-layer.md +36 -36
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
- package/docs/internals/internal-measures-and-scales.md +35 -35
- package/docs/internals/internal-query-path.md +46 -46
- package/docs/internals/internal-storage-memory.md +13 -13
- package/docs/internals/internal-storage-mongo.md +44 -44
- package/docs/internals/internal-storage-postgres.md +32 -0
- package/docs/internals/internal-storage-redis.md +63 -63
- package/docs/internals/internal-timeframe-bucketing.md +33 -33
- package/docs/use-cases/README.md +56 -56
- package/docs/use-cases/audit-security-log-counts.md +550 -550
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
- package/docs/use-cases/multi-tenant-isolation.md +203 -203
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
- package/docs/use-cases/processing-workflow-activity.md +165 -165
- package/docs/use-cases/product-feature-usage.md +240 -240
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
- package/package.json +184 -170
package/docs/guides/measures.md
CHANGED
|
@@ -1,71 +1,71 @@
|
|
|
1
|
-
# Guide: Measures
|
|
2
|
-
|
|
3
|
-
**Measures** are optional numeric fields per event (cost, latency, tokens, …) aggregated alongside counts.
|
|
4
|
-
|
|
5
|
-
## Step 1: Declare measures on the map
|
|
6
|
-
|
|
7
|
-
```ts
|
|
8
|
-
const map = {
|
|
9
|
-
counterId: "llm_usage",
|
|
10
|
-
scope: { path: "$.scope.id" },
|
|
11
|
-
eventTime: { path: "$.request.createdAt" },
|
|
12
|
-
subjects: [{ type: "app", path: "$.app.id", required: true }],
|
|
13
|
-
metrics: [{ path: "$.request.status", required: true }],
|
|
14
|
-
dimensions: [{ name: "model", path: "$.model.name" }],
|
|
15
|
-
delta: { default: 1 },
|
|
16
|
-
timeframes: ["day"],
|
|
17
|
-
measures: [
|
|
18
|
-
{ name: "cost", path: "$.usage.costCents", aggregations: ["sum"], scale: 100 },
|
|
19
|
-
{ name: "latencyMs", path: "$.usage.latencyMs", aggregations: ["sum", "count", "min", "max"] },
|
|
20
|
-
],
|
|
21
|
-
} as const;
|
|
22
|
-
```
|
|
23
|
-
|
|
24
|
-
### Aggregations
|
|
25
|
-
|
|
26
|
-
- **`sum`**, **`count`**, **`min`**, **`max`** are stored.
|
|
27
|
-
- **`avg`** is derived at query time when both **`sum`** and **`count`** exist for that measure.
|
|
28
|
-
|
|
29
|
-
### Scale
|
|
30
|
-
|
|
31
|
-
`scale` multiplies then truncates for fixed-point storage (for example cents → integer micros).
|
|
32
|
-
|
|
33
|
-
### Missing values
|
|
34
|
-
|
|
35
|
-
Use `onMissing: "skip"` (default) or `"drop"` / `"zero"` per measure.
|
|
36
|
-
|
|
37
|
-
### Unique / distinct counts are not additive sums
|
|
38
|
-
|
|
39
|
-
Countex **counts** and **measure aggregations** (`sum`, `count`, `min`, `max`) behave like standard additive counters over cells. They do **not** answer “how many **distinct** users appeared in this bucket?” unless you model that explicitly (for example one subject id per event and interpret cell counts accordingly). Do **not** sum counts across overlapping populations and treat the result as a unique count. For large-cardinality uniques, plan on a dedicated approach (for example optional future HLL-style storage — see deployment notes); vanilla cells are the wrong tool for deduplicated set cardinality across arbitrary queries.
|
|
40
|
-
|
|
41
|
-
## Step 2: Ingest as usual
|
|
42
|
-
|
|
43
|
-
Each fan-out increment carries the **same** measure snapshot for that event.
|
|
44
|
-
|
|
45
|
-
## Step 3: Query measures
|
|
46
|
-
|
|
47
|
-
```ts
|
|
48
|
-
const result = await countex.query({
|
|
49
|
-
scope: "org_1",
|
|
50
|
-
counterId: "llm_usage",
|
|
51
|
-
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
52
|
-
groupBy: ["subject", "metric"],
|
|
53
|
-
measures: ["cost", "latencyMs"],
|
|
54
|
-
format: "table",
|
|
55
|
-
});
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
Use `measures: "*"` to include all declared measures.
|
|
59
|
-
|
|
60
|
-
### Ordering
|
|
61
|
-
|
|
62
|
-
`orderBy` can target measure aggregations (for example sort by `latencyMs` max).
|
|
63
|
-
|
|
64
|
-
### Cache vs durable
|
|
65
|
-
|
|
66
|
-
Redis keeps **sum** and **count** for measures. **min** / **max** live on durable; the query planner skips cache when min/max are required.
|
|
67
|
-
|
|
68
|
-
## See also
|
|
69
|
-
|
|
70
|
-
- [Querying](./querying.md)
|
|
71
|
-
- Internal: [Measures & scales](../internals/internal-measures-and-scales.md)
|
|
1
|
+
# Guide: Measures
|
|
2
|
+
|
|
3
|
+
**Measures** are optional numeric fields per event (cost, latency, tokens, …) aggregated alongside counts.
|
|
4
|
+
|
|
5
|
+
## Step 1: Declare measures on the map
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const map = {
|
|
9
|
+
counterId: "llm_usage",
|
|
10
|
+
scope: { path: "$.scope.id" },
|
|
11
|
+
eventTime: { path: "$.request.createdAt" },
|
|
12
|
+
subjects: [{ type: "app", path: "$.app.id", required: true }],
|
|
13
|
+
metrics: [{ path: "$.request.status", required: true }],
|
|
14
|
+
dimensions: [{ name: "model", path: "$.model.name" }],
|
|
15
|
+
delta: { default: 1 },
|
|
16
|
+
timeframes: ["day"],
|
|
17
|
+
measures: [
|
|
18
|
+
{ name: "cost", path: "$.usage.costCents", aggregations: ["sum"], scale: 100 },
|
|
19
|
+
{ name: "latencyMs", path: "$.usage.latencyMs", aggregations: ["sum", "count", "min", "max"] },
|
|
20
|
+
],
|
|
21
|
+
} as const;
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
### Aggregations
|
|
25
|
+
|
|
26
|
+
- **`sum`**, **`count`**, **`min`**, **`max`** are stored.
|
|
27
|
+
- **`avg`** is derived at query time when both **`sum`** and **`count`** exist for that measure.
|
|
28
|
+
|
|
29
|
+
### Scale
|
|
30
|
+
|
|
31
|
+
`scale` multiplies then truncates for fixed-point storage (for example cents → integer micros).
|
|
32
|
+
|
|
33
|
+
### Missing values
|
|
34
|
+
|
|
35
|
+
Use `onMissing: "skip"` (default) or `"drop"` / `"zero"` per measure.
|
|
36
|
+
|
|
37
|
+
### Unique / distinct counts are not additive sums
|
|
38
|
+
|
|
39
|
+
Countex **counts** and **measure aggregations** (`sum`, `count`, `min`, `max`) behave like standard additive counters over cells. They do **not** answer “how many **distinct** users appeared in this bucket?” unless you model that explicitly (for example one subject id per event and interpret cell counts accordingly). Do **not** sum counts across overlapping populations and treat the result as a unique count. For large-cardinality uniques, plan on a dedicated approach (for example optional future HLL-style storage — see deployment notes); vanilla cells are the wrong tool for deduplicated set cardinality across arbitrary queries.
|
|
40
|
+
|
|
41
|
+
## Step 2: Ingest as usual
|
|
42
|
+
|
|
43
|
+
Each fan-out increment carries the **same** measure snapshot for that event.
|
|
44
|
+
|
|
45
|
+
## Step 3: Query measures
|
|
46
|
+
|
|
47
|
+
```ts
|
|
48
|
+
const result = await countex.query({
|
|
49
|
+
scope: "org_1",
|
|
50
|
+
counterId: "llm_usage",
|
|
51
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
52
|
+
groupBy: ["subject", "metric"],
|
|
53
|
+
measures: ["cost", "latencyMs"],
|
|
54
|
+
format: "table",
|
|
55
|
+
});
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Use `measures: "*"` to include all declared measures.
|
|
59
|
+
|
|
60
|
+
### Ordering
|
|
61
|
+
|
|
62
|
+
`orderBy` can target measure aggregations (for example sort by `latencyMs` max).
|
|
63
|
+
|
|
64
|
+
### Cache vs durable
|
|
65
|
+
|
|
66
|
+
Redis keeps **sum** and **count** for measures. **min** / **max** live on durable; the query planner skips cache when min/max are required.
|
|
67
|
+
|
|
68
|
+
## See also
|
|
69
|
+
|
|
70
|
+
- [Querying](./querying.md)
|
|
71
|
+
- Internal: [Measures & scales](../internals/internal-measures-and-scales.md)
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
# Migrating durable storage from MongoDB to PostgreSQL
|
|
2
|
+
|
|
3
|
+
Move the source of truth. Leave Redis in place. Counter maps, bucket keys, and cache keys do not change.
|
|
4
|
+
|
|
5
|
+
Do this while writers are stopped. A cutover that keeps ingesting into both stores will diverge, because both apply increments and neither can see the other's deltas.
|
|
6
|
+
|
|
7
|
+
PostgreSQL 15+ is required. Run `PostgresDurable.init()` (or `PostgresPipelineStore.connect()`) once so the schema exists before the copy.
|
|
8
|
+
|
|
9
|
+
## What stays
|
|
10
|
+
|
|
11
|
+
- Redis (or the in-process hot layer). Key layout does not encode which durable engine is behind it.
|
|
12
|
+
- Counter maps, dictionary id **numbers** already stored on cells, and query specs.
|
|
13
|
+
- The Mongo deployment until you have compared totals and pointed workers at Postgres. Mongo remains a supported adapter.
|
|
14
|
+
|
|
15
|
+
## `@x12i/countex` cells
|
|
16
|
+
|
|
17
|
+
Cells store `subjectId` and `metricId` as integers that were allocated from `countex_dict`. Copying cells without copying those ids, or calling `resolveIds` again on the new database, mints new ids and splits one subject across two rows.
|
|
18
|
+
|
|
19
|
+
Order:
|
|
20
|
+
|
|
21
|
+
1. Stop ingest workers. Flush them first (`countex.shutdown()` or `flush()`) so the Mongo copy includes the last buffer.
|
|
22
|
+
2. For each scope and counter, `streamCells` from `MongoDurable` over the retention window you still need, and `importSnapshotCells` into `PostgresDurable`. `importSnapshotCells` writes absolute counts. It does not add them to whatever is already there. The Postgres schema should be empty for that scope.
|
|
23
|
+
3. Copy `countex_dict` with the same `internalId` values. Do not allocate.
|
|
24
|
+
4. Set `seq` to `max(internal_id)` per scope so the next `resolveIds` continues above every copied id.
|
|
25
|
+
5. Copy `countex_evidence` if any map uses evidence. Order does not matter. Trim is enforced on later appends.
|
|
26
|
+
6. Start workers with `PostgresDurable` and the same Redis. If you do not trust cache coverage across the freeze, flush the Countex Redis keys for those scopes and let queries warm from Postgres. If Redis was not written during the freeze, leaving it is valid.
|
|
27
|
+
|
|
28
|
+
Sketch (host script, not a library CLI):
|
|
29
|
+
|
|
30
|
+
```ts
|
|
31
|
+
import { MongoClient } from "mongodb";
|
|
32
|
+
import { Pool } from "pg";
|
|
33
|
+
import { MongoDurable } from "@x12i/countex/adapters/mongo";
|
|
34
|
+
import { PostgresDurable } from "@x12i/countex/adapters/postgres";
|
|
35
|
+
|
|
36
|
+
const mongo = new MongoClient(process.env.MONGO_URI!);
|
|
37
|
+
await mongo.connect();
|
|
38
|
+
const pool = new Pool({ connectionString: process.env.POSTGRES_URL });
|
|
39
|
+
const from = new MongoDurable({ client: mongo, defaultDb: "countex" });
|
|
40
|
+
const to = new PostgresDurable({ pool, schema: "countex" });
|
|
41
|
+
await to.init();
|
|
42
|
+
|
|
43
|
+
const db = mongo.db("countex");
|
|
44
|
+
const batch: import("@x12i/countex").CounterCell[] = [];
|
|
45
|
+
for await (const cell of from.streamCells({
|
|
46
|
+
scope: "org_1",
|
|
47
|
+
counterId: "usage",
|
|
48
|
+
timeframe: { type: "day", from: "2024-01-01", to: "2026-12-31" },
|
|
49
|
+
})) {
|
|
50
|
+
batch.push(cell);
|
|
51
|
+
if (batch.length === 1000) {
|
|
52
|
+
await to.importSnapshotCells(batch.splice(0, batch.length));
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
if (batch.length) await to.importSnapshotCells(batch);
|
|
56
|
+
|
|
57
|
+
const dictRows = await db.collection("countex_dict").find({}).toArray();
|
|
58
|
+
for (const row of dictRows) {
|
|
59
|
+
await pool.query(
|
|
60
|
+
`INSERT INTO countex.dict (scope_id, kind, qualifier, source_key, internal_id)
|
|
61
|
+
VALUES ($1, $2, $3, $4, $5)
|
|
62
|
+
ON CONFLICT (scope_id, kind, qualifier, source_key)
|
|
63
|
+
DO UPDATE SET internal_id = EXCLUDED.internal_id`,
|
|
64
|
+
[row.scopeId, row.kind, row.qualifier ?? "", row.sourceKey, row.internalId],
|
|
65
|
+
);
|
|
66
|
+
}
|
|
67
|
+
const seqs = await db.collection("countex_seq").find({}).toArray();
|
|
68
|
+
for (const seq of seqs) {
|
|
69
|
+
const scopeId = String(seq._id).replace(/:dict$/, "");
|
|
70
|
+
await pool.query(
|
|
71
|
+
`INSERT INTO countex.seq AS s (scope_id, seq) VALUES ($1, $2)
|
|
72
|
+
ON CONFLICT (scope_id) DO UPDATE SET seq = GREATEST(s.seq, EXCLUDED.seq)`,
|
|
73
|
+
[scopeId, seq.seq],
|
|
74
|
+
);
|
|
75
|
+
}
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Repeat `streamCells` for every timeframe type you retain (`minute`, `hour`, `day`, `week`, `month`, `quarter`, `year`, `all`). A per-scope Mongo database (`dbResolver`) is one schema on the Postgres side, with `scope_id` on the row. Do not create a Postgres schema per tenant unless that host already does.
|
|
79
|
+
|
|
80
|
+
If dict rows and cells disagree, trust the cells' numeric ids and make dict match them. The sequence must be at least `max(internal_id)`, not `count(*)`.
|
|
81
|
+
|
|
82
|
+
Check before you open ingest:
|
|
83
|
+
|
|
84
|
+
```sql
|
|
85
|
+
SELECT count(*) AS cells, sum(count) AS count_sum
|
|
86
|
+
FROM countex.cells
|
|
87
|
+
WHERE scope_id = 'org_1' AND counter_id = 'usage';
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
Compare `cells` and `count_sum` to a Mongo aggregation on the same filter. Dictionary counts should match `countex_dict`.
|
|
91
|
+
|
|
92
|
+
## `@x12i/countex-pipelines`
|
|
93
|
+
|
|
94
|
+
Pipelines is a different database. Copy absolute counter values. Do not replay events unless you still have the source scan and are loading an empty tenant.
|
|
95
|
+
|
|
96
|
+
Mongo collections map to tables in schema `countex_pipelines` (override with `schema` / `COUNTEX_PIPELINES_SCHEMA`):
|
|
97
|
+
|
|
98
|
+
| Mongo collection | Postgres table |
|
|
99
|
+
|------------------|----------------|
|
|
100
|
+
| `countex_pipeline_records` | `countex_pipeline_records` |
|
|
101
|
+
| `countex_pipeline_dedup` | `countex_pipeline_dedup` |
|
|
102
|
+
| `countex_pipeline_subject_rollups` | `countex_pipeline_subject_rollups` |
|
|
103
|
+
| `countex_pipeline_pipeline_rollups` | `countex_pipeline_pipeline_rollups` |
|
|
104
|
+
| `countex_pipeline_dimension_rollups` | `countex_pipeline_dimension_rollups` |
|
|
105
|
+
| `countex_pipeline_dimension_subjects` | `countex_pipeline_dimension_subjects` |
|
|
106
|
+
|
|
107
|
+
Field names become snake_case (`tenantId` → `tenant_id`, `recordTotal` → `record_total`, `pipelineId` → `pipeline_id`, `subjectCount` → `subject_count`). `pipelines` and `dimensions` on the record row are jsonb. Copy dedup keys as they are. Those keys are what make a retried event a no-op. If you drop them, the next replay of the same status increments again.
|
|
108
|
+
|
|
109
|
+
`PostgresPipelineStore` applies each ingest in one transaction (dedup, record, rollups). Mongo applies those writes one call at a time. After cutover, a crash mid-ingest no longer leaves a record without its counters. That does not repair rows that already drifted in Mongo. Compare subject `record_total` to a count of `countex_pipeline_records` before you switch, and fix drift with a source rebuild if you have one.
|
|
110
|
+
|
|
111
|
+
Point the process at Postgres with `COUNTEX_POSTGRES_URL` or `postgresUrl`. If the process already has `MONGO_URI`, a generic `POSTGRES_URL` used by another database does **not** switch Countex. Set `COUNTEX_POSTGRES_URL` (or pass `postgresUrl`) so the move is explicit. `PipelinesEngine.create({ postgresUrl, mongoUri })` uses Postgres when both are set.
|
|
112
|
+
|
|
113
|
+
The hot layer (memory or Redis) can stay. After the freeze, either keep it or call `initialize({ tenantId, fromCounterStore: true })` so it reloads from Postgres.
|
|
114
|
+
|
|
115
|
+
## Cutover
|
|
116
|
+
|
|
117
|
+
1. Stop writers. Flush `@x12i/countex` buffers.
|
|
118
|
+
2. Run the copy above. Leave Mongo readable.
|
|
119
|
+
3. Compare cell counts, count sums, dictionary max id, and pipeline `record_total`.
|
|
120
|
+
4. Start writers on Postgres. Keep Redis if it was not written during the freeze. Otherwise delete the Countex key prefix and let coverage rebuild.
|
|
121
|
+
5. Watch flush errors and a durable read of a known scope. Mongo is the rollback target until this looks right: stop writers, point them back at Mongo, and rebuild Redis if any Postgres increments happened after the copy.
|
|
122
|
+
|
|
123
|
+
## See also
|
|
124
|
+
|
|
125
|
+
- [Postgres adapter](./adapter-postgres.md)
|
|
126
|
+
- [Mongo adapter](./adapter-mongo.md)
|
|
127
|
+
- [Deployment](../DEPLOYMENT.md)
|
package/docs/guides/preview.md
CHANGED
|
@@ -1,92 +1,92 @@
|
|
|
1
|
-
# Guide: Preview a map (dry-run)
|
|
2
|
-
|
|
3
|
-
`countex.preview(event, options?)` runs **extraction + cardinality** for one event against your maps and returns the increments it _would_ produce — without dictionary I/O, without buffering, without any storage writes. Use it to debug a counter map before ingesting millions of events.
|
|
4
|
-
|
|
5
|
-
## Step 1: Construct a Countex instance
|
|
6
|
-
|
|
7
|
-
```ts
|
|
8
|
-
import { createCountex } from "@x12i/countex";
|
|
9
|
-
import { MemoryDurable } from "@x12i/countex/adapters/memory";
|
|
10
|
-
|
|
11
|
-
const countex = createCountex({
|
|
12
|
-
maps: [accessMap],
|
|
13
|
-
durable: new MemoryDurable(),
|
|
14
|
-
flush: { everyMs: 0 },
|
|
15
|
-
});
|
|
16
|
-
```
|
|
17
|
-
|
|
18
|
-
A real durable adapter is fine too — preview never calls it.
|
|
19
|
-
|
|
20
|
-
## Step 2: Run preview
|
|
21
|
-
|
|
22
|
-
```ts
|
|
23
|
-
const previews = countex.preview(event, {
|
|
24
|
-
counterId: "access_by_subject_metric", // optional; defaults to all maps
|
|
25
|
-
});
|
|
26
|
-
```
|
|
27
|
-
|
|
28
|
-
`previews` is an array of `MapPreviewResult`, one per map.
|
|
29
|
-
|
|
30
|
-
## Result shape
|
|
31
|
-
|
|
32
|
-
```ts
|
|
33
|
-
interface MapPreviewResult {
|
|
34
|
-
counterId: string;
|
|
35
|
-
skipped: boolean;
|
|
36
|
-
extractionWarnings: ExtractionWarning[];
|
|
37
|
-
cells: PreviewIncrementCell[];
|
|
38
|
-
/** Increments dropped by cardinality guard. Preview runs silently — no `dropped` events. */
|
|
39
|
-
droppedAfterCardinality: number;
|
|
40
|
-
}
|
|
41
|
-
|
|
42
|
-
interface PreviewIncrementCell {
|
|
43
|
-
scopeId: string;
|
|
44
|
-
counterId: string;
|
|
45
|
-
timeframeType: TimeframeType;
|
|
46
|
-
timeframeKey: string;
|
|
47
|
-
subjectType: string;
|
|
48
|
-
subjectSourceKey: string;
|
|
49
|
-
metricSourceKey: string;
|
|
50
|
-
dimensions: Dimensions;
|
|
51
|
-
dimensionsHash: string;
|
|
52
|
-
delta: number;
|
|
53
|
-
measures: Record<string, { value: number; aggs: MeasureAgg[] }>;
|
|
54
|
-
}
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
`subjectSourceKey` and `metricSourceKey` are the raw strings extracted from the event — preview does not allocate dictionary ids. `dimensionsHash` matches the canonical form computed during real ingestion (see [internal-dimension-hash.md](../internals/internal-dimension-hash.md)).
|
|
58
|
-
|
|
59
|
-
## Counting fan-out
|
|
60
|
-
|
|
61
|
-
```ts
|
|
62
|
-
const total = previews.reduce((s, p) => s + p.cells.length, 0);
|
|
63
|
-
console.log(`This event would generate ${total} cell increments.`);
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
For the canonical example (1 actor × 2 metrics × 5 timeframes), `total === 10`.
|
|
67
|
-
|
|
68
|
-
## Diagnostics: warnings and skips
|
|
69
|
-
|
|
70
|
-
`extractionWarnings` includes things like `"measure_invalid"`, `"measure_missing"`, missing required dimensions, and bad path resolutions. Use them to spot a misconfigured map without having to ingest and watch event listeners.
|
|
71
|
-
|
|
72
|
-
```ts
|
|
73
|
-
for (const p of previews) {
|
|
74
|
-
if (p.skipped) console.warn(`map "${p.counterId}" skipped this event`);
|
|
75
|
-
for (const w of p.extractionWarnings) {
|
|
76
|
-
console.warn(`extraction warning: ${w.reason}`, w.meta);
|
|
77
|
-
}
|
|
78
|
-
}
|
|
79
|
-
```
|
|
80
|
-
|
|
81
|
-
## Caveats
|
|
82
|
-
|
|
83
|
-
- **Cardinality guard state is shared with the live engine.** Preview runs the same `CardinalityGuard.check`, so calling preview repeatedly counts toward the per-scope cardinality limits. For isolated dry-runs, construct a separate `Countex` instance just for previewing.
|
|
84
|
-
- **No idempotency check.** Preview ignores `idempotency` — same event can be previewed any number of times.
|
|
85
|
-
- **No measure scaling on output.** `measures[name].value` is the raw extracted number; on the live ingest path it would be multiplied by `measure.scale` before persistence.
|
|
86
|
-
- **No dropped/overflow events emitted.** Use `droppedAfterCardinality` and `extractionWarnings` instead.
|
|
87
|
-
|
|
88
|
-
## See also
|
|
89
|
-
|
|
90
|
-
- [Counter maps guide](./counter-maps.md)
|
|
91
|
-
- [Cardinality guide](./cardinality.md)
|
|
92
|
-
- Internal: [Mapping validation & extraction](../internals/internal-mapping-validation-and-extraction.md)
|
|
1
|
+
# Guide: Preview a map (dry-run)
|
|
2
|
+
|
|
3
|
+
`countex.preview(event, options?)` runs **extraction + cardinality** for one event against your maps and returns the increments it _would_ produce — without dictionary I/O, without buffering, without any storage writes. Use it to debug a counter map before ingesting millions of events.
|
|
4
|
+
|
|
5
|
+
## Step 1: Construct a Countex instance
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
import { createCountex } from "@x12i/countex";
|
|
9
|
+
import { MemoryDurable } from "@x12i/countex/adapters/memory";
|
|
10
|
+
|
|
11
|
+
const countex = createCountex({
|
|
12
|
+
maps: [accessMap],
|
|
13
|
+
durable: new MemoryDurable(),
|
|
14
|
+
flush: { everyMs: 0 },
|
|
15
|
+
});
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
A real durable adapter is fine too — preview never calls it.
|
|
19
|
+
|
|
20
|
+
## Step 2: Run preview
|
|
21
|
+
|
|
22
|
+
```ts
|
|
23
|
+
const previews = countex.preview(event, {
|
|
24
|
+
counterId: "access_by_subject_metric", // optional; defaults to all maps
|
|
25
|
+
});
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
`previews` is an array of `MapPreviewResult`, one per map.
|
|
29
|
+
|
|
30
|
+
## Result shape
|
|
31
|
+
|
|
32
|
+
```ts
|
|
33
|
+
interface MapPreviewResult {
|
|
34
|
+
counterId: string;
|
|
35
|
+
skipped: boolean;
|
|
36
|
+
extractionWarnings: ExtractionWarning[];
|
|
37
|
+
cells: PreviewIncrementCell[];
|
|
38
|
+
/** Increments dropped by cardinality guard. Preview runs silently — no `dropped` events. */
|
|
39
|
+
droppedAfterCardinality: number;
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
interface PreviewIncrementCell {
|
|
43
|
+
scopeId: string;
|
|
44
|
+
counterId: string;
|
|
45
|
+
timeframeType: TimeframeType;
|
|
46
|
+
timeframeKey: string;
|
|
47
|
+
subjectType: string;
|
|
48
|
+
subjectSourceKey: string;
|
|
49
|
+
metricSourceKey: string;
|
|
50
|
+
dimensions: Dimensions;
|
|
51
|
+
dimensionsHash: string;
|
|
52
|
+
delta: number;
|
|
53
|
+
measures: Record<string, { value: number; aggs: MeasureAgg[] }>;
|
|
54
|
+
}
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
`subjectSourceKey` and `metricSourceKey` are the raw strings extracted from the event — preview does not allocate dictionary ids. `dimensionsHash` matches the canonical form computed during real ingestion (see [internal-dimension-hash.md](../internals/internal-dimension-hash.md)).
|
|
58
|
+
|
|
59
|
+
## Counting fan-out
|
|
60
|
+
|
|
61
|
+
```ts
|
|
62
|
+
const total = previews.reduce((s, p) => s + p.cells.length, 0);
|
|
63
|
+
console.log(`This event would generate ${total} cell increments.`);
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
For the canonical example (1 actor × 2 metrics × 5 timeframes), `total === 10`.
|
|
67
|
+
|
|
68
|
+
## Diagnostics: warnings and skips
|
|
69
|
+
|
|
70
|
+
`extractionWarnings` includes things like `"measure_invalid"`, `"measure_missing"`, missing required dimensions, and bad path resolutions. Use them to spot a misconfigured map without having to ingest and watch event listeners.
|
|
71
|
+
|
|
72
|
+
```ts
|
|
73
|
+
for (const p of previews) {
|
|
74
|
+
if (p.skipped) console.warn(`map "${p.counterId}" skipped this event`);
|
|
75
|
+
for (const w of p.extractionWarnings) {
|
|
76
|
+
console.warn(`extraction warning: ${w.reason}`, w.meta);
|
|
77
|
+
}
|
|
78
|
+
}
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
## Caveats
|
|
82
|
+
|
|
83
|
+
- **Cardinality guard state is shared with the live engine.** Preview runs the same `CardinalityGuard.check`, so calling preview repeatedly counts toward the per-scope cardinality limits. For isolated dry-runs, construct a separate `Countex` instance just for previewing.
|
|
84
|
+
- **No idempotency check.** Preview ignores `idempotency` — same event can be previewed any number of times.
|
|
85
|
+
- **No measure scaling on output.** `measures[name].value` is the raw extracted number; on the live ingest path it would be multiplied by `measure.scale` before persistence.
|
|
86
|
+
- **No dropped/overflow events emitted.** Use `droppedAfterCardinality` and `extractionWarnings` instead.
|
|
87
|
+
|
|
88
|
+
## See also
|
|
89
|
+
|
|
90
|
+
- [Counter maps guide](./counter-maps.md)
|
|
91
|
+
- [Cardinality guide](./cardinality.md)
|
|
92
|
+
- Internal: [Mapping validation & extraction](../internals/internal-mapping-validation-and-extraction.md)
|