@x12i/countex 1.2.0 → 1.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (112) hide show
  1. package/README.md +80 -18
  2. package/dist/backup/manifest.d.ts +70 -0
  3. package/dist/backup/manifest.d.ts.map +1 -0
  4. package/dist/backup/manifest.js +131 -0
  5. package/dist/backup/manifest.js.map +1 -0
  6. package/dist/backup/snapshot-encoding.d.ts +8 -0
  7. package/dist/backup/snapshot-encoding.d.ts.map +1 -0
  8. package/dist/backup/snapshot-encoding.js +27 -0
  9. package/dist/backup/snapshot-encoding.js.map +1 -0
  10. package/dist/cli.js +16 -0
  11. package/dist/cli.js.map +1 -1
  12. package/dist/core/countex.d.ts +79 -1
  13. package/dist/core/countex.d.ts.map +1 -1
  14. package/dist/core/countex.js +254 -49
  15. package/dist/core/countex.js.map +1 -1
  16. package/dist/core/extraction.d.ts +1 -5
  17. package/dist/core/extraction.d.ts.map +1 -1
  18. package/dist/core/extraction.js.map +1 -1
  19. package/dist/core/mapping.d.ts +5 -0
  20. package/dist/core/mapping.d.ts.map +1 -1
  21. package/dist/core/mapping.js +27 -0
  22. package/dist/core/mapping.js.map +1 -1
  23. package/dist/index.d.ts +5 -2
  24. package/dist/index.d.ts.map +1 -1
  25. package/dist/index.js +15 -1
  26. package/dist/index.js.map +1 -1
  27. package/dist/query/formatter.d.ts +2 -2
  28. package/dist/query/formatter.d.ts.map +1 -1
  29. package/dist/query/formatter.js +9 -6
  30. package/dist/query/formatter.js.map +1 -1
  31. package/dist/query/planner.d.ts +15 -1
  32. package/dist/query/planner.d.ts.map +1 -1
  33. package/dist/query/planner.js +152 -3
  34. package/dist/query/planner.js.map +1 -1
  35. package/dist/simulation/context.d.ts +5 -0
  36. package/dist/simulation/context.d.ts.map +1 -0
  37. package/dist/simulation/context.js +56 -0
  38. package/dist/simulation/context.js.map +1 -0
  39. package/dist/simulation/events.d.ts +3 -0
  40. package/dist/simulation/events.d.ts.map +1 -0
  41. package/dist/simulation/events.js +67 -0
  42. package/dist/simulation/events.js.map +1 -0
  43. package/dist/simulation/index.d.ts +7 -0
  44. package/dist/simulation/index.d.ts.map +1 -0
  45. package/dist/simulation/index.js +16 -0
  46. package/dist/simulation/index.js.map +1 -0
  47. package/dist/simulation/prng.d.ts +4 -0
  48. package/dist/simulation/prng.d.ts.map +1 -0
  49. package/dist/simulation/prng.js +18 -0
  50. package/dist/simulation/prng.js.map +1 -0
  51. package/dist/simulation/resource-estimate.d.ts +6 -0
  52. package/dist/simulation/resource-estimate.d.ts.map +1 -0
  53. package/dist/simulation/resource-estimate.js +11 -0
  54. package/dist/simulation/resource-estimate.js.map +1 -0
  55. package/dist/simulation/simulator.d.ts +15 -0
  56. package/dist/simulation/simulator.d.ts.map +1 -0
  57. package/dist/simulation/simulator.js +67 -0
  58. package/dist/simulation/simulator.js.map +1 -0
  59. package/dist/simulation/tracker.d.ts +8 -0
  60. package/dist/simulation/tracker.d.ts.map +1 -0
  61. package/dist/simulation/tracker.js +18 -0
  62. package/dist/simulation/tracker.js.map +1 -0
  63. package/dist/simulation/types.d.ts +49 -0
  64. package/dist/simulation/types.d.ts.map +1 -0
  65. package/dist/simulation/types.js +21 -0
  66. package/dist/simulation/types.js.map +1 -0
  67. package/dist/storage/coverage.d.ts +13 -0
  68. package/dist/storage/coverage.d.ts.map +1 -0
  69. package/dist/storage/coverage.js +37 -0
  70. package/dist/storage/coverage.js.map +1 -0
  71. package/dist/storage/memory.d.ts +23 -5
  72. package/dist/storage/memory.d.ts.map +1 -1
  73. package/dist/storage/memory.js +162 -3
  74. package/dist/storage/memory.js.map +1 -1
  75. package/dist/storage/mongo.d.ts +5 -0
  76. package/dist/storage/mongo.d.ts.map +1 -1
  77. package/dist/storage/mongo.js +68 -1
  78. package/dist/storage/mongo.js.map +1 -1
  79. package/dist/storage/redis.d.ts +40 -11
  80. package/dist/storage/redis.d.ts.map +1 -1
  81. package/dist/storage/redis.js +202 -44
  82. package/dist/storage/redis.js.map +1 -1
  83. package/dist/types/index.d.ts +165 -7
  84. package/dist/types/index.d.ts.map +1 -1
  85. package/docs/DEPLOYMENT.md +125 -3
  86. package/docs/countex_gap_analysis_complete_checklist.md +1340 -0
  87. package/docs/guides/README.md +5 -0
  88. package/docs/guides/adapter-mongo.md +12 -0
  89. package/docs/guides/adapter-redis.md +46 -4
  90. package/docs/guides/backup-export.md +106 -10
  91. package/docs/guides/cli.md +8 -0
  92. package/docs/guides/counter-maps.md +14 -0
  93. package/docs/guides/insights-overview.md +6 -0
  94. package/docs/guides/measures.md +4 -0
  95. package/docs/guides/preview.md +92 -0
  96. package/docs/guides/querying.md +70 -0
  97. package/docs/guides/simulation-profiles.md +33 -0
  98. package/docs/guides/simulation-resource-estimation.md +30 -0
  99. package/docs/guides/simulation.md +36 -0
  100. package/docs/internals/README.md +1 -0
  101. package/docs/internals/internal-dimension-hash.md +34 -0
  102. package/docs/internals/internal-storage-mongo.md +24 -4
  103. package/docs/internals/internal-storage-redis.md +41 -9
  104. package/docs/use-cases/README.md +56 -0
  105. package/docs/use-cases/audit-security-log-counts.md +550 -0
  106. package/docs/use-cases/llm-cost-and-token-metering.md +255 -0
  107. package/docs/use-cases/multi-tenant-isolation.md +203 -0
  108. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -0
  109. package/docs/use-cases/processing-workflow-activity.md +165 -0
  110. package/docs/use-cases/product-feature-usage.md +240 -0
  111. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -0
  112. package/package.json +18 -2
@@ -0,0 +1,255 @@
1
+ # Technical playbook: LLM usage, cost, and latency metering
2
+
3
+ You operate an **LLM gateway** or application that emits **one JSON record per inference request**. Finance and reliability need:
4
+
5
+ - Request **counts** by app/model/status.
6
+ - **Token** totals (input/output).
7
+ - **Cost** in a currency-safe representation.
8
+ - **Latency** statistics (avg/min/max).
9
+
10
+ Countex stores **sparse aggregated cells** + **measure** fields; this playbook specifies exact map and query semantics.
11
+
12
+ ---
13
+
14
+ ## 1. Problem statement
15
+
16
+ ### 1.1 Measures vs counts
17
+
18
+ - **`delta` / count** — number of qualifying increments (here: one per request fan-out).
19
+ - **`measures`** — numeric aggregations merged into each cell (sum/min/max + implicit count for avg).
20
+
21
+ ### 1.2 Fixed-point cost storage
22
+
23
+ Floating dollar amounts are unsafe for atomic increments at scale. This playbook uses **`scale: 1_000_000`** to store **micro-dollars** as integers:
24
+
25
+ ```
26
+ stored_micros = trunc(event_dollars * 1_000_000)
27
+ ```
28
+
29
+ Query layers read **unscaled** values depending on adapter read path — Memory/Mongo adapters apply inverse scale on read where implemented; **always verify** in your integration tests for your exact adapter version.
30
+
31
+ ---
32
+
33
+ ## 2. Canonical payload
34
+
35
+ ```json
36
+ {
37
+ "scope": { "id": "org_123" },
38
+ "request": {
39
+ "id": "llm_req_771",
40
+ "createdAt": "2026-05-08T15:02:00.000Z",
41
+ "status": "success"
42
+ },
43
+ "app": { "id": "graphs-studio" },
44
+ "model": {
45
+ "provider": "openai",
46
+ "name": "gpt-4.1"
47
+ },
48
+ "usage": {
49
+ "inputTokens": 1200,
50
+ "outputTokens": 350,
51
+ "costDollars": 0.042,
52
+ "latencyMs": 1800
53
+ }
54
+ }
55
+ ```
56
+
57
+ ### 2.1 Required-field matrix
58
+
59
+ | Path | Role |
60
+ |------|------|
61
+ | `$.scope.id` | Tenant scope. |
62
+ | `$.request.createdAt` | Event time. |
63
+ | `$.app.id` | Subject (required). |
64
+ | `$.request.status` | Metric category (required). |
65
+ | `$.usage.*` | Measure inputs; governed per-measure `onMissing`. |
66
+
67
+ ---
68
+
69
+ ## 3. Primitives mapping
70
+
71
+ | Primitive | Path |
72
+ |-----------|------|
73
+ | scope | `$.scope.id` |
74
+ | eventTime | `$.request.createdAt` |
75
+ | subject `app` | `$.app.id` |
76
+ | metric | `$.request.status` |
77
+ | dimensions | `provider`, `model` |
78
+ | measures | see map |
79
+
80
+ ---
81
+
82
+ ## 4. Counter map (exact)
83
+
84
+ `src/countex/maps/llmGatewayMap.ts`:
85
+
86
+ ```ts
87
+ import type { CounterMap } from "@x12i/countex";
88
+
89
+ export const llmGatewayMap: CounterMap = {
90
+ counterId: "llm_gateway",
91
+
92
+ scope: { path: "$.scope.id" },
93
+ eventTime: { path: "$.request.createdAt" },
94
+
95
+ subjects: [{ type: "app", path: "$.app.id", required: true }],
96
+ metrics: [{ path: "$.request.status", required: true }],
97
+
98
+ dimensions: [
99
+ { name: "provider", path: "$.model.provider" },
100
+ { name: "model", path: "$.model.name" },
101
+ ],
102
+
103
+ delta: { default: 1 },
104
+ timeframes: ["hour", "day", "month"],
105
+
106
+ measures: [
107
+ { name: "inputTokens", path: "$.usage.inputTokens", aggregations: ["sum"] },
108
+ { name: "outputTokens", path: "$.usage.outputTokens", aggregations: ["sum"] },
109
+ {
110
+ name: "costMicros",
111
+ path: "$.usage.costDollars",
112
+ aggregations: ["sum"],
113
+ scale: 1_000_000,
114
+ },
115
+ {
116
+ name: "latencyMs",
117
+ path: "$.usage.latencyMs",
118
+ aggregations: ["sum", "count", "min", "max"],
119
+ },
120
+ ],
121
+ };
122
+ ```
123
+
124
+ ### 4.1 Aggregation semantics (exact)
125
+
126
+ - `latencyMs`: `sum` + `count` enable **avg = sum/count** at query time.
127
+ - `min` / `max`: stored on **durable**; cache may omit them — planner routes such queries to durable when needed (`cacheCanServe` in `src/query/planner.ts`).
128
+
129
+ ### 4.2 Effective aggregations
130
+
131
+ The engine may auto-add **`count`** to a measure when needed for avg semantics (`effectiveMeasureAggregations` in `src/core/extraction.ts`). Do not rely on suppressing `count` if you request min/max on the same measure.
132
+
133
+ ---
134
+
135
+ ## 5. Fan-out
136
+
137
+ One subject × one metric × **T** timeframes:
138
+
139
+ ```
140
+ R = T
141
+ ```
142
+
143
+ For three timeframes, **three cells** (before merge with identical keys).
144
+
145
+ **Measures replicate** onto each of those cells — cost is attributed per bucket.
146
+
147
+ ---
148
+
149
+ ## 6. Implementation steps
150
+
151
+ ### Step 6.1 — Construct Countex
152
+
153
+ Same lifecycle as other playbooks: `MongoDurable`, `RedisCache` optional, `createCountex({ maps: [llmGatewayMap], ... })`, `await init()`.
154
+
155
+ ### Step 6.2 — Ingest path
156
+
157
+ ```ts
158
+ await countex.ingest(gatewayJson);
159
+ ```
160
+
161
+ If `usage.costDollars` is missing and you did not set `onMissing`, measure warnings occur; subscribe:
162
+
163
+ ```ts
164
+ countex.on("measure_invalid", (p) => {
165
+ // log scopeId, counterId, measureName, reason
166
+ });
167
+ ```
168
+
169
+ ### Step 6.3 — Flush threshold guidance
170
+
171
+ LLM streams burst; prefer **`onOverflow: "block"`** under memory pressure so you do not silently drop deltas unless you explicitly chose `"drop"`.
172
+
173
+ ---
174
+
175
+ ## 7. Queries (exact specs)
176
+
177
+ ### 7.1 Cost drivers by model (day)
178
+
179
+ ```ts
180
+ const drivers = await countex.query({
181
+ scope: "org_123",
182
+ counterId: "llm_gateway",
183
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
184
+ groupBy: ["dim:model"],
185
+ measures: ["costMicros", "inputTokens", "outputTokens"],
186
+ format: "table",
187
+ orderBy: { measure: "costMicros", agg: "sum", dir: "desc" },
188
+ limit: 50,
189
+ });
190
+ ```
191
+
192
+ ### 7.2 Latency percentiles proxy
193
+
194
+ Countex stores **min/max/sum/count**, not full distributions. For p95 you still need raw traces or a histogram elsewhere. Use **`min`/`max`** for coarse SLO bounds.
195
+
196
+ ### 7.3 Force durable for min/max heavy dashboards
197
+
198
+ ```ts
199
+ bypassCache: true,
200
+ ```
201
+
202
+ when your SQL/BI tool issues arbitrary measure `"*"` pulls including min/max — or accept planner routing (merged/durable) via `result.source`.
203
+
204
+ ---
205
+
206
+ ## 8. Insights layer (optional second stage)
207
+
208
+ ```ts
209
+ import { createCountexInsights } from "@x12i/countex/insights";
210
+
211
+ const insights = createCountexInsights({
212
+ countex,
213
+ mapsByCounterId: new Map([["llm_gateway", llmGatewayMap]]),
214
+ });
215
+
216
+ const top = await insights.topN({
217
+ scope: "org_123",
218
+ counterId: "llm_gateway",
219
+ timeframe: { type: "month", from: "2026-05", to: "2026-05" },
220
+ groupBy: ["subject"],
221
+ measure: "costMicros.sum",
222
+ n: 15,
223
+ });
224
+ ```
225
+
226
+ **Measure selector string** `"costMicros.sum"` is parsed by insights (`parseMeasureSelector`).
227
+
228
+ ---
229
+
230
+ ## 9. Verification checklist
231
+
232
+ | Test | Steps |
233
+ |------|-------|
234
+ | Sum correctness | Ingest N identical events; query `costMicros.sum` ∝ N × stored increment. |
235
+ | Scale correctness | Compare adapter-read values against hand-calculated micro-dollar totals. |
236
+ | Min/max path | Query with `orderBy: { measure: "latencyMs", agg: "max", dir: "desc" }`; confirm `source` is not cache-only full lie — planner should bypass cache. |
237
+
238
+ ---
239
+
240
+ ## 10. Failure modes
241
+
242
+ | Symptom | Cause | Mitigation |
243
+ |---------|-------|------------|
244
+ | `precision_truncated` events | Non-representable micro-dollar after scale | Accept loss or use smaller currency unit. |
245
+ | Flat zeros | Wrong scope / timeframe range | Align bucket keys UTC. |
246
+ | Cache/durable mismatch on extrema | Expected by design | Document dashboards using min/max as durable-backed. |
247
+
248
+ ---
249
+
250
+ ## 11. References
251
+
252
+ - [`docs/guides/measures.md`](../guides/measures.md)
253
+ - [`docs/guides/querying.md`](../guides/querying.md)
254
+ - [`docs/internals/internal-query-path.md`](../internals/internal-query-path.md)
255
+ - `docs/INSIGHTS_LLM_COST_AND_TOKENS.md`
@@ -0,0 +1,203 @@
1
+ # Technical playbook: Multi-tenant isolation
2
+
3
+ You ship **B2B SaaS**. Every JSON event belongs to exactly one **tenant**. Regulatory and operational requirements demand:
4
+
5
+ - **No cross-tenant reads** from application bugs.
6
+ - Optionally **hard physical isolation** (per-tenant database).
7
+ - Predictable key namespaces in shared Redis clusters.
8
+
9
+ Countex always stores **`scopeId`** on cells; **you** must enforce that scope matches authenticated context and that storage layout matches policy.
10
+
11
+ ---
12
+
13
+ ## 1. Threat model (what this playbook addresses)
14
+
15
+ | Risk | Mitigation level |
16
+ |------|------------------|
17
+ | App passes wrong `scope` into `query()` | **Application code review** + always derive scope from auth token, never from client-supplied body alone. |
18
+ | Tenant A reads tenant B aggregates | Prevented if scope string wrong → queries return empty **but** dictionary ids could theoretically collide across scopes if you incorrectly reuse numeric ids in UI — **always pair ids with scope**. |
19
+ | Data residency | Mongo `dbResolver` per region + per tenant DB naming; Redis logical DB / ACLs. |
20
+
21
+ ---
22
+
23
+ ## 2. Non-negotiable rule
24
+
25
+ **Every `countex.query` and every event’s extracted scope must use the same canonical tenant id string** your identity system emits (e.g. `org_01JB…`, never slug vs id interchangeably).
26
+
27
+ ---
28
+
29
+ ## 3. Event contract
30
+
31
+ Canonical pattern:
32
+
33
+ ```json
34
+ {
35
+ "tenant": { "id": "org_abc" },
36
+ "at": "2026-05-08T12:00:00.000Z",
37
+ "userId": "user_42",
38
+ "action": "save"
39
+ }
40
+ ```
41
+
42
+ Map:
43
+
44
+ ```ts
45
+ import type { CounterMap } from "@x12i/countex";
46
+
47
+ export const tenantScopedMap: CounterMap = {
48
+ counterId: "usage",
49
+ scope: { path: "$.tenant.id" },
50
+ eventTime: { path: "$.at" },
51
+ subjects: [{ type: "user", path: "$.userId", required: true }],
52
+ metrics: [{ path: "$.action", required: true }],
53
+ dimensions: [],
54
+ delta: { default: 1 },
55
+ timeframes: ["day"],
56
+ };
57
+ ```
58
+
59
+ ---
60
+
61
+ ## 4. Storage isolation patterns
62
+
63
+ ### Pattern A — Single Mongo database, scope field discriminant
64
+
65
+ **When:** acceptable for early stage; compliance allows logical isolation only.
66
+
67
+ `MongoDurable` default `defaultDb` (e.g. `countex_prod`). Every document carries `scopeId`.
68
+
69
+ **Exact setup:**
70
+
71
+ ```ts
72
+ import { MongoClient } from "mongodb";
73
+ import { MongoDurable } from "@x12i/countex/adapters/mongo";
74
+
75
+ const mongo = new MongoClient(process.env.MONGO_URI!);
76
+ await mongo.connect();
77
+
78
+ const durable = new MongoDurable({
79
+ client: mongo,
80
+ defaultDb: "countex_prod",
81
+ });
82
+ ```
83
+
84
+ **Operational requirement:** Mongo queries outside Countex **must** filter `{ scopeId: <tenant> }` if you touch collections manually.
85
+
86
+ ### Pattern B — Per-tenant Mongo database (stronger)
87
+
88
+ **When:** customer contracts require separation at rest.
89
+
90
+ ```ts
91
+ const durable = new MongoDurable({
92
+ client: mongo,
93
+ dbResolver: ({ scope }) => {
94
+ // EXACT naming convention — enforce alphanumeric + safe chars only
95
+ const safe = scope.replace(/[^a-zA-Z0-9_-]/g, "_");
96
+ return `countex_${safe}`;
97
+ },
98
+ });
99
+ ```
100
+
101
+ **Exact consequences:**
102
+
103
+ 1. First write for a tenant creates DB `countex_<tenant>` with collections `countex_cells`, `countex_dict`, `countex_seq`.
104
+ 2. Dictionary ids **restart per database** — numeric `subjectId` in DB A has **no relation** to DB B.
105
+
106
+ ### Pattern C — Redis shared cluster
107
+
108
+ `RedisCache` keys:
109
+
110
+ ```
111
+ {prefix}:{scopeId}:{counterId}:{timeframeType}:{timeframeKey}
112
+ ```
113
+
114
+ **Exact action items:**
115
+
116
+ 1. Use **`keyPrefix`** per environment (`countex_prod`, `countex_staging`).
117
+ 2. Enable Redis **ACL users** per product shard if policy demands.
118
+ 3. TTL — `ttlSeconds` on adapter applies to cache hashes; tune ≥ analytics window.
119
+
120
+ ```ts
121
+ import IORedis from "ioredis";
122
+ import { RedisCache } from "@x12i/countex/adapters/redis";
123
+
124
+ const redis = new IORedis(process.env.REDIS_URL!);
125
+
126
+ const cache = new RedisCache({
127
+ client: redis,
128
+ keyPrefix: "countex_prod",
129
+ ttlSeconds: 92 * 24 * 3600,
130
+ });
131
+ ```
132
+
133
+ ---
134
+
135
+ ## 5. Application wiring (exact control flow)
136
+
137
+ ### Step 5.1 — Authenticate request
138
+
139
+ ```ts
140
+ const session = await authenticate(request);
141
+ const scope = session.tenantId; // MUST come from signed token / session store
142
+ ```
143
+
144
+ ### Step 5.2 — Reject scope mismatch
145
+
146
+ If JSON body includes `tenant.id`:
147
+
148
+ ```ts
149
+ const body = await request.json();
150
+ if (typeof body === "object" && body !== null && "tenant" in body) {
151
+ const tid = (body as { tenant?: { id?: string } }).tenant?.id;
152
+ if (tid !== undefined && tid !== scope) {
153
+ throw new Error("tenant mismatch"); // 403
154
+ }
155
+ }
156
+ ```
157
+
158
+ ### Step 5.3 — Ingest only after validation
159
+
160
+ ```ts
161
+ await countex.ingest(body);
162
+ ```
163
+
164
+ ### Step 5.4 — Query only with session scope
165
+
166
+ ```ts
167
+ const result = await countex.query({
168
+ scope, // never `body.tenant.id` alone
169
+ counterId: "usage",
170
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
171
+ format: "table",
172
+ });
173
+ ```
174
+
175
+ ---
176
+
177
+ ## 6. Testing matrix (exact cases)
178
+
179
+ | Case | Setup | Assert |
180
+ |------|-------|--------|
181
+ | Isolation | Ingest event scope `A`, query scope `B` | Empty or zero totals for `B`. |
182
+ | Resolver | Two tenants first write | Two Mongo databases exist when using Pattern B. |
183
+ | Redis collision | Same subject string different scopes | Distinct hash keys because `scopeId` differs. |
184
+
185
+ ---
186
+
187
+ ## 7. Failure modes
188
+
189
+ | Symptom | Cause | Fix |
190
+ |---------|-------|-----|
191
+ | “Missing data” after migrate | Changed scope string format | Backfill or alias old→new scope in ingress. |
192
+ | Exploding Mongo DB count | Resolver allows arbitrary strings | Sanitize tenant ids; block malicious lengths. |
193
+ | Wrong dashboard ids | Copied numeric ids across tenants | Never export dictionary ids without scope column. |
194
+
195
+ ---
196
+
197
+ ## 8. References
198
+
199
+ - Mongo adapter options: `MongoAdapterOptions` in `src/storage/mongo.ts`
200
+ - Redis key layout: header comment in `src/storage/redis.ts`
201
+ - [`docs/guides/adapter-mongo.md`](../guides/adapter-mongo.md)
202
+ - [`docs/guides/adapter-redis.md`](../guides/adapter-redis.md)
203
+ - [`docs/DEPLOYMENT.md`](../DEPLOYMENT.md)
@@ -0,0 +1,255 @@
1
+ # Technical playbook: Operational dashboards and rollups
2
+
3
+ You run **batch or streaming workers** that record pipeline outcomes (success/failure), region, pipeline name, etc. Leadership wants **daily rollups**, **hourly strips**, and **failure-focused tables** without standing up a separate OLAP cluster first.
4
+
5
+ This playbook specifies exact **`CounterMap`**, **`flush`** tuning, **`query`** specs, and **cache rebuild** after incidents.
6
+
7
+ ---
8
+
9
+ ## 1. Domain model
10
+
11
+ ### 1.1 Event
12
+
13
+ ```json
14
+ {
15
+ "orgId": "org_1",
16
+ "workerId": "worker-us-east-1a-07",
17
+ "pipelineName": "invoice_export",
18
+ "region": "us-east-1",
19
+ "status": "failed",
20
+ "finishedAt": "2026-05-08T14:22:00.000Z"
21
+ }
22
+ ```
23
+
24
+ ### 1.2 Questions to answer
25
+
26
+ - Failures per **pipeline** yesterday.
27
+ - Hourly **throughput** (count rate) for a single day.
28
+ - Optional: compare regions (dimension breakdown).
29
+
30
+ ---
31
+
32
+ ## 2. Counter map (exact)
33
+
34
+ `src/countex/maps/pipelineRunsMap.ts`:
35
+
36
+ ```ts
37
+ import type { CounterMap } from "@x12i/countex";
38
+
39
+ export const pipelineRunsMap: CounterMap = {
40
+ counterId: "pipeline_runs",
41
+
42
+ scope: { path: "$.orgId" },
43
+ eventTime: { path: "$.finishedAt" },
44
+
45
+ subjects: [{ type: "worker", path: "$.workerId", required: true }],
46
+ metrics: [{ path: "$.status", required: true }],
47
+
48
+ dimensions: [
49
+ { name: "pipeline", path: "$.pipelineName" },
50
+ { name: "region", path: "$.region" },
51
+ ],
52
+
53
+ delta: { default: 1 },
54
+ timeframes: ["hour", "day"],
55
+ };
56
+ ```
57
+
58
+ ### 2.1 Fan-out
59
+
60
+ `1 worker subject × 1 status metric × 2 timeframes = 2` increments per event (before buffer merge).
61
+
62
+ ---
63
+
64
+ ## 3. Countex construction (exact)
65
+
66
+ ```ts
67
+ import { createCountex } from "@x12i/countex";
68
+ import { MongoDurable } from "@x12i/countex/adapters/mongo";
69
+ import { RedisCache } from "@x12i/countex/adapters/redis";
70
+ import { pipelineRunsMap } from "./maps/pipelineRunsMap.js";
71
+
72
+ export function createOpsCountex(deps: { durable: MongoDurable; cache: RedisCache }) {
73
+ return createCountex({
74
+ maps: [pipelineRunsMap],
75
+ durable: deps.durable,
76
+ cache: deps.cache,
77
+ flush: {
78
+ everyMs: 30_000,
79
+ maxBufferedCells: 50_000,
80
+ maxBufferedEvents: 25_000,
81
+ flushOnShutdown: true,
82
+ onOverflow: "block",
83
+ },
84
+ cardinality: {
85
+ maxDimensionValuesPerName: 5_000,
86
+ onLimit: "overflow",
87
+ overflowValue: "__OTHER__",
88
+ },
89
+ });
90
+ }
91
+ ```
92
+
93
+ **Rationale:**
94
+
95
+ - `everyMs: 30_000` balances Mongo write batching vs dashboard freshness.
96
+ - `onOverflow: "block"` avoids silent loss under spike load.
97
+
98
+ ---
99
+
100
+ ## 4. Worker integration (exact steps)
101
+
102
+ ### Step 4.1 — On job completion
103
+
104
+ ```ts
105
+ await countex.ingest({
106
+ orgId,
107
+ workerId,
108
+ pipelineName,
109
+ region,
110
+ status,
111
+ finishedAt: new Date().toISOString(),
112
+ });
113
+ ```
114
+
115
+ ### Step 4.2 — Process shutdown
116
+
117
+ ```ts
118
+ await countex.shutdown(); // drains buffer when flushOnShutdown true
119
+ ```
120
+
121
+ ---
122
+
123
+ ## 5. Dashboard queries (copy-paste specs)
124
+
125
+ ### 5.1 Top failing pipelines (single UTC day)
126
+
127
+ In this map **`status` is the metric** (`$.status`), not a dimension. Restrict to failures with **`metrics: ["failed"]`** (metric source key):
128
+
129
+ ```ts
130
+ const failures = await countex.query({
131
+ scope: "org_1",
132
+ counterId: "pipeline_runs",
133
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
134
+ metrics: ["failed"],
135
+ groupBy: ["dim:pipeline"],
136
+ format: "table",
137
+ orderBy: "-count",
138
+ limit: 20,
139
+ });
140
+ ```
141
+
142
+ Numeric metric ids from the dictionary are also valid in `metrics: [41]` if you resolve them out-of-band.
143
+
144
+ ### 5.2 Hourly throughput (timeseries, UTC day)
145
+
146
+ Bucket keys for `hour` are `YYYY-MM-DDTHH` strings (UTC).
147
+
148
+ ```ts
149
+ const hourly = await countex.query({
150
+ scope: "org_1",
151
+ counterId: "pipeline_runs",
152
+ timeframe: { type: "hour", from: "2026-05-08T00", to: "2026-05-08T23" },
153
+ groupBy: [],
154
+ format: "timeseries",
155
+ });
156
+ ```
157
+
158
+ ### 5.3 Rate — runs per minute (timeseries)
159
+
160
+ ```ts
161
+ const rateTs = await countex.query({
162
+ scope: "org_1",
163
+ counterId: "pipeline_runs",
164
+ timeframe: { type: "hour", from: "2026-05-08T00", to: "2026-05-08T23" },
165
+ groupBy: [],
166
+ format: "timeseries",
167
+ rate: { of: "count", per: "minute" },
168
+ });
169
+ ```
170
+
171
+ **Constraint:** `rate` is **invalid** for timeframe type **`all`** (no defined bucket duration).
172
+
173
+ ### 5.4 Region breakdown (success vs failed)
174
+
175
+ Issue **two** queries or omit metric filter and **group by metric** — example grouping both dimensions:
176
+
177
+ ```ts
178
+ const byRegion = await countex.query({
179
+ scope: "org_1",
180
+ counterId: "pipeline_runs",
181
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
182
+ groupBy: ["dim:region", "metric"],
183
+ format: "table",
184
+ orderBy: "-count",
185
+ limit: 500,
186
+ });
187
+ ```
188
+
189
+ Include **`subjectType`** + **`subject`** in `groupBy` when you need stable interpretation of numeric ids.
190
+
191
+ ---
192
+
193
+ ## 6. Cache rebuild after Redis loss
194
+
195
+ Durable (Mongo) remains authoritative. Rewarm cache for a bounded window:
196
+
197
+ ```ts
198
+ import { rebuildCache } from "@x12i/countex";
199
+
200
+ const { cellsWarmed } = await rebuildCache(durable, cache, {
201
+ scope: "org_1",
202
+ counterId: "pipeline_runs",
203
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
204
+ }, { batchSize: 1000 });
205
+ ```
206
+
207
+ **Exact meaning:** streams `durable.streamCells` for that slice, batches `cache.warm`.
208
+
209
+ Optional cancellation:
210
+
211
+ ```ts
212
+ await rebuildCache(durable, cache, filter, {
213
+ batchSize: 1000,
214
+ signal: AbortSignal.timeout(600_000),
215
+ });
216
+ ```
217
+
218
+ ---
219
+
220
+ ## 7. Bucket window limits
221
+
222
+ `bucketRange` caps at **10,000** buckets. For `minute` resolution that is roughly **6.94 days** of continuous range. If your dashboard asks for more, either:
223
+
224
+ - Switch to **`hour`** / **`day`**, or
225
+ - Split queries into multiple jobs.
226
+
227
+ ---
228
+
229
+ ## 8. Verification checklist
230
+
231
+ | Step | Action | Pass criterion |
232
+ |------|--------|----------------|
233
+ | Writes | Ingest sample event, `await flush()` | Mongo cells include `counterId: pipeline_runs`. |
234
+ | Failures query | §5.1 | Only pipelines with failed increments appear (non-empty when data exists). |
235
+ | Timeseries ordering | §5.2 | Rows sorted by `timeframeKey` ascending in formatter output. |
236
+ | Rebuild | §6 | `cellsWarmed` > 0 for non-empty day; subsequent cache reads hit full coverage when within TTL. |
237
+
238
+ ---
239
+
240
+ ## 9. Failure modes
241
+
242
+ | Symptom | Cause | Mitigation |
243
+ |---------|-------|------------|
244
+ | Empty failures table | No `failed` metric rows | Confirm JSON uses exact string `failed` vs `failure`. |
245
+ | Query timeout | `groupBy` too wide | Narrow timeframe; add filters; reduce `maxBuffered*` unrelated — tune Mongo indexes. |
246
+ | Stale Redis after rebuild | TTL eviction | Increase `ttlSeconds` or rerun rebuild closer to query time. |
247
+
248
+ ---
249
+
250
+ ## 10. References
251
+
252
+ - [`docs/guides/querying.md`](../guides/querying.md)
253
+ - [`docs/guides/backup-export.md`](../guides/backup-export.md)
254
+ - [`docs/guides/ingestion-and-flush.md`](../guides/ingestion-and-flush.md)
255
+ - [`docs/guides/timeframes-and-bucketing.md`](../guides/timeframes-and-bucketing.md)