@x12i/countex 1.8.5 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +866 -860
  3. package/dist/cli.js +6 -6
  4. package/dist/index.d.ts +1 -0
  5. package/dist/index.d.ts.map +1 -1
  6. package/dist/index.js +1 -0
  7. package/dist/index.js.map +1 -1
  8. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  9. package/dist/insights/advisors/resource-advisor.js +5 -1
  10. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  11. package/dist/insights/types.d.ts +4 -0
  12. package/dist/insights/types.d.ts.map +1 -1
  13. package/dist/storage/postgres.d.ts +104 -0
  14. package/dist/storage/postgres.d.ts.map +1 -0
  15. package/dist/storage/postgres.js +1023 -0
  16. package/dist/storage/postgres.js.map +1 -0
  17. package/dist/types/index.d.ts +11 -5
  18. package/dist/types/index.d.ts.map +1 -1
  19. package/docs/DEPLOYMENT.md +399 -375
  20. package/docs/END_TO_END_EXAMPLE.md +316 -316
  21. package/docs/INSIGHTS.md +59 -59
  22. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  23. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  24. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  25. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  26. package/docs/guides/README.md +36 -34
  27. package/docs/guides/adapter-memory.md +60 -60
  28. package/docs/guides/adapter-mongo.md +64 -62
  29. package/docs/guides/adapter-postgres.md +90 -0
  30. package/docs/guides/adapter-redis.md +89 -89
  31. package/docs/guides/backup-export.md +135 -135
  32. package/docs/guides/cardinality.md +50 -50
  33. package/docs/guides/cli.md +53 -53
  34. package/docs/guides/client-events.md +65 -65
  35. package/docs/guides/counter-maps.md +160 -160
  36. package/docs/guides/idempotency.md +39 -39
  37. package/docs/guides/ingestion-and-flush.md +67 -67
  38. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  39. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  40. package/docs/guides/insights-comparison-and-trends.md +47 -47
  41. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  42. package/docs/guides/insights-overview.md +82 -82
  43. package/docs/guides/insights-ranking.md +81 -81
  44. package/docs/guides/measures.md +71 -71
  45. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  46. package/docs/guides/preview.md +92 -92
  47. package/docs/guides/querying.md +178 -178
  48. package/docs/guides/simulation-profiles.md +33 -33
  49. package/docs/guides/simulation-resource-estimation.md +31 -30
  50. package/docs/guides/simulation.md +36 -36
  51. package/docs/guides/timeframes-and-bucketing.md +55 -55
  52. package/docs/internals/README.md +21 -21
  53. package/docs/internals/internal-architecture-overview.md +65 -65
  54. package/docs/internals/internal-backup-stream.md +23 -23
  55. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  56. package/docs/internals/internal-cardinality.md +30 -30
  57. package/docs/internals/internal-dimension-hash.md +34 -34
  58. package/docs/internals/internal-idempotency.md +36 -36
  59. package/docs/internals/internal-ingest-pipeline.md +52 -52
  60. package/docs/internals/internal-insights-layer.md +36 -36
  61. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  62. package/docs/internals/internal-measures-and-scales.md +35 -35
  63. package/docs/internals/internal-query-path.md +46 -46
  64. package/docs/internals/internal-storage-memory.md +13 -13
  65. package/docs/internals/internal-storage-mongo.md +44 -44
  66. package/docs/internals/internal-storage-postgres.md +32 -0
  67. package/docs/internals/internal-storage-redis.md +63 -63
  68. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  69. package/docs/use-cases/README.md +56 -56
  70. package/docs/use-cases/audit-security-log-counts.md +550 -550
  71. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  72. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  73. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  74. package/docs/use-cases/processing-workflow-activity.md +165 -165
  75. package/docs/use-cases/product-feature-usage.md +240 -240
  76. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  77. package/package.json +184 -170
@@ -1,255 +1,255 @@
1
- # Technical playbook: Operational dashboards and rollups
2
-
3
- You run **batch or streaming workers** that record pipeline outcomes (success/failure), region, pipeline name, etc. Leadership wants **daily rollups**, **hourly strips**, and **failure-focused tables** without standing up a separate OLAP cluster first.
4
-
5
- This playbook specifies exact **`CounterMap`**, **`flush`** tuning, **`query`** specs, and **cache rebuild** after incidents.
6
-
7
- ---
8
-
9
- ## 1. Domain model
10
-
11
- ### 1.1 Event
12
-
13
- ```json
14
- {
15
- "orgId": "org_1",
16
- "workerId": "worker-us-east-1a-07",
17
- "pipelineName": "invoice_export",
18
- "region": "us-east-1",
19
- "status": "failed",
20
- "finishedAt": "2026-05-08T14:22:00.000Z"
21
- }
22
- ```
23
-
24
- ### 1.2 Questions to answer
25
-
26
- - Failures per **pipeline** yesterday.
27
- - Hourly **throughput** (count rate) for a single day.
28
- - Optional: compare regions (dimension breakdown).
29
-
30
- ---
31
-
32
- ## 2. Counter map (exact)
33
-
34
- `src/countex/maps/pipelineRunsMap.ts`:
35
-
36
- ```ts
37
- import type { CounterMap } from "@x12i/countex";
38
-
39
- export const pipelineRunsMap: CounterMap = {
40
- counterId: "pipeline_runs",
41
-
42
- scope: { path: "$.orgId" },
43
- eventTime: { path: "$.finishedAt" },
44
-
45
- subjects: [{ type: "worker", path: "$.workerId", required: true }],
46
- metrics: [{ path: "$.status", required: true }],
47
-
48
- dimensions: [
49
- { name: "pipeline", path: "$.pipelineName" },
50
- { name: "region", path: "$.region" },
51
- ],
52
-
53
- delta: { default: 1 },
54
- timeframes: ["hour", "day"],
55
- };
56
- ```
57
-
58
- ### 2.1 Fan-out
59
-
60
- `1 worker subject × 1 status metric × 2 timeframes = 2` increments per event (before buffer merge).
61
-
62
- ---
63
-
64
- ## 3. Countex construction (exact)
65
-
66
- ```ts
67
- import { createCountex } from "@x12i/countex";
68
- import { MongoDurable } from "@x12i/countex/adapters/mongo";
69
- import { RedisCache } from "@x12i/countex/adapters/redis";
70
- import { pipelineRunsMap } from "./maps/pipelineRunsMap.js";
71
-
72
- export function createOpsCountex(deps: { durable: MongoDurable; cache: RedisCache }) {
73
- return createCountex({
74
- maps: [pipelineRunsMap],
75
- durable: deps.durable,
76
- cache: deps.cache,
77
- flush: {
78
- everyMs: 30_000,
79
- maxBufferedCells: 50_000,
80
- maxBufferedEvents: 25_000,
81
- flushOnShutdown: true,
82
- onOverflow: "block",
83
- },
84
- cardinality: {
85
- maxDimensionValuesPerName: 5_000,
86
- onLimit: "overflow",
87
- overflowValue: "__OTHER__",
88
- },
89
- });
90
- }
91
- ```
92
-
93
- **Rationale:**
94
-
95
- - `everyMs: 30_000` balances Mongo write batching vs dashboard freshness.
96
- - `onOverflow: "block"` avoids silent loss under spike load.
97
-
98
- ---
99
-
100
- ## 4. Worker integration (exact steps)
101
-
102
- ### Step 4.1 — On job completion
103
-
104
- ```ts
105
- await countex.ingest({
106
- orgId,
107
- workerId,
108
- pipelineName,
109
- region,
110
- status,
111
- finishedAt: new Date().toISOString(),
112
- });
113
- ```
114
-
115
- ### Step 4.2 — Process shutdown
116
-
117
- ```ts
118
- await countex.shutdown(); // drains buffer when flushOnShutdown true
119
- ```
120
-
121
- ---
122
-
123
- ## 5. Dashboard queries (copy-paste specs)
124
-
125
- ### 5.1 Top failing pipelines (single UTC day)
126
-
127
- In this map **`status` is the metric** (`$.status`), not a dimension. Restrict to failures with **`metrics: ["failed"]`** (metric source key):
128
-
129
- ```ts
130
- const failures = await countex.query({
131
- scope: "org_1",
132
- counterId: "pipeline_runs",
133
- timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
134
- metrics: ["failed"],
135
- groupBy: ["dim:pipeline"],
136
- format: "table",
137
- orderBy: "-count",
138
- limit: 20,
139
- });
140
- ```
141
-
142
- Numeric metric ids from the dictionary are also valid in `metrics: [41]` if you resolve them out-of-band.
143
-
144
- ### 5.2 Hourly throughput (timeseries, UTC day)
145
-
146
- Bucket keys for `hour` are `YYYY-MM-DDTHH` strings (UTC).
147
-
148
- ```ts
149
- const hourly = await countex.query({
150
- scope: "org_1",
151
- counterId: "pipeline_runs",
152
- timeframe: { type: "hour", from: "2026-05-08T00", to: "2026-05-08T23" },
153
- groupBy: [],
154
- format: "timeseries",
155
- });
156
- ```
157
-
158
- ### 5.3 Rate — runs per minute (timeseries)
159
-
160
- ```ts
161
- const rateTs = await countex.query({
162
- scope: "org_1",
163
- counterId: "pipeline_runs",
164
- timeframe: { type: "hour", from: "2026-05-08T00", to: "2026-05-08T23" },
165
- groupBy: [],
166
- format: "timeseries",
167
- rate: { of: "count", per: "minute" },
168
- });
169
- ```
170
-
171
- **Constraint:** `rate` is **invalid** for timeframe type **`all`** (no defined bucket duration).
172
-
173
- ### 5.4 Region breakdown (success vs failed)
174
-
175
- Issue **two** queries or omit metric filter and **group by metric** — example grouping both dimensions:
176
-
177
- ```ts
178
- const byRegion = await countex.query({
179
- scope: "org_1",
180
- counterId: "pipeline_runs",
181
- timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
182
- groupBy: ["dim:region", "metric"],
183
- format: "table",
184
- orderBy: "-count",
185
- limit: 500,
186
- });
187
- ```
188
-
189
- Include **`subjectType`** + **`subject`** in `groupBy` when you need stable interpretation of numeric ids.
190
-
191
- ---
192
-
193
- ## 6. Cache rebuild after Redis loss
194
-
195
- Durable (Mongo) remains authoritative. Rewarm cache for a bounded window:
196
-
197
- ```ts
198
- import { rebuildCache } from "@x12i/countex";
199
-
200
- const { cellsWarmed } = await rebuildCache(durable, cache, {
201
- scope: "org_1",
202
- counterId: "pipeline_runs",
203
- timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
204
- }, { batchSize: 1000 });
205
- ```
206
-
207
- **Exact meaning:** streams `durable.streamCells` for that slice, batches `cache.warm`.
208
-
209
- Optional cancellation:
210
-
211
- ```ts
212
- await rebuildCache(durable, cache, filter, {
213
- batchSize: 1000,
214
- signal: AbortSignal.timeout(600_000),
215
- });
216
- ```
217
-
218
- ---
219
-
220
- ## 7. Bucket window limits
221
-
222
- `bucketRange` caps at **10,000** buckets. For `minute` resolution that is roughly **6.94 days** of continuous range. If your dashboard asks for more, either:
223
-
224
- - Switch to **`hour`** / **`day`**, or
225
- - Split queries into multiple jobs.
226
-
227
- ---
228
-
229
- ## 8. Verification checklist
230
-
231
- | Step | Action | Pass criterion |
232
- |------|--------|----------------|
233
- | Writes | Ingest sample event, `await flush()` | Mongo cells include `counterId: pipeline_runs`. |
234
- | Failures query | §5.1 | Only pipelines with failed increments appear (non-empty when data exists). |
235
- | Timeseries ordering | §5.2 | Rows sorted by `timeframeKey` ascending in formatter output. |
236
- | Rebuild | §6 | `cellsWarmed` > 0 for non-empty day; subsequent cache reads hit full coverage when within TTL. |
237
-
238
- ---
239
-
240
- ## 9. Failure modes
241
-
242
- | Symptom | Cause | Mitigation |
243
- |---------|-------|------------|
244
- | Empty failures table | No `failed` metric rows | Confirm JSON uses exact string `failed` vs `failure`. |
245
- | Query timeout | `groupBy` too wide | Narrow timeframe; add filters; reduce `maxBuffered*` unrelated — tune Mongo indexes. |
246
- | Stale Redis after rebuild | TTL eviction | Increase `ttlSeconds` or rerun rebuild closer to query time. |
247
-
248
- ---
249
-
250
- ## 10. References
251
-
252
- - [`docs/guides/querying.md`](../guides/querying.md)
253
- - [`docs/guides/backup-export.md`](../guides/backup-export.md)
254
- - [`docs/guides/ingestion-and-flush.md`](../guides/ingestion-and-flush.md)
255
- - [`docs/guides/timeframes-and-bucketing.md`](../guides/timeframes-and-bucketing.md)
1
+ # Technical playbook: Operational dashboards and rollups
2
+
3
+ You run **batch or streaming workers** that record pipeline outcomes (success/failure), region, pipeline name, etc. Leadership wants **daily rollups**, **hourly strips**, and **failure-focused tables** without standing up a separate OLAP cluster first.
4
+
5
+ This playbook specifies exact **`CounterMap`**, **`flush`** tuning, **`query`** specs, and **cache rebuild** after incidents.
6
+
7
+ ---
8
+
9
+ ## 1. Domain model
10
+
11
+ ### 1.1 Event
12
+
13
+ ```json
14
+ {
15
+ "orgId": "org_1",
16
+ "workerId": "worker-us-east-1a-07",
17
+ "pipelineName": "invoice_export",
18
+ "region": "us-east-1",
19
+ "status": "failed",
20
+ "finishedAt": "2026-05-08T14:22:00.000Z"
21
+ }
22
+ ```
23
+
24
+ ### 1.2 Questions to answer
25
+
26
+ - Failures per **pipeline** yesterday.
27
+ - Hourly **throughput** (count rate) for a single day.
28
+ - Optional: compare regions (dimension breakdown).
29
+
30
+ ---
31
+
32
+ ## 2. Counter map (exact)
33
+
34
+ `src/countex/maps/pipelineRunsMap.ts`:
35
+
36
+ ```ts
37
+ import type { CounterMap } from "@x12i/countex";
38
+
39
+ export const pipelineRunsMap: CounterMap = {
40
+ counterId: "pipeline_runs",
41
+
42
+ scope: { path: "$.orgId" },
43
+ eventTime: { path: "$.finishedAt" },
44
+
45
+ subjects: [{ type: "worker", path: "$.workerId", required: true }],
46
+ metrics: [{ path: "$.status", required: true }],
47
+
48
+ dimensions: [
49
+ { name: "pipeline", path: "$.pipelineName" },
50
+ { name: "region", path: "$.region" },
51
+ ],
52
+
53
+ delta: { default: 1 },
54
+ timeframes: ["hour", "day"],
55
+ };
56
+ ```
57
+
58
+ ### 2.1 Fan-out
59
+
60
+ `1 worker subject × 1 status metric × 2 timeframes = 2` increments per event (before buffer merge).
61
+
62
+ ---
63
+
64
+ ## 3. Countex construction (exact)
65
+
66
+ ```ts
67
+ import { createCountex } from "@x12i/countex";
68
+ import { MongoDurable } from "@x12i/countex/adapters/mongo";
69
+ import { RedisCache } from "@x12i/countex/adapters/redis";
70
+ import { pipelineRunsMap } from "./maps/pipelineRunsMap.js";
71
+
72
+ export function createOpsCountex(deps: { durable: MongoDurable; cache: RedisCache }) {
73
+ return createCountex({
74
+ maps: [pipelineRunsMap],
75
+ durable: deps.durable,
76
+ cache: deps.cache,
77
+ flush: {
78
+ everyMs: 30_000,
79
+ maxBufferedCells: 50_000,
80
+ maxBufferedEvents: 25_000,
81
+ flushOnShutdown: true,
82
+ onOverflow: "block",
83
+ },
84
+ cardinality: {
85
+ maxDimensionValuesPerName: 5_000,
86
+ onLimit: "overflow",
87
+ overflowValue: "__OTHER__",
88
+ },
89
+ });
90
+ }
91
+ ```
92
+
93
+ **Rationale:**
94
+
95
+ - `everyMs: 30_000` balances Mongo write batching vs dashboard freshness.
96
+ - `onOverflow: "block"` avoids silent loss under spike load.
97
+
98
+ ---
99
+
100
+ ## 4. Worker integration (exact steps)
101
+
102
+ ### Step 4.1 — On job completion
103
+
104
+ ```ts
105
+ await countex.ingest({
106
+ orgId,
107
+ workerId,
108
+ pipelineName,
109
+ region,
110
+ status,
111
+ finishedAt: new Date().toISOString(),
112
+ });
113
+ ```
114
+
115
+ ### Step 4.2 — Process shutdown
116
+
117
+ ```ts
118
+ await countex.shutdown(); // drains buffer when flushOnShutdown true
119
+ ```
120
+
121
+ ---
122
+
123
+ ## 5. Dashboard queries (copy-paste specs)
124
+
125
+ ### 5.1 Top failing pipelines (single UTC day)
126
+
127
+ In this map **`status` is the metric** (`$.status`), not a dimension. Restrict to failures with **`metrics: ["failed"]`** (metric source key):
128
+
129
+ ```ts
130
+ const failures = await countex.query({
131
+ scope: "org_1",
132
+ counterId: "pipeline_runs",
133
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
134
+ metrics: ["failed"],
135
+ groupBy: ["dim:pipeline"],
136
+ format: "table",
137
+ orderBy: "-count",
138
+ limit: 20,
139
+ });
140
+ ```
141
+
142
+ Numeric metric ids from the dictionary are also valid in `metrics: [41]` if you resolve them out-of-band.
143
+
144
+ ### 5.2 Hourly throughput (timeseries, UTC day)
145
+
146
+ Bucket keys for `hour` are `YYYY-MM-DDTHH` strings (UTC).
147
+
148
+ ```ts
149
+ const hourly = await countex.query({
150
+ scope: "org_1",
151
+ counterId: "pipeline_runs",
152
+ timeframe: { type: "hour", from: "2026-05-08T00", to: "2026-05-08T23" },
153
+ groupBy: [],
154
+ format: "timeseries",
155
+ });
156
+ ```
157
+
158
+ ### 5.3 Rate — runs per minute (timeseries)
159
+
160
+ ```ts
161
+ const rateTs = await countex.query({
162
+ scope: "org_1",
163
+ counterId: "pipeline_runs",
164
+ timeframe: { type: "hour", from: "2026-05-08T00", to: "2026-05-08T23" },
165
+ groupBy: [],
166
+ format: "timeseries",
167
+ rate: { of: "count", per: "minute" },
168
+ });
169
+ ```
170
+
171
+ **Constraint:** `rate` is **invalid** for timeframe type **`all`** (no defined bucket duration).
172
+
173
+ ### 5.4 Region breakdown (success vs failed)
174
+
175
+ Issue **two** queries or omit metric filter and **group by metric** — example grouping both dimensions:
176
+
177
+ ```ts
178
+ const byRegion = await countex.query({
179
+ scope: "org_1",
180
+ counterId: "pipeline_runs",
181
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
182
+ groupBy: ["dim:region", "metric"],
183
+ format: "table",
184
+ orderBy: "-count",
185
+ limit: 500,
186
+ });
187
+ ```
188
+
189
+ Include **`subjectType`** + **`subject`** in `groupBy` when you need stable interpretation of numeric ids.
190
+
191
+ ---
192
+
193
+ ## 6. Cache rebuild after Redis loss
194
+
195
+ Durable (Mongo) remains authoritative. Rewarm cache for a bounded window:
196
+
197
+ ```ts
198
+ import { rebuildCache } from "@x12i/countex";
199
+
200
+ const { cellsWarmed } = await rebuildCache(durable, cache, {
201
+ scope: "org_1",
202
+ counterId: "pipeline_runs",
203
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
204
+ }, { batchSize: 1000 });
205
+ ```
206
+
207
+ **Exact meaning:** streams `durable.streamCells` for that slice, batches `cache.warm`.
208
+
209
+ Optional cancellation:
210
+
211
+ ```ts
212
+ await rebuildCache(durable, cache, filter, {
213
+ batchSize: 1000,
214
+ signal: AbortSignal.timeout(600_000),
215
+ });
216
+ ```
217
+
218
+ ---
219
+
220
+ ## 7. Bucket window limits
221
+
222
+ `bucketRange` caps at **10,000** buckets. For `minute` resolution that is roughly **6.94 days** of continuous range. If your dashboard asks for more, either:
223
+
224
+ - Switch to **`hour`** / **`day`**, or
225
+ - Split queries into multiple jobs.
226
+
227
+ ---
228
+
229
+ ## 8. Verification checklist
230
+
231
+ | Step | Action | Pass criterion |
232
+ |------|--------|----------------|
233
+ | Writes | Ingest sample event, `await flush()` | Mongo cells include `counterId: pipeline_runs`. |
234
+ | Failures query | §5.1 | Only pipelines with failed increments appear (non-empty when data exists). |
235
+ | Timeseries ordering | §5.2 | Rows sorted by `timeframeKey` ascending in formatter output. |
236
+ | Rebuild | §6 | `cellsWarmed` > 0 for non-empty day; subsequent cache reads hit full coverage when within TTL. |
237
+
238
+ ---
239
+
240
+ ## 9. Failure modes
241
+
242
+ | Symptom | Cause | Mitigation |
243
+ |---------|-------|------------|
244
+ | Empty failures table | No `failed` metric rows | Confirm JSON uses exact string `failed` vs `failure`. |
245
+ | Query timeout | `groupBy` too wide | Narrow timeframe; add filters; reduce `maxBuffered*` unrelated — tune Mongo indexes. |
246
+ | Stale Redis after rebuild | TTL eviction | Increase `ttlSeconds` or rerun rebuild closer to query time. |
247
+
248
+ ---
249
+
250
+ ## 10. References
251
+
252
+ - [`docs/guides/querying.md`](../guides/querying.md)
253
+ - [`docs/guides/backup-export.md`](../guides/backup-export.md)
254
+ - [`docs/guides/ingestion-and-flush.md`](../guides/ingestion-and-flush.md)
255
+ - [`docs/guides/timeframes-and-bucketing.md`](../guides/timeframes-and-bucketing.md)