@x12i/countex 1.8.5 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +866 -860
  3. package/dist/cli.js +6 -6
  4. package/dist/index.d.ts +1 -0
  5. package/dist/index.d.ts.map +1 -1
  6. package/dist/index.js +1 -0
  7. package/dist/index.js.map +1 -1
  8. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  9. package/dist/insights/advisors/resource-advisor.js +5 -1
  10. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  11. package/dist/insights/types.d.ts +4 -0
  12. package/dist/insights/types.d.ts.map +1 -1
  13. package/dist/storage/postgres.d.ts +104 -0
  14. package/dist/storage/postgres.d.ts.map +1 -0
  15. package/dist/storage/postgres.js +1023 -0
  16. package/dist/storage/postgres.js.map +1 -0
  17. package/dist/types/index.d.ts +11 -5
  18. package/dist/types/index.d.ts.map +1 -1
  19. package/docs/DEPLOYMENT.md +399 -375
  20. package/docs/END_TO_END_EXAMPLE.md +316 -316
  21. package/docs/INSIGHTS.md +59 -59
  22. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  23. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  24. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  25. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  26. package/docs/guides/README.md +36 -34
  27. package/docs/guides/adapter-memory.md +60 -60
  28. package/docs/guides/adapter-mongo.md +64 -62
  29. package/docs/guides/adapter-postgres.md +90 -0
  30. package/docs/guides/adapter-redis.md +89 -89
  31. package/docs/guides/backup-export.md +135 -135
  32. package/docs/guides/cardinality.md +50 -50
  33. package/docs/guides/cli.md +53 -53
  34. package/docs/guides/client-events.md +65 -65
  35. package/docs/guides/counter-maps.md +160 -160
  36. package/docs/guides/idempotency.md +39 -39
  37. package/docs/guides/ingestion-and-flush.md +67 -67
  38. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  39. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  40. package/docs/guides/insights-comparison-and-trends.md +47 -47
  41. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  42. package/docs/guides/insights-overview.md +82 -82
  43. package/docs/guides/insights-ranking.md +81 -81
  44. package/docs/guides/measures.md +71 -71
  45. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  46. package/docs/guides/preview.md +92 -92
  47. package/docs/guides/querying.md +178 -178
  48. package/docs/guides/simulation-profiles.md +33 -33
  49. package/docs/guides/simulation-resource-estimation.md +31 -30
  50. package/docs/guides/simulation.md +36 -36
  51. package/docs/guides/timeframes-and-bucketing.md +55 -55
  52. package/docs/internals/README.md +21 -21
  53. package/docs/internals/internal-architecture-overview.md +65 -65
  54. package/docs/internals/internal-backup-stream.md +23 -23
  55. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  56. package/docs/internals/internal-cardinality.md +30 -30
  57. package/docs/internals/internal-dimension-hash.md +34 -34
  58. package/docs/internals/internal-idempotency.md +36 -36
  59. package/docs/internals/internal-ingest-pipeline.md +52 -52
  60. package/docs/internals/internal-insights-layer.md +36 -36
  61. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  62. package/docs/internals/internal-measures-and-scales.md +35 -35
  63. package/docs/internals/internal-query-path.md +46 -46
  64. package/docs/internals/internal-storage-memory.md +13 -13
  65. package/docs/internals/internal-storage-mongo.md +44 -44
  66. package/docs/internals/internal-storage-postgres.md +32 -0
  67. package/docs/internals/internal-storage-redis.md +63 -63
  68. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  69. package/docs/use-cases/README.md +56 -56
  70. package/docs/use-cases/audit-security-log-counts.md +550 -550
  71. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  72. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  73. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  74. package/docs/use-cases/processing-workflow-activity.md +165 -165
  75. package/docs/use-cases/product-feature-usage.md +240 -240
  76. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  77. package/package.json +184 -170
@@ -1,255 +1,255 @@
1
- # Technical playbook: LLM usage, cost, and latency metering
2
-
3
- You operate an **LLM gateway** or application that emits **one JSON record per inference request**. Finance and reliability need:
4
-
5
- - Request **counts** by app/model/status.
6
- - **Token** totals (input/output).
7
- - **Cost** in a currency-safe representation.
8
- - **Latency** statistics (avg/min/max).
9
-
10
- Countex stores **sparse aggregated cells** + **measure** fields; this playbook specifies exact map and query semantics.
11
-
12
- ---
13
-
14
- ## 1. Problem statement
15
-
16
- ### 1.1 Measures vs counts
17
-
18
- - **`delta` / count** — number of qualifying increments (here: one per request fan-out).
19
- - **`measures`** — numeric aggregations merged into each cell (sum/min/max + implicit count for avg).
20
-
21
- ### 1.2 Fixed-point cost storage
22
-
23
- Floating dollar amounts are unsafe for atomic increments at scale. This playbook uses **`scale: 1_000_000`** to store **micro-dollars** as integers:
24
-
25
- ```
26
- stored_micros = trunc(event_dollars * 1_000_000)
27
- ```
28
-
29
- Query layers read **unscaled** values depending on adapter read path — Memory/Mongo adapters apply inverse scale on read where implemented; **always verify** in your integration tests for your exact adapter version.
30
-
31
- ---
32
-
33
- ## 2. Canonical payload
34
-
35
- ```json
36
- {
37
- "scope": { "id": "org_123" },
38
- "request": {
39
- "id": "llm_req_771",
40
- "createdAt": "2026-05-08T15:02:00.000Z",
41
- "status": "success"
42
- },
43
- "app": { "id": "app-1" },
44
- "model": {
45
- "provider": "openai",
46
- "name": "gpt-4.1"
47
- },
48
- "usage": {
49
- "inputTokens": 1200,
50
- "outputTokens": 350,
51
- "costDollars": 0.042,
52
- "latencyMs": 1800
53
- }
54
- }
55
- ```
56
-
57
- ### 2.1 Required-field matrix
58
-
59
- | Path | Role |
60
- |------|------|
61
- | `$.scope.id` | Tenant scope. |
62
- | `$.request.createdAt` | Event time. |
63
- | `$.app.id` | Subject (required). |
64
- | `$.request.status` | Metric category (required). |
65
- | `$.usage.*` | Measure inputs; governed per-measure `onMissing`. |
66
-
67
- ---
68
-
69
- ## 3. Primitives mapping
70
-
71
- | Primitive | Path |
72
- |-----------|------|
73
- | scope | `$.scope.id` |
74
- | eventTime | `$.request.createdAt` |
75
- | subject `app` | `$.app.id` |
76
- | metric | `$.request.status` |
77
- | dimensions | `provider`, `model` |
78
- | measures | see map |
79
-
80
- ---
81
-
82
- ## 4. Counter map (exact)
83
-
84
- `src/countex/maps/llmGatewayMap.ts`:
85
-
86
- ```ts
87
- import type { CounterMap } from "@x12i/countex";
88
-
89
- export const llmGatewayMap: CounterMap = {
90
- counterId: "llm_gateway",
91
-
92
- scope: { path: "$.scope.id" },
93
- eventTime: { path: "$.request.createdAt" },
94
-
95
- subjects: [{ type: "app", path: "$.app.id", required: true }],
96
- metrics: [{ path: "$.request.status", required: true }],
97
-
98
- dimensions: [
99
- { name: "provider", path: "$.model.provider" },
100
- { name: "model", path: "$.model.name" },
101
- ],
102
-
103
- delta: { default: 1 },
104
- timeframes: ["hour", "day", "month"],
105
-
106
- measures: [
107
- { name: "inputTokens", path: "$.usage.inputTokens", aggregations: ["sum"] },
108
- { name: "outputTokens", path: "$.usage.outputTokens", aggregations: ["sum"] },
109
- {
110
- name: "costMicros",
111
- path: "$.usage.costDollars",
112
- aggregations: ["sum"],
113
- scale: 1_000_000,
114
- },
115
- {
116
- name: "latencyMs",
117
- path: "$.usage.latencyMs",
118
- aggregations: ["sum", "count", "min", "max"],
119
- },
120
- ],
121
- };
122
- ```
123
-
124
- ### 4.1 Aggregation semantics (exact)
125
-
126
- - `latencyMs`: `sum` + `count` enable **avg = sum/count** at query time.
127
- - `min` / `max`: stored on **durable**; cache may omit them — planner routes such queries to durable when needed (`cacheCanServe` in `src/query/planner.ts`).
128
-
129
- ### 4.2 Effective aggregations
130
-
131
- The engine may auto-add **`count`** to a measure when needed for avg semantics (`effectiveMeasureAggregations` in `src/core/extraction.ts`). Do not rely on suppressing `count` if you request min/max on the same measure.
132
-
133
- ---
134
-
135
- ## 5. Fan-out
136
-
137
- One subject × one metric × **T** timeframes:
138
-
139
- ```
140
- R = T
141
- ```
142
-
143
- For three timeframes, **three cells** (before merge with identical keys).
144
-
145
- **Measures replicate** onto each of those cells — cost is attributed per bucket.
146
-
147
- ---
148
-
149
- ## 6. Implementation steps
150
-
151
- ### Step 6.1 — Construct Countex
152
-
153
- Same lifecycle as other playbooks: `MongoDurable`, `RedisCache` optional, `createCountex({ maps: [llmGatewayMap], ... })`, `await init()`.
154
-
155
- ### Step 6.2 — Ingest path
156
-
157
- ```ts
158
- await countex.ingest(gatewayJson);
159
- ```
160
-
161
- If `usage.costDollars` is missing and you did not set `onMissing`, measure warnings occur; subscribe:
162
-
163
- ```ts
164
- countex.on("measure_invalid", (p) => {
165
- // log scopeId, counterId, measureName, reason
166
- });
167
- ```
168
-
169
- ### Step 6.3 — Flush threshold guidance
170
-
171
- LLM streams burst; prefer **`onOverflow: "block"`** under memory pressure so you do not silently drop deltas unless you explicitly chose `"drop"`.
172
-
173
- ---
174
-
175
- ## 7. Queries (exact specs)
176
-
177
- ### 7.1 Cost drivers by model (day)
178
-
179
- ```ts
180
- const drivers = await countex.query({
181
- scope: "org_123",
182
- counterId: "llm_gateway",
183
- timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
184
- groupBy: ["dim:model"],
185
- measures: ["costMicros", "inputTokens", "outputTokens"],
186
- format: "table",
187
- orderBy: { measure: "costMicros", agg: "sum", dir: "desc" },
188
- limit: 50,
189
- });
190
- ```
191
-
192
- ### 7.2 Latency percentiles proxy
193
-
194
- Countex stores **min/max/sum/count**, not full distributions. For p95 you still need raw traces or a histogram elsewhere. Use **`min`/`max`** for coarse SLO bounds.
195
-
196
- ### 7.3 Force durable for min/max heavy dashboards
197
-
198
- ```ts
199
- bypassCache: true,
200
- ```
201
-
202
- when your SQL/BI tool issues arbitrary measure `"*"` pulls including min/max — or accept planner routing (merged/durable) via `result.source`.
203
-
204
- ---
205
-
206
- ## 8. Insights layer (optional second stage)
207
-
208
- ```ts
209
- import { createCountexInsights } from "@x12i/countex/insights";
210
-
211
- const insights = createCountexInsights({
212
- countex,
213
- mapsByCounterId: new Map([["llm_gateway", llmGatewayMap]]),
214
- });
215
-
216
- const top = await insights.topN({
217
- scope: "org_123",
218
- counterId: "llm_gateway",
219
- timeframe: { type: "month", from: "2026-05", to: "2026-05" },
220
- groupBy: ["subject"],
221
- measure: "costMicros.sum",
222
- n: 15,
223
- });
224
- ```
225
-
226
- **Measure selector string** `"costMicros.sum"` is parsed by insights (`parseMeasureSelector`).
227
-
228
- ---
229
-
230
- ## 9. Verification checklist
231
-
232
- | Test | Steps |
233
- |------|-------|
234
- | Sum correctness | Ingest N identical events; query `costMicros.sum` ∝ N × stored increment. |
235
- | Scale correctness | Compare adapter-read values against hand-calculated micro-dollar totals. |
236
- | Min/max path | Query with `orderBy: { measure: "latencyMs", agg: "max", dir: "desc" }`; confirm `source` is not cache-only full lie — planner should bypass cache. |
237
-
238
- ---
239
-
240
- ## 10. Failure modes
241
-
242
- | Symptom | Cause | Mitigation |
243
- |---------|-------|------------|
244
- | `precision_truncated` events | Non-representable micro-dollar after scale | Accept loss or use smaller currency unit. |
245
- | Flat zeros | Wrong scope / timeframe range | Align bucket keys UTC. |
246
- | Cache/durable mismatch on extrema | Expected by design | Document dashboards using min/max as durable-backed. |
247
-
248
- ---
249
-
250
- ## 11. References
251
-
252
- - [`docs/guides/measures.md`](../guides/measures.md)
253
- - [`docs/guides/querying.md`](../guides/querying.md)
254
- - [`docs/internals/internal-query-path.md`](../internals/internal-query-path.md)
255
- - `docs/INSIGHTS_LLM_COST_AND_TOKENS.md`
1
+ # Technical playbook: LLM usage, cost, and latency metering
2
+
3
+ You operate an **LLM gateway** or application that emits **one JSON record per inference request**. Finance and reliability need:
4
+
5
+ - Request **counts** by app/model/status.
6
+ - **Token** totals (input/output).
7
+ - **Cost** in a currency-safe representation.
8
+ - **Latency** statistics (avg/min/max).
9
+
10
+ Countex stores **sparse aggregated cells** + **measure** fields; this playbook specifies exact map and query semantics.
11
+
12
+ ---
13
+
14
+ ## 1. Problem statement
15
+
16
+ ### 1.1 Measures vs counts
17
+
18
+ - **`delta` / count** — number of qualifying increments (here: one per request fan-out).
19
+ - **`measures`** — numeric aggregations merged into each cell (sum/min/max + implicit count for avg).
20
+
21
+ ### 1.2 Fixed-point cost storage
22
+
23
+ Floating dollar amounts are unsafe for atomic increments at scale. This playbook uses **`scale: 1_000_000`** to store **micro-dollars** as integers:
24
+
25
+ ```
26
+ stored_micros = trunc(event_dollars * 1_000_000)
27
+ ```
28
+
29
+ Query layers read **unscaled** values depending on adapter read path — Memory/Mongo adapters apply inverse scale on read where implemented; **always verify** in your integration tests for your exact adapter version.
30
+
31
+ ---
32
+
33
+ ## 2. Canonical payload
34
+
35
+ ```json
36
+ {
37
+ "scope": { "id": "org_123" },
38
+ "request": {
39
+ "id": "llm_req_771",
40
+ "createdAt": "2026-05-08T15:02:00.000Z",
41
+ "status": "success"
42
+ },
43
+ "app": { "id": "app-1" },
44
+ "model": {
45
+ "provider": "openai",
46
+ "name": "gpt-4.1"
47
+ },
48
+ "usage": {
49
+ "inputTokens": 1200,
50
+ "outputTokens": 350,
51
+ "costDollars": 0.042,
52
+ "latencyMs": 1800
53
+ }
54
+ }
55
+ ```
56
+
57
+ ### 2.1 Required-field matrix
58
+
59
+ | Path | Role |
60
+ |------|------|
61
+ | `$.scope.id` | Tenant scope. |
62
+ | `$.request.createdAt` | Event time. |
63
+ | `$.app.id` | Subject (required). |
64
+ | `$.request.status` | Metric category (required). |
65
+ | `$.usage.*` | Measure inputs; governed per-measure `onMissing`. |
66
+
67
+ ---
68
+
69
+ ## 3. Primitives mapping
70
+
71
+ | Primitive | Path |
72
+ |-----------|------|
73
+ | scope | `$.scope.id` |
74
+ | eventTime | `$.request.createdAt` |
75
+ | subject `app` | `$.app.id` |
76
+ | metric | `$.request.status` |
77
+ | dimensions | `provider`, `model` |
78
+ | measures | see map |
79
+
80
+ ---
81
+
82
+ ## 4. Counter map (exact)
83
+
84
+ `src/countex/maps/llmGatewayMap.ts`:
85
+
86
+ ```ts
87
+ import type { CounterMap } from "@x12i/countex";
88
+
89
+ export const llmGatewayMap: CounterMap = {
90
+ counterId: "llm_gateway",
91
+
92
+ scope: { path: "$.scope.id" },
93
+ eventTime: { path: "$.request.createdAt" },
94
+
95
+ subjects: [{ type: "app", path: "$.app.id", required: true }],
96
+ metrics: [{ path: "$.request.status", required: true }],
97
+
98
+ dimensions: [
99
+ { name: "provider", path: "$.model.provider" },
100
+ { name: "model", path: "$.model.name" },
101
+ ],
102
+
103
+ delta: { default: 1 },
104
+ timeframes: ["hour", "day", "month"],
105
+
106
+ measures: [
107
+ { name: "inputTokens", path: "$.usage.inputTokens", aggregations: ["sum"] },
108
+ { name: "outputTokens", path: "$.usage.outputTokens", aggregations: ["sum"] },
109
+ {
110
+ name: "costMicros",
111
+ path: "$.usage.costDollars",
112
+ aggregations: ["sum"],
113
+ scale: 1_000_000,
114
+ },
115
+ {
116
+ name: "latencyMs",
117
+ path: "$.usage.latencyMs",
118
+ aggregations: ["sum", "count", "min", "max"],
119
+ },
120
+ ],
121
+ };
122
+ ```
123
+
124
+ ### 4.1 Aggregation semantics (exact)
125
+
126
+ - `latencyMs`: `sum` + `count` enable **avg = sum/count** at query time.
127
+ - `min` / `max`: stored on **durable**; cache may omit them — planner routes such queries to durable when needed (`cacheCanServe` in `src/query/planner.ts`).
128
+
129
+ ### 4.2 Effective aggregations
130
+
131
+ The engine may auto-add **`count`** to a measure when needed for avg semantics (`effectiveMeasureAggregations` in `src/core/extraction.ts`). Do not rely on suppressing `count` if you request min/max on the same measure.
132
+
133
+ ---
134
+
135
+ ## 5. Fan-out
136
+
137
+ One subject × one metric × **T** timeframes:
138
+
139
+ ```
140
+ R = T
141
+ ```
142
+
143
+ For three timeframes, **three cells** (before merge with identical keys).
144
+
145
+ **Measures replicate** onto each of those cells — cost is attributed per bucket.
146
+
147
+ ---
148
+
149
+ ## 6. Implementation steps
150
+
151
+ ### Step 6.1 — Construct Countex
152
+
153
+ Same lifecycle as other playbooks: `MongoDurable`, `RedisCache` optional, `createCountex({ maps: [llmGatewayMap], ... })`, `await init()`.
154
+
155
+ ### Step 6.2 — Ingest path
156
+
157
+ ```ts
158
+ await countex.ingest(gatewayJson);
159
+ ```
160
+
161
+ If `usage.costDollars` is missing and you did not set `onMissing`, measure warnings occur; subscribe:
162
+
163
+ ```ts
164
+ countex.on("measure_invalid", (p) => {
165
+ // log scopeId, counterId, measureName, reason
166
+ });
167
+ ```
168
+
169
+ ### Step 6.3 — Flush threshold guidance
170
+
171
+ LLM streams burst; prefer **`onOverflow: "block"`** under memory pressure so you do not silently drop deltas unless you explicitly chose `"drop"`.
172
+
173
+ ---
174
+
175
+ ## 7. Queries (exact specs)
176
+
177
+ ### 7.1 Cost drivers by model (day)
178
+
179
+ ```ts
180
+ const drivers = await countex.query({
181
+ scope: "org_123",
182
+ counterId: "llm_gateway",
183
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
184
+ groupBy: ["dim:model"],
185
+ measures: ["costMicros", "inputTokens", "outputTokens"],
186
+ format: "table",
187
+ orderBy: { measure: "costMicros", agg: "sum", dir: "desc" },
188
+ limit: 50,
189
+ });
190
+ ```
191
+
192
+ ### 7.2 Latency percentiles proxy
193
+
194
+ Countex stores **min/max/sum/count**, not full distributions. For p95 you still need raw traces or a histogram elsewhere. Use **`min`/`max`** for coarse SLO bounds.
195
+
196
+ ### 7.3 Force durable for min/max heavy dashboards
197
+
198
+ ```ts
199
+ bypassCache: true,
200
+ ```
201
+
202
+ when your SQL/BI tool issues arbitrary measure `"*"` pulls including min/max — or accept planner routing (merged/durable) via `result.source`.
203
+
204
+ ---
205
+
206
+ ## 8. Insights layer (optional second stage)
207
+
208
+ ```ts
209
+ import { createCountexInsights } from "@x12i/countex/insights";
210
+
211
+ const insights = createCountexInsights({
212
+ countex,
213
+ mapsByCounterId: new Map([["llm_gateway", llmGatewayMap]]),
214
+ });
215
+
216
+ const top = await insights.topN({
217
+ scope: "org_123",
218
+ counterId: "llm_gateway",
219
+ timeframe: { type: "month", from: "2026-05", to: "2026-05" },
220
+ groupBy: ["subject"],
221
+ measure: "costMicros.sum",
222
+ n: 15,
223
+ });
224
+ ```
225
+
226
+ **Measure selector string** `"costMicros.sum"` is parsed by insights (`parseMeasureSelector`).
227
+
228
+ ---
229
+
230
+ ## 9. Verification checklist
231
+
232
+ | Test | Steps |
233
+ |------|-------|
234
+ | Sum correctness | Ingest N identical events; query `costMicros.sum` ∝ N × stored increment. |
235
+ | Scale correctness | Compare adapter-read values against hand-calculated micro-dollar totals. |
236
+ | Min/max path | Query with `orderBy: { measure: "latencyMs", agg: "max", dir: "desc" }`; confirm `source` is not cache-only full lie — planner should bypass cache. |
237
+
238
+ ---
239
+
240
+ ## 10. Failure modes
241
+
242
+ | Symptom | Cause | Mitigation |
243
+ |---------|-------|------------|
244
+ | `precision_truncated` events | Non-representable micro-dollar after scale | Accept loss or use smaller currency unit. |
245
+ | Flat zeros | Wrong scope / timeframe range | Align bucket keys UTC. |
246
+ | Cache/durable mismatch on extrema | Expected by design | Document dashboards using min/max as durable-backed. |
247
+
248
+ ---
249
+
250
+ ## 11. References
251
+
252
+ - [`docs/guides/measures.md`](../guides/measures.md)
253
+ - [`docs/guides/querying.md`](../guides/querying.md)
254
+ - [`docs/internals/internal-query-path.md`](../internals/internal-query-path.md)
255
+ - `docs/INSIGHTS_LLM_COST_AND_TOKENS.md`