@x12i/countex 1.8.5 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +866 -860
- package/dist/cli.js +6 -6
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
- package/dist/insights/advisors/resource-advisor.js +5 -1
- package/dist/insights/advisors/resource-advisor.js.map +1 -1
- package/dist/insights/types.d.ts +4 -0
- package/dist/insights/types.d.ts.map +1 -1
- package/dist/storage/postgres.d.ts +104 -0
- package/dist/storage/postgres.d.ts.map +1 -0
- package/dist/storage/postgres.js +1023 -0
- package/dist/storage/postgres.js.map +1 -0
- package/dist/types/index.d.ts +11 -5
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +399 -375
- package/docs/END_TO_END_EXAMPLE.md +316 -316
- package/docs/INSIGHTS.md +59 -59
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
- package/docs/guides/README.md +36 -34
- package/docs/guides/adapter-memory.md +60 -60
- package/docs/guides/adapter-mongo.md +64 -62
- package/docs/guides/adapter-postgres.md +90 -0
- package/docs/guides/adapter-redis.md +89 -89
- package/docs/guides/backup-export.md +135 -135
- package/docs/guides/cardinality.md +50 -50
- package/docs/guides/cli.md +53 -53
- package/docs/guides/client-events.md +65 -65
- package/docs/guides/counter-maps.md +160 -160
- package/docs/guides/idempotency.md +39 -39
- package/docs/guides/ingestion-and-flush.md +67 -67
- package/docs/guides/insights-advisors-and-optimization.md +39 -39
- package/docs/guides/insights-anomalies-and-rules.md +48 -48
- package/docs/guides/insights-comparison-and-trends.md +47 -47
- package/docs/guides/insights-forecasting-and-budget.md +61 -61
- package/docs/guides/insights-overview.md +82 -82
- package/docs/guides/insights-ranking.md +81 -81
- package/docs/guides/measures.md +71 -71
- package/docs/guides/migrating-mongo-to-postgres.md +127 -0
- package/docs/guides/preview.md +92 -92
- package/docs/guides/querying.md +178 -178
- package/docs/guides/simulation-profiles.md +33 -33
- package/docs/guides/simulation-resource-estimation.md +31 -30
- package/docs/guides/simulation.md +36 -36
- package/docs/guides/timeframes-and-bucketing.md +55 -55
- package/docs/internals/README.md +21 -21
- package/docs/internals/internal-architecture-overview.md +65 -65
- package/docs/internals/internal-backup-stream.md +23 -23
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
- package/docs/internals/internal-cardinality.md +30 -30
- package/docs/internals/internal-dimension-hash.md +34 -34
- package/docs/internals/internal-idempotency.md +36 -36
- package/docs/internals/internal-ingest-pipeline.md +52 -52
- package/docs/internals/internal-insights-layer.md +36 -36
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
- package/docs/internals/internal-measures-and-scales.md +35 -35
- package/docs/internals/internal-query-path.md +46 -46
- package/docs/internals/internal-storage-memory.md +13 -13
- package/docs/internals/internal-storage-mongo.md +44 -44
- package/docs/internals/internal-storage-postgres.md +32 -0
- package/docs/internals/internal-storage-redis.md +63 -63
- package/docs/internals/internal-timeframe-bucketing.md +33 -33
- package/docs/use-cases/README.md +56 -56
- package/docs/use-cases/audit-security-log-counts.md +550 -550
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
- package/docs/use-cases/multi-tenant-isolation.md +203 -203
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
- package/docs/use-cases/processing-workflow-activity.md +165 -165
- package/docs/use-cases/product-feature-usage.md +240 -240
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
- package/package.json +184 -170
|
@@ -1,255 +1,255 @@
|
|
|
1
|
-
# Technical playbook: LLM usage, cost, and latency metering
|
|
2
|
-
|
|
3
|
-
You operate an **LLM gateway** or application that emits **one JSON record per inference request**. Finance and reliability need:
|
|
4
|
-
|
|
5
|
-
- Request **counts** by app/model/status.
|
|
6
|
-
- **Token** totals (input/output).
|
|
7
|
-
- **Cost** in a currency-safe representation.
|
|
8
|
-
- **Latency** statistics (avg/min/max).
|
|
9
|
-
|
|
10
|
-
Countex stores **sparse aggregated cells** + **measure** fields; this playbook specifies exact map and query semantics.
|
|
11
|
-
|
|
12
|
-
---
|
|
13
|
-
|
|
14
|
-
## 1. Problem statement
|
|
15
|
-
|
|
16
|
-
### 1.1 Measures vs counts
|
|
17
|
-
|
|
18
|
-
- **`delta` / count** — number of qualifying increments (here: one per request fan-out).
|
|
19
|
-
- **`measures`** — numeric aggregations merged into each cell (sum/min/max + implicit count for avg).
|
|
20
|
-
|
|
21
|
-
### 1.2 Fixed-point cost storage
|
|
22
|
-
|
|
23
|
-
Floating dollar amounts are unsafe for atomic increments at scale. This playbook uses **`scale: 1_000_000`** to store **micro-dollars** as integers:
|
|
24
|
-
|
|
25
|
-
```
|
|
26
|
-
stored_micros = trunc(event_dollars * 1_000_000)
|
|
27
|
-
```
|
|
28
|
-
|
|
29
|
-
Query layers read **unscaled** values depending on adapter read path — Memory/Mongo adapters apply inverse scale on read where implemented; **always verify** in your integration tests for your exact adapter version.
|
|
30
|
-
|
|
31
|
-
---
|
|
32
|
-
|
|
33
|
-
## 2. Canonical payload
|
|
34
|
-
|
|
35
|
-
```json
|
|
36
|
-
{
|
|
37
|
-
"scope": { "id": "org_123" },
|
|
38
|
-
"request": {
|
|
39
|
-
"id": "llm_req_771",
|
|
40
|
-
"createdAt": "2026-05-08T15:02:00.000Z",
|
|
41
|
-
"status": "success"
|
|
42
|
-
},
|
|
43
|
-
"app": { "id": "app-1" },
|
|
44
|
-
"model": {
|
|
45
|
-
"provider": "openai",
|
|
46
|
-
"name": "gpt-4.1"
|
|
47
|
-
},
|
|
48
|
-
"usage": {
|
|
49
|
-
"inputTokens": 1200,
|
|
50
|
-
"outputTokens": 350,
|
|
51
|
-
"costDollars": 0.042,
|
|
52
|
-
"latencyMs": 1800
|
|
53
|
-
}
|
|
54
|
-
}
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
### 2.1 Required-field matrix
|
|
58
|
-
|
|
59
|
-
| Path | Role |
|
|
60
|
-
|------|------|
|
|
61
|
-
| `$.scope.id` | Tenant scope. |
|
|
62
|
-
| `$.request.createdAt` | Event time. |
|
|
63
|
-
| `$.app.id` | Subject (required). |
|
|
64
|
-
| `$.request.status` | Metric category (required). |
|
|
65
|
-
| `$.usage.*` | Measure inputs; governed per-measure `onMissing`. |
|
|
66
|
-
|
|
67
|
-
---
|
|
68
|
-
|
|
69
|
-
## 3. Primitives mapping
|
|
70
|
-
|
|
71
|
-
| Primitive | Path |
|
|
72
|
-
|-----------|------|
|
|
73
|
-
| scope | `$.scope.id` |
|
|
74
|
-
| eventTime | `$.request.createdAt` |
|
|
75
|
-
| subject `app` | `$.app.id` |
|
|
76
|
-
| metric | `$.request.status` |
|
|
77
|
-
| dimensions | `provider`, `model` |
|
|
78
|
-
| measures | see map |
|
|
79
|
-
|
|
80
|
-
---
|
|
81
|
-
|
|
82
|
-
## 4. Counter map (exact)
|
|
83
|
-
|
|
84
|
-
`src/countex/maps/llmGatewayMap.ts`:
|
|
85
|
-
|
|
86
|
-
```ts
|
|
87
|
-
import type { CounterMap } from "@x12i/countex";
|
|
88
|
-
|
|
89
|
-
export const llmGatewayMap: CounterMap = {
|
|
90
|
-
counterId: "llm_gateway",
|
|
91
|
-
|
|
92
|
-
scope: { path: "$.scope.id" },
|
|
93
|
-
eventTime: { path: "$.request.createdAt" },
|
|
94
|
-
|
|
95
|
-
subjects: [{ type: "app", path: "$.app.id", required: true }],
|
|
96
|
-
metrics: [{ path: "$.request.status", required: true }],
|
|
97
|
-
|
|
98
|
-
dimensions: [
|
|
99
|
-
{ name: "provider", path: "$.model.provider" },
|
|
100
|
-
{ name: "model", path: "$.model.name" },
|
|
101
|
-
],
|
|
102
|
-
|
|
103
|
-
delta: { default: 1 },
|
|
104
|
-
timeframes: ["hour", "day", "month"],
|
|
105
|
-
|
|
106
|
-
measures: [
|
|
107
|
-
{ name: "inputTokens", path: "$.usage.inputTokens", aggregations: ["sum"] },
|
|
108
|
-
{ name: "outputTokens", path: "$.usage.outputTokens", aggregations: ["sum"] },
|
|
109
|
-
{
|
|
110
|
-
name: "costMicros",
|
|
111
|
-
path: "$.usage.costDollars",
|
|
112
|
-
aggregations: ["sum"],
|
|
113
|
-
scale: 1_000_000,
|
|
114
|
-
},
|
|
115
|
-
{
|
|
116
|
-
name: "latencyMs",
|
|
117
|
-
path: "$.usage.latencyMs",
|
|
118
|
-
aggregations: ["sum", "count", "min", "max"],
|
|
119
|
-
},
|
|
120
|
-
],
|
|
121
|
-
};
|
|
122
|
-
```
|
|
123
|
-
|
|
124
|
-
### 4.1 Aggregation semantics (exact)
|
|
125
|
-
|
|
126
|
-
- `latencyMs`: `sum` + `count` enable **avg = sum/count** at query time.
|
|
127
|
-
- `min` / `max`: stored on **durable**; cache may omit them — planner routes such queries to durable when needed (`cacheCanServe` in `src/query/planner.ts`).
|
|
128
|
-
|
|
129
|
-
### 4.2 Effective aggregations
|
|
130
|
-
|
|
131
|
-
The engine may auto-add **`count`** to a measure when needed for avg semantics (`effectiveMeasureAggregations` in `src/core/extraction.ts`). Do not rely on suppressing `count` if you request min/max on the same measure.
|
|
132
|
-
|
|
133
|
-
---
|
|
134
|
-
|
|
135
|
-
## 5. Fan-out
|
|
136
|
-
|
|
137
|
-
One subject × one metric × **T** timeframes:
|
|
138
|
-
|
|
139
|
-
```
|
|
140
|
-
R = T
|
|
141
|
-
```
|
|
142
|
-
|
|
143
|
-
For three timeframes, **three cells** (before merge with identical keys).
|
|
144
|
-
|
|
145
|
-
**Measures replicate** onto each of those cells — cost is attributed per bucket.
|
|
146
|
-
|
|
147
|
-
---
|
|
148
|
-
|
|
149
|
-
## 6. Implementation steps
|
|
150
|
-
|
|
151
|
-
### Step 6.1 — Construct Countex
|
|
152
|
-
|
|
153
|
-
Same lifecycle as other playbooks: `MongoDurable`, `RedisCache` optional, `createCountex({ maps: [llmGatewayMap], ... })`, `await init()`.
|
|
154
|
-
|
|
155
|
-
### Step 6.2 — Ingest path
|
|
156
|
-
|
|
157
|
-
```ts
|
|
158
|
-
await countex.ingest(gatewayJson);
|
|
159
|
-
```
|
|
160
|
-
|
|
161
|
-
If `usage.costDollars` is missing and you did not set `onMissing`, measure warnings occur; subscribe:
|
|
162
|
-
|
|
163
|
-
```ts
|
|
164
|
-
countex.on("measure_invalid", (p) => {
|
|
165
|
-
// log scopeId, counterId, measureName, reason
|
|
166
|
-
});
|
|
167
|
-
```
|
|
168
|
-
|
|
169
|
-
### Step 6.3 — Flush threshold guidance
|
|
170
|
-
|
|
171
|
-
LLM streams burst; prefer **`onOverflow: "block"`** under memory pressure so you do not silently drop deltas unless you explicitly chose `"drop"`.
|
|
172
|
-
|
|
173
|
-
---
|
|
174
|
-
|
|
175
|
-
## 7. Queries (exact specs)
|
|
176
|
-
|
|
177
|
-
### 7.1 Cost drivers by model (day)
|
|
178
|
-
|
|
179
|
-
```ts
|
|
180
|
-
const drivers = await countex.query({
|
|
181
|
-
scope: "org_123",
|
|
182
|
-
counterId: "llm_gateway",
|
|
183
|
-
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
184
|
-
groupBy: ["dim:model"],
|
|
185
|
-
measures: ["costMicros", "inputTokens", "outputTokens"],
|
|
186
|
-
format: "table",
|
|
187
|
-
orderBy: { measure: "costMicros", agg: "sum", dir: "desc" },
|
|
188
|
-
limit: 50,
|
|
189
|
-
});
|
|
190
|
-
```
|
|
191
|
-
|
|
192
|
-
### 7.2 Latency percentiles proxy
|
|
193
|
-
|
|
194
|
-
Countex stores **min/max/sum/count**, not full distributions. For p95 you still need raw traces or a histogram elsewhere. Use **`min`/`max`** for coarse SLO bounds.
|
|
195
|
-
|
|
196
|
-
### 7.3 Force durable for min/max heavy dashboards
|
|
197
|
-
|
|
198
|
-
```ts
|
|
199
|
-
bypassCache: true,
|
|
200
|
-
```
|
|
201
|
-
|
|
202
|
-
when your SQL/BI tool issues arbitrary measure `"*"` pulls including min/max — or accept planner routing (merged/durable) via `result.source`.
|
|
203
|
-
|
|
204
|
-
---
|
|
205
|
-
|
|
206
|
-
## 8. Insights layer (optional second stage)
|
|
207
|
-
|
|
208
|
-
```ts
|
|
209
|
-
import { createCountexInsights } from "@x12i/countex/insights";
|
|
210
|
-
|
|
211
|
-
const insights = createCountexInsights({
|
|
212
|
-
countex,
|
|
213
|
-
mapsByCounterId: new Map([["llm_gateway", llmGatewayMap]]),
|
|
214
|
-
});
|
|
215
|
-
|
|
216
|
-
const top = await insights.topN({
|
|
217
|
-
scope: "org_123",
|
|
218
|
-
counterId: "llm_gateway",
|
|
219
|
-
timeframe: { type: "month", from: "2026-05", to: "2026-05" },
|
|
220
|
-
groupBy: ["subject"],
|
|
221
|
-
measure: "costMicros.sum",
|
|
222
|
-
n: 15,
|
|
223
|
-
});
|
|
224
|
-
```
|
|
225
|
-
|
|
226
|
-
**Measure selector string** `"costMicros.sum"` is parsed by insights (`parseMeasureSelector`).
|
|
227
|
-
|
|
228
|
-
---
|
|
229
|
-
|
|
230
|
-
## 9. Verification checklist
|
|
231
|
-
|
|
232
|
-
| Test | Steps |
|
|
233
|
-
|------|-------|
|
|
234
|
-
| Sum correctness | Ingest N identical events; query `costMicros.sum` ∝ N × stored increment. |
|
|
235
|
-
| Scale correctness | Compare adapter-read values against hand-calculated micro-dollar totals. |
|
|
236
|
-
| Min/max path | Query with `orderBy: { measure: "latencyMs", agg: "max", dir: "desc" }`; confirm `source` is not cache-only full lie — planner should bypass cache. |
|
|
237
|
-
|
|
238
|
-
---
|
|
239
|
-
|
|
240
|
-
## 10. Failure modes
|
|
241
|
-
|
|
242
|
-
| Symptom | Cause | Mitigation |
|
|
243
|
-
|---------|-------|------------|
|
|
244
|
-
| `precision_truncated` events | Non-representable micro-dollar after scale | Accept loss or use smaller currency unit. |
|
|
245
|
-
| Flat zeros | Wrong scope / timeframe range | Align bucket keys UTC. |
|
|
246
|
-
| Cache/durable mismatch on extrema | Expected by design | Document dashboards using min/max as durable-backed. |
|
|
247
|
-
|
|
248
|
-
---
|
|
249
|
-
|
|
250
|
-
## 11. References
|
|
251
|
-
|
|
252
|
-
- [`docs/guides/measures.md`](../guides/measures.md)
|
|
253
|
-
- [`docs/guides/querying.md`](../guides/querying.md)
|
|
254
|
-
- [`docs/internals/internal-query-path.md`](../internals/internal-query-path.md)
|
|
255
|
-
- `docs/INSIGHTS_LLM_COST_AND_TOKENS.md`
|
|
1
|
+
# Technical playbook: LLM usage, cost, and latency metering
|
|
2
|
+
|
|
3
|
+
You operate an **LLM gateway** or application that emits **one JSON record per inference request**. Finance and reliability need:
|
|
4
|
+
|
|
5
|
+
- Request **counts** by app/model/status.
|
|
6
|
+
- **Token** totals (input/output).
|
|
7
|
+
- **Cost** in a currency-safe representation.
|
|
8
|
+
- **Latency** statistics (avg/min/max).
|
|
9
|
+
|
|
10
|
+
Countex stores **sparse aggregated cells** + **measure** fields; this playbook specifies exact map and query semantics.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## 1. Problem statement
|
|
15
|
+
|
|
16
|
+
### 1.1 Measures vs counts
|
|
17
|
+
|
|
18
|
+
- **`delta` / count** — number of qualifying increments (here: one per request fan-out).
|
|
19
|
+
- **`measures`** — numeric aggregations merged into each cell (sum/min/max + implicit count for avg).
|
|
20
|
+
|
|
21
|
+
### 1.2 Fixed-point cost storage
|
|
22
|
+
|
|
23
|
+
Floating dollar amounts are unsafe for atomic increments at scale. This playbook uses **`scale: 1_000_000`** to store **micro-dollars** as integers:
|
|
24
|
+
|
|
25
|
+
```
|
|
26
|
+
stored_micros = trunc(event_dollars * 1_000_000)
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Query layers read **unscaled** values depending on adapter read path — Memory/Mongo adapters apply inverse scale on read where implemented; **always verify** in your integration tests for your exact adapter version.
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## 2. Canonical payload
|
|
34
|
+
|
|
35
|
+
```json
|
|
36
|
+
{
|
|
37
|
+
"scope": { "id": "org_123" },
|
|
38
|
+
"request": {
|
|
39
|
+
"id": "llm_req_771",
|
|
40
|
+
"createdAt": "2026-05-08T15:02:00.000Z",
|
|
41
|
+
"status": "success"
|
|
42
|
+
},
|
|
43
|
+
"app": { "id": "app-1" },
|
|
44
|
+
"model": {
|
|
45
|
+
"provider": "openai",
|
|
46
|
+
"name": "gpt-4.1"
|
|
47
|
+
},
|
|
48
|
+
"usage": {
|
|
49
|
+
"inputTokens": 1200,
|
|
50
|
+
"outputTokens": 350,
|
|
51
|
+
"costDollars": 0.042,
|
|
52
|
+
"latencyMs": 1800
|
|
53
|
+
}
|
|
54
|
+
}
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### 2.1 Required-field matrix
|
|
58
|
+
|
|
59
|
+
| Path | Role |
|
|
60
|
+
|------|------|
|
|
61
|
+
| `$.scope.id` | Tenant scope. |
|
|
62
|
+
| `$.request.createdAt` | Event time. |
|
|
63
|
+
| `$.app.id` | Subject (required). |
|
|
64
|
+
| `$.request.status` | Metric category (required). |
|
|
65
|
+
| `$.usage.*` | Measure inputs; governed per-measure `onMissing`. |
|
|
66
|
+
|
|
67
|
+
---
|
|
68
|
+
|
|
69
|
+
## 3. Primitives mapping
|
|
70
|
+
|
|
71
|
+
| Primitive | Path |
|
|
72
|
+
|-----------|------|
|
|
73
|
+
| scope | `$.scope.id` |
|
|
74
|
+
| eventTime | `$.request.createdAt` |
|
|
75
|
+
| subject `app` | `$.app.id` |
|
|
76
|
+
| metric | `$.request.status` |
|
|
77
|
+
| dimensions | `provider`, `model` |
|
|
78
|
+
| measures | see map |
|
|
79
|
+
|
|
80
|
+
---
|
|
81
|
+
|
|
82
|
+
## 4. Counter map (exact)
|
|
83
|
+
|
|
84
|
+
`src/countex/maps/llmGatewayMap.ts`:
|
|
85
|
+
|
|
86
|
+
```ts
|
|
87
|
+
import type { CounterMap } from "@x12i/countex";
|
|
88
|
+
|
|
89
|
+
export const llmGatewayMap: CounterMap = {
|
|
90
|
+
counterId: "llm_gateway",
|
|
91
|
+
|
|
92
|
+
scope: { path: "$.scope.id" },
|
|
93
|
+
eventTime: { path: "$.request.createdAt" },
|
|
94
|
+
|
|
95
|
+
subjects: [{ type: "app", path: "$.app.id", required: true }],
|
|
96
|
+
metrics: [{ path: "$.request.status", required: true }],
|
|
97
|
+
|
|
98
|
+
dimensions: [
|
|
99
|
+
{ name: "provider", path: "$.model.provider" },
|
|
100
|
+
{ name: "model", path: "$.model.name" },
|
|
101
|
+
],
|
|
102
|
+
|
|
103
|
+
delta: { default: 1 },
|
|
104
|
+
timeframes: ["hour", "day", "month"],
|
|
105
|
+
|
|
106
|
+
measures: [
|
|
107
|
+
{ name: "inputTokens", path: "$.usage.inputTokens", aggregations: ["sum"] },
|
|
108
|
+
{ name: "outputTokens", path: "$.usage.outputTokens", aggregations: ["sum"] },
|
|
109
|
+
{
|
|
110
|
+
name: "costMicros",
|
|
111
|
+
path: "$.usage.costDollars",
|
|
112
|
+
aggregations: ["sum"],
|
|
113
|
+
scale: 1_000_000,
|
|
114
|
+
},
|
|
115
|
+
{
|
|
116
|
+
name: "latencyMs",
|
|
117
|
+
path: "$.usage.latencyMs",
|
|
118
|
+
aggregations: ["sum", "count", "min", "max"],
|
|
119
|
+
},
|
|
120
|
+
],
|
|
121
|
+
};
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### 4.1 Aggregation semantics (exact)
|
|
125
|
+
|
|
126
|
+
- `latencyMs`: `sum` + `count` enable **avg = sum/count** at query time.
|
|
127
|
+
- `min` / `max`: stored on **durable**; cache may omit them — planner routes such queries to durable when needed (`cacheCanServe` in `src/query/planner.ts`).
|
|
128
|
+
|
|
129
|
+
### 4.2 Effective aggregations
|
|
130
|
+
|
|
131
|
+
The engine may auto-add **`count`** to a measure when needed for avg semantics (`effectiveMeasureAggregations` in `src/core/extraction.ts`). Do not rely on suppressing `count` if you request min/max on the same measure.
|
|
132
|
+
|
|
133
|
+
---
|
|
134
|
+
|
|
135
|
+
## 5. Fan-out
|
|
136
|
+
|
|
137
|
+
One subject × one metric × **T** timeframes:
|
|
138
|
+
|
|
139
|
+
```
|
|
140
|
+
R = T
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
For three timeframes, **three cells** (before merge with identical keys).
|
|
144
|
+
|
|
145
|
+
**Measures replicate** onto each of those cells — cost is attributed per bucket.
|
|
146
|
+
|
|
147
|
+
---
|
|
148
|
+
|
|
149
|
+
## 6. Implementation steps
|
|
150
|
+
|
|
151
|
+
### Step 6.1 — Construct Countex
|
|
152
|
+
|
|
153
|
+
Same lifecycle as other playbooks: `MongoDurable`, `RedisCache` optional, `createCountex({ maps: [llmGatewayMap], ... })`, `await init()`.
|
|
154
|
+
|
|
155
|
+
### Step 6.2 — Ingest path
|
|
156
|
+
|
|
157
|
+
```ts
|
|
158
|
+
await countex.ingest(gatewayJson);
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
If `usage.costDollars` is missing and you did not set `onMissing`, measure warnings occur; subscribe:
|
|
162
|
+
|
|
163
|
+
```ts
|
|
164
|
+
countex.on("measure_invalid", (p) => {
|
|
165
|
+
// log scopeId, counterId, measureName, reason
|
|
166
|
+
});
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
### Step 6.3 — Flush threshold guidance
|
|
170
|
+
|
|
171
|
+
LLM streams burst; prefer **`onOverflow: "block"`** under memory pressure so you do not silently drop deltas unless you explicitly chose `"drop"`.
|
|
172
|
+
|
|
173
|
+
---
|
|
174
|
+
|
|
175
|
+
## 7. Queries (exact specs)
|
|
176
|
+
|
|
177
|
+
### 7.1 Cost drivers by model (day)
|
|
178
|
+
|
|
179
|
+
```ts
|
|
180
|
+
const drivers = await countex.query({
|
|
181
|
+
scope: "org_123",
|
|
182
|
+
counterId: "llm_gateway",
|
|
183
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
184
|
+
groupBy: ["dim:model"],
|
|
185
|
+
measures: ["costMicros", "inputTokens", "outputTokens"],
|
|
186
|
+
format: "table",
|
|
187
|
+
orderBy: { measure: "costMicros", agg: "sum", dir: "desc" },
|
|
188
|
+
limit: 50,
|
|
189
|
+
});
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
### 7.2 Latency percentiles proxy
|
|
193
|
+
|
|
194
|
+
Countex stores **min/max/sum/count**, not full distributions. For p95 you still need raw traces or a histogram elsewhere. Use **`min`/`max`** for coarse SLO bounds.
|
|
195
|
+
|
|
196
|
+
### 7.3 Force durable for min/max heavy dashboards
|
|
197
|
+
|
|
198
|
+
```ts
|
|
199
|
+
bypassCache: true,
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
when your SQL/BI tool issues arbitrary measure `"*"` pulls including min/max — or accept planner routing (merged/durable) via `result.source`.
|
|
203
|
+
|
|
204
|
+
---
|
|
205
|
+
|
|
206
|
+
## 8. Insights layer (optional second stage)
|
|
207
|
+
|
|
208
|
+
```ts
|
|
209
|
+
import { createCountexInsights } from "@x12i/countex/insights";
|
|
210
|
+
|
|
211
|
+
const insights = createCountexInsights({
|
|
212
|
+
countex,
|
|
213
|
+
mapsByCounterId: new Map([["llm_gateway", llmGatewayMap]]),
|
|
214
|
+
});
|
|
215
|
+
|
|
216
|
+
const top = await insights.topN({
|
|
217
|
+
scope: "org_123",
|
|
218
|
+
counterId: "llm_gateway",
|
|
219
|
+
timeframe: { type: "month", from: "2026-05", to: "2026-05" },
|
|
220
|
+
groupBy: ["subject"],
|
|
221
|
+
measure: "costMicros.sum",
|
|
222
|
+
n: 15,
|
|
223
|
+
});
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
**Measure selector string** `"costMicros.sum"` is parsed by insights (`parseMeasureSelector`).
|
|
227
|
+
|
|
228
|
+
---
|
|
229
|
+
|
|
230
|
+
## 9. Verification checklist
|
|
231
|
+
|
|
232
|
+
| Test | Steps |
|
|
233
|
+
|------|-------|
|
|
234
|
+
| Sum correctness | Ingest N identical events; query `costMicros.sum` ∝ N × stored increment. |
|
|
235
|
+
| Scale correctness | Compare adapter-read values against hand-calculated micro-dollar totals. |
|
|
236
|
+
| Min/max path | Query with `orderBy: { measure: "latencyMs", agg: "max", dir: "desc" }`; confirm `source` is not cache-only full lie — planner should bypass cache. |
|
|
237
|
+
|
|
238
|
+
---
|
|
239
|
+
|
|
240
|
+
## 10. Failure modes
|
|
241
|
+
|
|
242
|
+
| Symptom | Cause | Mitigation |
|
|
243
|
+
|---------|-------|------------|
|
|
244
|
+
| `precision_truncated` events | Non-representable micro-dollar after scale | Accept loss or use smaller currency unit. |
|
|
245
|
+
| Flat zeros | Wrong scope / timeframe range | Align bucket keys UTC. |
|
|
246
|
+
| Cache/durable mismatch on extrema | Expected by design | Document dashboards using min/max as durable-backed. |
|
|
247
|
+
|
|
248
|
+
---
|
|
249
|
+
|
|
250
|
+
## 11. References
|
|
251
|
+
|
|
252
|
+
- [`docs/guides/measures.md`](../guides/measures.md)
|
|
253
|
+
- [`docs/guides/querying.md`](../guides/querying.md)
|
|
254
|
+
- [`docs/internals/internal-query-path.md`](../internals/internal-query-path.md)
|
|
255
|
+
- `docs/INSIGHTS_LLM_COST_AND_TOKENS.md`
|