@x12i/countex 1.8.5 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +866 -860
- package/dist/cli.js +6 -6
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
- package/dist/insights/advisors/resource-advisor.js +5 -1
- package/dist/insights/advisors/resource-advisor.js.map +1 -1
- package/dist/insights/types.d.ts +4 -0
- package/dist/insights/types.d.ts.map +1 -1
- package/dist/storage/postgres.d.ts +104 -0
- package/dist/storage/postgres.d.ts.map +1 -0
- package/dist/storage/postgres.js +1023 -0
- package/dist/storage/postgres.js.map +1 -0
- package/dist/types/index.d.ts +11 -5
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +399 -375
- package/docs/END_TO_END_EXAMPLE.md +316 -316
- package/docs/INSIGHTS.md +59 -59
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
- package/docs/guides/README.md +36 -34
- package/docs/guides/adapter-memory.md +60 -60
- package/docs/guides/adapter-mongo.md +64 -62
- package/docs/guides/adapter-postgres.md +90 -0
- package/docs/guides/adapter-redis.md +89 -89
- package/docs/guides/backup-export.md +135 -135
- package/docs/guides/cardinality.md +50 -50
- package/docs/guides/cli.md +53 -53
- package/docs/guides/client-events.md +65 -65
- package/docs/guides/counter-maps.md +160 -160
- package/docs/guides/idempotency.md +39 -39
- package/docs/guides/ingestion-and-flush.md +67 -67
- package/docs/guides/insights-advisors-and-optimization.md +39 -39
- package/docs/guides/insights-anomalies-and-rules.md +48 -48
- package/docs/guides/insights-comparison-and-trends.md +47 -47
- package/docs/guides/insights-forecasting-and-budget.md +61 -61
- package/docs/guides/insights-overview.md +82 -82
- package/docs/guides/insights-ranking.md +81 -81
- package/docs/guides/measures.md +71 -71
- package/docs/guides/migrating-mongo-to-postgres.md +127 -0
- package/docs/guides/preview.md +92 -92
- package/docs/guides/querying.md +178 -178
- package/docs/guides/simulation-profiles.md +33 -33
- package/docs/guides/simulation-resource-estimation.md +31 -30
- package/docs/guides/simulation.md +36 -36
- package/docs/guides/timeframes-and-bucketing.md +55 -55
- package/docs/internals/README.md +21 -21
- package/docs/internals/internal-architecture-overview.md +65 -65
- package/docs/internals/internal-backup-stream.md +23 -23
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
- package/docs/internals/internal-cardinality.md +30 -30
- package/docs/internals/internal-dimension-hash.md +34 -34
- package/docs/internals/internal-idempotency.md +36 -36
- package/docs/internals/internal-ingest-pipeline.md +52 -52
- package/docs/internals/internal-insights-layer.md +36 -36
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
- package/docs/internals/internal-measures-and-scales.md +35 -35
- package/docs/internals/internal-query-path.md +46 -46
- package/docs/internals/internal-storage-memory.md +13 -13
- package/docs/internals/internal-storage-mongo.md +44 -44
- package/docs/internals/internal-storage-postgres.md +32 -0
- package/docs/internals/internal-storage-redis.md +63 -63
- package/docs/internals/internal-timeframe-bucketing.md +33 -33
- package/docs/use-cases/README.md +56 -56
- package/docs/use-cases/audit-security-log-counts.md +550 -550
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
- package/docs/use-cases/multi-tenant-isolation.md +203 -203
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
- package/docs/use-cases/processing-workflow-activity.md +165 -165
- package/docs/use-cases/product-feature-usage.md +240 -240
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
- package/package.json +184 -170
|
@@ -1,255 +1,255 @@
|
|
|
1
|
-
# Technical playbook: Operational dashboards and rollups
|
|
2
|
-
|
|
3
|
-
You run **batch or streaming workers** that record pipeline outcomes (success/failure), region, pipeline name, etc. Leadership wants **daily rollups**, **hourly strips**, and **failure-focused tables** without standing up a separate OLAP cluster first.
|
|
4
|
-
|
|
5
|
-
This playbook specifies exact **`CounterMap`**, **`flush`** tuning, **`query`** specs, and **cache rebuild** after incidents.
|
|
6
|
-
|
|
7
|
-
---
|
|
8
|
-
|
|
9
|
-
## 1. Domain model
|
|
10
|
-
|
|
11
|
-
### 1.1 Event
|
|
12
|
-
|
|
13
|
-
```json
|
|
14
|
-
{
|
|
15
|
-
"orgId": "org_1",
|
|
16
|
-
"workerId": "worker-us-east-1a-07",
|
|
17
|
-
"pipelineName": "invoice_export",
|
|
18
|
-
"region": "us-east-1",
|
|
19
|
-
"status": "failed",
|
|
20
|
-
"finishedAt": "2026-05-08T14:22:00.000Z"
|
|
21
|
-
}
|
|
22
|
-
```
|
|
23
|
-
|
|
24
|
-
### 1.2 Questions to answer
|
|
25
|
-
|
|
26
|
-
- Failures per **pipeline** yesterday.
|
|
27
|
-
- Hourly **throughput** (count rate) for a single day.
|
|
28
|
-
- Optional: compare regions (dimension breakdown).
|
|
29
|
-
|
|
30
|
-
---
|
|
31
|
-
|
|
32
|
-
## 2. Counter map (exact)
|
|
33
|
-
|
|
34
|
-
`src/countex/maps/pipelineRunsMap.ts`:
|
|
35
|
-
|
|
36
|
-
```ts
|
|
37
|
-
import type { CounterMap } from "@x12i/countex";
|
|
38
|
-
|
|
39
|
-
export const pipelineRunsMap: CounterMap = {
|
|
40
|
-
counterId: "pipeline_runs",
|
|
41
|
-
|
|
42
|
-
scope: { path: "$.orgId" },
|
|
43
|
-
eventTime: { path: "$.finishedAt" },
|
|
44
|
-
|
|
45
|
-
subjects: [{ type: "worker", path: "$.workerId", required: true }],
|
|
46
|
-
metrics: [{ path: "$.status", required: true }],
|
|
47
|
-
|
|
48
|
-
dimensions: [
|
|
49
|
-
{ name: "pipeline", path: "$.pipelineName" },
|
|
50
|
-
{ name: "region", path: "$.region" },
|
|
51
|
-
],
|
|
52
|
-
|
|
53
|
-
delta: { default: 1 },
|
|
54
|
-
timeframes: ["hour", "day"],
|
|
55
|
-
};
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
### 2.1 Fan-out
|
|
59
|
-
|
|
60
|
-
`1 worker subject × 1 status metric × 2 timeframes = 2` increments per event (before buffer merge).
|
|
61
|
-
|
|
62
|
-
---
|
|
63
|
-
|
|
64
|
-
## 3. Countex construction (exact)
|
|
65
|
-
|
|
66
|
-
```ts
|
|
67
|
-
import { createCountex } from "@x12i/countex";
|
|
68
|
-
import { MongoDurable } from "@x12i/countex/adapters/mongo";
|
|
69
|
-
import { RedisCache } from "@x12i/countex/adapters/redis";
|
|
70
|
-
import { pipelineRunsMap } from "./maps/pipelineRunsMap.js";
|
|
71
|
-
|
|
72
|
-
export function createOpsCountex(deps: { durable: MongoDurable; cache: RedisCache }) {
|
|
73
|
-
return createCountex({
|
|
74
|
-
maps: [pipelineRunsMap],
|
|
75
|
-
durable: deps.durable,
|
|
76
|
-
cache: deps.cache,
|
|
77
|
-
flush: {
|
|
78
|
-
everyMs: 30_000,
|
|
79
|
-
maxBufferedCells: 50_000,
|
|
80
|
-
maxBufferedEvents: 25_000,
|
|
81
|
-
flushOnShutdown: true,
|
|
82
|
-
onOverflow: "block",
|
|
83
|
-
},
|
|
84
|
-
cardinality: {
|
|
85
|
-
maxDimensionValuesPerName: 5_000,
|
|
86
|
-
onLimit: "overflow",
|
|
87
|
-
overflowValue: "__OTHER__",
|
|
88
|
-
},
|
|
89
|
-
});
|
|
90
|
-
}
|
|
91
|
-
```
|
|
92
|
-
|
|
93
|
-
**Rationale:**
|
|
94
|
-
|
|
95
|
-
- `everyMs: 30_000` balances Mongo write batching vs dashboard freshness.
|
|
96
|
-
- `onOverflow: "block"` avoids silent loss under spike load.
|
|
97
|
-
|
|
98
|
-
---
|
|
99
|
-
|
|
100
|
-
## 4. Worker integration (exact steps)
|
|
101
|
-
|
|
102
|
-
### Step 4.1 — On job completion
|
|
103
|
-
|
|
104
|
-
```ts
|
|
105
|
-
await countex.ingest({
|
|
106
|
-
orgId,
|
|
107
|
-
workerId,
|
|
108
|
-
pipelineName,
|
|
109
|
-
region,
|
|
110
|
-
status,
|
|
111
|
-
finishedAt: new Date().toISOString(),
|
|
112
|
-
});
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
### Step 4.2 — Process shutdown
|
|
116
|
-
|
|
117
|
-
```ts
|
|
118
|
-
await countex.shutdown(); // drains buffer when flushOnShutdown true
|
|
119
|
-
```
|
|
120
|
-
|
|
121
|
-
---
|
|
122
|
-
|
|
123
|
-
## 5. Dashboard queries (copy-paste specs)
|
|
124
|
-
|
|
125
|
-
### 5.1 Top failing pipelines (single UTC day)
|
|
126
|
-
|
|
127
|
-
In this map **`status` is the metric** (`$.status`), not a dimension. Restrict to failures with **`metrics: ["failed"]`** (metric source key):
|
|
128
|
-
|
|
129
|
-
```ts
|
|
130
|
-
const failures = await countex.query({
|
|
131
|
-
scope: "org_1",
|
|
132
|
-
counterId: "pipeline_runs",
|
|
133
|
-
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
134
|
-
metrics: ["failed"],
|
|
135
|
-
groupBy: ["dim:pipeline"],
|
|
136
|
-
format: "table",
|
|
137
|
-
orderBy: "-count",
|
|
138
|
-
limit: 20,
|
|
139
|
-
});
|
|
140
|
-
```
|
|
141
|
-
|
|
142
|
-
Numeric metric ids from the dictionary are also valid in `metrics: [41]` if you resolve them out-of-band.
|
|
143
|
-
|
|
144
|
-
### 5.2 Hourly throughput (timeseries, UTC day)
|
|
145
|
-
|
|
146
|
-
Bucket keys for `hour` are `YYYY-MM-DDTHH` strings (UTC).
|
|
147
|
-
|
|
148
|
-
```ts
|
|
149
|
-
const hourly = await countex.query({
|
|
150
|
-
scope: "org_1",
|
|
151
|
-
counterId: "pipeline_runs",
|
|
152
|
-
timeframe: { type: "hour", from: "2026-05-08T00", to: "2026-05-08T23" },
|
|
153
|
-
groupBy: [],
|
|
154
|
-
format: "timeseries",
|
|
155
|
-
});
|
|
156
|
-
```
|
|
157
|
-
|
|
158
|
-
### 5.3 Rate — runs per minute (timeseries)
|
|
159
|
-
|
|
160
|
-
```ts
|
|
161
|
-
const rateTs = await countex.query({
|
|
162
|
-
scope: "org_1",
|
|
163
|
-
counterId: "pipeline_runs",
|
|
164
|
-
timeframe: { type: "hour", from: "2026-05-08T00", to: "2026-05-08T23" },
|
|
165
|
-
groupBy: [],
|
|
166
|
-
format: "timeseries",
|
|
167
|
-
rate: { of: "count", per: "minute" },
|
|
168
|
-
});
|
|
169
|
-
```
|
|
170
|
-
|
|
171
|
-
**Constraint:** `rate` is **invalid** for timeframe type **`all`** (no defined bucket duration).
|
|
172
|
-
|
|
173
|
-
### 5.4 Region breakdown (success vs failed)
|
|
174
|
-
|
|
175
|
-
Issue **two** queries or omit metric filter and **group by metric** — example grouping both dimensions:
|
|
176
|
-
|
|
177
|
-
```ts
|
|
178
|
-
const byRegion = await countex.query({
|
|
179
|
-
scope: "org_1",
|
|
180
|
-
counterId: "pipeline_runs",
|
|
181
|
-
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
182
|
-
groupBy: ["dim:region", "metric"],
|
|
183
|
-
format: "table",
|
|
184
|
-
orderBy: "-count",
|
|
185
|
-
limit: 500,
|
|
186
|
-
});
|
|
187
|
-
```
|
|
188
|
-
|
|
189
|
-
Include **`subjectType`** + **`subject`** in `groupBy` when you need stable interpretation of numeric ids.
|
|
190
|
-
|
|
191
|
-
---
|
|
192
|
-
|
|
193
|
-
## 6. Cache rebuild after Redis loss
|
|
194
|
-
|
|
195
|
-
Durable (Mongo) remains authoritative. Rewarm cache for a bounded window:
|
|
196
|
-
|
|
197
|
-
```ts
|
|
198
|
-
import { rebuildCache } from "@x12i/countex";
|
|
199
|
-
|
|
200
|
-
const { cellsWarmed } = await rebuildCache(durable, cache, {
|
|
201
|
-
scope: "org_1",
|
|
202
|
-
counterId: "pipeline_runs",
|
|
203
|
-
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
204
|
-
}, { batchSize: 1000 });
|
|
205
|
-
```
|
|
206
|
-
|
|
207
|
-
**Exact meaning:** streams `durable.streamCells` for that slice, batches `cache.warm`.
|
|
208
|
-
|
|
209
|
-
Optional cancellation:
|
|
210
|
-
|
|
211
|
-
```ts
|
|
212
|
-
await rebuildCache(durable, cache, filter, {
|
|
213
|
-
batchSize: 1000,
|
|
214
|
-
signal: AbortSignal.timeout(600_000),
|
|
215
|
-
});
|
|
216
|
-
```
|
|
217
|
-
|
|
218
|
-
---
|
|
219
|
-
|
|
220
|
-
## 7. Bucket window limits
|
|
221
|
-
|
|
222
|
-
`bucketRange` caps at **10,000** buckets. For `minute` resolution that is roughly **6.94 days** of continuous range. If your dashboard asks for more, either:
|
|
223
|
-
|
|
224
|
-
- Switch to **`hour`** / **`day`**, or
|
|
225
|
-
- Split queries into multiple jobs.
|
|
226
|
-
|
|
227
|
-
---
|
|
228
|
-
|
|
229
|
-
## 8. Verification checklist
|
|
230
|
-
|
|
231
|
-
| Step | Action | Pass criterion |
|
|
232
|
-
|------|--------|----------------|
|
|
233
|
-
| Writes | Ingest sample event, `await flush()` | Mongo cells include `counterId: pipeline_runs`. |
|
|
234
|
-
| Failures query | §5.1 | Only pipelines with failed increments appear (non-empty when data exists). |
|
|
235
|
-
| Timeseries ordering | §5.2 | Rows sorted by `timeframeKey` ascending in formatter output. |
|
|
236
|
-
| Rebuild | §6 | `cellsWarmed` > 0 for non-empty day; subsequent cache reads hit full coverage when within TTL. |
|
|
237
|
-
|
|
238
|
-
---
|
|
239
|
-
|
|
240
|
-
## 9. Failure modes
|
|
241
|
-
|
|
242
|
-
| Symptom | Cause | Mitigation |
|
|
243
|
-
|---------|-------|------------|
|
|
244
|
-
| Empty failures table | No `failed` metric rows | Confirm JSON uses exact string `failed` vs `failure`. |
|
|
245
|
-
| Query timeout | `groupBy` too wide | Narrow timeframe; add filters; reduce `maxBuffered*` unrelated — tune Mongo indexes. |
|
|
246
|
-
| Stale Redis after rebuild | TTL eviction | Increase `ttlSeconds` or rerun rebuild closer to query time. |
|
|
247
|
-
|
|
248
|
-
---
|
|
249
|
-
|
|
250
|
-
## 10. References
|
|
251
|
-
|
|
252
|
-
- [`docs/guides/querying.md`](../guides/querying.md)
|
|
253
|
-
- [`docs/guides/backup-export.md`](../guides/backup-export.md)
|
|
254
|
-
- [`docs/guides/ingestion-and-flush.md`](../guides/ingestion-and-flush.md)
|
|
255
|
-
- [`docs/guides/timeframes-and-bucketing.md`](../guides/timeframes-and-bucketing.md)
|
|
1
|
+
# Technical playbook: Operational dashboards and rollups
|
|
2
|
+
|
|
3
|
+
You run **batch or streaming workers** that record pipeline outcomes (success/failure), region, pipeline name, etc. Leadership wants **daily rollups**, **hourly strips**, and **failure-focused tables** without standing up a separate OLAP cluster first.
|
|
4
|
+
|
|
5
|
+
This playbook specifies exact **`CounterMap`**, **`flush`** tuning, **`query`** specs, and **cache rebuild** after incidents.
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## 1. Domain model
|
|
10
|
+
|
|
11
|
+
### 1.1 Event
|
|
12
|
+
|
|
13
|
+
```json
|
|
14
|
+
{
|
|
15
|
+
"orgId": "org_1",
|
|
16
|
+
"workerId": "worker-us-east-1a-07",
|
|
17
|
+
"pipelineName": "invoice_export",
|
|
18
|
+
"region": "us-east-1",
|
|
19
|
+
"status": "failed",
|
|
20
|
+
"finishedAt": "2026-05-08T14:22:00.000Z"
|
|
21
|
+
}
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
### 1.2 Questions to answer
|
|
25
|
+
|
|
26
|
+
- Failures per **pipeline** yesterday.
|
|
27
|
+
- Hourly **throughput** (count rate) for a single day.
|
|
28
|
+
- Optional: compare regions (dimension breakdown).
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## 2. Counter map (exact)
|
|
33
|
+
|
|
34
|
+
`src/countex/maps/pipelineRunsMap.ts`:
|
|
35
|
+
|
|
36
|
+
```ts
|
|
37
|
+
import type { CounterMap } from "@x12i/countex";
|
|
38
|
+
|
|
39
|
+
export const pipelineRunsMap: CounterMap = {
|
|
40
|
+
counterId: "pipeline_runs",
|
|
41
|
+
|
|
42
|
+
scope: { path: "$.orgId" },
|
|
43
|
+
eventTime: { path: "$.finishedAt" },
|
|
44
|
+
|
|
45
|
+
subjects: [{ type: "worker", path: "$.workerId", required: true }],
|
|
46
|
+
metrics: [{ path: "$.status", required: true }],
|
|
47
|
+
|
|
48
|
+
dimensions: [
|
|
49
|
+
{ name: "pipeline", path: "$.pipelineName" },
|
|
50
|
+
{ name: "region", path: "$.region" },
|
|
51
|
+
],
|
|
52
|
+
|
|
53
|
+
delta: { default: 1 },
|
|
54
|
+
timeframes: ["hour", "day"],
|
|
55
|
+
};
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
### 2.1 Fan-out
|
|
59
|
+
|
|
60
|
+
`1 worker subject × 1 status metric × 2 timeframes = 2` increments per event (before buffer merge).
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
## 3. Countex construction (exact)
|
|
65
|
+
|
|
66
|
+
```ts
|
|
67
|
+
import { createCountex } from "@x12i/countex";
|
|
68
|
+
import { MongoDurable } from "@x12i/countex/adapters/mongo";
|
|
69
|
+
import { RedisCache } from "@x12i/countex/adapters/redis";
|
|
70
|
+
import { pipelineRunsMap } from "./maps/pipelineRunsMap.js";
|
|
71
|
+
|
|
72
|
+
export function createOpsCountex(deps: { durable: MongoDurable; cache: RedisCache }) {
|
|
73
|
+
return createCountex({
|
|
74
|
+
maps: [pipelineRunsMap],
|
|
75
|
+
durable: deps.durable,
|
|
76
|
+
cache: deps.cache,
|
|
77
|
+
flush: {
|
|
78
|
+
everyMs: 30_000,
|
|
79
|
+
maxBufferedCells: 50_000,
|
|
80
|
+
maxBufferedEvents: 25_000,
|
|
81
|
+
flushOnShutdown: true,
|
|
82
|
+
onOverflow: "block",
|
|
83
|
+
},
|
|
84
|
+
cardinality: {
|
|
85
|
+
maxDimensionValuesPerName: 5_000,
|
|
86
|
+
onLimit: "overflow",
|
|
87
|
+
overflowValue: "__OTHER__",
|
|
88
|
+
},
|
|
89
|
+
});
|
|
90
|
+
}
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
**Rationale:**
|
|
94
|
+
|
|
95
|
+
- `everyMs: 30_000` balances Mongo write batching vs dashboard freshness.
|
|
96
|
+
- `onOverflow: "block"` avoids silent loss under spike load.
|
|
97
|
+
|
|
98
|
+
---
|
|
99
|
+
|
|
100
|
+
## 4. Worker integration (exact steps)
|
|
101
|
+
|
|
102
|
+
### Step 4.1 — On job completion
|
|
103
|
+
|
|
104
|
+
```ts
|
|
105
|
+
await countex.ingest({
|
|
106
|
+
orgId,
|
|
107
|
+
workerId,
|
|
108
|
+
pipelineName,
|
|
109
|
+
region,
|
|
110
|
+
status,
|
|
111
|
+
finishedAt: new Date().toISOString(),
|
|
112
|
+
});
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### Step 4.2 — Process shutdown
|
|
116
|
+
|
|
117
|
+
```ts
|
|
118
|
+
await countex.shutdown(); // drains buffer when flushOnShutdown true
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
---
|
|
122
|
+
|
|
123
|
+
## 5. Dashboard queries (copy-paste specs)
|
|
124
|
+
|
|
125
|
+
### 5.1 Top failing pipelines (single UTC day)
|
|
126
|
+
|
|
127
|
+
In this map **`status` is the metric** (`$.status`), not a dimension. Restrict to failures with **`metrics: ["failed"]`** (metric source key):
|
|
128
|
+
|
|
129
|
+
```ts
|
|
130
|
+
const failures = await countex.query({
|
|
131
|
+
scope: "org_1",
|
|
132
|
+
counterId: "pipeline_runs",
|
|
133
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
134
|
+
metrics: ["failed"],
|
|
135
|
+
groupBy: ["dim:pipeline"],
|
|
136
|
+
format: "table",
|
|
137
|
+
orderBy: "-count",
|
|
138
|
+
limit: 20,
|
|
139
|
+
});
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
Numeric metric ids from the dictionary are also valid in `metrics: [41]` if you resolve them out-of-band.
|
|
143
|
+
|
|
144
|
+
### 5.2 Hourly throughput (timeseries, UTC day)
|
|
145
|
+
|
|
146
|
+
Bucket keys for `hour` are `YYYY-MM-DDTHH` strings (UTC).
|
|
147
|
+
|
|
148
|
+
```ts
|
|
149
|
+
const hourly = await countex.query({
|
|
150
|
+
scope: "org_1",
|
|
151
|
+
counterId: "pipeline_runs",
|
|
152
|
+
timeframe: { type: "hour", from: "2026-05-08T00", to: "2026-05-08T23" },
|
|
153
|
+
groupBy: [],
|
|
154
|
+
format: "timeseries",
|
|
155
|
+
});
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
### 5.3 Rate — runs per minute (timeseries)
|
|
159
|
+
|
|
160
|
+
```ts
|
|
161
|
+
const rateTs = await countex.query({
|
|
162
|
+
scope: "org_1",
|
|
163
|
+
counterId: "pipeline_runs",
|
|
164
|
+
timeframe: { type: "hour", from: "2026-05-08T00", to: "2026-05-08T23" },
|
|
165
|
+
groupBy: [],
|
|
166
|
+
format: "timeseries",
|
|
167
|
+
rate: { of: "count", per: "minute" },
|
|
168
|
+
});
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
**Constraint:** `rate` is **invalid** for timeframe type **`all`** (no defined bucket duration).
|
|
172
|
+
|
|
173
|
+
### 5.4 Region breakdown (success vs failed)
|
|
174
|
+
|
|
175
|
+
Issue **two** queries or omit metric filter and **group by metric** — example grouping both dimensions:
|
|
176
|
+
|
|
177
|
+
```ts
|
|
178
|
+
const byRegion = await countex.query({
|
|
179
|
+
scope: "org_1",
|
|
180
|
+
counterId: "pipeline_runs",
|
|
181
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
182
|
+
groupBy: ["dim:region", "metric"],
|
|
183
|
+
format: "table",
|
|
184
|
+
orderBy: "-count",
|
|
185
|
+
limit: 500,
|
|
186
|
+
});
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
Include **`subjectType`** + **`subject`** in `groupBy` when you need stable interpretation of numeric ids.
|
|
190
|
+
|
|
191
|
+
---
|
|
192
|
+
|
|
193
|
+
## 6. Cache rebuild after Redis loss
|
|
194
|
+
|
|
195
|
+
Durable (Mongo) remains authoritative. Rewarm cache for a bounded window:
|
|
196
|
+
|
|
197
|
+
```ts
|
|
198
|
+
import { rebuildCache } from "@x12i/countex";
|
|
199
|
+
|
|
200
|
+
const { cellsWarmed } = await rebuildCache(durable, cache, {
|
|
201
|
+
scope: "org_1",
|
|
202
|
+
counterId: "pipeline_runs",
|
|
203
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
204
|
+
}, { batchSize: 1000 });
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
**Exact meaning:** streams `durable.streamCells` for that slice, batches `cache.warm`.
|
|
208
|
+
|
|
209
|
+
Optional cancellation:
|
|
210
|
+
|
|
211
|
+
```ts
|
|
212
|
+
await rebuildCache(durable, cache, filter, {
|
|
213
|
+
batchSize: 1000,
|
|
214
|
+
signal: AbortSignal.timeout(600_000),
|
|
215
|
+
});
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
---
|
|
219
|
+
|
|
220
|
+
## 7. Bucket window limits
|
|
221
|
+
|
|
222
|
+
`bucketRange` caps at **10,000** buckets. For `minute` resolution that is roughly **6.94 days** of continuous range. If your dashboard asks for more, either:
|
|
223
|
+
|
|
224
|
+
- Switch to **`hour`** / **`day`**, or
|
|
225
|
+
- Split queries into multiple jobs.
|
|
226
|
+
|
|
227
|
+
---
|
|
228
|
+
|
|
229
|
+
## 8. Verification checklist
|
|
230
|
+
|
|
231
|
+
| Step | Action | Pass criterion |
|
|
232
|
+
|------|--------|----------------|
|
|
233
|
+
| Writes | Ingest sample event, `await flush()` | Mongo cells include `counterId: pipeline_runs`. |
|
|
234
|
+
| Failures query | §5.1 | Only pipelines with failed increments appear (non-empty when data exists). |
|
|
235
|
+
| Timeseries ordering | §5.2 | Rows sorted by `timeframeKey` ascending in formatter output. |
|
|
236
|
+
| Rebuild | §6 | `cellsWarmed` > 0 for non-empty day; subsequent cache reads hit full coverage when within TTL. |
|
|
237
|
+
|
|
238
|
+
---
|
|
239
|
+
|
|
240
|
+
## 9. Failure modes
|
|
241
|
+
|
|
242
|
+
| Symptom | Cause | Mitigation |
|
|
243
|
+
|---------|-------|------------|
|
|
244
|
+
| Empty failures table | No `failed` metric rows | Confirm JSON uses exact string `failed` vs `failure`. |
|
|
245
|
+
| Query timeout | `groupBy` too wide | Narrow timeframe; add filters; reduce `maxBuffered*` unrelated — tune Mongo indexes. |
|
|
246
|
+
| Stale Redis after rebuild | TTL eviction | Increase `ttlSeconds` or rerun rebuild closer to query time. |
|
|
247
|
+
|
|
248
|
+
---
|
|
249
|
+
|
|
250
|
+
## 10. References
|
|
251
|
+
|
|
252
|
+
- [`docs/guides/querying.md`](../guides/querying.md)
|
|
253
|
+
- [`docs/guides/backup-export.md`](../guides/backup-export.md)
|
|
254
|
+
- [`docs/guides/ingestion-and-flush.md`](../guides/ingestion-and-flush.md)
|
|
255
|
+
- [`docs/guides/timeframes-and-bucketing.md`](../guides/timeframes-and-bucketing.md)
|