@x12i/countex 1.8.5 → 1.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +867 -860
- package/dist/cli.js +6 -6
- package/dist/core/countex.d.ts +61 -1
- package/dist/core/countex.d.ts.map +1 -1
- package/dist/core/countex.js +241 -0
- package/dist/core/countex.js.map +1 -1
- package/dist/core/managed.d.ts +200 -0
- package/dist/core/managed.d.ts.map +1 -0
- package/dist/core/managed.js +431 -0
- package/dist/core/managed.js.map +1 -0
- package/dist/core/timeframe.d.ts +5 -0
- package/dist/core/timeframe.d.ts.map +1 -1
- package/dist/core/timeframe.js +36 -0
- package/dist/core/timeframe.js.map +1 -1
- package/dist/index.d.ts +6 -2
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +9 -1
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
- package/dist/insights/advisors/resource-advisor.js +5 -1
- package/dist/insights/advisors/resource-advisor.js.map +1 -1
- package/dist/insights/types.d.ts +4 -0
- package/dist/insights/types.d.ts.map +1 -1
- package/dist/storage/managed-memory.d.ts +33 -0
- package/dist/storage/managed-memory.d.ts.map +1 -0
- package/dist/storage/managed-memory.js +211 -0
- package/dist/storage/managed-memory.js.map +1 -0
- package/dist/storage/memory.d.ts +7 -4
- package/dist/storage/memory.d.ts.map +1 -1
- package/dist/storage/memory.js.map +1 -1
- package/dist/storage/postgres.d.ts +123 -0
- package/dist/storage/postgres.d.ts.map +1 -0
- package/dist/storage/postgres.js +1325 -0
- package/dist/storage/postgres.js.map +1 -0
- package/dist/storage/redis.d.ts +6 -0
- package/dist/storage/redis.d.ts.map +1 -1
- package/dist/storage/redis.js +21 -0
- package/dist/storage/redis.js.map +1 -1
- package/dist/types/index.d.ts +63 -5
- package/dist/types/index.d.ts.map +1 -1
- package/dist/utils/errors.d.ts +12 -0
- package/dist/utils/errors.d.ts.map +1 -1
- package/dist/utils/errors.js +22 -1
- package/dist/utils/errors.js.map +1 -1
- package/docs/DEPLOYMENT.md +399 -375
- package/docs/END_TO_END_EXAMPLE.md +316 -316
- package/docs/INSIGHTS.md +59 -59
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
- package/docs/guides/README.md +37 -34
- package/docs/guides/adapter-memory.md +60 -60
- package/docs/guides/adapter-mongo.md +64 -62
- package/docs/guides/adapter-postgres.md +92 -0
- package/docs/guides/adapter-redis.md +89 -89
- package/docs/guides/backup-export.md +135 -135
- package/docs/guides/cardinality.md +50 -50
- package/docs/guides/cli.md +53 -53
- package/docs/guides/client-events.md +65 -65
- package/docs/guides/counter-maps.md +160 -160
- package/docs/guides/idempotency.md +39 -39
- package/docs/guides/ingestion-and-flush.md +67 -67
- package/docs/guides/insights-advisors-and-optimization.md +39 -39
- package/docs/guides/insights-anomalies-and-rules.md +48 -48
- package/docs/guides/insights-comparison-and-trends.md +47 -47
- package/docs/guides/insights-forecasting-and-budget.md +61 -61
- package/docs/guides/insights-overview.md +82 -82
- package/docs/guides/insights-ranking.md +81 -81
- package/docs/guides/managed-commit.md +119 -0
- package/docs/guides/measures.md +72 -71
- package/docs/guides/migrating-mongo-to-postgres.md +127 -0
- package/docs/guides/preview.md +92 -92
- package/docs/guides/querying.md +178 -178
- package/docs/guides/simulation-profiles.md +33 -33
- package/docs/guides/simulation-resource-estimation.md +31 -30
- package/docs/guides/simulation.md +36 -36
- package/docs/guides/timeframes-and-bucketing.md +55 -55
- package/docs/internals/README.md +21 -21
- package/docs/internals/internal-architecture-overview.md +65 -65
- package/docs/internals/internal-backup-stream.md +23 -23
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
- package/docs/internals/internal-cardinality.md +30 -30
- package/docs/internals/internal-dimension-hash.md +34 -34
- package/docs/internals/internal-idempotency.md +36 -36
- package/docs/internals/internal-ingest-pipeline.md +52 -52
- package/docs/internals/internal-insights-layer.md +36 -36
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
- package/docs/internals/internal-measures-and-scales.md +35 -35
- package/docs/internals/internal-query-path.md +46 -46
- package/docs/internals/internal-storage-memory.md +13 -13
- package/docs/internals/internal-storage-mongo.md +44 -44
- package/docs/internals/internal-storage-postgres.md +32 -0
- package/docs/internals/internal-storage-redis.md +63 -63
- package/docs/internals/internal-timeframe-bucketing.md +33 -33
- package/docs/use-cases/README.md +56 -56
- package/docs/use-cases/audit-security-log-counts.md +550 -550
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
- package/docs/use-cases/multi-tenant-isolation.md +203 -203
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
- package/docs/use-cases/processing-workflow-activity.md +165 -165
- package/docs/use-cases/product-feature-usage.md +240 -240
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
- package/package.json +190 -170
|
@@ -1,165 +1,165 @@
|
|
|
1
|
-
# Technical playbook: Processing and workflow activity
|
|
2
|
-
|
|
3
|
-
You operate **pipelines**, **queues**, **agents**, or **batch jobs**. Events describe steps, statuses, errors, and durations. You want rollups by job/workflow, task type, outcome, and timeframe — without Countex knowing your orchestrator brand.
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
## 1. Problem statement
|
|
8
|
-
|
|
9
|
-
### 1.1 Outcomes
|
|
10
|
-
|
|
11
|
-
- Per **tenant** (`scope`), count activity by **job or run** (`subject`) and **step/task kind** (`metric`).
|
|
12
|
-
- Slice by **status**, **worker pool**, or **region** (`dimensions`).
|
|
13
|
-
- Track **duration** and **retry counts** as **measures** where useful.
|
|
14
|
-
|
|
15
|
-
### 1.2 Constraints
|
|
16
|
-
|
|
17
|
-
- Step names (`metrics`) should stay **bounded** — register allowed task types like an enum.
|
|
18
|
-
- Job ids may be high cardinality; dictionary maps string job ids per scope.
|
|
19
|
-
|
|
20
|
-
---
|
|
21
|
-
|
|
22
|
-
## 2. Primitives mapping
|
|
23
|
-
|
|
24
|
-
| Primitive | JSON path | Notes |
|
|
25
|
-
|-----------|-----------|--------|
|
|
26
|
-
| scope | `$.tenant.id` | Isolation boundary. |
|
|
27
|
-
| eventTime | `$.run.recordedAt` | UTC bucketing. |
|
|
28
|
-
| subject `job` | `$.run.jobId` | Required — one counter row per job run key. |
|
|
29
|
-
| metric | `$.task.name` | Task/step identifier (single path or controlled fan-out). |
|
|
30
|
-
| dim `status` | `$.task.status` | Literal dimension (`success`, `failed`, …). |
|
|
31
|
-
| dim `pool` | `$.worker.pool` | Optional. |
|
|
32
|
-
| delta | `1` | One logical step completion per event (adjust if one event batches many steps). |
|
|
33
|
-
| measure `durationMs` | `$.task.durationMs` | Optional `sum` / `count` / `min` / `max`. |
|
|
34
|
-
|
|
35
|
-
---
|
|
36
|
-
|
|
37
|
-
## 3. Canonical payload
|
|
38
|
-
|
|
39
|
-
```json
|
|
40
|
-
{
|
|
41
|
-
"tenant": { "id": "org_42" },
|
|
42
|
-
"run": {
|
|
43
|
-
"jobId": "job_9f3a",
|
|
44
|
-
"recordedAt": "2026-05-08T18:22:01.000Z"
|
|
45
|
-
},
|
|
46
|
-
"task": {
|
|
47
|
-
"name": "extract_documents",
|
|
48
|
-
"status": "success",
|
|
49
|
-
"durationMs": 842
|
|
50
|
-
},
|
|
51
|
-
"worker": { "pool": "default" }
|
|
52
|
-
}
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
### 3.1 Required-field matrix
|
|
56
|
-
|
|
57
|
-
| Path | Required | Effect if absent |
|
|
58
|
-
|------|----------|------------------|
|
|
59
|
-
| `$.tenant.id` | Yes | Skip for map. |
|
|
60
|
-
| `$.run.recordedAt` | Yes | Skip. |
|
|
61
|
-
| `$.run.jobId` | Yes | Skip (`required: true` subject). |
|
|
62
|
-
| `$.task.name` | Yes | Skip (`metrics.required: true`). |
|
|
63
|
-
| `$.task.status` | No | Optional dimension omitted if missing. |
|
|
64
|
-
| `$.worker.pool` | No | Same. |
|
|
65
|
-
| `$.task.durationMs` | No | Measure skipped per `onMissing`. |
|
|
66
|
-
|
|
67
|
-
---
|
|
68
|
-
|
|
69
|
-
## 4. Counter map (exact)
|
|
70
|
-
|
|
71
|
-
```ts
|
|
72
|
-
import type { CounterMap } from "@x12i/countex";
|
|
73
|
-
|
|
74
|
-
export const processingWorkflowMap: CounterMap = {
|
|
75
|
-
counterId: "processing_workflow",
|
|
76
|
-
|
|
77
|
-
scope: { path: "$.tenant.id" },
|
|
78
|
-
eventTime: { path: "$.run.recordedAt" },
|
|
79
|
-
|
|
80
|
-
subjects: [{ type: "job", path: "$.run.jobId", required: true }],
|
|
81
|
-
|
|
82
|
-
metrics: [{ path: "$.task.name", required: true }],
|
|
83
|
-
|
|
84
|
-
dimensions: [
|
|
85
|
-
{ name: "status", path: "$.task.status", required: false },
|
|
86
|
-
{ name: "pool", path: "$.worker.pool", required: false },
|
|
87
|
-
],
|
|
88
|
-
|
|
89
|
-
delta: { default: 1 },
|
|
90
|
-
timeframes: ["hour", "day", "month"],
|
|
91
|
-
|
|
92
|
-
measures: [
|
|
93
|
-
{
|
|
94
|
-
name: "durationMs",
|
|
95
|
-
path: "$.task.durationMs",
|
|
96
|
-
aggregations: ["sum", "count", "min", "max"],
|
|
97
|
-
onMissing: "skip",
|
|
98
|
-
},
|
|
99
|
-
],
|
|
100
|
-
};
|
|
101
|
-
```
|
|
102
|
-
|
|
103
|
-
---
|
|
104
|
-
|
|
105
|
-
## 5. Fan-out arithmetic
|
|
106
|
-
|
|
107
|
-
One subject × one metric × `timeframes.length` increments per event unless paths fan out.
|
|
108
|
-
|
|
109
|
-
Example: **1 × 1 × 3 = 3** cells updated per event for `hour`, `day`, `month`.
|
|
110
|
-
|
|
111
|
-
---
|
|
112
|
-
|
|
113
|
-
## 6. Implementation steps
|
|
114
|
-
|
|
115
|
-
1. Validate payloads at your ingress (schema or optional `validateMap(map, samples)` with fixtures).
|
|
116
|
-
2. `createCountex({ maps: [processingWorkflowMap], durable, cache })`.
|
|
117
|
-
3. `ingest(event)` on each completed step or aggregated batch boundary.
|
|
118
|
-
4. `flush()` on interval or shutdown.
|
|
119
|
-
|
|
120
|
-
---
|
|
121
|
-
|
|
122
|
-
## 7. Query specifications
|
|
123
|
-
|
|
124
|
-
```ts
|
|
125
|
-
await countex.query({
|
|
126
|
-
scope: "org_42",
|
|
127
|
-
counterId: "processing_workflow",
|
|
128
|
-
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
129
|
-
groupBy: ["metric"],
|
|
130
|
-
measures: ["durationMs"],
|
|
131
|
-
format: "table",
|
|
132
|
-
});
|
|
133
|
-
```
|
|
134
|
-
|
|
135
|
-
---
|
|
136
|
-
|
|
137
|
-
## 8. Production hardening
|
|
138
|
-
|
|
139
|
-
- **Idempotency** if the same step may be reported twice (dedupe on `event.id`).
|
|
140
|
-
- **Cardinality** on `task.name` — alert when unknown names appear.
|
|
141
|
-
- **Minute/hour** buckets only if you need SLA dashboards; they multiply storage.
|
|
142
|
-
|
|
143
|
-
---
|
|
144
|
-
|
|
145
|
-
## 9. Verification
|
|
146
|
-
|
|
147
|
-
- Dry-run mapping with **`preview`** on sample payloads (see package release notes / API).
|
|
148
|
-
- Compare simulator **`audit-log`** / **`processing-activity`** profiles against staging counters.
|
|
149
|
-
|
|
150
|
-
---
|
|
151
|
-
|
|
152
|
-
## 10. Failure modes
|
|
153
|
-
|
|
154
|
-
| Symptom | Cause | Mitigation |
|
|
155
|
-
|---------|-------|------------|
|
|
156
|
-
| Missing rollups | Required paths absent | Fix producer or relax `required`. |
|
|
157
|
-
| Cardinality blow-up | Unbounded `task.name` | Registry + overflow dimension policy. |
|
|
158
|
-
| Slow queries | Too-wide timeframe + huge groups | Narrow `bucketRange`, pre-aggregate in map if needed. |
|
|
159
|
-
|
|
160
|
-
---
|
|
161
|
-
|
|
162
|
-
## Related
|
|
163
|
-
|
|
164
|
-
- [Product & feature activity](./product-feature-usage.md) — analogous pattern for user-facing events.
|
|
165
|
-
- [Simulation profiles](../guides/simulation-profiles.md) — `processing-activity` profile.
|
|
1
|
+
# Technical playbook: Processing and workflow activity
|
|
2
|
+
|
|
3
|
+
You operate **pipelines**, **queues**, **agents**, or **batch jobs**. Events describe steps, statuses, errors, and durations. You want rollups by job/workflow, task type, outcome, and timeframe — without Countex knowing your orchestrator brand.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## 1. Problem statement
|
|
8
|
+
|
|
9
|
+
### 1.1 Outcomes
|
|
10
|
+
|
|
11
|
+
- Per **tenant** (`scope`), count activity by **job or run** (`subject`) and **step/task kind** (`metric`).
|
|
12
|
+
- Slice by **status**, **worker pool**, or **region** (`dimensions`).
|
|
13
|
+
- Track **duration** and **retry counts** as **measures** where useful.
|
|
14
|
+
|
|
15
|
+
### 1.2 Constraints
|
|
16
|
+
|
|
17
|
+
- Step names (`metrics`) should stay **bounded** — register allowed task types like an enum.
|
|
18
|
+
- Job ids may be high cardinality; dictionary maps string job ids per scope.
|
|
19
|
+
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## 2. Primitives mapping
|
|
23
|
+
|
|
24
|
+
| Primitive | JSON path | Notes |
|
|
25
|
+
|-----------|-----------|--------|
|
|
26
|
+
| scope | `$.tenant.id` | Isolation boundary. |
|
|
27
|
+
| eventTime | `$.run.recordedAt` | UTC bucketing. |
|
|
28
|
+
| subject `job` | `$.run.jobId` | Required — one counter row per job run key. |
|
|
29
|
+
| metric | `$.task.name` | Task/step identifier (single path or controlled fan-out). |
|
|
30
|
+
| dim `status` | `$.task.status` | Literal dimension (`success`, `failed`, …). |
|
|
31
|
+
| dim `pool` | `$.worker.pool` | Optional. |
|
|
32
|
+
| delta | `1` | One logical step completion per event (adjust if one event batches many steps). |
|
|
33
|
+
| measure `durationMs` | `$.task.durationMs` | Optional `sum` / `count` / `min` / `max`. |
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## 3. Canonical payload
|
|
38
|
+
|
|
39
|
+
```json
|
|
40
|
+
{
|
|
41
|
+
"tenant": { "id": "org_42" },
|
|
42
|
+
"run": {
|
|
43
|
+
"jobId": "job_9f3a",
|
|
44
|
+
"recordedAt": "2026-05-08T18:22:01.000Z"
|
|
45
|
+
},
|
|
46
|
+
"task": {
|
|
47
|
+
"name": "extract_documents",
|
|
48
|
+
"status": "success",
|
|
49
|
+
"durationMs": 842
|
|
50
|
+
},
|
|
51
|
+
"worker": { "pool": "default" }
|
|
52
|
+
}
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### 3.1 Required-field matrix
|
|
56
|
+
|
|
57
|
+
| Path | Required | Effect if absent |
|
|
58
|
+
|------|----------|------------------|
|
|
59
|
+
| `$.tenant.id` | Yes | Skip for map. |
|
|
60
|
+
| `$.run.recordedAt` | Yes | Skip. |
|
|
61
|
+
| `$.run.jobId` | Yes | Skip (`required: true` subject). |
|
|
62
|
+
| `$.task.name` | Yes | Skip (`metrics.required: true`). |
|
|
63
|
+
| `$.task.status` | No | Optional dimension omitted if missing. |
|
|
64
|
+
| `$.worker.pool` | No | Same. |
|
|
65
|
+
| `$.task.durationMs` | No | Measure skipped per `onMissing`. |
|
|
66
|
+
|
|
67
|
+
---
|
|
68
|
+
|
|
69
|
+
## 4. Counter map (exact)
|
|
70
|
+
|
|
71
|
+
```ts
|
|
72
|
+
import type { CounterMap } from "@x12i/countex";
|
|
73
|
+
|
|
74
|
+
export const processingWorkflowMap: CounterMap = {
|
|
75
|
+
counterId: "processing_workflow",
|
|
76
|
+
|
|
77
|
+
scope: { path: "$.tenant.id" },
|
|
78
|
+
eventTime: { path: "$.run.recordedAt" },
|
|
79
|
+
|
|
80
|
+
subjects: [{ type: "job", path: "$.run.jobId", required: true }],
|
|
81
|
+
|
|
82
|
+
metrics: [{ path: "$.task.name", required: true }],
|
|
83
|
+
|
|
84
|
+
dimensions: [
|
|
85
|
+
{ name: "status", path: "$.task.status", required: false },
|
|
86
|
+
{ name: "pool", path: "$.worker.pool", required: false },
|
|
87
|
+
],
|
|
88
|
+
|
|
89
|
+
delta: { default: 1 },
|
|
90
|
+
timeframes: ["hour", "day", "month"],
|
|
91
|
+
|
|
92
|
+
measures: [
|
|
93
|
+
{
|
|
94
|
+
name: "durationMs",
|
|
95
|
+
path: "$.task.durationMs",
|
|
96
|
+
aggregations: ["sum", "count", "min", "max"],
|
|
97
|
+
onMissing: "skip",
|
|
98
|
+
},
|
|
99
|
+
],
|
|
100
|
+
};
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
---
|
|
104
|
+
|
|
105
|
+
## 5. Fan-out arithmetic
|
|
106
|
+
|
|
107
|
+
One subject × one metric × `timeframes.length` increments per event unless paths fan out.
|
|
108
|
+
|
|
109
|
+
Example: **1 × 1 × 3 = 3** cells updated per event for `hour`, `day`, `month`.
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
|
113
|
+
## 6. Implementation steps
|
|
114
|
+
|
|
115
|
+
1. Validate payloads at your ingress (schema or optional `validateMap(map, samples)` with fixtures).
|
|
116
|
+
2. `createCountex({ maps: [processingWorkflowMap], durable, cache })`.
|
|
117
|
+
3. `ingest(event)` on each completed step or aggregated batch boundary.
|
|
118
|
+
4. `flush()` on interval or shutdown.
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
## 7. Query specifications
|
|
123
|
+
|
|
124
|
+
```ts
|
|
125
|
+
await countex.query({
|
|
126
|
+
scope: "org_42",
|
|
127
|
+
counterId: "processing_workflow",
|
|
128
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
129
|
+
groupBy: ["metric"],
|
|
130
|
+
measures: ["durationMs"],
|
|
131
|
+
format: "table",
|
|
132
|
+
});
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## 8. Production hardening
|
|
138
|
+
|
|
139
|
+
- **Idempotency** if the same step may be reported twice (dedupe on `event.id`).
|
|
140
|
+
- **Cardinality** on `task.name` — alert when unknown names appear.
|
|
141
|
+
- **Minute/hour** buckets only if you need SLA dashboards; they multiply storage.
|
|
142
|
+
|
|
143
|
+
---
|
|
144
|
+
|
|
145
|
+
## 9. Verification
|
|
146
|
+
|
|
147
|
+
- Dry-run mapping with **`preview`** on sample payloads (see package release notes / API).
|
|
148
|
+
- Compare simulator **`audit-log`** / **`processing-activity`** profiles against staging counters.
|
|
149
|
+
|
|
150
|
+
---
|
|
151
|
+
|
|
152
|
+
## 10. Failure modes
|
|
153
|
+
|
|
154
|
+
| Symptom | Cause | Mitigation |
|
|
155
|
+
|---------|-------|------------|
|
|
156
|
+
| Missing rollups | Required paths absent | Fix producer or relax `required`. |
|
|
157
|
+
| Cardinality blow-up | Unbounded `task.name` | Registry + overflow dimension policy. |
|
|
158
|
+
| Slow queries | Too-wide timeframe + huge groups | Narrow `bucketRange`, pre-aggregate in map if needed. |
|
|
159
|
+
|
|
160
|
+
---
|
|
161
|
+
|
|
162
|
+
## Related
|
|
163
|
+
|
|
164
|
+
- [Product & feature activity](./product-feature-usage.md) — analogous pattern for user-facing events.
|
|
165
|
+
- [Simulation profiles](../guides/simulation-profiles.md) — `processing-activity` profile.
|