@x12i/countex 1.8.3 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +866 -858
  3. package/dist/cli.js +6 -6
  4. package/dist/index.d.ts +1 -0
  5. package/dist/index.d.ts.map +1 -1
  6. package/dist/index.js +1 -0
  7. package/dist/index.js.map +1 -1
  8. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  9. package/dist/insights/advisors/resource-advisor.js +5 -1
  10. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  11. package/dist/insights/types.d.ts +4 -0
  12. package/dist/insights/types.d.ts.map +1 -1
  13. package/dist/storage/postgres.d.ts +104 -0
  14. package/dist/storage/postgres.d.ts.map +1 -0
  15. package/dist/storage/postgres.js +1023 -0
  16. package/dist/storage/postgres.js.map +1 -0
  17. package/dist/types/index.d.ts +11 -5
  18. package/dist/types/index.d.ts.map +1 -1
  19. package/docs/DEPLOYMENT.md +399 -375
  20. package/docs/END_TO_END_EXAMPLE.md +316 -316
  21. package/docs/INSIGHTS.md +59 -59
  22. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  23. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  24. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  25. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  26. package/docs/guides/README.md +36 -34
  27. package/docs/guides/adapter-memory.md +60 -60
  28. package/docs/guides/adapter-mongo.md +64 -62
  29. package/docs/guides/adapter-postgres.md +90 -0
  30. package/docs/guides/adapter-redis.md +89 -89
  31. package/docs/guides/backup-export.md +135 -135
  32. package/docs/guides/cardinality.md +50 -50
  33. package/docs/guides/cli.md +53 -53
  34. package/docs/guides/client-events.md +65 -65
  35. package/docs/guides/counter-maps.md +160 -160
  36. package/docs/guides/idempotency.md +39 -39
  37. package/docs/guides/ingestion-and-flush.md +67 -67
  38. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  39. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  40. package/docs/guides/insights-comparison-and-trends.md +47 -47
  41. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  42. package/docs/guides/insights-overview.md +82 -82
  43. package/docs/guides/insights-ranking.md +81 -81
  44. package/docs/guides/measures.md +71 -71
  45. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  46. package/docs/guides/preview.md +92 -92
  47. package/docs/guides/querying.md +178 -178
  48. package/docs/guides/simulation-profiles.md +33 -33
  49. package/docs/guides/simulation-resource-estimation.md +31 -30
  50. package/docs/guides/simulation.md +36 -36
  51. package/docs/guides/timeframes-and-bucketing.md +55 -55
  52. package/docs/internals/README.md +21 -21
  53. package/docs/internals/internal-architecture-overview.md +65 -65
  54. package/docs/internals/internal-backup-stream.md +23 -23
  55. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  56. package/docs/internals/internal-cardinality.md +30 -30
  57. package/docs/internals/internal-dimension-hash.md +34 -34
  58. package/docs/internals/internal-idempotency.md +36 -36
  59. package/docs/internals/internal-ingest-pipeline.md +52 -52
  60. package/docs/internals/internal-insights-layer.md +36 -36
  61. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  62. package/docs/internals/internal-measures-and-scales.md +35 -35
  63. package/docs/internals/internal-query-path.md +46 -46
  64. package/docs/internals/internal-storage-memory.md +13 -13
  65. package/docs/internals/internal-storage-mongo.md +44 -44
  66. package/docs/internals/internal-storage-postgres.md +32 -0
  67. package/docs/internals/internal-storage-redis.md +63 -63
  68. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  69. package/docs/use-cases/README.md +56 -56
  70. package/docs/use-cases/audit-security-log-counts.md +550 -550
  71. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  72. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  73. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  74. package/docs/use-cases/processing-workflow-activity.md +165 -165
  75. package/docs/use-cases/product-feature-usage.md +240 -240
  76. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  77. package/package.json +184 -169
@@ -1,165 +1,165 @@
1
- # Technical playbook: Processing and workflow activity
2
-
3
- You operate **pipelines**, **queues**, **agents**, or **batch jobs**. Events describe steps, statuses, errors, and durations. You want rollups by job/workflow, task type, outcome, and timeframe — without Countex knowing your orchestrator brand.
4
-
5
- ---
6
-
7
- ## 1. Problem statement
8
-
9
- ### 1.1 Outcomes
10
-
11
- - Per **tenant** (`scope`), count activity by **job or run** (`subject`) and **step/task kind** (`metric`).
12
- - Slice by **status**, **worker pool**, or **region** (`dimensions`).
13
- - Track **duration** and **retry counts** as **measures** where useful.
14
-
15
- ### 1.2 Constraints
16
-
17
- - Step names (`metrics`) should stay **bounded** — register allowed task types like an enum.
18
- - Job ids may be high cardinality; dictionary maps string job ids per scope.
19
-
20
- ---
21
-
22
- ## 2. Primitives mapping
23
-
24
- | Primitive | JSON path | Notes |
25
- |-----------|-----------|--------|
26
- | scope | `$.tenant.id` | Isolation boundary. |
27
- | eventTime | `$.run.recordedAt` | UTC bucketing. |
28
- | subject `job` | `$.run.jobId` | Required — one counter row per job run key. |
29
- | metric | `$.task.name` | Task/step identifier (single path or controlled fan-out). |
30
- | dim `status` | `$.task.status` | Literal dimension (`success`, `failed`, …). |
31
- | dim `pool` | `$.worker.pool` | Optional. |
32
- | delta | `1` | One logical step completion per event (adjust if one event batches many steps). |
33
- | measure `durationMs` | `$.task.durationMs` | Optional `sum` / `count` / `min` / `max`. |
34
-
35
- ---
36
-
37
- ## 3. Canonical payload
38
-
39
- ```json
40
- {
41
- "tenant": { "id": "org_42" },
42
- "run": {
43
- "jobId": "job_9f3a",
44
- "recordedAt": "2026-05-08T18:22:01.000Z"
45
- },
46
- "task": {
47
- "name": "extract_documents",
48
- "status": "success",
49
- "durationMs": 842
50
- },
51
- "worker": { "pool": "default" }
52
- }
53
- ```
54
-
55
- ### 3.1 Required-field matrix
56
-
57
- | Path | Required | Effect if absent |
58
- |------|----------|------------------|
59
- | `$.tenant.id` | Yes | Skip for map. |
60
- | `$.run.recordedAt` | Yes | Skip. |
61
- | `$.run.jobId` | Yes | Skip (`required: true` subject). |
62
- | `$.task.name` | Yes | Skip (`metrics.required: true`). |
63
- | `$.task.status` | No | Optional dimension omitted if missing. |
64
- | `$.worker.pool` | No | Same. |
65
- | `$.task.durationMs` | No | Measure skipped per `onMissing`. |
66
-
67
- ---
68
-
69
- ## 4. Counter map (exact)
70
-
71
- ```ts
72
- import type { CounterMap } from "@x12i/countex";
73
-
74
- export const processingWorkflowMap: CounterMap = {
75
- counterId: "processing_workflow",
76
-
77
- scope: { path: "$.tenant.id" },
78
- eventTime: { path: "$.run.recordedAt" },
79
-
80
- subjects: [{ type: "job", path: "$.run.jobId", required: true }],
81
-
82
- metrics: [{ path: "$.task.name", required: true }],
83
-
84
- dimensions: [
85
- { name: "status", path: "$.task.status", required: false },
86
- { name: "pool", path: "$.worker.pool", required: false },
87
- ],
88
-
89
- delta: { default: 1 },
90
- timeframes: ["hour", "day", "month"],
91
-
92
- measures: [
93
- {
94
- name: "durationMs",
95
- path: "$.task.durationMs",
96
- aggregations: ["sum", "count", "min", "max"],
97
- onMissing: "skip",
98
- },
99
- ],
100
- };
101
- ```
102
-
103
- ---
104
-
105
- ## 5. Fan-out arithmetic
106
-
107
- One subject × one metric × `timeframes.length` increments per event unless paths fan out.
108
-
109
- Example: **1 × 1 × 3 = 3** cells updated per event for `hour`, `day`, `month`.
110
-
111
- ---
112
-
113
- ## 6. Implementation steps
114
-
115
- 1. Validate payloads at your ingress (schema or optional `validateMap(map, samples)` with fixtures).
116
- 2. `createCountex({ maps: [processingWorkflowMap], durable, cache })`.
117
- 3. `ingest(event)` on each completed step or aggregated batch boundary.
118
- 4. `flush()` on interval or shutdown.
119
-
120
- ---
121
-
122
- ## 7. Query specifications
123
-
124
- ```ts
125
- await countex.query({
126
- scope: "org_42",
127
- counterId: "processing_workflow",
128
- timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
129
- groupBy: ["metric"],
130
- measures: ["durationMs"],
131
- format: "table",
132
- });
133
- ```
134
-
135
- ---
136
-
137
- ## 8. Production hardening
138
-
139
- - **Idempotency** if the same step may be reported twice (dedupe on `event.id`).
140
- - **Cardinality** on `task.name` — alert when unknown names appear.
141
- - **Minute/hour** buckets only if you need SLA dashboards; they multiply storage.
142
-
143
- ---
144
-
145
- ## 9. Verification
146
-
147
- - Dry-run mapping with **`preview`** on sample payloads (see package release notes / API).
148
- - Compare simulator **`audit-log`** / **`processing-activity`** profiles against staging counters.
149
-
150
- ---
151
-
152
- ## 10. Failure modes
153
-
154
- | Symptom | Cause | Mitigation |
155
- |---------|-------|------------|
156
- | Missing rollups | Required paths absent | Fix producer or relax `required`. |
157
- | Cardinality blow-up | Unbounded `task.name` | Registry + overflow dimension policy. |
158
- | Slow queries | Too-wide timeframe + huge groups | Narrow `bucketRange`, pre-aggregate in map if needed. |
159
-
160
- ---
161
-
162
- ## Related
163
-
164
- - [Product & feature activity](./product-feature-usage.md) — analogous pattern for user-facing events.
165
- - [Simulation profiles](../guides/simulation-profiles.md) — `processing-activity` profile.
1
+ # Technical playbook: Processing and workflow activity
2
+
3
+ You operate **pipelines**, **queues**, **agents**, or **batch jobs**. Events describe steps, statuses, errors, and durations. You want rollups by job/workflow, task type, outcome, and timeframe — without Countex knowing your orchestrator brand.
4
+
5
+ ---
6
+
7
+ ## 1. Problem statement
8
+
9
+ ### 1.1 Outcomes
10
+
11
+ - Per **tenant** (`scope`), count activity by **job or run** (`subject`) and **step/task kind** (`metric`).
12
+ - Slice by **status**, **worker pool**, or **region** (`dimensions`).
13
+ - Track **duration** and **retry counts** as **measures** where useful.
14
+
15
+ ### 1.2 Constraints
16
+
17
+ - Step names (`metrics`) should stay **bounded** — register allowed task types like an enum.
18
+ - Job ids may be high cardinality; dictionary maps string job ids per scope.
19
+
20
+ ---
21
+
22
+ ## 2. Primitives mapping
23
+
24
+ | Primitive | JSON path | Notes |
25
+ |-----------|-----------|--------|
26
+ | scope | `$.tenant.id` | Isolation boundary. |
27
+ | eventTime | `$.run.recordedAt` | UTC bucketing. |
28
+ | subject `job` | `$.run.jobId` | Required — one counter row per job run key. |
29
+ | metric | `$.task.name` | Task/step identifier (single path or controlled fan-out). |
30
+ | dim `status` | `$.task.status` | Literal dimension (`success`, `failed`, …). |
31
+ | dim `pool` | `$.worker.pool` | Optional. |
32
+ | delta | `1` | One logical step completion per event (adjust if one event batches many steps). |
33
+ | measure `durationMs` | `$.task.durationMs` | Optional `sum` / `count` / `min` / `max`. |
34
+
35
+ ---
36
+
37
+ ## 3. Canonical payload
38
+
39
+ ```json
40
+ {
41
+ "tenant": { "id": "org_42" },
42
+ "run": {
43
+ "jobId": "job_9f3a",
44
+ "recordedAt": "2026-05-08T18:22:01.000Z"
45
+ },
46
+ "task": {
47
+ "name": "extract_documents",
48
+ "status": "success",
49
+ "durationMs": 842
50
+ },
51
+ "worker": { "pool": "default" }
52
+ }
53
+ ```
54
+
55
+ ### 3.1 Required-field matrix
56
+
57
+ | Path | Required | Effect if absent |
58
+ |------|----------|------------------|
59
+ | `$.tenant.id` | Yes | Skip for map. |
60
+ | `$.run.recordedAt` | Yes | Skip. |
61
+ | `$.run.jobId` | Yes | Skip (`required: true` subject). |
62
+ | `$.task.name` | Yes | Skip (`metrics.required: true`). |
63
+ | `$.task.status` | No | Optional dimension omitted if missing. |
64
+ | `$.worker.pool` | No | Same. |
65
+ | `$.task.durationMs` | No | Measure skipped per `onMissing`. |
66
+
67
+ ---
68
+
69
+ ## 4. Counter map (exact)
70
+
71
+ ```ts
72
+ import type { CounterMap } from "@x12i/countex";
73
+
74
+ export const processingWorkflowMap: CounterMap = {
75
+ counterId: "processing_workflow",
76
+
77
+ scope: { path: "$.tenant.id" },
78
+ eventTime: { path: "$.run.recordedAt" },
79
+
80
+ subjects: [{ type: "job", path: "$.run.jobId", required: true }],
81
+
82
+ metrics: [{ path: "$.task.name", required: true }],
83
+
84
+ dimensions: [
85
+ { name: "status", path: "$.task.status", required: false },
86
+ { name: "pool", path: "$.worker.pool", required: false },
87
+ ],
88
+
89
+ delta: { default: 1 },
90
+ timeframes: ["hour", "day", "month"],
91
+
92
+ measures: [
93
+ {
94
+ name: "durationMs",
95
+ path: "$.task.durationMs",
96
+ aggregations: ["sum", "count", "min", "max"],
97
+ onMissing: "skip",
98
+ },
99
+ ],
100
+ };
101
+ ```
102
+
103
+ ---
104
+
105
+ ## 5. Fan-out arithmetic
106
+
107
+ One subject × one metric × `timeframes.length` increments per event unless paths fan out.
108
+
109
+ Example: **1 × 1 × 3 = 3** cells updated per event for `hour`, `day`, `month`.
110
+
111
+ ---
112
+
113
+ ## 6. Implementation steps
114
+
115
+ 1. Validate payloads at your ingress (schema or optional `validateMap(map, samples)` with fixtures).
116
+ 2. `createCountex({ maps: [processingWorkflowMap], durable, cache })`.
117
+ 3. `ingest(event)` on each completed step or aggregated batch boundary.
118
+ 4. `flush()` on interval or shutdown.
119
+
120
+ ---
121
+
122
+ ## 7. Query specifications
123
+
124
+ ```ts
125
+ await countex.query({
126
+ scope: "org_42",
127
+ counterId: "processing_workflow",
128
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
129
+ groupBy: ["metric"],
130
+ measures: ["durationMs"],
131
+ format: "table",
132
+ });
133
+ ```
134
+
135
+ ---
136
+
137
+ ## 8. Production hardening
138
+
139
+ - **Idempotency** if the same step may be reported twice (dedupe on `event.id`).
140
+ - **Cardinality** on `task.name` — alert when unknown names appear.
141
+ - **Minute/hour** buckets only if you need SLA dashboards; they multiply storage.
142
+
143
+ ---
144
+
145
+ ## 9. Verification
146
+
147
+ - Dry-run mapping with **`preview`** on sample payloads (see package release notes / API).
148
+ - Compare simulator **`audit-log`** / **`processing-activity`** profiles against staging counters.
149
+
150
+ ---
151
+
152
+ ## 10. Failure modes
153
+
154
+ | Symptom | Cause | Mitigation |
155
+ |---------|-------|------------|
156
+ | Missing rollups | Required paths absent | Fix producer or relax `required`. |
157
+ | Cardinality blow-up | Unbounded `task.name` | Registry + overflow dimension policy. |
158
+ | Slow queries | Too-wide timeframe + huge groups | Narrow `bucketRange`, pre-aggregate in map if needed. |
159
+
160
+ ---
161
+
162
+ ## Related
163
+
164
+ - [Product & feature activity](./product-feature-usage.md) — analogous pattern for user-facing events.
165
+ - [Simulation profiles](../guides/simulation-profiles.md) — `processing-activity` profile.