@x12i/countex 1.8.5 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +866 -860
  3. package/dist/cli.js +6 -6
  4. package/dist/index.d.ts +1 -0
  5. package/dist/index.d.ts.map +1 -1
  6. package/dist/index.js +1 -0
  7. package/dist/index.js.map +1 -1
  8. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  9. package/dist/insights/advisors/resource-advisor.js +5 -1
  10. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  11. package/dist/insights/types.d.ts +4 -0
  12. package/dist/insights/types.d.ts.map +1 -1
  13. package/dist/storage/postgres.d.ts +104 -0
  14. package/dist/storage/postgres.d.ts.map +1 -0
  15. package/dist/storage/postgres.js +1023 -0
  16. package/dist/storage/postgres.js.map +1 -0
  17. package/dist/types/index.d.ts +11 -5
  18. package/dist/types/index.d.ts.map +1 -1
  19. package/docs/DEPLOYMENT.md +399 -375
  20. package/docs/END_TO_END_EXAMPLE.md +316 -316
  21. package/docs/INSIGHTS.md +59 -59
  22. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  23. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  24. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  25. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  26. package/docs/guides/README.md +36 -34
  27. package/docs/guides/adapter-memory.md +60 -60
  28. package/docs/guides/adapter-mongo.md +64 -62
  29. package/docs/guides/adapter-postgres.md +90 -0
  30. package/docs/guides/adapter-redis.md +89 -89
  31. package/docs/guides/backup-export.md +135 -135
  32. package/docs/guides/cardinality.md +50 -50
  33. package/docs/guides/cli.md +53 -53
  34. package/docs/guides/client-events.md +65 -65
  35. package/docs/guides/counter-maps.md +160 -160
  36. package/docs/guides/idempotency.md +39 -39
  37. package/docs/guides/ingestion-and-flush.md +67 -67
  38. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  39. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  40. package/docs/guides/insights-comparison-and-trends.md +47 -47
  41. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  42. package/docs/guides/insights-overview.md +82 -82
  43. package/docs/guides/insights-ranking.md +81 -81
  44. package/docs/guides/measures.md +71 -71
  45. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  46. package/docs/guides/preview.md +92 -92
  47. package/docs/guides/querying.md +178 -178
  48. package/docs/guides/simulation-profiles.md +33 -33
  49. package/docs/guides/simulation-resource-estimation.md +31 -30
  50. package/docs/guides/simulation.md +36 -36
  51. package/docs/guides/timeframes-and-bucketing.md +55 -55
  52. package/docs/internals/README.md +21 -21
  53. package/docs/internals/internal-architecture-overview.md +65 -65
  54. package/docs/internals/internal-backup-stream.md +23 -23
  55. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  56. package/docs/internals/internal-cardinality.md +30 -30
  57. package/docs/internals/internal-dimension-hash.md +34 -34
  58. package/docs/internals/internal-idempotency.md +36 -36
  59. package/docs/internals/internal-ingest-pipeline.md +52 -52
  60. package/docs/internals/internal-insights-layer.md +36 -36
  61. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  62. package/docs/internals/internal-measures-and-scales.md +35 -35
  63. package/docs/internals/internal-query-path.md +46 -46
  64. package/docs/internals/internal-storage-memory.md +13 -13
  65. package/docs/internals/internal-storage-mongo.md +44 -44
  66. package/docs/internals/internal-storage-postgres.md +32 -0
  67. package/docs/internals/internal-storage-redis.md +63 -63
  68. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  69. package/docs/use-cases/README.md +56 -56
  70. package/docs/use-cases/audit-security-log-counts.md +550 -550
  71. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  72. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  73. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  74. package/docs/use-cases/processing-workflow-activity.md +165 -165
  75. package/docs/use-cases/product-feature-usage.md +240 -240
  76. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  77. package/package.json +184 -170
@@ -1,188 +1,188 @@
1
- # Technical playbook: Idempotent webhooks and queues
2
-
3
- HTTP webhooks and brokers (SQS, Pub/Sub, Rabbit) usually deliver **at least once**. Retries and consumer crashes re-deliver the same logical message. Without dedupe, Countex **double-counts**.
4
-
5
- This playbook specifies **exact** configuration and handler semantics using Countex **idempotency** + Redis **`markEventSeen`**.
6
-
7
- ---
8
-
9
- ## 1. Preconditions
10
-
11
- | Requirement | Detail |
12
- |-------------|--------|
13
- | Node.js | ≥ 18 |
14
- | Packages | `@x12i/countex`, `mongodb` (durable), `ioredis` (cache + dedupe) |
15
- | Stable id | Unique string per logical delivery, available **before** ingest |
16
-
17
- ---
18
-
19
- ## 2. Idempotency mechanics (internal contract)
20
-
21
- 1. During `extractIncrements`, Countex reads `eventIdPath` → string `eventId` (optional warning if missing).
22
- 2. Before buffering, `ingestBatch` calls `cache.markEventSeen(scope, dedupeKey, ttl)` where:
23
-
24
- ```
25
- dedupeKey = `${counterId}:${eventId}`
26
- ```
27
-
28
- See `src/core/countex.ts` — dedupe is **per map** so two maps on same raw event do not share keys incorrectly.
29
-
30
- 3. If `markEventSeen` returns **`false`**, increments for that map slice are cleared and event **`duplicate`** fires.
31
-
32
- 4. Redis adapter stores `countex:{scope}:dedupe:{eventId}` — note the adapter receives **already-namespaced** `eventId` from Countex (includes `counterId:` prefix).
33
-
34
- ---
35
-
36
- ## 3. Canonical payload
37
-
38
- ```json
39
- {
40
- "tenant": { "id": "org_1" },
41
- "delivery": { "id": "wh_8f3a2c9e1d004f2a" },
42
- "at": "2026-05-08T18:00:00.000Z",
43
- "userId": "u_1",
44
- "kind": "invoice.paid"
45
- }
46
- ```
47
-
48
- **Invariant:** `delivery.id` is unique **for all time** or unique within TTL horizon — if ids recycle after TTL expiry, duplicates may re-count.
49
-
50
- ---
51
-
52
- ## 4. Counter map (exact)
53
-
54
- `src/countex/maps/billingWebhookMap.ts`:
55
-
56
- ```ts
57
- import type { CounterMap } from "@x12i/countex";
58
-
59
- export const billingWebhookMap: CounterMap = {
60
- counterId: "billing_events",
61
-
62
- scope: { path: "$.tenant.id" },
63
- eventTime: { path: "$.at" },
64
-
65
- subjects: [{ type: "user", path: "$.userId", required: true }],
66
- metrics: [{ path: "$.kind", required: true }],
67
-
68
- delta: { default: 1 },
69
- timeframes: ["day"],
70
-
71
- idempotency: {
72
- enabled: true,
73
- eventIdPath: "$.delivery.id",
74
- ttlSeconds: 604800, // 7d — MUST exceed max retry horizon
75
- },
76
- };
77
- ```
78
-
79
- ### 4.1 Choosing `ttlSeconds`
80
-
81
- Set:
82
-
83
- ```
84
- ttlSeconds ≥ (max broker retention) + (max consumer lag) + safety_margin
85
- ```
86
-
87
- Example: SQS visibility timeout retries for 4 days → use ≥ 5–7 days.
88
-
89
- ---
90
-
91
- ## 5. Factory (exact)
92
-
93
- ```ts
94
- import { MongoClient } from "mongodb";
95
- import IORedis from "ioredis";
96
- import { createCountex } from "@x12i/countex";
97
- import { MongoDurable } from "@x12i/countex/adapters/mongo";
98
- import { RedisCache } from "@x12i/countex/adapters/redis";
99
- import { billingWebhookMap } from "./maps/billingWebhookMap.js";
100
-
101
- export async function createWebhookCountex() {
102
- const mongo = new MongoClient(process.env.MONGO_URI!);
103
- await mongo.connect();
104
- const redis = new IORedis(process.env.REDIS_URL!);
105
-
106
- const durable = new MongoDurable({ client: mongo, dbResolver: ({ scope }) => `countex_${scope}` });
107
- const cache = new RedisCache({ client: redis, keyPrefix: "countex", ttlSeconds: 14 * 24 * 3600 });
108
-
109
- const countex = createCountex({
110
- maps: [billingWebhookMap],
111
- durable,
112
- cache,
113
- flush: { everyMs: 5_000, maxBufferedCells: 20_000, maxBufferedEvents: 10_000, flushOnShutdown: true, onOverflow: "block" },
114
- });
115
-
116
- await countex.init();
117
- return { countex, mongo, redis };
118
- }
119
- ```
120
-
121
- **Critical:** Without `cache` implementing `markEventSeen`, idempotency **does nothing** — duplicates still buffer.
122
-
123
- ---
124
-
125
- ## 6. Consumer handler (exact steps)
126
-
127
- ### Step 6.1 — Parse
128
-
129
- ```ts
130
- const payload = JSON.parse(messageBody) as unknown;
131
- ```
132
-
133
- ### Step 6.2 — Validate schema (your code)
134
-
135
- Reject before ingest if `delivery.id` missing while idempotency enabled — otherwise you get warnings and **non-deduped** increments.
136
-
137
- ### Step 6.3 — Ingest
138
-
139
- ```ts
140
- await countex.ingest(payload);
141
- ```
142
-
143
- ### Step 6.4 — Observe duplicates
144
-
145
- ```ts
146
- countex.on("duplicate", ({ scopeId, eventId }) => {
147
- // eventId here is the NAMESPACED key "billing_events:wh_8f3a2c..."
148
- });
149
- ```
150
-
151
- Return **200 OK / ack message** even on duplicate — delivery succeeded idempotently.
152
-
153
- ### Step 6.5 — Durability boundary
154
-
155
- Either:
156
-
157
- - Rely on **`flush.everyMs`**, or
158
- - **`await countex.flush()`** after batch if webhook framework requires strong durability before ack (latency tradeoff).
159
-
160
- **Ack ordering policy:** industry-standard is **process → flush OR tolerate timer → ack**; never ack before deserialize if you cannot handle poison pills.
161
-
162
- ---
163
-
164
- ## 7. Verification tests
165
-
166
- | Test | Procedure | Expected |
167
- |------|-----------|----------|
168
- | Dedupe works | Same payload twice | Second `duplicate`; Mongo count unchanged. |
169
- | TTL expiry | Simulate key delete | Third delivery increments again — expected. |
170
- | Multi-map | Two maps, same raw `delivery.id` | Distinct dedupe keys (`counterId:` prefix) → both run once each. |
171
-
172
- ---
173
-
174
- ## 8. Failure modes
175
-
176
- | Symptom | Cause | Fix |
177
- |---------|-------|-----|
178
- | Duplicates still increment | Memory-only cache adapter | Use `RedisCache` or implement `markEventSeen`. |
179
- | Redis down | `markEventSeen` throws | Decide fail-open vs fail-closed; today ingest path uses cache — surface errors to DLQ. |
180
- | Mass replays after TTL | TTL too short | Raise `ttlSeconds`. |
181
-
182
- ---
183
-
184
- ## 9. References
185
-
186
- - [`docs/guides/idempotency.md`](../guides/idempotency.md)
187
- - [`docs/internals/internal-idempotency.md`](../internals/internal-idempotency.md)
188
- - `src/storage/redis.ts` dedupe key format
1
+ # Technical playbook: Idempotent webhooks and queues
2
+
3
+ HTTP webhooks and brokers (SQS, Pub/Sub, Rabbit) usually deliver **at least once**. Retries and consumer crashes re-deliver the same logical message. Without dedupe, Countex **double-counts**.
4
+
5
+ This playbook specifies **exact** configuration and handler semantics using Countex **idempotency** + Redis **`markEventSeen`**.
6
+
7
+ ---
8
+
9
+ ## 1. Preconditions
10
+
11
+ | Requirement | Detail |
12
+ |-------------|--------|
13
+ | Node.js | ≥ 18 |
14
+ | Packages | `@x12i/countex`, `mongodb` (durable), `ioredis` (cache + dedupe) |
15
+ | Stable id | Unique string per logical delivery, available **before** ingest |
16
+
17
+ ---
18
+
19
+ ## 2. Idempotency mechanics (internal contract)
20
+
21
+ 1. During `extractIncrements`, Countex reads `eventIdPath` → string `eventId` (optional warning if missing).
22
+ 2. Before buffering, `ingestBatch` calls `cache.markEventSeen(scope, dedupeKey, ttl)` where:
23
+
24
+ ```
25
+ dedupeKey = `${counterId}:${eventId}`
26
+ ```
27
+
28
+ See `src/core/countex.ts` — dedupe is **per map** so two maps on same raw event do not share keys incorrectly.
29
+
30
+ 3. If `markEventSeen` returns **`false`**, increments for that map slice are cleared and event **`duplicate`** fires.
31
+
32
+ 4. Redis adapter stores `countex:{scope}:dedupe:{eventId}` — note the adapter receives **already-namespaced** `eventId` from Countex (includes `counterId:` prefix).
33
+
34
+ ---
35
+
36
+ ## 3. Canonical payload
37
+
38
+ ```json
39
+ {
40
+ "tenant": { "id": "org_1" },
41
+ "delivery": { "id": "wh_8f3a2c9e1d004f2a" },
42
+ "at": "2026-05-08T18:00:00.000Z",
43
+ "userId": "u_1",
44
+ "kind": "invoice.paid"
45
+ }
46
+ ```
47
+
48
+ **Invariant:** `delivery.id` is unique **for all time** or unique within TTL horizon — if ids recycle after TTL expiry, duplicates may re-count.
49
+
50
+ ---
51
+
52
+ ## 4. Counter map (exact)
53
+
54
+ `src/countex/maps/billingWebhookMap.ts`:
55
+
56
+ ```ts
57
+ import type { CounterMap } from "@x12i/countex";
58
+
59
+ export const billingWebhookMap: CounterMap = {
60
+ counterId: "billing_events",
61
+
62
+ scope: { path: "$.tenant.id" },
63
+ eventTime: { path: "$.at" },
64
+
65
+ subjects: [{ type: "user", path: "$.userId", required: true }],
66
+ metrics: [{ path: "$.kind", required: true }],
67
+
68
+ delta: { default: 1 },
69
+ timeframes: ["day"],
70
+
71
+ idempotency: {
72
+ enabled: true,
73
+ eventIdPath: "$.delivery.id",
74
+ ttlSeconds: 604800, // 7d — MUST exceed max retry horizon
75
+ },
76
+ };
77
+ ```
78
+
79
+ ### 4.1 Choosing `ttlSeconds`
80
+
81
+ Set:
82
+
83
+ ```
84
+ ttlSeconds ≥ (max broker retention) + (max consumer lag) + safety_margin
85
+ ```
86
+
87
+ Example: SQS visibility timeout retries for 4 days → use ≥ 5–7 days.
88
+
89
+ ---
90
+
91
+ ## 5. Factory (exact)
92
+
93
+ ```ts
94
+ import { MongoClient } from "mongodb";
95
+ import IORedis from "ioredis";
96
+ import { createCountex } from "@x12i/countex";
97
+ import { MongoDurable } from "@x12i/countex/adapters/mongo";
98
+ import { RedisCache } from "@x12i/countex/adapters/redis";
99
+ import { billingWebhookMap } from "./maps/billingWebhookMap.js";
100
+
101
+ export async function createWebhookCountex() {
102
+ const mongo = new MongoClient(process.env.MONGO_URI!);
103
+ await mongo.connect();
104
+ const redis = new IORedis(process.env.REDIS_URL!);
105
+
106
+ const durable = new MongoDurable({ client: mongo, dbResolver: ({ scope }) => `countex_${scope}` });
107
+ const cache = new RedisCache({ client: redis, keyPrefix: "countex", ttlSeconds: 14 * 24 * 3600 });
108
+
109
+ const countex = createCountex({
110
+ maps: [billingWebhookMap],
111
+ durable,
112
+ cache,
113
+ flush: { everyMs: 5_000, maxBufferedCells: 20_000, maxBufferedEvents: 10_000, flushOnShutdown: true, onOverflow: "block" },
114
+ });
115
+
116
+ await countex.init();
117
+ return { countex, mongo, redis };
118
+ }
119
+ ```
120
+
121
+ **Critical:** Without `cache` implementing `markEventSeen`, idempotency **does nothing** — duplicates still buffer.
122
+
123
+ ---
124
+
125
+ ## 6. Consumer handler (exact steps)
126
+
127
+ ### Step 6.1 — Parse
128
+
129
+ ```ts
130
+ const payload = JSON.parse(messageBody) as unknown;
131
+ ```
132
+
133
+ ### Step 6.2 — Validate schema (your code)
134
+
135
+ Reject before ingest if `delivery.id` missing while idempotency enabled — otherwise you get warnings and **non-deduped** increments.
136
+
137
+ ### Step 6.3 — Ingest
138
+
139
+ ```ts
140
+ await countex.ingest(payload);
141
+ ```
142
+
143
+ ### Step 6.4 — Observe duplicates
144
+
145
+ ```ts
146
+ countex.on("duplicate", ({ scopeId, eventId }) => {
147
+ // eventId here is the NAMESPACED key "billing_events:wh_8f3a2c..."
148
+ });
149
+ ```
150
+
151
+ Return **200 OK / ack message** even on duplicate — delivery succeeded idempotently.
152
+
153
+ ### Step 6.5 — Durability boundary
154
+
155
+ Either:
156
+
157
+ - Rely on **`flush.everyMs`**, or
158
+ - **`await countex.flush()`** after batch if webhook framework requires strong durability before ack (latency tradeoff).
159
+
160
+ **Ack ordering policy:** industry-standard is **process → flush OR tolerate timer → ack**; never ack before deserialize if you cannot handle poison pills.
161
+
162
+ ---
163
+
164
+ ## 7. Verification tests
165
+
166
+ | Test | Procedure | Expected |
167
+ |------|-----------|----------|
168
+ | Dedupe works | Same payload twice | Second `duplicate`; Mongo count unchanged. |
169
+ | TTL expiry | Simulate key delete | Third delivery increments again — expected. |
170
+ | Multi-map | Two maps, same raw `delivery.id` | Distinct dedupe keys (`counterId:` prefix) → both run once each. |
171
+
172
+ ---
173
+
174
+ ## 8. Failure modes
175
+
176
+ | Symptom | Cause | Fix |
177
+ |---------|-------|-----|
178
+ | Duplicates still increment | Memory-only cache adapter | Use `RedisCache` or implement `markEventSeen`. |
179
+ | Redis down | `markEventSeen` throws | Decide fail-open vs fail-closed; today ingest path uses cache — surface errors to DLQ. |
180
+ | Mass replays after TTL | TTL too short | Raise `ttlSeconds`. |
181
+
182
+ ---
183
+
184
+ ## 9. References
185
+
186
+ - [`docs/guides/idempotency.md`](../guides/idempotency.md)
187
+ - [`docs/internals/internal-idempotency.md`](../internals/internal-idempotency.md)
188
+ - `src/storage/redis.ts` dedupe key format