@x12i/countex 1.8.5 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +866 -860
  3. package/dist/cli.js +6 -6
  4. package/dist/index.d.ts +1 -0
  5. package/dist/index.d.ts.map +1 -1
  6. package/dist/index.js +1 -0
  7. package/dist/index.js.map +1 -1
  8. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  9. package/dist/insights/advisors/resource-advisor.js +5 -1
  10. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  11. package/dist/insights/types.d.ts +4 -0
  12. package/dist/insights/types.d.ts.map +1 -1
  13. package/dist/storage/postgres.d.ts +104 -0
  14. package/dist/storage/postgres.d.ts.map +1 -0
  15. package/dist/storage/postgres.js +1023 -0
  16. package/dist/storage/postgres.js.map +1 -0
  17. package/dist/types/index.d.ts +11 -5
  18. package/dist/types/index.d.ts.map +1 -1
  19. package/docs/DEPLOYMENT.md +399 -375
  20. package/docs/END_TO_END_EXAMPLE.md +316 -316
  21. package/docs/INSIGHTS.md +59 -59
  22. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  23. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  24. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  25. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  26. package/docs/guides/README.md +36 -34
  27. package/docs/guides/adapter-memory.md +60 -60
  28. package/docs/guides/adapter-mongo.md +64 -62
  29. package/docs/guides/adapter-postgres.md +90 -0
  30. package/docs/guides/adapter-redis.md +89 -89
  31. package/docs/guides/backup-export.md +135 -135
  32. package/docs/guides/cardinality.md +50 -50
  33. package/docs/guides/cli.md +53 -53
  34. package/docs/guides/client-events.md +65 -65
  35. package/docs/guides/counter-maps.md +160 -160
  36. package/docs/guides/idempotency.md +39 -39
  37. package/docs/guides/ingestion-and-flush.md +67 -67
  38. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  39. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  40. package/docs/guides/insights-comparison-and-trends.md +47 -47
  41. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  42. package/docs/guides/insights-overview.md +82 -82
  43. package/docs/guides/insights-ranking.md +81 -81
  44. package/docs/guides/measures.md +71 -71
  45. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  46. package/docs/guides/preview.md +92 -92
  47. package/docs/guides/querying.md +178 -178
  48. package/docs/guides/simulation-profiles.md +33 -33
  49. package/docs/guides/simulation-resource-estimation.md +31 -30
  50. package/docs/guides/simulation.md +36 -36
  51. package/docs/guides/timeframes-and-bucketing.md +55 -55
  52. package/docs/internals/README.md +21 -21
  53. package/docs/internals/internal-architecture-overview.md +65 -65
  54. package/docs/internals/internal-backup-stream.md +23 -23
  55. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  56. package/docs/internals/internal-cardinality.md +30 -30
  57. package/docs/internals/internal-dimension-hash.md +34 -34
  58. package/docs/internals/internal-idempotency.md +36 -36
  59. package/docs/internals/internal-ingest-pipeline.md +52 -52
  60. package/docs/internals/internal-insights-layer.md +36 -36
  61. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  62. package/docs/internals/internal-measures-and-scales.md +35 -35
  63. package/docs/internals/internal-query-path.md +46 -46
  64. package/docs/internals/internal-storage-memory.md +13 -13
  65. package/docs/internals/internal-storage-mongo.md +44 -44
  66. package/docs/internals/internal-storage-postgres.md +32 -0
  67. package/docs/internals/internal-storage-redis.md +63 -63
  68. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  69. package/docs/use-cases/README.md +56 -56
  70. package/docs/use-cases/audit-security-log-counts.md +550 -550
  71. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  72. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  73. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  74. package/docs/use-cases/processing-workflow-activity.md +165 -165
  75. package/docs/use-cases/product-feature-usage.md +240 -240
  76. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  77. package/package.json +184 -170
@@ -1,203 +1,203 @@
1
- # Technical playbook: Multi-tenant isolation
2
-
3
- You ship **B2B SaaS**. Every JSON event belongs to exactly one **tenant**. Regulatory and operational requirements demand:
4
-
5
- - **No cross-tenant reads** from application bugs.
6
- - Optionally **hard physical isolation** (per-tenant database).
7
- - Predictable key namespaces in shared Redis clusters.
8
-
9
- Countex always stores **`scopeId`** on cells; **you** must enforce that scope matches authenticated context and that storage layout matches policy.
10
-
11
- ---
12
-
13
- ## 1. Threat model (what this playbook addresses)
14
-
15
- | Risk | Mitigation level |
16
- |------|------------------|
17
- | App passes wrong `scope` into `query()` | **Application code review** + always derive scope from auth token, never from client-supplied body alone. |
18
- | Tenant A reads tenant B aggregates | Prevented if scope string wrong → queries return empty **but** dictionary ids could theoretically collide across scopes if you incorrectly reuse numeric ids in UI — **always pair ids with scope**. |
19
- | Data residency | Mongo `dbResolver` per region + per tenant DB naming; Redis logical DB / ACLs. |
20
-
21
- ---
22
-
23
- ## 2. Non-negotiable rule
24
-
25
- **Every `countex.query` and every event’s extracted scope must use the same canonical tenant id string** your identity system emits (e.g. `org_01JB…`, never slug vs id interchangeably).
26
-
27
- ---
28
-
29
- ## 3. Event contract
30
-
31
- Canonical pattern:
32
-
33
- ```json
34
- {
35
- "tenant": { "id": "org_abc" },
36
- "at": "2026-05-08T12:00:00.000Z",
37
- "userId": "user_42",
38
- "action": "save"
39
- }
40
- ```
41
-
42
- Map:
43
-
44
- ```ts
45
- import type { CounterMap } from "@x12i/countex";
46
-
47
- export const tenantScopedMap: CounterMap = {
48
- counterId: "usage",
49
- scope: { path: "$.tenant.id" },
50
- eventTime: { path: "$.at" },
51
- subjects: [{ type: "user", path: "$.userId", required: true }],
52
- metrics: [{ path: "$.action", required: true }],
53
- dimensions: [],
54
- delta: { default: 1 },
55
- timeframes: ["day"],
56
- };
57
- ```
58
-
59
- ---
60
-
61
- ## 4. Storage isolation patterns
62
-
63
- ### Pattern A — Single Mongo database, scope field discriminant
64
-
65
- **When:** acceptable for early stage; compliance allows logical isolation only.
66
-
67
- `MongoDurable` default `defaultDb` (e.g. `countex_prod`). Every document carries `scopeId`.
68
-
69
- **Exact setup:**
70
-
71
- ```ts
72
- import { MongoClient } from "mongodb";
73
- import { MongoDurable } from "@x12i/countex/adapters/mongo";
74
-
75
- const mongo = new MongoClient(process.env.MONGO_URI!);
76
- await mongo.connect();
77
-
78
- const durable = new MongoDurable({
79
- client: mongo,
80
- defaultDb: "countex_prod",
81
- });
82
- ```
83
-
84
- **Operational requirement:** Mongo queries outside Countex **must** filter `{ scopeId: <tenant> }` if you touch collections manually.
85
-
86
- ### Pattern B — Per-tenant Mongo database (stronger)
87
-
88
- **When:** customer contracts require separation at rest.
89
-
90
- ```ts
91
- const durable = new MongoDurable({
92
- client: mongo,
93
- dbResolver: ({ scope }) => {
94
- // EXACT naming convention — enforce alphanumeric + safe chars only
95
- const safe = scope.replace(/[^a-zA-Z0-9_-]/g, "_");
96
- return `countex_${safe}`;
97
- },
98
- });
99
- ```
100
-
101
- **Exact consequences:**
102
-
103
- 1. First write for a tenant creates DB `countex_<tenant>` with collections `countex_cells`, `countex_dict`, `countex_seq`.
104
- 2. Dictionary ids **restart per database** — numeric `subjectId` in DB A has **no relation** to DB B.
105
-
106
- ### Pattern C — Redis shared cluster
107
-
108
- `RedisCache` keys:
109
-
110
- ```
111
- {prefix}:{scopeId}:{counterId}:{timeframeType}:{timeframeKey}
112
- ```
113
-
114
- **Exact action items:**
115
-
116
- 1. Use **`keyPrefix`** per environment (`countex_prod`, `countex_staging`).
117
- 2. Enable Redis **ACL users** per product shard if policy demands.
118
- 3. TTL — `ttlSeconds` on adapter applies to cache hashes; tune ≥ analytics window.
119
-
120
- ```ts
121
- import IORedis from "ioredis";
122
- import { RedisCache } from "@x12i/countex/adapters/redis";
123
-
124
- const redis = new IORedis(process.env.REDIS_URL!);
125
-
126
- const cache = new RedisCache({
127
- client: redis,
128
- keyPrefix: "countex_prod",
129
- ttlSeconds: 92 * 24 * 3600,
130
- });
131
- ```
132
-
133
- ---
134
-
135
- ## 5. Application wiring (exact control flow)
136
-
137
- ### Step 5.1 — Authenticate request
138
-
139
- ```ts
140
- const session = await authenticate(request);
141
- const scope = session.tenantId; // MUST come from signed token / session store
142
- ```
143
-
144
- ### Step 5.2 — Reject scope mismatch
145
-
146
- If JSON body includes `tenant.id`:
147
-
148
- ```ts
149
- const body = await request.json();
150
- if (typeof body === "object" && body !== null && "tenant" in body) {
151
- const tid = (body as { tenant?: { id?: string } }).tenant?.id;
152
- if (tid !== undefined && tid !== scope) {
153
- throw new Error("tenant mismatch"); // 403
154
- }
155
- }
156
- ```
157
-
158
- ### Step 5.3 — Ingest only after validation
159
-
160
- ```ts
161
- await countex.ingest(body);
162
- ```
163
-
164
- ### Step 5.4 — Query only with session scope
165
-
166
- ```ts
167
- const result = await countex.query({
168
- scope, // never `body.tenant.id` alone
169
- counterId: "usage",
170
- timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
171
- format: "table",
172
- });
173
- ```
174
-
175
- ---
176
-
177
- ## 6. Testing matrix (exact cases)
178
-
179
- | Case | Setup | Assert |
180
- |------|-------|--------|
181
- | Isolation | Ingest event scope `A`, query scope `B` | Empty or zero totals for `B`. |
182
- | Resolver | Two tenants first write | Two Mongo databases exist when using Pattern B. |
183
- | Redis collision | Same subject string different scopes | Distinct hash keys because `scopeId` differs. |
184
-
185
- ---
186
-
187
- ## 7. Failure modes
188
-
189
- | Symptom | Cause | Fix |
190
- |---------|-------|-----|
191
- | “Missing data” after migrate | Changed scope string format | Backfill or alias old→new scope in ingress. |
192
- | Exploding Mongo DB count | Resolver allows arbitrary strings | Sanitize tenant ids; block malicious lengths. |
193
- | Wrong dashboard ids | Copied numeric ids across tenants | Never export dictionary ids without scope column. |
194
-
195
- ---
196
-
197
- ## 8. References
198
-
199
- - Mongo adapter options: `MongoAdapterOptions` in `src/storage/mongo.ts`
200
- - Redis key layout: header comment in `src/storage/redis.ts`
201
- - [`docs/guides/adapter-mongo.md`](../guides/adapter-mongo.md)
202
- - [`docs/guides/adapter-redis.md`](../guides/adapter-redis.md)
203
- - [`docs/DEPLOYMENT.md`](../DEPLOYMENT.md)
1
+ # Technical playbook: Multi-tenant isolation
2
+
3
+ You ship **B2B SaaS**. Every JSON event belongs to exactly one **tenant**. Regulatory and operational requirements demand:
4
+
5
+ - **No cross-tenant reads** from application bugs.
6
+ - Optionally **hard physical isolation** (per-tenant database).
7
+ - Predictable key namespaces in shared Redis clusters.
8
+
9
+ Countex always stores **`scopeId`** on cells; **you** must enforce that scope matches authenticated context and that storage layout matches policy.
10
+
11
+ ---
12
+
13
+ ## 1. Threat model (what this playbook addresses)
14
+
15
+ | Risk | Mitigation level |
16
+ |------|------------------|
17
+ | App passes wrong `scope` into `query()` | **Application code review** + always derive scope from auth token, never from client-supplied body alone. |
18
+ | Tenant A reads tenant B aggregates | Prevented if scope string wrong → queries return empty **but** dictionary ids could theoretically collide across scopes if you incorrectly reuse numeric ids in UI — **always pair ids with scope**. |
19
+ | Data residency | Mongo `dbResolver` per region + per tenant DB naming; Redis logical DB / ACLs. |
20
+
21
+ ---
22
+
23
+ ## 2. Non-negotiable rule
24
+
25
+ **Every `countex.query` and every event’s extracted scope must use the same canonical tenant id string** your identity system emits (e.g. `org_01JB…`, never slug vs id interchangeably).
26
+
27
+ ---
28
+
29
+ ## 3. Event contract
30
+
31
+ Canonical pattern:
32
+
33
+ ```json
34
+ {
35
+ "tenant": { "id": "org_abc" },
36
+ "at": "2026-05-08T12:00:00.000Z",
37
+ "userId": "user_42",
38
+ "action": "save"
39
+ }
40
+ ```
41
+
42
+ Map:
43
+
44
+ ```ts
45
+ import type { CounterMap } from "@x12i/countex";
46
+
47
+ export const tenantScopedMap: CounterMap = {
48
+ counterId: "usage",
49
+ scope: { path: "$.tenant.id" },
50
+ eventTime: { path: "$.at" },
51
+ subjects: [{ type: "user", path: "$.userId", required: true }],
52
+ metrics: [{ path: "$.action", required: true }],
53
+ dimensions: [],
54
+ delta: { default: 1 },
55
+ timeframes: ["day"],
56
+ };
57
+ ```
58
+
59
+ ---
60
+
61
+ ## 4. Storage isolation patterns
62
+
63
+ ### Pattern A — Single Mongo database, scope field discriminant
64
+
65
+ **When:** acceptable for early stage; compliance allows logical isolation only.
66
+
67
+ `MongoDurable` default `defaultDb` (e.g. `countex_prod`). Every document carries `scopeId`.
68
+
69
+ **Exact setup:**
70
+
71
+ ```ts
72
+ import { MongoClient } from "mongodb";
73
+ import { MongoDurable } from "@x12i/countex/adapters/mongo";
74
+
75
+ const mongo = new MongoClient(process.env.MONGO_URI!);
76
+ await mongo.connect();
77
+
78
+ const durable = new MongoDurable({
79
+ client: mongo,
80
+ defaultDb: "countex_prod",
81
+ });
82
+ ```
83
+
84
+ **Operational requirement:** Mongo queries outside Countex **must** filter `{ scopeId: <tenant> }` if you touch collections manually.
85
+
86
+ ### Pattern B — Per-tenant Mongo database (stronger)
87
+
88
+ **When:** customer contracts require separation at rest.
89
+
90
+ ```ts
91
+ const durable = new MongoDurable({
92
+ client: mongo,
93
+ dbResolver: ({ scope }) => {
94
+ // EXACT naming convention — enforce alphanumeric + safe chars only
95
+ const safe = scope.replace(/[^a-zA-Z0-9_-]/g, "_");
96
+ return `countex_${safe}`;
97
+ },
98
+ });
99
+ ```
100
+
101
+ **Exact consequences:**
102
+
103
+ 1. First write for a tenant creates DB `countex_<tenant>` with collections `countex_cells`, `countex_dict`, `countex_seq`.
104
+ 2. Dictionary ids **restart per database** — numeric `subjectId` in DB A has **no relation** to DB B.
105
+
106
+ ### Pattern C — Redis shared cluster
107
+
108
+ `RedisCache` keys:
109
+
110
+ ```
111
+ {prefix}:{scopeId}:{counterId}:{timeframeType}:{timeframeKey}
112
+ ```
113
+
114
+ **Exact action items:**
115
+
116
+ 1. Use **`keyPrefix`** per environment (`countex_prod`, `countex_staging`).
117
+ 2. Enable Redis **ACL users** per product shard if policy demands.
118
+ 3. TTL — `ttlSeconds` on adapter applies to cache hashes; tune ≥ analytics window.
119
+
120
+ ```ts
121
+ import IORedis from "ioredis";
122
+ import { RedisCache } from "@x12i/countex/adapters/redis";
123
+
124
+ const redis = new IORedis(process.env.REDIS_URL!);
125
+
126
+ const cache = new RedisCache({
127
+ client: redis,
128
+ keyPrefix: "countex_prod",
129
+ ttlSeconds: 92 * 24 * 3600,
130
+ });
131
+ ```
132
+
133
+ ---
134
+
135
+ ## 5. Application wiring (exact control flow)
136
+
137
+ ### Step 5.1 — Authenticate request
138
+
139
+ ```ts
140
+ const session = await authenticate(request);
141
+ const scope = session.tenantId; // MUST come from signed token / session store
142
+ ```
143
+
144
+ ### Step 5.2 — Reject scope mismatch
145
+
146
+ If JSON body includes `tenant.id`:
147
+
148
+ ```ts
149
+ const body = await request.json();
150
+ if (typeof body === "object" && body !== null && "tenant" in body) {
151
+ const tid = (body as { tenant?: { id?: string } }).tenant?.id;
152
+ if (tid !== undefined && tid !== scope) {
153
+ throw new Error("tenant mismatch"); // 403
154
+ }
155
+ }
156
+ ```
157
+
158
+ ### Step 5.3 — Ingest only after validation
159
+
160
+ ```ts
161
+ await countex.ingest(body);
162
+ ```
163
+
164
+ ### Step 5.4 — Query only with session scope
165
+
166
+ ```ts
167
+ const result = await countex.query({
168
+ scope, // never `body.tenant.id` alone
169
+ counterId: "usage",
170
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
171
+ format: "table",
172
+ });
173
+ ```
174
+
175
+ ---
176
+
177
+ ## 6. Testing matrix (exact cases)
178
+
179
+ | Case | Setup | Assert |
180
+ |------|-------|--------|
181
+ | Isolation | Ingest event scope `A`, query scope `B` | Empty or zero totals for `B`. |
182
+ | Resolver | Two tenants first write | Two Mongo databases exist when using Pattern B. |
183
+ | Redis collision | Same subject string different scopes | Distinct hash keys because `scopeId` differs. |
184
+
185
+ ---
186
+
187
+ ## 7. Failure modes
188
+
189
+ | Symptom | Cause | Fix |
190
+ |---------|-------|-----|
191
+ | “Missing data” after migrate | Changed scope string format | Backfill or alias old→new scope in ingress. |
192
+ | Exploding Mongo DB count | Resolver allows arbitrary strings | Sanitize tenant ids; block malicious lengths. |
193
+ | Wrong dashboard ids | Copied numeric ids across tenants | Never export dictionary ids without scope column. |
194
+
195
+ ---
196
+
197
+ ## 8. References
198
+
199
+ - Mongo adapter options: `MongoAdapterOptions` in `src/storage/mongo.ts`
200
+ - Redis key layout: header comment in `src/storage/redis.ts`
201
+ - [`docs/guides/adapter-mongo.md`](../guides/adapter-mongo.md)
202
+ - [`docs/guides/adapter-redis.md`](../guides/adapter-redis.md)
203
+ - [`docs/DEPLOYMENT.md`](../DEPLOYMENT.md)