@x12i/countex 1.8.5 → 1.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (105) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +867 -860
  3. package/dist/cli.js +6 -6
  4. package/dist/core/countex.d.ts +61 -1
  5. package/dist/core/countex.d.ts.map +1 -1
  6. package/dist/core/countex.js +241 -0
  7. package/dist/core/countex.js.map +1 -1
  8. package/dist/core/managed.d.ts +200 -0
  9. package/dist/core/managed.d.ts.map +1 -0
  10. package/dist/core/managed.js +431 -0
  11. package/dist/core/managed.js.map +1 -0
  12. package/dist/core/timeframe.d.ts +5 -0
  13. package/dist/core/timeframe.d.ts.map +1 -1
  14. package/dist/core/timeframe.js +36 -0
  15. package/dist/core/timeframe.js.map +1 -1
  16. package/dist/index.d.ts +6 -2
  17. package/dist/index.d.ts.map +1 -1
  18. package/dist/index.js +9 -1
  19. package/dist/index.js.map +1 -1
  20. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  21. package/dist/insights/advisors/resource-advisor.js +5 -1
  22. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  23. package/dist/insights/types.d.ts +4 -0
  24. package/dist/insights/types.d.ts.map +1 -1
  25. package/dist/storage/managed-memory.d.ts +33 -0
  26. package/dist/storage/managed-memory.d.ts.map +1 -0
  27. package/dist/storage/managed-memory.js +211 -0
  28. package/dist/storage/managed-memory.js.map +1 -0
  29. package/dist/storage/memory.d.ts +7 -4
  30. package/dist/storage/memory.d.ts.map +1 -1
  31. package/dist/storage/memory.js.map +1 -1
  32. package/dist/storage/postgres.d.ts +123 -0
  33. package/dist/storage/postgres.d.ts.map +1 -0
  34. package/dist/storage/postgres.js +1325 -0
  35. package/dist/storage/postgres.js.map +1 -0
  36. package/dist/storage/redis.d.ts +6 -0
  37. package/dist/storage/redis.d.ts.map +1 -1
  38. package/dist/storage/redis.js +21 -0
  39. package/dist/storage/redis.js.map +1 -1
  40. package/dist/types/index.d.ts +63 -5
  41. package/dist/types/index.d.ts.map +1 -1
  42. package/dist/utils/errors.d.ts +12 -0
  43. package/dist/utils/errors.d.ts.map +1 -1
  44. package/dist/utils/errors.js +22 -1
  45. package/dist/utils/errors.js.map +1 -1
  46. package/docs/DEPLOYMENT.md +399 -375
  47. package/docs/END_TO_END_EXAMPLE.md +316 -316
  48. package/docs/INSIGHTS.md +59 -59
  49. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  50. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  51. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  52. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  53. package/docs/guides/README.md +37 -34
  54. package/docs/guides/adapter-memory.md +60 -60
  55. package/docs/guides/adapter-mongo.md +64 -62
  56. package/docs/guides/adapter-postgres.md +92 -0
  57. package/docs/guides/adapter-redis.md +89 -89
  58. package/docs/guides/backup-export.md +135 -135
  59. package/docs/guides/cardinality.md +50 -50
  60. package/docs/guides/cli.md +53 -53
  61. package/docs/guides/client-events.md +65 -65
  62. package/docs/guides/counter-maps.md +160 -160
  63. package/docs/guides/idempotency.md +39 -39
  64. package/docs/guides/ingestion-and-flush.md +67 -67
  65. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  66. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  67. package/docs/guides/insights-comparison-and-trends.md +47 -47
  68. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  69. package/docs/guides/insights-overview.md +82 -82
  70. package/docs/guides/insights-ranking.md +81 -81
  71. package/docs/guides/managed-commit.md +119 -0
  72. package/docs/guides/measures.md +72 -71
  73. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  74. package/docs/guides/preview.md +92 -92
  75. package/docs/guides/querying.md +178 -178
  76. package/docs/guides/simulation-profiles.md +33 -33
  77. package/docs/guides/simulation-resource-estimation.md +31 -30
  78. package/docs/guides/simulation.md +36 -36
  79. package/docs/guides/timeframes-and-bucketing.md +55 -55
  80. package/docs/internals/README.md +21 -21
  81. package/docs/internals/internal-architecture-overview.md +65 -65
  82. package/docs/internals/internal-backup-stream.md +23 -23
  83. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  84. package/docs/internals/internal-cardinality.md +30 -30
  85. package/docs/internals/internal-dimension-hash.md +34 -34
  86. package/docs/internals/internal-idempotency.md +36 -36
  87. package/docs/internals/internal-ingest-pipeline.md +52 -52
  88. package/docs/internals/internal-insights-layer.md +36 -36
  89. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  90. package/docs/internals/internal-measures-and-scales.md +35 -35
  91. package/docs/internals/internal-query-path.md +46 -46
  92. package/docs/internals/internal-storage-memory.md +13 -13
  93. package/docs/internals/internal-storage-mongo.md +44 -44
  94. package/docs/internals/internal-storage-postgres.md +32 -0
  95. package/docs/internals/internal-storage-redis.md +63 -63
  96. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  97. package/docs/use-cases/README.md +56 -56
  98. package/docs/use-cases/audit-security-log-counts.md +550 -550
  99. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  100. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  101. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  102. package/docs/use-cases/processing-workflow-activity.md +165 -165
  103. package/docs/use-cases/product-feature-usage.md +240 -240
  104. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  105. package/package.json +190 -170
@@ -1,375 +1,399 @@
1
- # Deploying and releasing `@x12i/countex`
2
-
3
- This guide has two parts:
4
-
5
- 1. **Publishing the library** to the public npm registry (`@x12i/countex`).
6
- 2. **Running it in production** in your own service (MongoDB durable store, optional cache — Redis, in-memory, or durable-backed reads — plus process lifecycle, configuration).
7
-
8
- After `npm install @x12i/countex`, these files ship inside the package (see **Finding this document** at the end).
9
-
10
- ---
11
-
12
- ## Part 1 — Publish the package to npm
13
-
14
- ### 1. Prerequisites
15
-
16
- - **Node.js** 18 or newer (matches `engines` in `package.json`).
17
- - An **npm account** with permission to publish under the **`@x12i` scope**. If the scope is new, create an npm organization (or user scope) and add trusted publishers.
18
- - **Git** remote aligned with `repository` in `package.json` (for tags and optional CI).
19
-
20
- ### 2. Machine setup
21
-
22
- 1. Log in locally (interactive):
23
-
24
- ```bash
25
- npm login
26
- ```
27
-
28
- Or configure a **granular access token** in **`~/.npmrc`** on the machine or in CI secrets — never commit tokens into the repository. The repo’s `.gitignore` excludes `.npmrc` for that reason.
29
-
30
- 2. Confirm who you are:
31
-
32
- ```bash
33
- npm whoami
34
- ```
35
-
36
- ### 3. Pre-release checklist
37
-
38
- Run from the package root (the folder that contains `package.json`):
39
-
40
- | Step | Command / action |
41
- |------|------------------|
42
- | Install dev deps | `npm install` |
43
- | Tests | `npm test` |
44
- | Optional integration | `npm run test:integration` |
45
- | Clean build | `npm run build` (or rely on publish hook below) |
46
- | Dry-run tarball | `npm publish --dry-run` |
47
-
48
- `package.json` includes **`prepublishOnly`**: `npm run clean && npm run build`. That means every **`npm publish`** rebuilds `dist/` so the published tarball always matches source.
49
-
50
- `publishConfig` is set to **`access: "public"`** so the scoped name `@x12i/countex` is published as a **public** package.
51
-
52
- ### 4. Versioning
53
-
54
- Follow **semver**:
55
-
56
- - **`0.x`** — API still stabilizing; breaking changes are expected until `1.0.0`.
57
- - Bump **`version`** in `package.json` (or use `npm version patch|minor|major`), commit, and tag if your workflow uses tags.
58
-
59
- Example:
60
-
61
- ```bash
62
- npm version patch
63
- git push origin main --follow-tags
64
- ```
65
-
66
- ### 5. Publish
67
-
68
- ```bash
69
- npm publish
70
- ```
71
-
72
- First-time scoped publish: if anything blocks you, confirm **`publishConfig.access`** is `"public"` (already set in this repo).
73
-
74
- ### 6. Verify on the registry
75
-
76
- ```bash
77
- npm view @x12i/countex version
78
- npm view @x12i/countex dist.tarball
79
- ```
80
-
81
- Smoke-test in an empty directory:
82
-
83
- ```bash
84
- mkdir /tmp/countex-smoke && cd /tmp/countex-smoke
85
- npm init -y
86
- npm install @x12i/countex
87
- node -e "console.log(require('@x12i/countex/package.json').version)"
88
- ```
89
-
90
- ### 7. Optional — CI publishing (outline)
91
-
92
- Typical pattern:
93
-
94
- 1. On release tag or `release` workflow, check out the repo.
95
- 2. Use a **trusted publishing** / **OIDC** setup with npm (recommended) or a **scoped automation token** stored as a secret.
96
- 3. Run `npm ci`, `npm test`, then `npm publish`.
97
-
98
- Exact YAML depends on your host (GitHub Actions, GitLab CI, etc.). Keep tokens out of logs.
99
-
100
- ---
101
-
102
- ## Part 2 — Deploy an application that uses Countex
103
-
104
- Countex is a **library**, not a standalone server. You embed it in a long-running worker, API, or stream consumer.
105
-
106
- ### 1. Install in your app
107
-
108
- ```bash
109
- npm install @x12i/countex
110
- ```
111
-
112
- If you use **MongoDB** and/or **Redis** adapters:
113
-
114
- ```bash
115
- npm install mongodb ioredis
116
- ```
117
-
118
- For **Mongo + in-process cache or durable-read cache only**, `ioredis` is not required:
119
-
120
- ```bash
121
- npm install mongodb
122
- ```
123
-
124
- Those are **peer dependencies**: they must be installed in **your** application; they are not bundled inside `@x12i/countex`.
125
-
126
- ### 2. Imports
127
-
128
- | Need | Import |
129
- |------|--------|
130
- | Engine factory | `import { createCountex } from "@x12i/countex"` |
131
- | In-memory adapters | `import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory"` |
132
- | Mongo durable | `import { MongoDurable } from "@x12i/countex/adapters/mongo"` |
133
- | Redis cache | `import { RedisCache } from "@x12i/countex/adapters/redis"` |
134
- | Durable-backed reads (no Redis) | `import { DurableReadCache } from "@x12i/countex/adapters/durable-cache"` |
135
-
136
- ### 3. MongoDB (durable storage)
137
-
138
- **Role:** Source of truth for counter cells and dictionary ids (`resolveIds`). Writes use atomic **`$inc`**-style upserts per cell.
139
-
140
- **Setup:**
141
-
142
- 1. Provide a **`MongoClient`** from the `mongodb` driver (you own the client lifecycle — connect before `createCountex`, disconnect after shutdown).
143
- 2. Choose how databases map to **scopes**:
144
- - **Single DB:** omit `dbResolver` and set `defaultDb` (default name `countex`).
145
- - **DB per tenant:** use `dbResolver: ({ scope }) => \`countex_${scope}\`` (sanitize names for Mongo rules).
146
-
147
- **Collections** (defaults):
148
-
149
- - **`countex_cells`** — aggregate counts.
150
- - **`countex_dict`** — maps `(scope, kind, qualifier, sourceKey)` → numeric ids for subjects/metrics.
151
- - **`countex_seq`** — sequence for allocating ids.
152
-
153
- On **`init()`**, the adapter **ensures indexes** on each DB it touches (unless **`skipIndexInit: true`** for externally managed migrations).
154
-
155
- **Operations:**
156
-
157
- - Call **`await countex.init()`** once after construction so default DB indexes exist (when not using only lazy per-scope DBs before first flush — lazy paths still ensure indexes on first write).
158
- - Under load, Mongo must be reachable with latency suitable for your flush interval; durable flush failures cause buffered deltas to be **re-merged** into memory.
159
-
160
- ### 4. Redis (cache)
161
-
162
- **Role:** Hot reads and optional **idempotency** (`markEventSeen`). Cache apply failures during flush are **non-fatal**; durable remains authoritative.
163
-
164
- **Redis is optional.** For a **single process** you may use **`MemoryCache`** with `MongoDurable` (hot state in RAM; `init()` warns not to use this with multiple workers on the same data). For **low query volume** without a hot tier, use **`DurableReadCache`** from `@x12i/countex/adapters/durable-cache`, which reads from durable storage on each query (`init()` warns about latency and process-local idempotency). See [Memory adapters guide](./guides/adapter-memory.md).
165
-
166
- **Setup (when using Redis):**
167
-
168
- 1. Provide an **`ioredis`** client instance.
169
- 2. Set **`keyPrefix`** to avoid collisions with other apps on shared Redis.
170
- 3. Set **`ttlSeconds`** for cache eviction consistent with your retention (example code uses ~3 months), or pass `retention.redis` on `createCountex` to forward per-timeframe TTLs to the cache adapter.
171
-
172
- **Required Redis configuration:**
173
-
174
- - **`maxmemory-policy noeviction`** — silent eviction breaks per-bucket coverage assumptions: a half-evicted bucket would still report `:cov.status=complete` while the cells hash is gone, producing incorrect "full" coverage reads. Use `noeviction` and provision Redis memory above your hot working set; rely on TTLs (not LRU) for expiry.
175
- - Plan memory headroom = (cells per hot bucket × bucket count for hottest timeframe × per-cell overhead) + 25%.
176
-
177
- **Idempotency:** If your map defines `idempotency` with `eventIdPath`, Countex calls **`cache.markEventSeen`** before buffering. Ensure Redis is available for dedupe semantics you rely on.
178
-
179
- ### 5. Environment variables (recommended)
180
-
181
- Define explicitly in your deployment (Kubernetes secrets, systemd `Environment=`, etc.):
182
-
183
- | Variable | Used by | Purpose |
184
- |----------|---------|---------|
185
- | `MONGO_URI` | Your app | Mongo connection string |
186
- | `REDIS_URL` | Your app | Redis connection string |
187
-
188
- Example pattern:
189
-
190
- ```ts
191
- const mongo = new MongoClient(process.env.MONGO_URI ?? "mongodb://localhost:27017");
192
- const redis = new IORedis(process.env.REDIS_URL ?? "redis://localhost:6379");
193
- ```
194
-
195
- Never commit real URIs or tokens.
196
-
197
- ### 6. Process lifecycle
198
-
199
- Recommended order:
200
-
201
- 1. Construct **`MongoClient`** / **`IORedis`** and connect.
202
- 2. **`createCountex({ ... })`** with `MongoDurable` / `RedisCache`.
203
- 3. **`await countex.init()`** — storage initialization.
204
- 4. **Ingest loop** — `await countex.ingest(event)` or **`ingestBatch`** from Kafka/NATS/HTTP/etc.
205
- 5. **Flush** — periodic timer may already run via `flush.everyMs`; also call **`await countex.flush()`** before exit or on graceful shutdown.
206
- 6. **`await countex.shutdown()`** — stops flush timer, optional final flush (`flushOnShutdown`), then **`close`** hooks on adapters (Mongo client is **not** closed by `MongoDurable`; you close it after shutdown).
207
-
208
- Handle **`SIGTERM` / `SIGINT`** in servers so shutdown flushes buffered counts.
209
-
210
- ### 7. Flush and backpressure
211
-
212
- Configure under **`flush`**:
213
-
214
- - **`everyMs`** — background flush interval (`0` disables timer; you flush manually).
215
- - **`maxBufferedCells` / `maxBufferedEvents`** — thresholds triggering flush.
216
- - **`onOverflow`** — `"block"` (await flush) vs `"drop"` (fire-and-forget flush under pressure — understand data risk).
217
-
218
- Tune for **peak event rate**, **Mongo write throughput**, and **acceptable latency** before counts appear in storage.
219
-
220
- ### 8. Observability
221
-
222
- Subscribe to **`Countex`** events (`countex.on(...)`) for:
223
-
224
- - **`flush`** — cells flushed, duration.
225
- - **`flush_error`** — durable failure (buffer re-merged).
226
- - **`duplicate`** — idempotent skip.
227
- - **`cardinality_warning`**, **`overflow`**, **`dropped`** — mapping/cardinality/dictionary issues.
228
- - **`measure_invalid`** — per-measure extraction failures (missing, non-numeric, non-finite, negative-for-unsigned, precision-truncated).
229
-
230
- Wire these to your logger/metrics stack.
231
-
232
- **Recommended monitoring signals:**
233
-
234
- | Signal | What to watch |
235
- |---|---|
236
- | `flush.cells` per minute | Workload volume |
237
- | `flush.durationMs` p50 / p95 | Mongo write latency under load |
238
- | `flush_error` rate | Durable failover; deltas re-merged into the buffer |
239
- | `dropped.reason` cardinality vs cardinality:* | Map cardinality budgets |
240
- | `duplicate` rate | Idempotency dedupe effectiveness; webhook replay rate |
241
- | `countex.diag().bufferCells` | Backpressure signal — rising = flush can't keep up |
242
- | Redis `maxmemory_used` | Stay below maxmemory; rising eviction count means policy drift |
243
- | Mongo `countex_cells` doc count growth per timeframe | Capacity planning, retention triggers |
244
- | Query `result.diagnostics.cacheCoverage` distribution | Cache effectiveness; lots of `partial`/`miss` means undersized hot window |
245
-
246
- **Worker crash-loss window:** the only state lost on a hard worker crash is **unflushed deltas in memory**. Bound this window with `flush.everyMs`, `flush.maxBufferedCells`, and `flush.maxBufferedEvents`. With `flush.everyMs: 60_000` and `maxBufferedCells: 100_000`, expected loss is `min(60s of writes, 100k cells)`. Tune lower for high-volume / low-tolerance workloads.
247
-
248
- ### 9. Retention (Mongo + Redis)
249
-
250
- Prefer the unified **`CountexConfig.retention`** policy:
251
-
252
- ```ts
253
- const countex = createCountex({
254
- // …
255
- retention: {
256
- redis: {
257
- minute: "2d",
258
- hour: "30d",
259
- day: "92d",
260
- week: "52w",
261
- month: "24mo",
262
- quarter: "12q",
263
- year: "7y",
264
- all: "forever",
265
- },
266
- // Mongo entries are documented today, not enforced.
267
- mongo: { day: "365d", month: "60mo", all: "forever" },
268
- },
269
- });
270
- ```
271
-
272
- - **Redis** is ephemeral: the engine forwards `retention.redis` to the cache adapter via `setRetentionPolicy(...)`, overriding any constructor-time `ttlSecondsByTimeframe`. After TTL expiry, the bucket disappears, reads fall back to Mongo, and the planner re-warms the cache. Combined with `noeviction`, this is the only authoritative way buckets leave Redis.
273
- - **MongoDB** is the system of record. Plan archival or TTL **per your compliance window** — Countex does not delete Mongo rows automatically. Operators implement Mongo retention themselves:
274
-
275
- ```js
276
- // Option 1: TTL index on updatedAt (per-collection, drops cells after N seconds).
277
- db.countex_cells.createIndex({ updatedAt: 1 }, { expireAfterSeconds: 365 * 86400 });
278
-
279
- // Option 2: scheduled prune by timeframeKey range (per-counter retention window).
280
- db.countex_cells.deleteMany({
281
- counterId: "events",
282
- timeframeType: "day",
283
- timeframeKey: { $lt: "2025-05-01" },
284
- });
285
- ```
286
-
287
- Large tenants may use **per-scope databases** via `dbResolver` and drop/archive databases independently.
288
-
289
- Cell documents written by `MongoDurable` include optional **`schemaVersion`** for future migrations (`MONGO_CELL_SCHEMA_VERSION` export).
290
-
291
- ### 10. Self-managed sizing (back-of-napkin)
292
-
293
- | Signal | Rule of thumb |
294
- |--------|----------------|
295
- | Dictionary growth | One doc per distinct `(scope, subject/metric sourceKey)` seen in ingest. |
296
- | Cell growth | Fan-out × distinct buckets × subjects × metrics × dimension hashes. |
297
- | Flush load | `(cells per flush batch)` ÷ `flush.everyMs` × write cost of Mongo bulk upserts. |
298
- | Redis memory | Roughly proportional to **hot buckets × fields per hash**; bounded by TTL. |
299
-
300
- Use [`docs/guides/simulation-resource-estimation.md`](./guides/simulation-resource-estimation.md) and workload playbooks under [`docs/use-cases/`](./use-cases/README.md) for concrete scenarios.
301
-
302
- ### 11. Backup verify / restore
303
-
304
- Stream **`exportCells`**, attach a **`BackupManifest`** (`computeCellsFingerprint` + `createBackupManifest`), and **`verifyBackupAgainstManifest`** before trusting cold storage. **`restoreSnapshotCells`** applies absolute snapshots to adapters that implement **`importSnapshotCells`** (Mongo + memory today). Re-warm Redis afterward. See [`docs/guides/backup-export.md`](./guides/backup-export.md).
305
-
306
- ### 12. Tenant isolation
307
-
308
- Cross-link: [`docs/use-cases/multi-tenant-isolation.md`](./use-cases/multi-tenant-isolation.md) — pin **`scope`** on every query, isolate Mongo via **`dbResolver`**, and use separate Redis key prefixes per environment.
309
-
310
- **Tenant DB naming rules** (when using `dbResolver`):
311
-
312
- - Mongo DB names are 1–63 characters, must not contain `/ \ . " $ * < > : | ?` or null bytes, and on Windows are also case-insensitive (so two scopes that differ only in case collide). Sanitize **before** passing to `dbResolver`:
313
-
314
- ```ts
315
- function sanitizeScope(s: string): string {
316
- return s.toLowerCase().replace(/[^a-z0-9_-]/g, "_").slice(0, 50);
317
- }
318
-
319
- const durable = new MongoDurable({
320
- client: mongo,
321
- dbResolver: ({ scope }) => `countex_${sanitizeScope(scope)}`,
322
- });
323
- ```
324
-
325
- - For very high tenant counts (>1k tenants), prefer a **single shared DB with indexed `scopeId`** over DB-per-tenant — Mongo has per-DB metadata overhead (`countex_cells`, `countex_dict`, `countex_seq` ≈ 6 indexes per tenant DB).
326
- - For untrusted scope sources (raw user input, URLs, emails) prefer **hashing** rather than sanitizing:
327
-
328
- ```ts
329
- import crypto from "node:crypto";
330
- const hashScope = (s: string) => crypto.createHash("sha1").update(s).digest("hex").slice(0, 16);
331
- dbResolver: ({ scope }) => `countex_${hashScope(scope)}`;
332
- ```
333
-
334
- - Sanity-check that two distinct sanitized names cannot collide in your dataset (e.g. `org/123` and `org_123` both → `org_123`). Hash if collision is plausible.
335
-
336
- ### 13. Failure modes (short)
337
-
338
- | Situation | Behavior |
339
- |-----------|----------|
340
- | Redis down during flush | Warning; durable OK; reads may hit Mongo |
341
- | Mongo down during flush | Error; deltas **returned to buffer** (with brief durable retries) |
342
- | Duplicate event id | Increments skipped for that map/event |
343
-
344
- ---
345
-
346
- ## Architecture decisions (operational snapshot)
347
-
348
- | Topic | Decision for this package |
349
- |-------|---------------------------|
350
- | Monorepo vs split packages | **Single npm package** `@x12i/countex` with subpath exports (`/insights`, adapters, docs). |
351
- | Dictionary ids | **`number`** per scope (Mongo sequence); BigInt not required at current scale. |
352
- | Percentiles / histograms | **Not in core cells**; insights stay summary/ranking-oriented unless you add explicit measure storage later. |
353
- | Backup restore | **Replace snapshot** via `importSnapshotCells` + manifest checksum; not logical event replay. |
354
- | Timezones | Bucketing is **UTC** (`bucketKey`); convert in your pipeline before `eventTime` if needed. |
355
-
356
- ---
357
-
358
- ## Finding this document after install
359
-
360
- From a project that depends on `@x12i/countex`:
361
-
362
- - **On disk:** `node_modules/@x12i/countex/docs/DEPLOYMENT.md`
363
- - **Stable package subpaths** (for tooling that resolves `exports`):
364
-
365
- - `@x12i/countex/docs/deployment`
366
- - `@x12i/countex/docs/end-to-end-example`
367
-
368
- Your bundler or runtime may not import `.md` as code; open the resolved file path or read it from disk.
369
-
370
- ---
371
-
372
- ## Related
373
-
374
- - **`END_TO_END_EXAMPLE.md`** — tutorial-style walkthrough from install to query.
375
- - Repository **README** — API overview and quick start.
1
+ # Deploying and releasing `@x12i/countex`
2
+
3
+ This guide has two parts:
4
+
5
+ 1. **Publishing the library** to the public npm registry (`@x12i/countex`).
6
+ 2. **Running it in production** in your own service (MongoDB durable store, optional cache — Redis, in-memory, or durable-backed reads — plus process lifecycle, configuration).
7
+
8
+ After `npm install @x12i/countex`, these files ship inside the package (see **Finding this document** at the end).
9
+
10
+ ---
11
+
12
+ ## Part 1 — Publish the package to npm
13
+
14
+ ### 1. Prerequisites
15
+
16
+ - **Node.js** 18 or newer (matches `engines` in `package.json`).
17
+ - An **npm account** with permission to publish under the **`@x12i` scope**. If the scope is new, create an npm organization (or user scope) and add trusted publishers.
18
+ - **Git** remote aligned with `repository` in `package.json` (for tags and optional CI).
19
+
20
+ ### 2. Machine setup
21
+
22
+ 1. Log in locally (interactive):
23
+
24
+ ```bash
25
+ npm login
26
+ ```
27
+
28
+ Or configure a **granular access token** in **`~/.npmrc`** on the machine or in CI secrets — never commit tokens into the repository. The repo’s `.gitignore` excludes `.npmrc` for that reason.
29
+
30
+ 2. Confirm who you are:
31
+
32
+ ```bash
33
+ npm whoami
34
+ ```
35
+
36
+ ### 3. Pre-release checklist
37
+
38
+ Run from the package root (the folder that contains `package.json`):
39
+
40
+ | Step | Command / action |
41
+ |------|------------------|
42
+ | Install dev deps | `npm install` |
43
+ | Tests | `npm test` |
44
+ | Optional integration | `npm run test:integration` |
45
+ | Clean build | `npm run build` (or rely on publish hook below) |
46
+ | Dry-run tarball | `npm publish --dry-run` |
47
+
48
+ `package.json` includes **`prepublishOnly`**: `npm run clean && npm run build`. That means every **`npm publish`** rebuilds `dist/` so the published tarball always matches source.
49
+
50
+ `publishConfig` is set to **`access: "public"`** so the scoped name `@x12i/countex` is published as a **public** package.
51
+
52
+ ### 4. Versioning
53
+
54
+ Follow **semver**:
55
+
56
+ - **`0.x`** — API still stabilizing; breaking changes are expected until `1.0.0`.
57
+ - Bump **`version`** in `package.json` (or use `npm version patch|minor|major`), commit, and tag if your workflow uses tags.
58
+
59
+ Example:
60
+
61
+ ```bash
62
+ npm version patch
63
+ git push origin main --follow-tags
64
+ ```
65
+
66
+ ### 5. Publish
67
+
68
+ ```bash
69
+ npm publish
70
+ ```
71
+
72
+ First-time scoped publish: if anything blocks you, confirm **`publishConfig.access`** is `"public"` (already set in this repo).
73
+
74
+ ### 6. Verify on the registry
75
+
76
+ ```bash
77
+ npm view @x12i/countex version
78
+ npm view @x12i/countex dist.tarball
79
+ ```
80
+
81
+ Smoke-test in an empty directory:
82
+
83
+ ```bash
84
+ mkdir /tmp/countex-smoke && cd /tmp/countex-smoke
85
+ npm init -y
86
+ npm install @x12i/countex
87
+ node -e "console.log(require('@x12i/countex/package.json').version)"
88
+ ```
89
+
90
+ ### 7. Optional — CI publishing (outline)
91
+
92
+ Typical pattern:
93
+
94
+ 1. On release tag or `release` workflow, check out the repo.
95
+ 2. Use a **trusted publishing** / **OIDC** setup with npm (recommended) or a **scoped automation token** stored as a secret.
96
+ 3. Run `npm ci`, `npm test`, then `npm publish`.
97
+
98
+ Exact YAML depends on your host (GitHub Actions, GitLab CI, etc.). Keep tokens out of logs.
99
+
100
+ ---
101
+
102
+ ## Part 2 — Deploy an application that uses Countex
103
+
104
+ Countex is a **library**, not a standalone server. You embed it in a long-running worker, API, or stream consumer.
105
+
106
+ ### 1. Install in your app
107
+
108
+ ```bash
109
+ npm install @x12i/countex
110
+ ```
111
+
112
+ If you use **MongoDB** and/or **Redis** adapters:
113
+
114
+ ```bash
115
+ npm install mongodb ioredis
116
+ ```
117
+
118
+ For **PostgreSQL** durable storage:
119
+
120
+ ```bash
121
+ npm install pg ioredis
122
+ ```
123
+
124
+ For **Mongo + in-process cache or durable-read cache only**, `ioredis` is not required:
125
+
126
+ ```bash
127
+ npm install mongodb
128
+ ```
129
+
130
+ Those are **peer dependencies**: they must be installed in **your** application; they are not bundled inside `@x12i/countex`.
131
+
132
+ ### 2. Imports
133
+
134
+ | Need | Import |
135
+ |------|--------|
136
+ | Engine factory | `import { createCountex } from "@x12i/countex"` |
137
+ | In-memory adapters | `import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory"` |
138
+ | Mongo durable | `import { MongoDurable } from "@x12i/countex/adapters/mongo"` |
139
+ | Postgres durable | `import { PostgresDurable } from "@x12i/countex/adapters/postgres"` |
140
+ | Redis cache | `import { RedisCache } from "@x12i/countex/adapters/redis"` |
141
+ | Durable-backed reads (no Redis) | `import { DurableReadCache } from "@x12i/countex/adapters/durable-cache"` |
142
+
143
+ ### 3. MongoDB (durable storage)
144
+
145
+ **Role:** Source of truth for counter cells and dictionary ids (`resolveIds`). Writes use atomic **`$inc`**-style upserts per cell.
146
+
147
+ **Setup:**
148
+
149
+ 1. Provide a **`MongoClient`** from the `mongodb` driver (you own the client lifecycle — connect before `createCountex`, disconnect after shutdown).
150
+ 2. Choose how databases map to **scopes**:
151
+ - **Single DB:** omit `dbResolver` and set `defaultDb` (default name `countex`).
152
+ - **DB per tenant:** use `dbResolver: ({ scope }) => \`countex_${scope}\`` (sanitize names for Mongo rules).
153
+
154
+ **Collections** (defaults):
155
+
156
+ - **`countex_cells`** — aggregate counts.
157
+ - **`countex_dict`** — maps `(scope, kind, qualifier, sourceKey)` → numeric ids for subjects/metrics.
158
+ - **`countex_seq`** — sequence for allocating ids.
159
+
160
+ On **`init()`**, the adapter **ensures indexes** on each DB it touches (unless **`skipIndexInit: true`** for externally managed migrations).
161
+
162
+ **Operations:**
163
+
164
+ - Call **`await countex.init()`** once after construction so default DB indexes exist (when not using only lazy per-scope DBs before first flush — lazy paths still ensure indexes on first write).
165
+ - Under load, Mongo must be reachable with latency suitable for your flush interval; durable flush failures cause buffered deltas to be **re-merged** into memory.
166
+
167
+ ### 3a. PostgreSQL (durable storage)
168
+
169
+ **Role:** Same source of truth as MongoDB: counter cells and dictionary ids. Writes are batched `INSERT … ON CONFLICT` increments. Redis remains the hot cache.
170
+
171
+ **Setup:**
172
+
173
+ 1. PostgreSQL **15+**. Provide a **`pg.Pool`** (you own the pool — Countex does not call `pool.end()`).
174
+ 2. Use one schema (default `countex`). `scope_id` is already on every row. A schema per tenant is optional and usually the wrong default.
175
+ 3. In front of many Node workers, put **PgBouncer** in transaction mode. Postgres connections are heavier than Mongo driver connections.
176
+ 4. Call **`await countex.init()`** so the adapter creates `cells` (minute/hour/default partitions), `dict`, `seq`, `evidence`, and `merge_measures`.
177
+
178
+ **Retention** is operator-owned (`retention.postgres` is documentation). Drop or detach old minute/hour partitions, or delete by `timeframe_key`. Keep `autovacuum_vacuum_scale_factor` well below the default on `cells` so increment bloat does not erase the smaller row size.
179
+
180
+ Moving an existing Mongo deployment: [migrating-mongo-to-postgres.md](./guides/migrating-mongo-to-postgres.md). Copy dictionary ids. Do not call `resolveIds` to renumber them.
181
+
182
+ Guide: [adapter-postgres.md](./guides/adapter-postgres.md).
183
+
184
+ ### 4. Redis (cache)
185
+
186
+ **Role:** Hot reads and optional **idempotency** (`markEventSeen`). Cache apply failures during flush are **non-fatal**; durable remains authoritative.
187
+
188
+ **Redis is optional.** For a **single process** you may use **`MemoryCache`** with `MongoDurable` (hot state in RAM; `init()` warns not to use this with multiple workers on the same data). For **low query volume** without a hot tier, use **`DurableReadCache`** from `@x12i/countex/adapters/durable-cache`, which reads from durable storage on each query (`init()` warns about latency and process-local idempotency). See [Memory adapters guide](./guides/adapter-memory.md).
189
+
190
+ **Setup (when using Redis):**
191
+
192
+ 1. Provide an **`ioredis`** client instance.
193
+ 2. Set **`keyPrefix`** to avoid collisions with other apps on shared Redis.
194
+ 3. Set **`ttlSeconds`** for cache eviction consistent with your retention (example code uses ~3 months), or pass `retention.redis` on `createCountex` to forward per-timeframe TTLs to the cache adapter.
195
+
196
+ **Required Redis configuration:**
197
+
198
+ - **`maxmemory-policy noeviction`** — silent eviction breaks per-bucket coverage assumptions: a half-evicted bucket would still report `:cov.status=complete` while the cells hash is gone, producing incorrect "full" coverage reads. Use `noeviction` and provision Redis memory above your hot working set; rely on TTLs (not LRU) for expiry.
199
+ - Plan memory headroom = (cells per hot bucket × bucket count for hottest timeframe × per-cell overhead) + 25%.
200
+
201
+ **Idempotency:** If your map defines `idempotency` with `eventIdPath`, Countex calls **`cache.markEventSeen`** before buffering. Ensure Redis is available for dedupe semantics you rely on.
202
+
203
+ ### 5. Environment variables (recommended)
204
+
205
+ Define explicitly in your deployment (Kubernetes secrets, systemd `Environment=`, etc.):
206
+
207
+ | Variable | Used by | Purpose |
208
+ |----------|---------|---------|
209
+ | `MONGO_URI` | Your app | Mongo connection string |
210
+ | `REDIS_URL` | Your app | Redis connection string |
211
+
212
+ Example pattern:
213
+
214
+ ```ts
215
+ const mongo = new MongoClient(process.env.MONGO_URI ?? "mongodb://localhost:27017");
216
+ const redis = new IORedis(process.env.REDIS_URL ?? "redis://localhost:6379");
217
+ ```
218
+
219
+ Never commit real URIs or tokens.
220
+
221
+ ### 6. Process lifecycle
222
+
223
+ Recommended order:
224
+
225
+ 1. Construct **`MongoClient`** / **`IORedis`** and connect.
226
+ 2. **`createCountex({ ... })`** with `MongoDurable` / `RedisCache`.
227
+ 3. **`await countex.init()`** — storage initialization.
228
+ 4. **Ingest loop** — `await countex.ingest(event)` or **`ingestBatch`** from Kafka/NATS/HTTP/etc.
229
+ 5. **Flush** — periodic timer may already run via `flush.everyMs`; also call **`await countex.flush()`** before exit or on graceful shutdown.
230
+ 6. **`await countex.shutdown()`** — stops flush timer, optional final flush (`flushOnShutdown`), then **`close`** hooks on adapters (Mongo client is **not** closed by `MongoDurable`; you close it after shutdown).
231
+
232
+ Handle **`SIGTERM` / `SIGINT`** in servers so shutdown flushes buffered counts.
233
+
234
+ ### 7. Flush and backpressure
235
+
236
+ Configure under **`flush`**:
237
+
238
+ - **`everyMs`** — background flush interval (`0` disables timer; you flush manually).
239
+ - **`maxBufferedCells` / `maxBufferedEvents`** — thresholds triggering flush.
240
+ - **`onOverflow`** — `"block"` (await flush) vs `"drop"` (fire-and-forget flush under pressure — understand data risk).
241
+
242
+ Tune for **peak event rate**, **Mongo write throughput**, and **acceptable latency** before counts appear in storage.
243
+
244
+ ### 8. Observability
245
+
246
+ Subscribe to **`Countex`** events (`countex.on(...)`) for:
247
+
248
+ - **`flush`** — cells flushed, duration.
249
+ - **`flush_error`** — durable failure (buffer re-merged).
250
+ - **`duplicate`** — idempotent skip.
251
+ - **`cardinality_warning`**, **`overflow`**, **`dropped`** — mapping/cardinality/dictionary issues.
252
+ - **`measure_invalid`** — per-measure extraction failures (missing, non-numeric, non-finite, negative-for-unsigned, precision-truncated).
253
+
254
+ Wire these to your logger/metrics stack.
255
+
256
+ **Recommended monitoring signals:**
257
+
258
+ | Signal | What to watch |
259
+ |---|---|
260
+ | `flush.cells` per minute | Workload volume |
261
+ | `flush.durationMs` p50 / p95 | Mongo write latency under load |
262
+ | `flush_error` rate | Durable failover; deltas re-merged into the buffer |
263
+ | `dropped.reason` cardinality vs cardinality:* | Map cardinality budgets |
264
+ | `duplicate` rate | Idempotency dedupe effectiveness; webhook replay rate |
265
+ | `countex.diag().bufferCells` | Backpressure signal — rising = flush can't keep up |
266
+ | Redis `maxmemory_used` | Stay below maxmemory; rising eviction count means policy drift |
267
+ | Mongo `countex_cells` doc count growth per timeframe | Capacity planning, retention triggers |
268
+ | Query `result.diagnostics.cacheCoverage` distribution | Cache effectiveness; lots of `partial`/`miss` means undersized hot window |
269
+
270
+ **Worker crash-loss window:** the only state lost on a hard worker crash is **unflushed deltas in memory**. Bound this window with `flush.everyMs`, `flush.maxBufferedCells`, and `flush.maxBufferedEvents`. With `flush.everyMs: 60_000` and `maxBufferedCells: 100_000`, expected loss is `min(60s of writes, 100k cells)`. Tune lower for high-volume / low-tolerance workloads.
271
+
272
+ ### 9. Retention (Mongo + Redis)
273
+
274
+ Prefer the unified **`CountexConfig.retention`** policy:
275
+
276
+ ```ts
277
+ const countex = createCountex({
278
+ // …
279
+ retention: {
280
+ redis: {
281
+ minute: "2d",
282
+ hour: "30d",
283
+ day: "92d",
284
+ week: "52w",
285
+ month: "24mo",
286
+ quarter: "12q",
287
+ year: "7y",
288
+ all: "forever",
289
+ },
290
+ // Mongo entries are documented today, not enforced.
291
+ mongo: { day: "365d", month: "60mo", all: "forever" },
292
+ },
293
+ });
294
+ ```
295
+
296
+ - **Redis** is ephemeral: the engine forwards `retention.redis` to the cache adapter via `setRetentionPolicy(...)`, overriding any constructor-time `ttlSecondsByTimeframe`. After TTL expiry, the bucket disappears, reads fall back to Mongo, and the planner re-warms the cache. Combined with `noeviction`, this is the only authoritative way buckets leave Redis.
297
+ - **MongoDB** is the system of record. Plan archival or TTL **per your compliance window** — Countex does not delete Mongo rows automatically. Operators implement Mongo retention themselves:
298
+
299
+ ```js
300
+ // Option 1: TTL index on updatedAt (per-collection, drops cells after N seconds).
301
+ db.countex_cells.createIndex({ updatedAt: 1 }, { expireAfterSeconds: 365 * 86400 });
302
+
303
+ // Option 2: scheduled prune by timeframeKey range (per-counter retention window).
304
+ db.countex_cells.deleteMany({
305
+ counterId: "events",
306
+ timeframeType: "day",
307
+ timeframeKey: { $lt: "2025-05-01" },
308
+ });
309
+ ```
310
+
311
+ Large tenants may use **per-scope databases** via `dbResolver` and drop/archive databases independently.
312
+
313
+ Cell documents written by `MongoDurable` include optional **`schemaVersion`** for future migrations (`MONGO_CELL_SCHEMA_VERSION` export).
314
+
315
+ ### 10. Self-managed sizing (back-of-napkin)
316
+
317
+ | Signal | Rule of thumb |
318
+ |--------|----------------|
319
+ | Dictionary growth | One doc per distinct `(scope, subject/metric sourceKey)` seen in ingest. |
320
+ | Cell growth | Fan-out × distinct buckets × subjects × metrics × dimension hashes. |
321
+ | Flush load | `(cells per flush batch)` ÷ `flush.everyMs` × write cost of Mongo bulk upserts. |
322
+ | Redis memory | Roughly proportional to **hot buckets × fields per hash**; bounded by TTL. |
323
+
324
+ Use [`docs/guides/simulation-resource-estimation.md`](./guides/simulation-resource-estimation.md) and workload playbooks under [`docs/use-cases/`](./use-cases/README.md) for concrete scenarios.
325
+
326
+ ### 11. Backup verify / restore
327
+
328
+ Stream **`exportCells`**, attach a **`BackupManifest`** (`computeCellsFingerprint` + `createBackupManifest`), and **`verifyBackupAgainstManifest`** before trusting cold storage. **`restoreSnapshotCells`** applies absolute snapshots to adapters that implement **`importSnapshotCells`** (Mongo + memory today). Re-warm Redis afterward. See [`docs/guides/backup-export.md`](./guides/backup-export.md).
329
+
330
+ ### 12. Tenant isolation
331
+
332
+ Cross-link: [`docs/use-cases/multi-tenant-isolation.md`](./use-cases/multi-tenant-isolation.md) — pin **`scope`** on every query, isolate Mongo via **`dbResolver`**, and use separate Redis key prefixes per environment.
333
+
334
+ **Tenant DB naming rules** (when using `dbResolver`):
335
+
336
+ - Mongo DB names are 1–63 characters, must not contain `/ \ . " $ * < > : | ?` or null bytes, and on Windows are also case-insensitive (so two scopes that differ only in case collide). Sanitize **before** passing to `dbResolver`:
337
+
338
+ ```ts
339
+ function sanitizeScope(s: string): string {
340
+ return s.toLowerCase().replace(/[^a-z0-9_-]/g, "_").slice(0, 50);
341
+ }
342
+
343
+ const durable = new MongoDurable({
344
+ client: mongo,
345
+ dbResolver: ({ scope }) => `countex_${sanitizeScope(scope)}`,
346
+ });
347
+ ```
348
+
349
+ - For very high tenant counts (>1k tenants), prefer a **single shared DB with indexed `scopeId`** over DB-per-tenant — Mongo has per-DB metadata overhead (`countex_cells`, `countex_dict`, `countex_seq` ≈ 6 indexes per tenant DB).
350
+ - For untrusted scope sources (raw user input, URLs, emails) prefer **hashing** rather than sanitizing:
351
+
352
+ ```ts
353
+ import crypto from "node:crypto";
354
+ const hashScope = (s: string) => crypto.createHash("sha1").update(s).digest("hex").slice(0, 16);
355
+ dbResolver: ({ scope }) => `countex_${hashScope(scope)}`;
356
+ ```
357
+
358
+ - Sanity-check that two distinct sanitized names cannot collide in your dataset (e.g. `org/123` and `org_123` both → `org_123`). Hash if collision is plausible.
359
+
360
+ ### 13. Failure modes (short)
361
+
362
+ | Situation | Behavior |
363
+ |-----------|----------|
364
+ | Redis down during flush | Warning; durable OK; reads may hit Mongo |
365
+ | Mongo down during flush | Error; deltas **returned to buffer** (with brief durable retries) |
366
+ | Duplicate event id | Increments skipped for that map/event |
367
+
368
+ ---
369
+
370
+ ## Architecture decisions (operational snapshot)
371
+
372
+ | Topic | Decision for this package |
373
+ |-------|---------------------------|
374
+ | Monorepo vs split packages | **Single npm package** `@x12i/countex` with subpath exports (`/insights`, adapters, docs). |
375
+ | Dictionary ids | **`number`** per scope (Mongo sequence); BigInt not required at current scale. |
376
+ | Percentiles / histograms | **Not in core cells**; insights stay summary/ranking-oriented unless you add explicit measure storage later. |
377
+ | Backup restore | **Replace snapshot** via `importSnapshotCells` + manifest checksum; not logical event replay. |
378
+ | Timezones | Bucketing is **UTC** (`bucketKey`); convert in your pipeline before `eventTime` if needed. |
379
+
380
+ ---
381
+
382
+ ## Finding this document after install
383
+
384
+ From a project that depends on `@x12i/countex`:
385
+
386
+ - **On disk:** `node_modules/@x12i/countex/docs/DEPLOYMENT.md`
387
+ - **Stable package subpaths** (for tooling that resolves `exports`):
388
+
389
+ - `@x12i/countex/docs/deployment`
390
+ - `@x12i/countex/docs/end-to-end-example`
391
+
392
+ Your bundler or runtime may not import `.md` as code; open the resolved file path or read it from disk.
393
+
394
+ ---
395
+
396
+ ## Related
397
+
398
+ - **`END_TO_END_EXAMPLE.md`** — tutorial-style walkthrough from install to query.
399
+ - Repository **README** — API overview and quick start.