@x12i/countex 1.8.5 → 1.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (105) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +867 -860
  3. package/dist/cli.js +6 -6
  4. package/dist/core/countex.d.ts +61 -1
  5. package/dist/core/countex.d.ts.map +1 -1
  6. package/dist/core/countex.js +241 -0
  7. package/dist/core/countex.js.map +1 -1
  8. package/dist/core/managed.d.ts +200 -0
  9. package/dist/core/managed.d.ts.map +1 -0
  10. package/dist/core/managed.js +431 -0
  11. package/dist/core/managed.js.map +1 -0
  12. package/dist/core/timeframe.d.ts +5 -0
  13. package/dist/core/timeframe.d.ts.map +1 -1
  14. package/dist/core/timeframe.js +36 -0
  15. package/dist/core/timeframe.js.map +1 -1
  16. package/dist/index.d.ts +6 -2
  17. package/dist/index.d.ts.map +1 -1
  18. package/dist/index.js +9 -1
  19. package/dist/index.js.map +1 -1
  20. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  21. package/dist/insights/advisors/resource-advisor.js +5 -1
  22. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  23. package/dist/insights/types.d.ts +4 -0
  24. package/dist/insights/types.d.ts.map +1 -1
  25. package/dist/storage/managed-memory.d.ts +33 -0
  26. package/dist/storage/managed-memory.d.ts.map +1 -0
  27. package/dist/storage/managed-memory.js +211 -0
  28. package/dist/storage/managed-memory.js.map +1 -0
  29. package/dist/storage/memory.d.ts +7 -4
  30. package/dist/storage/memory.d.ts.map +1 -1
  31. package/dist/storage/memory.js.map +1 -1
  32. package/dist/storage/postgres.d.ts +123 -0
  33. package/dist/storage/postgres.d.ts.map +1 -0
  34. package/dist/storage/postgres.js +1325 -0
  35. package/dist/storage/postgres.js.map +1 -0
  36. package/dist/storage/redis.d.ts +6 -0
  37. package/dist/storage/redis.d.ts.map +1 -1
  38. package/dist/storage/redis.js +21 -0
  39. package/dist/storage/redis.js.map +1 -1
  40. package/dist/types/index.d.ts +63 -5
  41. package/dist/types/index.d.ts.map +1 -1
  42. package/dist/utils/errors.d.ts +12 -0
  43. package/dist/utils/errors.d.ts.map +1 -1
  44. package/dist/utils/errors.js +22 -1
  45. package/dist/utils/errors.js.map +1 -1
  46. package/docs/DEPLOYMENT.md +399 -375
  47. package/docs/END_TO_END_EXAMPLE.md +316 -316
  48. package/docs/INSIGHTS.md +59 -59
  49. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  50. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  51. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  52. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  53. package/docs/guides/README.md +37 -34
  54. package/docs/guides/adapter-memory.md +60 -60
  55. package/docs/guides/adapter-mongo.md +64 -62
  56. package/docs/guides/adapter-postgres.md +92 -0
  57. package/docs/guides/adapter-redis.md +89 -89
  58. package/docs/guides/backup-export.md +135 -135
  59. package/docs/guides/cardinality.md +50 -50
  60. package/docs/guides/cli.md +53 -53
  61. package/docs/guides/client-events.md +65 -65
  62. package/docs/guides/counter-maps.md +160 -160
  63. package/docs/guides/idempotency.md +39 -39
  64. package/docs/guides/ingestion-and-flush.md +67 -67
  65. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  66. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  67. package/docs/guides/insights-comparison-and-trends.md +47 -47
  68. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  69. package/docs/guides/insights-overview.md +82 -82
  70. package/docs/guides/insights-ranking.md +81 -81
  71. package/docs/guides/managed-commit.md +119 -0
  72. package/docs/guides/measures.md +72 -71
  73. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  74. package/docs/guides/preview.md +92 -92
  75. package/docs/guides/querying.md +178 -178
  76. package/docs/guides/simulation-profiles.md +33 -33
  77. package/docs/guides/simulation-resource-estimation.md +31 -30
  78. package/docs/guides/simulation.md +36 -36
  79. package/docs/guides/timeframes-and-bucketing.md +55 -55
  80. package/docs/internals/README.md +21 -21
  81. package/docs/internals/internal-architecture-overview.md +65 -65
  82. package/docs/internals/internal-backup-stream.md +23 -23
  83. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  84. package/docs/internals/internal-cardinality.md +30 -30
  85. package/docs/internals/internal-dimension-hash.md +34 -34
  86. package/docs/internals/internal-idempotency.md +36 -36
  87. package/docs/internals/internal-ingest-pipeline.md +52 -52
  88. package/docs/internals/internal-insights-layer.md +36 -36
  89. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  90. package/docs/internals/internal-measures-and-scales.md +35 -35
  91. package/docs/internals/internal-query-path.md +46 -46
  92. package/docs/internals/internal-storage-memory.md +13 -13
  93. package/docs/internals/internal-storage-mongo.md +44 -44
  94. package/docs/internals/internal-storage-postgres.md +32 -0
  95. package/docs/internals/internal-storage-redis.md +63 -63
  96. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  97. package/docs/use-cases/README.md +56 -56
  98. package/docs/use-cases/audit-security-log-counts.md +550 -550
  99. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  100. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  101. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  102. package/docs/use-cases/processing-workflow-activity.md +165 -165
  103. package/docs/use-cases/product-feature-usage.md +240 -240
  104. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  105. package/package.json +190 -170
@@ -1,34 +1,37 @@
1
- # Countex feature guides
2
-
3
- Step-by-step guides for using each major feature of `@x12i/countex`. Each document stands alone.
4
-
5
- These files are published with the npm package and exposed as stable import subpaths (see root `README.md` and `package.json` `exports`).
6
-
7
- | Guide | Package subpath |
8
- |-------|-----------------|
9
- | [Counter maps](./counter-maps.md) | `@x12i/countex/docs/guides/counter-maps` |
10
- | [Ingestion and flush](./ingestion-and-flush.md) | `@x12i/countex/docs/guides/ingestion-and-flush` |
11
- | [Timeframes](./timeframes-and-bucketing.md) | `@x12i/countex/docs/guides/timeframes-and-bucketing` |
12
- | [Measures](./measures.md) | `@x12i/countex/docs/guides/measures` |
13
- | [Querying](./querying.md) | `@x12i/countex/docs/guides/querying` |
14
- | [Memory adapters](./adapter-memory.md) | `@x12i/countex/docs/guides/adapter-memory` |
15
- | [Mongo durable](./adapter-mongo.md) | `@x12i/countex/docs/guides/adapter-mongo` |
16
- | [Redis cache](./adapter-redis.md) | `@x12i/countex/docs/guides/adapter-redis` |
17
- | [Cardinality](./cardinality.md) | `@x12i/countex/docs/guides/cardinality` |
18
- | [Idempotency](./idempotency.md) | `@x12i/countex/docs/guides/idempotency` |
19
- | [Backup export](./backup-export.md) | `@x12i/countex/docs/guides/backup-export` |
20
- | [Client events](./client-events.md) | `@x12i/countex/docs/guides/client-events` |
21
- | [Insights overview](./insights-overview.md) | `@x12i/countex/docs/guides/insights-overview` |
22
- | [Insights ranking](./insights-ranking.md) | `@x12i/countex/docs/guides/insights-ranking` |
23
- | [Insights comparison & trends](./insights-comparison-and-trends.md) | `@x12i/countex/docs/guides/insights-comparison-and-trends` |
24
- | [Insights anomalies & rules](./insights-anomalies-and-rules.md) | `@x12i/countex/docs/guides/insights-anomalies-and-rules` |
25
- | [Insights forecasting & budget](./insights-forecasting-and-budget.md) | `@x12i/countex/docs/guides/insights-forecasting-and-budget` |
26
- | [Insights advisors](./insights-advisors-and-optimization.md) | `@x12i/countex/docs/guides/insights-advisors-and-optimization` |
27
- | [CLI](./cli.md) | `@x12i/countex/docs/guides/cli` |
28
- | [Simulation overview](./simulation.md) | `@x12i/countex/docs/guides/simulation` |
29
- | [Simulation profiles](./simulation-profiles.md) | `@x12i/countex/docs/guides/simulation-profiles` |
30
- | [Simulation resource estimation](./simulation-resource-estimation.md) | `@x12i/countex/docs/guides/simulation-resource-estimation` |
31
-
32
- Scenario walkthroughs: [`docs/use-cases/`](../use-cases/README.md) (also exposed as `@x12i/countex/docs/use-cases/*`).
33
-
34
- Additional narrative docs (also under `docs/`): `DEPLOYMENT.md`, `END_TO_END_EXAMPLE.md`, `INSIGHTS*.md`.
1
+ # Countex feature guides
2
+
3
+ Step-by-step guides for using each major feature of `@x12i/countex`. Each document stands alone.
4
+
5
+ These files are published with the npm package and exposed as stable import subpaths (see root `README.md` and `package.json` `exports`).
6
+
7
+ | Guide | Package subpath |
8
+ |-------|-----------------|
9
+ | [Counter maps](./counter-maps.md) | `@x12i/countex/docs/guides/counter-maps` |
10
+ | [Ingestion and flush](./ingestion-and-flush.md) | `@x12i/countex/docs/guides/ingestion-and-flush` |
11
+ | [Timeframes](./timeframes-and-bucketing.md) | `@x12i/countex/docs/guides/timeframes-and-bucketing` |
12
+ | [Measures](./measures.md) | `@x12i/countex/docs/guides/measures` |
13
+ | [Querying](./querying.md) | `@x12i/countex/docs/guides/querying` |
14
+ | [Memory adapters](./adapter-memory.md) | `@x12i/countex/docs/guides/adapter-memory` |
15
+ | [Mongo durable](./adapter-mongo.md) | `@x12i/countex/docs/guides/adapter-mongo` |
16
+ | [Postgres durable](./adapter-postgres.md) | `@x12i/countex/docs/guides/adapter-postgres` |
17
+ | [Mongo to Postgres](./migrating-mongo-to-postgres.md) | `@x12i/countex/docs/guides/migrating-mongo-to-postgres` |
18
+ | [Redis cache](./adapter-redis.md) | `@x12i/countex/docs/guides/adapter-redis` |
19
+ | [Cardinality](./cardinality.md) | `@x12i/countex/docs/guides/cardinality` |
20
+ | [Idempotency](./idempotency.md) | `@x12i/countex/docs/guides/idempotency` |
21
+ | [Managed commit](./managed-commit.md) | `@x12i/countex/docs/guides/managed-commit` |
22
+ | [Backup export](./backup-export.md) | `@x12i/countex/docs/guides/backup-export` |
23
+ | [Client events](./client-events.md) | `@x12i/countex/docs/guides/client-events` |
24
+ | [Insights overview](./insights-overview.md) | `@x12i/countex/docs/guides/insights-overview` |
25
+ | [Insights ranking](./insights-ranking.md) | `@x12i/countex/docs/guides/insights-ranking` |
26
+ | [Insights comparison & trends](./insights-comparison-and-trends.md) | `@x12i/countex/docs/guides/insights-comparison-and-trends` |
27
+ | [Insights anomalies & rules](./insights-anomalies-and-rules.md) | `@x12i/countex/docs/guides/insights-anomalies-and-rules` |
28
+ | [Insights forecasting & budget](./insights-forecasting-and-budget.md) | `@x12i/countex/docs/guides/insights-forecasting-and-budget` |
29
+ | [Insights advisors](./insights-advisors-and-optimization.md) | `@x12i/countex/docs/guides/insights-advisors-and-optimization` |
30
+ | [CLI](./cli.md) | `@x12i/countex/docs/guides/cli` |
31
+ | [Simulation overview](./simulation.md) | `@x12i/countex/docs/guides/simulation` |
32
+ | [Simulation profiles](./simulation-profiles.md) | `@x12i/countex/docs/guides/simulation-profiles` |
33
+ | [Simulation resource estimation](./simulation-resource-estimation.md) | `@x12i/countex/docs/guides/simulation-resource-estimation` |
34
+
35
+ Scenario walkthroughs: [`docs/use-cases/`](../use-cases/README.md) (also exposed as `@x12i/countex/docs/use-cases/*`).
36
+
37
+ Additional narrative docs (also under `docs/`): `DEPLOYMENT.md`, `END_TO_END_EXAMPLE.md`, `INSIGHTS*.md`.
@@ -1,60 +1,60 @@
1
- # Guide: Memory adapters
2
-
3
- In-process **`MemoryDurable`** and **`MemoryCache`** implement the storage contracts for tests and local development.
4
-
5
- ## Step 1: Import
6
-
7
- ```ts
8
- import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory";
9
- ```
10
-
11
- ## Step 2: Construct Countex
12
-
13
- ```ts
14
- import { createCountex } from "@x12i/countex";
15
-
16
- const countex = createCountex({
17
- maps: [myMap],
18
- durable: new MemoryDurable(),
19
- cache: new MemoryCache(),
20
- });
21
- ```
22
-
23
- ## Step 3: Use like production
24
-
25
- ```ts
26
- await countex.ingest(event);
27
- await countex.flush();
28
- const q = await countex.query({ /* ... */ });
29
- ```
30
-
31
- No network, no peer dependencies.
32
-
33
- ## Mongo + `MemoryCache` (single process, no Redis)
34
-
35
- You can pair **`MongoDurable`** with **`MemoryCache`** so hot reads stay in RAM while the database remains the source of truth. That only works correctly when **one process** owns ingestion and queries for those scopes: the cache is not shared across workers, and idempotency markers are process-local.
36
-
37
- `await countex.init()` emits a **warning** when this combination is detected (see `Countex` lifecycle).
38
-
39
- ## `DurableReadCache` (no Redis; read from Mongo each query)
40
-
41
- For **low query volume** or relaxed latency, `@x12i/countex/adapters/durable-cache` provides **`DurableReadCache`**, which implements `CacheStorage` by delegating **`readCells`** to the same durable instance. There is no separate hot tier: every query hits durable storage once (no second fallback read in the planner). `applyDeltas` is a no-op. Optional idempotency still uses in-process `markEventSeen` unless you add a shared dedupe layer.
42
-
43
- `init()` logs a warning when this adapter is used.
44
-
45
- ## When to use
46
-
47
- - Unit tests
48
- - Demos and notebooks
49
- - CI without Docker services
50
-
51
- ## Limitations
52
-
53
- - State dies with the process.
54
- - Not suitable for multi-process concurrency testing at scale.
55
-
56
- ## See also
57
-
58
- - [Mongo adapter](./adapter-mongo.md)
59
- - [Redis adapter](./adapter-redis.md)
60
- - Internal: [Memory storage](../internals/internal-storage-memory.md)
1
+ # Guide: Memory adapters
2
+
3
+ In-process **`MemoryDurable`** and **`MemoryCache`** implement the storage contracts for tests and local development.
4
+
5
+ ## Step 1: Import
6
+
7
+ ```ts
8
+ import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory";
9
+ ```
10
+
11
+ ## Step 2: Construct Countex
12
+
13
+ ```ts
14
+ import { createCountex } from "@x12i/countex";
15
+
16
+ const countex = createCountex({
17
+ maps: [myMap],
18
+ durable: new MemoryDurable(),
19
+ cache: new MemoryCache(),
20
+ });
21
+ ```
22
+
23
+ ## Step 3: Use like production
24
+
25
+ ```ts
26
+ await countex.ingest(event);
27
+ await countex.flush();
28
+ const q = await countex.query({ /* ... */ });
29
+ ```
30
+
31
+ No network, no peer dependencies.
32
+
33
+ ## Mongo + `MemoryCache` (single process, no Redis)
34
+
35
+ You can pair **`MongoDurable`** with **`MemoryCache`** so hot reads stay in RAM while the database remains the source of truth. That only works correctly when **one process** owns ingestion and queries for those scopes: the cache is not shared across workers, and idempotency markers are process-local.
36
+
37
+ `await countex.init()` emits a **warning** when this combination is detected (see `Countex` lifecycle).
38
+
39
+ ## `DurableReadCache` (no Redis; read from Mongo each query)
40
+
41
+ For **low query volume** or relaxed latency, `@x12i/countex/adapters/durable-cache` provides **`DurableReadCache`**, which implements `CacheStorage` by delegating **`readCells`** to the same durable instance. There is no separate hot tier: every query hits durable storage once (no second fallback read in the planner). `applyDeltas` is a no-op. Optional idempotency still uses in-process `markEventSeen` unless you add a shared dedupe layer.
42
+
43
+ `init()` logs a warning when this adapter is used.
44
+
45
+ ## When to use
46
+
47
+ - Unit tests
48
+ - Demos and notebooks
49
+ - CI without Docker services
50
+
51
+ ## Limitations
52
+
53
+ - State dies with the process.
54
+ - Not suitable for multi-process concurrency testing at scale.
55
+
56
+ ## See also
57
+
58
+ - [Mongo adapter](./adapter-mongo.md)
59
+ - [Redis adapter](./adapter-redis.md)
60
+ - Internal: [Memory storage](../internals/internal-storage-memory.md)
@@ -1,62 +1,64 @@
1
- # Guide: MongoDB durable adapter
2
-
3
- `MongoDurable` persists cells and the subject/metric dictionary to MongoDB.
4
-
5
- ## Step 1: Install peer dependency
6
-
7
- ```bash
8
- npm install mongodb
9
- ```
10
-
11
- ## Step 2: Create a client (you own lifecycle)
12
-
13
- ```ts
14
- import { MongoClient } from "mongodb";
15
- import { MongoDurable } from "@x12i/countex/adapters/mongo";
16
- import { createCountex } from "@x12i/countex";
17
-
18
- const mongo = new MongoClient(process.env.MONGO_URI!);
19
- await mongo.connect();
20
-
21
- const durable = new MongoDurable({
22
- client: mongo,
23
- dbResolver: ({ scope }) => `countex_${scope}`,
24
- });
25
-
26
- const countex = createCountex({
27
- maps: [myMap],
28
- durable,
29
- cache: /* optional Redis */,
30
- });
31
-
32
- await countex.init();
33
- ```
34
-
35
- ## Step 3: Collections and indexes
36
-
37
- On first write the adapter ensures collections such as **`countex_cells`** and **`countex_dict`**, plus indexes for identity and range scans. Per-scope databases isolate tenant data when using `dbResolver`.
38
-
39
- ## Step 4: Flush semantics
40
-
41
- Flushes call `applyDeltas` with atomic increment operations so parallel writers can safely aggregate.
42
-
43
- ## Dictionary allocation (`resolveIds`) and concurrency
44
-
45
- Dictionary rows map **`(scopeId, kind, qualifier, sourceKey)` → `internalId`** (integer). The adapter:
46
-
47
- 1. **Reads** existing rows for the batch.
48
- 2. **Allocates** a contiguous block of ids from a per-scope sequence document (`countex_seq`).
49
- 3. **Inserts** new rows with `insertMany({ ordered: false })`.
50
-
51
- A **unique index** (`uniq_dict_key` on `scopeId`, `kind`, `qualifier`, `sourceKey`) prevents duplicate keys. If two workers race on the same new key, one insert wins and the other hits a duplicate-key error; the adapter **re-reads** and continues. Always create indexes via `init()` / first write (unless `skipIndexInit`).
52
-
53
- Sequence allocation uses MongoDB **`findOneAndUpdate` `$inc`**, which is atomic per sequence document. Very large parallel bursts should still be safe; id blocks may interleave across workers but remain unique.
54
-
55
- ## Step 5: Shutdown
56
-
57
- Close Countex, then close the Mongo client when your process exits.
58
-
59
- ## See also
60
-
61
- - [Deployment](../DEPLOYMENT.md)
62
- - Internal: [Mongo durable](../internals/internal-storage-mongo.md)
1
+ # Guide: MongoDB durable adapter
2
+
3
+ `MongoDurable` persists cells and the subject/metric dictionary to MongoDB.
4
+
5
+ ## Step 1: Install peer dependency
6
+
7
+ ```bash
8
+ npm install mongodb
9
+ ```
10
+
11
+ ## Step 2: Create a client (you own lifecycle)
12
+
13
+ ```ts
14
+ import { MongoClient } from "mongodb";
15
+ import { MongoDurable } from "@x12i/countex/adapters/mongo";
16
+ import { createCountex } from "@x12i/countex";
17
+
18
+ const mongo = new MongoClient(process.env.MONGO_URI!);
19
+ await mongo.connect();
20
+
21
+ const durable = new MongoDurable({
22
+ client: mongo,
23
+ dbResolver: ({ scope }) => `countex_${scope}`,
24
+ });
25
+
26
+ const countex = createCountex({
27
+ maps: [myMap],
28
+ durable,
29
+ cache: /* optional Redis */,
30
+ });
31
+
32
+ await countex.init();
33
+ ```
34
+
35
+ ## Step 3: Collections and indexes
36
+
37
+ On first write the adapter ensures collections such as **`countex_cells`** and **`countex_dict`**, plus indexes for identity and range scans. Per-scope databases isolate tenant data when using `dbResolver`.
38
+
39
+ ## Step 4: Flush semantics
40
+
41
+ Flushes call `applyDeltas` with atomic increment operations so parallel writers can safely aggregate.
42
+
43
+ ## Dictionary allocation (`resolveIds`) and concurrency
44
+
45
+ Dictionary rows map **`(scopeId, kind, qualifier, sourceKey)` → `internalId`** (integer). The adapter:
46
+
47
+ 1. **Reads** existing rows for the batch.
48
+ 2. **Allocates** a contiguous block of ids from a per-scope sequence document (`countex_seq`).
49
+ 3. **Inserts** new rows with `insertMany({ ordered: false })`.
50
+
51
+ A **unique index** (`uniq_dict_key` on `scopeId`, `kind`, `qualifier`, `sourceKey`) prevents duplicate keys. If two workers race on the same new key, one insert wins and the other hits a duplicate-key error; the adapter **re-reads** and continues. Always create indexes via `init()` / first write (unless `skipIndexInit`).
52
+
53
+ Sequence allocation uses MongoDB **`findOneAndUpdate` `$inc`**, which is atomic per sequence document. Very large parallel bursts should still be safe; id blocks may interleave across workers but remain unique.
54
+
55
+ ## Step 5: Shutdown
56
+
57
+ Close Countex, then close the Mongo client when your process exits.
58
+
59
+ ## See also
60
+
61
+ - [PostgreSQL adapter](./adapter-postgres.md) — same contract, different durable engine
62
+ - [Migrating MongoDB to PostgreSQL](./migrating-mongo-to-postgres.md)
63
+ - [Deployment](../DEPLOYMENT.md)
64
+ - Internal: [Mongo durable](../internals/internal-storage-mongo.md)
@@ -0,0 +1,92 @@
1
+ # Guide: PostgreSQL durable adapter
2
+
3
+ `PostgresDurable` persists cells and the subject/metric dictionary to PostgreSQL. It implements the same `DurableStorage` contract as `MongoDurable`. Dashboards still read Redis when the cache covers the query. Postgres replaces the durable store, not the hot cache.
4
+
5
+ The managed commit profile (`commit` / `commitBatch`) uses this adapter. Event identity and cell effects commit in one transaction. See [Managed commit](./managed-commit.md).
6
+
7
+ Use PostgreSQL 15 or newer. Unique constraints on the partitioned `cells` table need that version. Put PgBouncer in transaction mode in front of many Node workers. Each `pg` connection costs several megabytes. A pool per worker without a pooler spends the memory you saved by leaving Mongo.
8
+
9
+ ## Step 1: Install the peer dependency
10
+
11
+ ```bash
12
+ npm install pg
13
+ ```
14
+
15
+ ## Step 2: Create a pool (you own the lifecycle)
16
+
17
+ ```ts
18
+ import { Pool } from "pg";
19
+ import { PostgresDurable } from "@x12i/countex/adapters/postgres";
20
+ import { createCountex } from "@x12i/countex";
21
+
22
+ const pool = new Pool({ connectionString: process.env.POSTGRES_URL });
23
+
24
+ const durable = new PostgresDurable({
25
+ pool,
26
+ schema: "countex",
27
+ });
28
+
29
+ const countex = createCountex({
30
+ maps: [myMap],
31
+ durable,
32
+ cache: /* optional Redis */,
33
+ });
34
+
35
+ await countex.init();
36
+ ```
37
+
38
+ `PostgresDurable` does not call `pool.end()`. Close Countex, then end the pool when the process exits.
39
+
40
+ ## Step 3: Schema
41
+
42
+ `init()` creates one schema (default `countex`) and these relations:
43
+
44
+ | Relation | Role |
45
+ |----------|------|
46
+ | `cells` | Counter rows. List-partitioned: `minute`, `hour`, and a default partition for day, week, month, quarter, year, and `all`. |
47
+ | `dict` | `(scope_id, kind, qualifier, source_key) → internal_id` |
48
+ | `seq` | Per-scope id block, same gap-tolerant allocation as Mongo `countex_seq` |
49
+ | `evidence` | Optional exemplar rows, trimmed to `maxSamplesAtCapture` |
50
+
51
+ `scope_id` is a column on every row. Do not create a schema per tenant. That bloats the catalog. `schemaResolver` exists for hosts that already isolate tenants that way. Prefer one schema.
52
+
53
+ Cell identity is a unique constraint on `(counter_id, scope_id, timeframe_type, timeframe_key, subject_type, subject_id, metric_id, dimensions_hash)`. Increments do not change those columns, so updates can be HOT. Partitions are created with `fillfactor = 85` and `autovacuum_vacuum_scale_factor = 0.05` so dead tuples from counter updates do not grow the table back to Mongo size. Keep autovacuum on.
54
+
55
+ Minute and hour volume is dropped by detaching or dropping that partition, or by deleting a `timeframe_key` range. `retention.postgres` on `createCountex` documents the window. The engine does not enforce it, same as `retention.mongo`.
56
+
57
+ Set `skipIndexInit` only when an external migration creates this schema, including `merge_measures(jsonb, jsonb, jsonb, boolean)`.
58
+
59
+ ## Step 4: Flush semantics
60
+
61
+ `applyDeltas` collapses duplicate identities in the batch, sorts them so concurrent workers lock rows in the same order, and runs one `INSERT … ON CONFLICT DO UPDATE` per chunk (default 1000 rows). The statement adds `count` and merges measures inside the conflict update:
62
+
63
+ - `sum` and `count` add
64
+ - `min` takes the lesser value, `max` the greater
65
+ - `scale` is stored on insert and left alone on later increments
66
+
67
+ That matches Mongo `$inc`, `$min`, `$max`, and `$setOnInsert`. Doing the math in Node and then writing the row would lose increments under concurrent flushes.
68
+
69
+ The whole batch is one transaction. A thrown error rolls back, and the engine can retry the same snapshot without double-counting.
70
+
71
+ `importSnapshotCells` replaces `count` and measures. It is the restore path, not the ingest path.
72
+
73
+ ## Dictionary allocation (`resolveIds`) and concurrency
74
+
75
+ Same rule as Mongo: one internal id per `(scope, kind, qualifier, sourceKey)`.
76
+
77
+ 1. Read existing dict rows for the batch.
78
+ 2. Allocate a contiguous block with `INSERT … ON CONFLICT DO UPDATE` on `seq`, returning the new high-water mark.
79
+ 3. Insert new rows with `ON CONFLICT DO NOTHING`.
80
+ 4. Re-read. If another worker won the race, their id is the one that sticks. Unused ids from the losing block are gaps, not duplicates.
81
+
82
+ ## Step 5: What this does not change
83
+
84
+ Redis key layout, coverage, and idempotency (`markEventSeen`) stay on the cache adapter. Query `groupBy` still runs in the engine after `readCells` returns cells. A warm dashboard does not get faster because the durable engine changed. Ingest throughput stays in the same band as a healthy Mongo `$inc` batch. The gain is a smaller row, cheaper retention, and one less database process when Mongo existed only for Countex.
85
+
86
+ A packed cell is estimated at **250 bytes** in `estimateResourcesFromActuals` (`postgresBytesPerCell`, default 250, next to the 750-byte Mongo estimate). Profile your own dimension payloads before you size disks.
87
+
88
+ ## See also
89
+
90
+ - [Migrating MongoDB to PostgreSQL](./migrating-mongo-to-postgres.md)
91
+ - [Deployment](../DEPLOYMENT.md)
92
+ - Internal: [Postgres durable](../internals/internal-storage-postgres.md)
@@ -1,89 +1,89 @@
1
- # Guide: Redis cache adapter
2
-
3
- `RedisCache` stores hot counter cells for fast reads. It pairs with a durable store (typically Mongo) and tracks per-bucket coverage so the query planner only treats fully-warmed buckets as cache-authoritative.
4
-
5
- ## Step 1: Install peer dependency
6
-
7
- ```bash
8
- npm install ioredis
9
- ```
10
-
11
- ## Step 2: Wire client and adapter
12
-
13
- ```ts
14
- import IORedis from "ioredis";
15
- import { RedisCache } from "@x12i/countex/adapters/redis";
16
-
17
- const redis = new IORedis(process.env.REDIS_URL!);
18
-
19
- const cache = new RedisCache({
20
- client: redis,
21
- keyPrefix: "countex",
22
- ttlSeconds: 92 * 24 * 3600,
23
- // Optional: shorter hot buckets, longer rollups
24
- ttlSecondsByTimeframe: {
25
- minute: 48 * 3600,
26
- hour: 14 * 24 * 3600,
27
- day: 92 * 24 * 3600,
28
- },
29
- // Optional opaque map version stamped onto bucket coverage on warm()
30
- mapVersion: 1,
31
- });
32
-
33
- const countex = createCountex({
34
- maps: [myMap],
35
- durable,
36
- cache,
37
- });
38
-
39
- await countex.init();
40
- ```
41
-
42
- ## Step 3: Unified retention via `CountexConfig.retention`
43
-
44
- Prefer the engine-level `retention` config so the same policy is enforced regardless of which cache adapter is used:
45
-
46
- ```ts
47
- const countex = createCountex({
48
- maps: [myMap],
49
- durable,
50
- cache,
51
- retention: {
52
- redis: {
53
- minute: "2d",
54
- hour: "30d",
55
- day: "92d",
56
- week: "52w",
57
- month: "24mo",
58
- quarter: "12q",
59
- year: "7y",
60
- all: "forever",
61
- },
62
- // Mongo retention is operator-managed today (TTL index or scheduled prune).
63
- },
64
- });
65
- ```
66
-
67
- The engine forwards the resolved per-timeframe TTLs to `cache.setRetentionPolicy(...)` at construction. Engine policy overrides the adapter's own `ttlSecondsByTimeframe`.
68
-
69
- Accepted duration strings: `<n><unit>` where unit is `s|m|h|d|w|mo|q|y`, or `"forever"` for no expiry. Plain numbers are seconds.
70
-
71
- ## Step 4: Operations
72
-
73
- - **Production Redis must use `maxmemory-policy noeviction`.** Silent eviction breaks the per-bucket "complete" coverage assumption — a half-evicted bucket can no longer be served from cache, and silent eviction means the `:cov` key may report `complete` while the cells hash is gone.
74
- - Cache misses fall back to durable reads; results may warm the cache afterward.
75
- - Cache flush failures during write path are **non-fatal** (durable is source of truth).
76
-
77
- ## Step 5: Idempotency hook
78
-
79
- When maps declare **idempotency**, the cache adapter implements **`markEventSeen`** so duplicates are skipped before buffering.
80
-
81
- ## Step 6: Per-bucket coverage
82
-
83
- `RedisCache` maintains a `:cov` HASH per bucket with `status` (`complete | partial | rebuilding`), `lastWarmAt`, `cellCount`, `schemaVersion`, and optional `mapVersion`. The query planner reads it to know which buckets the cache can answer with confidence; details in [internals/internal-storage-redis.md](../internals/internal-storage-redis.md).
84
-
85
- ## See also
86
-
87
- - [Idempotency](./idempotency.md)
88
- - [Querying](./querying.md) (cache vs durable for min/max measures)
89
- - Internal: [Redis cache](../internals/internal-storage-redis.md)
1
+ # Guide: Redis cache adapter
2
+
3
+ `RedisCache` stores hot counter cells for fast reads. It pairs with a durable store (typically Mongo) and tracks per-bucket coverage so the query planner only treats fully-warmed buckets as cache-authoritative.
4
+
5
+ ## Step 1: Install peer dependency
6
+
7
+ ```bash
8
+ npm install ioredis
9
+ ```
10
+
11
+ ## Step 2: Wire client and adapter
12
+
13
+ ```ts
14
+ import IORedis from "ioredis";
15
+ import { RedisCache } from "@x12i/countex/adapters/redis";
16
+
17
+ const redis = new IORedis(process.env.REDIS_URL!);
18
+
19
+ const cache = new RedisCache({
20
+ client: redis,
21
+ keyPrefix: "countex",
22
+ ttlSeconds: 92 * 24 * 3600,
23
+ // Optional: shorter hot buckets, longer rollups
24
+ ttlSecondsByTimeframe: {
25
+ minute: 48 * 3600,
26
+ hour: 14 * 24 * 3600,
27
+ day: 92 * 24 * 3600,
28
+ },
29
+ // Optional opaque map version stamped onto bucket coverage on warm()
30
+ mapVersion: 1,
31
+ });
32
+
33
+ const countex = createCountex({
34
+ maps: [myMap],
35
+ durable,
36
+ cache,
37
+ });
38
+
39
+ await countex.init();
40
+ ```
41
+
42
+ ## Step 3: Unified retention via `CountexConfig.retention`
43
+
44
+ Prefer the engine-level `retention` config so the same policy is enforced regardless of which cache adapter is used:
45
+
46
+ ```ts
47
+ const countex = createCountex({
48
+ maps: [myMap],
49
+ durable,
50
+ cache,
51
+ retention: {
52
+ redis: {
53
+ minute: "2d",
54
+ hour: "30d",
55
+ day: "92d",
56
+ week: "52w",
57
+ month: "24mo",
58
+ quarter: "12q",
59
+ year: "7y",
60
+ all: "forever",
61
+ },
62
+ // Mongo retention is operator-managed today (TTL index or scheduled prune).
63
+ },
64
+ });
65
+ ```
66
+
67
+ The engine forwards the resolved per-timeframe TTLs to `cache.setRetentionPolicy(...)` at construction. Engine policy overrides the adapter's own `ttlSecondsByTimeframe`.
68
+
69
+ Accepted duration strings: `<n><unit>` where unit is `s|m|h|d|w|mo|q|y`, or `"forever"` for no expiry. Plain numbers are seconds.
70
+
71
+ ## Step 4: Operations
72
+
73
+ - **Production Redis must use `maxmemory-policy noeviction`.** Silent eviction breaks the per-bucket "complete" coverage assumption — a half-evicted bucket can no longer be served from cache, and silent eviction means the `:cov` key may report `complete` while the cells hash is gone.
74
+ - Cache misses fall back to durable reads; results may warm the cache afterward.
75
+ - Cache flush failures during write path are **non-fatal** (durable is source of truth).
76
+
77
+ ## Step 5: Idempotency hook
78
+
79
+ When maps declare **idempotency**, the cache adapter implements **`markEventSeen`** so duplicates are skipped before buffering.
80
+
81
+ ## Step 6: Per-bucket coverage
82
+
83
+ `RedisCache` maintains a `:cov` HASH per bucket with `status` (`complete | partial | rebuilding`), `lastWarmAt`, `cellCount`, `schemaVersion`, and optional `mapVersion`. The query planner reads it to know which buckets the cache can answer with confidence; details in [internals/internal-storage-redis.md](../internals/internal-storage-redis.md).
84
+
85
+ ## See also
86
+
87
+ - [Idempotency](./idempotency.md)
88
+ - [Querying](./querying.md) (cache vs durable for min/max measures)
89
+ - Internal: [Redis cache](../internals/internal-storage-redis.md)