@x12i/countex 1.8.5 → 1.10.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (105) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +867 -860
  3. package/dist/cli.js +6 -6
  4. package/dist/core/countex.d.ts +61 -1
  5. package/dist/core/countex.d.ts.map +1 -1
  6. package/dist/core/countex.js +241 -0
  7. package/dist/core/countex.js.map +1 -1
  8. package/dist/core/managed.d.ts +200 -0
  9. package/dist/core/managed.d.ts.map +1 -0
  10. package/dist/core/managed.js +431 -0
  11. package/dist/core/managed.js.map +1 -0
  12. package/dist/core/timeframe.d.ts +5 -0
  13. package/dist/core/timeframe.d.ts.map +1 -1
  14. package/dist/core/timeframe.js +36 -0
  15. package/dist/core/timeframe.js.map +1 -1
  16. package/dist/index.d.ts +6 -2
  17. package/dist/index.d.ts.map +1 -1
  18. package/dist/index.js +9 -1
  19. package/dist/index.js.map +1 -1
  20. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  21. package/dist/insights/advisors/resource-advisor.js +5 -1
  22. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  23. package/dist/insights/types.d.ts +4 -0
  24. package/dist/insights/types.d.ts.map +1 -1
  25. package/dist/storage/managed-memory.d.ts +33 -0
  26. package/dist/storage/managed-memory.d.ts.map +1 -0
  27. package/dist/storage/managed-memory.js +211 -0
  28. package/dist/storage/managed-memory.js.map +1 -0
  29. package/dist/storage/memory.d.ts +7 -4
  30. package/dist/storage/memory.d.ts.map +1 -1
  31. package/dist/storage/memory.js.map +1 -1
  32. package/dist/storage/postgres.d.ts +123 -0
  33. package/dist/storage/postgres.d.ts.map +1 -0
  34. package/dist/storage/postgres.js +1325 -0
  35. package/dist/storage/postgres.js.map +1 -0
  36. package/dist/storage/redis.d.ts +6 -0
  37. package/dist/storage/redis.d.ts.map +1 -1
  38. package/dist/storage/redis.js +21 -0
  39. package/dist/storage/redis.js.map +1 -1
  40. package/dist/types/index.d.ts +63 -5
  41. package/dist/types/index.d.ts.map +1 -1
  42. package/dist/utils/errors.d.ts +12 -0
  43. package/dist/utils/errors.d.ts.map +1 -1
  44. package/dist/utils/errors.js +22 -1
  45. package/dist/utils/errors.js.map +1 -1
  46. package/docs/DEPLOYMENT.md +399 -375
  47. package/docs/END_TO_END_EXAMPLE.md +316 -316
  48. package/docs/INSIGHTS.md +59 -59
  49. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  50. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  51. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  52. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  53. package/docs/guides/README.md +37 -34
  54. package/docs/guides/adapter-memory.md +60 -60
  55. package/docs/guides/adapter-mongo.md +64 -62
  56. package/docs/guides/adapter-postgres.md +92 -0
  57. package/docs/guides/adapter-redis.md +89 -89
  58. package/docs/guides/backup-export.md +135 -135
  59. package/docs/guides/cardinality.md +50 -50
  60. package/docs/guides/cli.md +53 -53
  61. package/docs/guides/client-events.md +65 -65
  62. package/docs/guides/counter-maps.md +160 -160
  63. package/docs/guides/idempotency.md +39 -39
  64. package/docs/guides/ingestion-and-flush.md +67 -67
  65. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  66. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  67. package/docs/guides/insights-comparison-and-trends.md +47 -47
  68. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  69. package/docs/guides/insights-overview.md +82 -82
  70. package/docs/guides/insights-ranking.md +81 -81
  71. package/docs/guides/managed-commit.md +119 -0
  72. package/docs/guides/measures.md +72 -71
  73. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  74. package/docs/guides/preview.md +92 -92
  75. package/docs/guides/querying.md +178 -178
  76. package/docs/guides/simulation-profiles.md +33 -33
  77. package/docs/guides/simulation-resource-estimation.md +31 -30
  78. package/docs/guides/simulation.md +36 -36
  79. package/docs/guides/timeframes-and-bucketing.md +55 -55
  80. package/docs/internals/README.md +21 -21
  81. package/docs/internals/internal-architecture-overview.md +65 -65
  82. package/docs/internals/internal-backup-stream.md +23 -23
  83. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  84. package/docs/internals/internal-cardinality.md +30 -30
  85. package/docs/internals/internal-dimension-hash.md +34 -34
  86. package/docs/internals/internal-idempotency.md +36 -36
  87. package/docs/internals/internal-ingest-pipeline.md +52 -52
  88. package/docs/internals/internal-insights-layer.md +36 -36
  89. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  90. package/docs/internals/internal-measures-and-scales.md +35 -35
  91. package/docs/internals/internal-query-path.md +46 -46
  92. package/docs/internals/internal-storage-memory.md +13 -13
  93. package/docs/internals/internal-storage-mongo.md +44 -44
  94. package/docs/internals/internal-storage-postgres.md +32 -0
  95. package/docs/internals/internal-storage-redis.md +63 -63
  96. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  97. package/docs/use-cases/README.md +56 -56
  98. package/docs/use-cases/audit-security-log-counts.md +550 -550
  99. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  100. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  101. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  102. package/docs/use-cases/processing-workflow-activity.md +165 -165
  103. package/docs/use-cases/product-feature-usage.md +240 -240
  104. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  105. package/package.json +190 -170
package/README.md CHANGED
@@ -1,860 +1,867 @@
1
- # @x12i/countex
2
-
3
- > Generic sparse counter matrices for JSON events.
4
- > Dynamic counting, aggregation, and retrieval from mapped JSON inputs.
5
-
6
- `@x12i/countex` is a **generic mapped-event counter**: a source-agnostic counting engine. You give it any JSON event, a declarative mapping config, and storage adapters; it projects events into sparse counter cells, batches deltas locally, and persists them atomically to a durable store (MongoDB) and a hot cache (Redis).
7
-
8
- **Package policy:** Countex is generic. It never references a specific host product, client name, schema, or storage layout. Host-specific behavior belongs in the host application, not in this package.
9
-
10
- It does **not** know about users, organizations, documents, labels, or events. It only knows generic primitives: **scope, subject, metric, dimensions, timeframe, delta**. Whatever your domain calls those things — Countex doesn't care.
11
-
12
- **Terminology:** `metric` is the generic Countex name for the thing being counted or categorized. In your domain that may be a label, status, event name, operation, outcome, class, model result, or business category — the `metrics` array in a map simply lists JSON paths that resolve to those categories. Storage and APIs refer to dictionary **`sourceKey`** (the string extracted from the event); avoid confusing that with application-level UUIDs unless your map uses UUIDs as keys.
13
-
14
- **Process memory vs retained state:** Each worker keeps **unflushed deltas** in an in-memory buffer. After flush, authoritative aggregates live in **durable** storage (MongoDB); **Redis** holds a hot cache. Restarting a worker loses only buffered deltas that were not flushed yet — tune flush intervals accordingly.
15
-
16
- ```
17
- event JSON ──► map ──► raw increments ──► buffer ──► flush ──► Mongo (durable)
18
- └───► Redis (hot cache)
19
-
20
- ◄── query ◄── Redis (full coverage) or ◄── Mongo (fallback) ──◄
21
- ```
22
-
23
- ---
24
-
25
- ## Install
26
-
27
- ```bash
28
- npm install @x12i/countex
29
- # Bring your own clients — peer deps, both optional:
30
- npm install mongodb ioredis
31
- ```
32
-
33
- Node 18+. Zero runtime dependencies in the core package.
34
-
35
- ### Documentation shipped with the package
36
-
37
- These guides are included in the published tarball under `docs/` and exposed as stable import subpaths (resolve to `.md` files for editors and tooling):
38
-
39
- | Topic | Package subpath | File |
40
- |-------|-----------------|------|
41
- | Release & production deployment | `@x12i/countex/docs/deployment` | `docs/DEPLOYMENT.md` |
42
- | Tutorial — install → map → ingest → query | `@x12i/countex/docs/end-to-end-example` | `docs/END_TO_END_EXAMPLE.md` |
43
- | Feature guides index | `@x12i/countex/docs/guides` | `docs/guides/README.md` |
44
- | Counter maps | `@x12i/countex/docs/guides/counter-maps` | `docs/guides/counter-maps.md` |
45
- | Preview a map (dry-run) | `@x12i/countex/docs/guides/preview` | `docs/guides/preview.md` |
46
- | Ingestion & flush | `@x12i/countex/docs/guides/ingestion-and-flush` | `docs/guides/ingestion-and-flush.md` |
47
- | Timeframes | `@x12i/countex/docs/guides/timeframes-and-bucketing` | `docs/guides/timeframes-and-bucketing.md` |
48
- | Measures | `@x12i/countex/docs/guides/measures` | `docs/guides/measures.md` |
49
- | Querying | `@x12i/countex/docs/guides/querying` | `docs/guides/querying.md` |
50
- | Memory adapters | `@x12i/countex/docs/guides/adapter-memory` | `docs/guides/adapter-memory.md` |
51
- | Mongo durable | `@x12i/countex/docs/guides/adapter-mongo` | `docs/guides/adapter-mongo.md` |
52
- | Redis cache | `@x12i/countex/docs/guides/adapter-redis` | `docs/guides/adapter-redis.md` |
53
- | Cardinality | `@x12i/countex/docs/guides/cardinality` | `docs/guides/cardinality.md` |
54
- | Idempotency | `@x12i/countex/docs/guides/idempotency` | `docs/guides/idempotency.md` |
55
- | Backup export | `@x12i/countex/docs/guides/backup-export` | `docs/guides/backup-export.md` |
56
- | Client events | `@x12i/countex/docs/guides/client-events` | `docs/guides/client-events.md` |
57
- | Insights overview | `@x12i/countex/docs/guides/insights-overview` | `docs/guides/insights-overview.md` |
58
- | Insights ranking | `@x12i/countex/docs/guides/insights-ranking` | `docs/guides/insights-ranking.md` |
59
- | Insights comparison & trends | `@x12i/countex/docs/guides/insights-comparison-and-trends` | `docs/guides/insights-comparison-and-trends.md` |
60
- | Insights anomalies & rules | `@x12i/countex/docs/guides/insights-anomalies-and-rules` | `docs/guides/insights-anomalies-and-rules.md` |
61
- | Insights forecasting & budget | `@x12i/countex/docs/guides/insights-forecasting-and-budget` | `docs/guides/insights-forecasting-and-budget.md` |
62
- | Insights advisors | `@x12i/countex/docs/guides/insights-advisors-and-optimization` | `docs/guides/insights-advisors-and-optimization.md` |
63
- | CLI | `@x12i/countex/docs/guides/cli` | `docs/guides/cli.md` |
64
- | Simulation runtime (JS API) | `@x12i/countex/simulation` | `dist/simulation/` |
65
- | Simulation overview | `@x12i/countex/docs/guides/simulation` | `docs/guides/simulation.md` |
66
- | Simulation profiles | `@x12i/countex/docs/guides/simulation-profiles` | `docs/guides/simulation-profiles.md` |
67
- | Simulation resource estimation | `@x12i/countex/docs/guides/simulation-resource-estimation` | `docs/guides/simulation-resource-estimation.md` |
68
-
69
- **Use cases** (technical playbooks: contracts, fan-out math, exact `QuerySpec`s, production wiring, verification — see [`docs/use-cases/README.md`](docs/use-cases/README.md)):
70
-
71
- | Topic | Package subpath | File |
72
- |-------|-----------------|------|
73
- | Use cases index | `@x12i/countex/docs/use-cases` | `docs/use-cases/README.md` |
74
- | Audit & security logs (volume, situations, distinct entities, logical event totals) | `@x12i/countex/docs/use-cases/audit-security-log-counts` | `docs/use-cases/audit-security-log-counts.md` |
75
- | Product & feature activity | `@x12i/countex/docs/use-cases/product-feature-usage` | `docs/use-cases/product-feature-usage.md` |
76
- | LLM cost & token metering | `@x12i/countex/docs/use-cases/llm-cost-and-token-metering` | `docs/use-cases/llm-cost-and-token-metering.md` |
77
- | Multi-tenant isolation | `@x12i/countex/docs/use-cases/multi-tenant-isolation` | `docs/use-cases/multi-tenant-isolation.md` |
78
- | Idempotent webhooks & queues | `@x12i/countex/docs/use-cases/webhook-idempotent-ingestion` | `docs/use-cases/webhook-idempotent-ingestion.md` |
79
- | Ops dashboards & rollups | `@x12i/countex/docs/use-cases/ops-dashboard-daily-rollups` | `docs/use-cases/ops-dashboard-daily-rollups.md` |
80
- | Processing & workflow activity | `@x12i/countex/docs/use-cases/processing-workflow-activity` | `docs/use-cases/processing-workflow-activity.md` |
81
-
82
- **Internals** (how the engine works — diagrams and traces) live under `docs/internals/` in the tarball; browse [`docs/internals/README.md`](docs/internals/README.md).
83
-
84
- After install: `node_modules/@x12i/countex/docs/`.
85
-
86
- ---
87
-
88
- ## The mental model
89
-
90
- Countex turns mapped JSON events into sparse aggregate cells.
91
-
92
- A counter cell is uniquely identified by:
93
-
94
- ```
95
- (scope, counterId, timeframeType, timeframeKey, subjectType, subjectId, metricId, dimensionsHash)
96
- ```
97
-
98
- Each event can produce one or many cell increments. A counter increment is defined by:
99
-
100
- - **scope** — your tenant / org / account boundary
101
- - **counterId** — which counter family this is (e.g. `access_by_subject_metric`)
102
- - **timeframe** — `minute` / `hour` / `day` / `week` / `month` / `quarter` / `year` / `all`
103
- - **subject** — the thing being counted against (e.g. an actor or a target)
104
- - **metric** — the count category / label
105
- - **dimensions** — arbitrary additional breakdowns
106
- - **delta** — `+1`, `-1`, or any integer
107
-
108
- Countex buffers increments locally, flushes atomic deltas to durable + cache, and serves queries from cache with durable fallback.
109
-
110
- ### Measures and rates
111
-
112
- Maps may declare optional **`measures`**: named numeric fields per event (cost, tokens, latency, …) with aggregations `sum`, `count`, `min`, and `max`. **`avg`** is computed at query time when both `sum` and `count` exist on the measure. Use **`scale`** on a measure for fixed-point storage (e.g. cents). Extracted values attach to each fan-out increment the same way as **`delta`** (see RFC-style docs for fan-out semantics).
113
-
114
- Queries accept **`measures`** (explicit list or `"*"`), **`orderBy`** on a measure aggregation, and optional **`rate`** (count or a measure’s sum divided by elapsed time in the queried window; for `format: "timeseries"`, per bucket). **`rate`** is invalid for timeframe **`all`**.
115
-
116
- **Redis cache** stores **sum** and **count** per measure only; **min** / **max** are durable-backed. If a query needs min/max (including `orderBy` on them, or requesting measures that maintain min/max when using `"*"`), the planner reads from **durable** first so cache cannot drift.
117
-
118
- **`bucketRange`** caps at **10,000** buckets; at **`minute`** resolution that is about **6.94 days** — keep queries within that bound or narrow the range.
119
-
120
- ### Percentiles & histograms
121
-
122
- Core cells store **additive** measure aggregates (`sum / count / min / max`) per sparse cell. They do **not** store distributions, so percentile-style answers (p50/p90/p95/p99) can only come from min/max approximation or from histogram/sketch storage you add yourself. Insights advisors that mention percentile-shaped recommendations are heuristic; see [`docs/guides/insights-overview.md`](docs/guides/insights-overview.md) §"Distributions" and [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) "Architecture decisions".
123
-
124
- ---
125
-
126
- ## What kinds of events Countex is for
127
-
128
- Countex is not tied to one event source or domain. It is a **generic mapped-event counter**: any JSON-like input stream where useful aggregates can be produced from paths in the event.
129
-
130
- Typical sources include:
131
-
132
- | Source family | Examples | Common Countex projection |
133
- |---|---|---|
134
- | Audit and security logs | Microsoft 365 audit logs, Google Workspace logs, Okta logs, cloud audit logs | actor/resource/action/situation/severity |
135
- | Product and user activity | Woopra, Segment, PostHog-style events, application activity logs | user/account/feature/event/page/funnel |
136
- | Processing and workflow activity | ETL runs, background jobs, agent graphs, workflow engines, queue processors | job/task/node/status/error/runtime |
137
- | LLM and AI service usage | Model gateway logs, OpenAI/Anthropic calls, internal LLM services | app/user/model/provider/tokens/cost/latency/result |
138
- | Business/domain events | CRM, support tickets, payments, commerce, lifecycle events | customer/object/state/category/outcome |
139
-
140
- Countex only needs a map. The map tells Countex where to find:
141
-
142
- - the scope boundary
143
- - the event time
144
- - the subjects
145
- - the metrics
146
- - the dimensions
147
- - the delta/count
148
- - optional numeric measures such as cost, tokens, latency, or duration
149
-
150
- The original event can come from a log, webhook, queue message, analytics event, system activity, processing trace, or generated test fixture. Countex does not care about the event source as long as the map can project it into generic counter primitives.
151
-
152
- ### Audit-log style event
153
-
154
- ```json
155
- {
156
- "tenant": { "id": "org_123" },
157
- "event": {
158
- "id": "audit_001",
159
- "createdAt": "2026-05-08T14:33:00Z",
160
- "operation": "FileAccessed",
161
- "situation": "external"
162
- },
163
- "actor": { "id": "user_42", "type": "user" },
164
- "resource": { "id": "file_99", "type": "document" },
165
- "labels": ["PII", "Financial"]
166
- }
167
- ```
168
-
169
- Counts:
170
-
171
- ```text
172
- actor/resource × label × operation/situation/resourceType × timeframe
173
- ```
174
-
175
- ### Product-activity style event
176
-
177
- ```json
178
- {
179
- "account": { "id": "acct_123" },
180
- "user": { "id": "user_42" },
181
- "event": {
182
- "name": "dashboard_viewed",
183
- "createdAt": "2026-05-08T14:33:00Z"
184
- },
185
- "product": {
186
- "feature": "risk_dashboard",
187
- "plan": "enterprise"
188
- }
189
- }
190
- ```
191
-
192
- Counts:
193
-
194
- ```text
195
- user/account × activity metric × feature/plan × timeframe
196
- ```
197
-
198
- ### LLM-processing style event
199
-
200
- ```json
201
- {
202
- "scope": { "id": "org_123" },
203
- "request": {
204
- "id": "llm_req_001",
205
- "createdAt": "2026-05-08T14:33:00Z",
206
- "status": "success"
207
- },
208
- "app": { "id": "app-1" },
209
- "model": {
210
- "provider": "openai",
211
- "name": "gpt-4.1"
212
- },
213
- "usage": {
214
- "inputTokens": 1200,
215
- "outputTokens": 350,
216
- "costCents": 4,
217
- "latencyMs": 1800
218
- }
219
- }
220
- ```
221
-
222
- Counts/measures:
223
-
224
- ```text
225
- app/model/provider/status × timeframe
226
- sum inputTokens
227
- sum outputTokens
228
- sum costCents
229
- avg latencyMs
230
- count requests
231
- ```
232
-
233
- ---
234
-
235
- ## Quick start
236
-
237
- ```ts
238
- import { createCountex } from "@x12i/countex";
239
- import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory";
240
-
241
- const accessMap = {
242
- counterId: "access_by_subject_metric",
243
-
244
- scope: { path: "$.tenant.id" },
245
- eventTime: { path: "$.event.createdAt" },
246
-
247
- subjects: [
248
- { type: "actor", path: "$.actor.id", required: true },
249
- { type: "target", path: "$.target.id", required: false },
250
- ],
251
-
252
- metrics: [
253
- { path: "$.labels[*]", required: true },
254
- ],
255
-
256
- dimensions: [
257
- { name: "eventType", path: "$.event.type" },
258
- { name: "situation", path: "$.event.situation" },
259
- { name: "targetType", path: "$.target.type" },
260
- ],
261
-
262
- delta: { default: 1 },
263
- timeframes: ["day", "week", "month", "quarter", "year"],
264
- } as const;
265
-
266
- const countex = createCountex({
267
- maps: [accessMap],
268
- durable: new MemoryDurable(),
269
- cache: new MemoryCache(),
270
- });
271
-
272
- await countex.ingest({
273
- tenant: { id: "org_123" },
274
- event: { type: "share", situation: "external", createdAt: "2026-05-08T14:33:00Z" },
275
- actor: { id: "user_42" },
276
- target: { id: "doc_99", type: "document" },
277
- labels: ["PCI", "PII"],
278
- });
279
-
280
- await countex.flush();
281
-
282
- const result = await countex.query({
283
- scope: "org_123",
284
- counterId: "access_by_subject_metric",
285
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
286
- groupBy: ["subject", "metric", "dim:eventType"],
287
- format: "table",
288
- orderBy: "-count",
289
- limit: 100,
290
- });
291
-
292
- console.log(result.rows);
293
-
294
- await countex.shutdown();
295
- ```
296
-
297
- That single ingest produces:
298
-
299
- - 1 actor + 1 target = **2 subjects**
300
- - 2 labels = **2 metrics**
301
- - 5 timeframes
302
- - → **2 × 2 × 5 = 20 cell increments**
303
-
304
- Each cell uses an atomic `$inc` on durable and `HINCRBY` on the cache, so 20 ingest workers can safely run in parallel.
305
-
306
- ### Preview a map (dry-run)
307
-
308
- Use `countex.preview(event, { counterId? })` to inspect what one event would produce without any I/O — no dictionary allocation, no buffering, no writes. Returns one `MapPreviewResult` per matching map, each with `cells: PreviewIncrementCell[]` (subjects, metrics, dimensions, dimensionsHash, timeframe keys, deltas, measures), `extractionWarnings`, and `droppedAfterCardinality`. See [docs/guides/preview.md](docs/guides/preview.md).
309
-
310
- ---
311
-
312
- ## Production setup (Mongo + Redis)
313
-
314
- Do **not** derive Mongo database names directly from raw tenant strings (length, `$`, `/`, and Unicode can break DB naming rules or leak structure). Hash or normalize first:
315
-
316
- ```ts
317
- import { createHash } from "node:crypto";
318
- import { MongoClient } from "mongodb";
319
- import IORedis from "ioredis";
320
- import { createCountex } from "@x12i/countex";
321
- import { MongoDurable } from "@x12i/countex/adapters/mongo";
322
- import { RedisCache } from "@x12i/countex/adapters/redis";
323
-
324
- function hashScopeId(scope: string): string {
325
- return createHash("sha256").update(scope, "utf8").digest("hex").slice(0, 16);
326
- }
327
-
328
- const mongo = new MongoClient(process.env.MONGO_URI!);
329
- await mongo.connect();
330
- const redis = new IORedis(process.env.REDIS_URL!);
331
-
332
- const countex = createCountex({
333
- maps: [accessMap],
334
- durable: new MongoDurable({
335
- client: mongo,
336
- dbResolver: ({ scope }) => `countex_${hashScopeId(scope)}`,
337
- }),
338
- cache: new RedisCache({
339
- client: redis,
340
- keyPrefix: "countex",
341
- // Optional adapter default TTLs; when `retention.redis` is set below, the
342
- // engine resolves those strings to seconds and forwards them via
343
- // `setRetentionPolicy`, overriding adapter defaults for matching timeframes.
344
- }),
345
- retention: {
346
- redis: {
347
- minute: "48h",
348
- hour: "14d",
349
- day: "92d",
350
- week: "52w",
351
- month: "24mo",
352
- quarter: "24mo",
353
- year: "forever",
354
- all: "forever",
355
- },
356
- // Durable TTL / prune is not enforced by the engine — operators apply Mongo
357
- // TTL indexes or scheduled jobs. `retention.mongo` is optional per-timeframe
358
- // documentation (same duration grammar as redis); see types / DEPLOYMENT.
359
- },
360
- flush: {
361
- everyMs: 60_000,
362
- maxBufferedCells: 100_000,
363
- maxBufferedEvents: 50_000,
364
- flushOnShutdown: true,
365
- },
366
- cardinality: {
367
- maxDimensionValuesPerName: 10_000,
368
- onLimit: "overflow",
369
- overflowValue: "__OTHER__",
370
- },
371
- });
372
-
373
- await countex.init();
374
-
375
- process.on("SIGTERM", async () => { await countex.shutdown(); });
376
- ```
377
-
378
- ---
379
-
380
- ## Mapping reference
381
-
382
- A `CounterMap` is the heart of the package. Everything Countex knows about your data is in here.
383
-
384
- | Field | Type | Notes |
385
- |---------------|----------------------------|-------|
386
- | `counterId` | `string` | Stable identifier for this counter family. |
387
- | `scope` | `{ path }` | Path into the event for the tenant/scope id. Required. |
388
- | `eventTime` | `{ path }` | Path to a Date / ISO string / epoch ms. Drives bucket selection. |
389
- | `subjects` | `SubjectMapping[]` | One or many. Each has `type` and `path`; multi-value paths fan out. |
390
- | `metrics` | `MetricMapping[]` | Same fanout semantics. `$.labels[*]` is common. |
391
- | `measures` | `MeasureMapping[]` | Optional numeric fields per event (`sum` / `count` / `min` / `max`); see [Measures](./docs/guides/measures.md). |
392
- | `dimensions` | `DimensionMapping[]` | Named extra breakdowns. Optional by default. |
393
- | `delta` | `{ path?, default? }` | Numeric value at runtime. Default `1`. Negative deltas are fine. |
394
- | `timeframes` | `TimeframeType[]` | Subset of `minute` / `hour` / `day` / `week` / `month` / `quarter` / `year` / `all`. |
395
- | `idempotency` | `IdempotencyPolicy` | Optional; needs a cache adapter that implements `markEventSeen`. |
396
- | `cardinality` | `CardinalityPolicy` | Optional; per-map override of engine-level defaults. |
397
- | `options` | `MappingOptions` | `skipMissingOptionalDimensions` (default true), `normalizeValues` (default true). |
398
-
399
- ### Path syntax
400
-
401
- A small JSONPath subset:
402
-
403
- | Syntax | Meaning |
404
- |-------------------|--------------------------------------------------|
405
- | `$` | Root. |
406
- | `.name` | Property. |
407
- | `['name']` | Property (bracket form, allows arbitrary chars). |
408
- | `[N]` | Numeric array index (negatives allowed). |
409
- | `[*]` | Wildcard — fan out across array elems / object values. |
410
-
411
- Examples:
412
-
413
- ```
414
- $.tenant.id
415
- $.actor.id
416
- $.labels[*]
417
- $.items[*].id
418
- $['weird.key.with.dots']
419
- ```
420
-
421
- ### Fanout
422
-
423
- A single event with 2 subjects and 3 labels (metrics) and 5 selected timeframes produces **2 × 3 × 5 = 30** atomic increments. This is intentional — read your mapping carefully. Countex emits a `dropped` event when cardinality protection bites, so you'll know.
424
-
425
- ---
426
-
427
- ## Storage adapters
428
-
429
- Countex defines two contracts and ships three adapters.
430
-
431
- ```ts
432
- interface DurableStorage {
433
- applyDeltas(deltas: PendingDelta[]): Promise<void>;
434
- readCells(spec: QuerySpec): Promise<CounterCell[]>;
435
- resolveIds(lookups: DictionaryLookup[]): Promise<Map<string, number>>;
436
- streamCells(filter): AsyncIterable<CounterCell>;
437
- init?(): Promise<void>;
438
- close?(): Promise<void>;
439
- }
440
-
441
- interface CacheStorage {
442
- applyDeltas(deltas: PendingDelta[]): Promise<void>;
443
- readCells(spec: QuerySpec): Promise<{ cells: CounterCell[]; coverage: "full" | "partial" | "miss" }>;
444
- warm(cells: CounterCell[]): Promise<void>;
445
- markEventSeen?(scopeId, eventId, ttlSeconds): Promise<boolean>;
446
- init?(): Promise<void>;
447
- close?(): Promise<void>;
448
- }
449
- ```
450
-
451
- ### Built-in adapters
452
-
453
- | Module | Purpose |
454
- |-----------------------------------|--------------------------------------|
455
- | `@x12i/countex/adapters/memory` | In-process durable + cache; tests / local dev. `MemoryCache` + Mongo is single-process only (`init` warns). |
456
- | `@x12i/countex/adapters/mongo` | Durable. Requires `mongodb` peer. |
457
- | `@x12i/countex/adapters/redis` | Hot cache. Requires `ioredis` peer. |
458
- | `@x12i/countex/adapters/durable-cache` | No Redis: each query reads durable once. Low volume / one machine. |
459
-
460
- Bring your own clients — Countex doesn't open or close them. That keeps lifecycle in the host app's hands.
461
-
462
- ### Mongo schema
463
-
464
- `countex_cells`:
465
-
466
- ```json
467
- {
468
- "scopeId": "org_123",
469
- "counterId": "access_by_subject_metric",
470
- "timeframeType": "day",
471
- "timeframeKey": "2026-05-08",
472
- "subjectType": "actor",
473
- "subjectId": 10001,
474
- "metricId": 41,
475
- "dimensionsHash": "a8f21c0b1d2e",
476
- "dimensions": { "eventType": "share", "situation": "external" },
477
- "count": 42,
478
- "createdAt": "...",
479
- "updatedAt": "..."
480
- }
481
- ```
482
-
483
- Indexes (created automatically on first write):
484
-
485
- - `uniq_cell_identity` (unique, on the full identity tuple)
486
- - `by_timeframe` — for range scans
487
- - `by_subject` — for subject filters
488
- - `by_metric` — for metric filters
489
-
490
- `countex_dict` — the per-scope subject/metric dictionary, with monotonic ids allocated via `countex_seq`. Allocation is **concurrency-safe by construction**: read existing → atomic sequence `$inc` → `insertMany({ ordered: false })` → on duplicate-key, re-read. See [`docs/guides/adapter-mongo.md`](docs/guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency" and [`docs/internals/internal-storage-mongo.md`](docs/internals/internal-storage-mongo.md).
491
-
492
- `dimensionsHash` is canonicalized so any two workers produce the same hash for the same logical dimension tuple: keys sorted lexicographically, values type-tagged (`n: / b: / i: / s:`), serialized, hashed (SHA-1, truncated to 12 hex). See [`docs/internals/internal-dimension-hash.md`](docs/internals/internal-dimension-hash.md).
493
-
494
- ### Redis layout
495
-
496
- ```
497
- countex:{scope}:{counterId}:{timeframeType}:{timeframeKey} → HASH
498
- field = "{subjectType}|{subjectId}|{metricId}|{dimensionsHash}"
499
- value = count
500
-
501
- countex:{scope}:{counterId}:{timeframeType}:{timeframeKey}:cov → HASH (coverage meta)
502
- status = complete | partial | rebuilding (see BucketCoverageMeta; reads may infer missing)
503
- lastWarmAt = epoch ms (string)
504
- mapVersion = optional opaque integer (string)
505
- schemaVersion = cache layout version (string)
506
- cellCount = cells last warmed into this bucket (string, when known)
507
-
508
- countex:{scope}:{counterId}:...:{timeframeKey}:meta:{field} → JSON of dimensions
509
-
510
- countex:{scope}:dedupe:{eventId} → "1" with TTL
511
- ```
512
-
513
- One hash per `(scope, counterId, bucket)` keeps Redis key counts small and per-bucket reads to a single `HGETALL`. **Do not** treat “cell hash is non-empty” as full cache coverage — use the **`:cov`** hash (`status === complete` after `warm`, `partial` after incremental `applyDeltas`, `rebuilding` during cache rebuild). See [`docs/guides/adapter-redis.md`](docs/guides/adapter-redis.md) and [`docs/internals/internal-storage-redis.md`](docs/internals/internal-storage-redis.md).
514
-
515
- ---
516
-
517
- ## Query API
518
-
519
- ```ts
520
- const result = await countex.query({
521
- scope: "org_123",
522
- counterId: "access_by_subject_metric",
523
-
524
- timeframe: {
525
- type: "day",
526
- from: "2026-05-01",
527
- to: "2026-05-08",
528
- },
529
-
530
- // Filters — all optional (subject.ids / metrics: internal numbers and/or dictionary source strings)
531
- subject: { type: "actor", ids: [10001, 10002, "user_uuid_a"] },
532
- metrics: [41, 42, "PCI"],
533
- dimensions: { eventType: ["share", "download"], situation: ["external"] },
534
-
535
- // Shape
536
- groupBy: ["subject", "metric", "dim:eventType"],
537
- format: "table", // "cells" | "table" | "matrix" | "timeseries"
538
- orderBy: "-count",
539
- limit: 100,
540
-
541
- // Optional: use dictionary source keys in grouped keys / cells instead of numeric ids only
542
- resolveLabels: true,
543
-
544
- // Diagnostics
545
- bypassCache: false,
546
- signal: AbortSignal.timeout(5_000),
547
- });
548
- ```
549
-
550
- **Filters:** `subject.ids` and `metrics` may mix **internal numeric ids** (from ingest) and **string source keys** (UUIDs, labels, and so on). String subject keys require `subject.type`. Set **`resolveLabels: true`** to return source keys in grouped `keys.subject` / `keys.metric`, and `subjectSourceKey` / `metricSourceKey` on **`format: "cells"`** rows. See [`docs/guides/querying.md`](docs/guides/querying.md).
551
-
552
- `explainQuery` runs the planner **without** materializing cells. It returns a **`QueryPlan`**: expanded `bucketsRequested`, `cacheEligible` / `cacheReason`, predicted `durableFallbackPredicted`, `cacheBucketsExpectedHit` (when the cache adapter supports `probeBucketCoverage`), heuristic `plannedCellsEstimate`, and cost hints (`expensive`, `groupByCost`). Use it to preflight dashboards and expensive `groupBy` shapes.
553
-
554
- ```ts
555
- const plan = await countex.explainQuery({
556
- scope: "org_123",
557
- counterId: "access_by_subject_metric",
558
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
559
- groupBy: ["subject", "metric"],
560
- format: "table",
561
- });
562
- ```
563
-
564
- Result:
565
-
566
- ```ts
567
- {
568
- format: "table",
569
- rows: [
570
- { keys: { subject: 10001, metric: 41, "dim:eventType": "share" }, count: 142 },
571
- // With resolveLabels: true (and known dictionary rows), subject/metric keys are strings:
572
- // { keys: { subject: "user_42", metric: "PCI", "dim:eventType": "share" }, count: 142 },
573
- ...
574
- ],
575
- total: 1234,
576
- source: "cache" | "durable" | "merged",
577
- }
578
- ```
579
-
580
- String filters are resolved to internal ids before reads; unknown source keys are omitted from the filter. With **`resolveLabels: true`**, **`format: "cells"`** rows include **`subjectSourceKey`** and **`metricSourceKey`** in addition to numeric ids.
581
-
582
- `groupBy` accepts:
583
- - `"subject"`, `"subjectType"`, `"metric"`, `"timeframeKey"`
584
- - `"dim:NAME"` — group by a dimension value
585
-
586
- ---
587
-
588
- ## Insights and advisors
589
-
590
- Countex can optionally analyze stored counters and measures to produce higher-level insights. The core engine answers “what happened?” by maintaining sparse aggregate cells. The insights layer answers “what matters?” by ranking, comparing periods, detecting anomalies, forecasting usage, and emitting evidence-backed recommendations.
591
-
592
- Typical uses:
593
-
594
- - Top / bottom N subjects, metrics, or dimensions; top/bottom percent of groups; Pareto and contribution views
595
- - Period-over-period comparison and simple trends on timeseries
596
- - Explainable anomaly detection (robust z-score, thresholds, missing activity)
597
- - Rule evaluation with normalized recommendations
598
- - Cost / budget / max-token style advisors (generic measures only — no domain lock-in)
599
- - Map and cardinality health when you pass the counter map into `createCountexInsights`
600
- - Resource projections from observed cells via `estimateResourcesFromActuals` (requires the durable adapter reference)
601
-
602
- **Safety:** Insights honor `maxBuckets` (aligned with the core 10,000 bucket cap) and `maxGroups`. Grouped queries load all matching cells for the timeframe before aggregation, so narrow filters when possible.
603
-
604
- **Sparse counters:** Bottom-N style rankings omit zero rows by default; Countex only stores cells that received increments.
605
-
606
- ```ts
607
- import { createCountexInsights } from "@x12i/countex/insights";
608
-
609
- const insights = createCountexInsights({
610
- countex,
611
- durable, // optional; required for estimateResourcesFromActuals → streamCells
612
- mapsByCounterId: new Map(maps.map((m) => [m.counterId, m])),
613
- });
614
-
615
- const topCostDrivers = await insights.topPercent({
616
- scope: "org_123",
617
- counterId: "llm_usage",
618
- timeframe: { type: "month", from: "2026-05", to: "2026-05" },
619
- groupBy: ["subject"],
620
- measure: "costMicros.sum",
621
- percent: 20,
622
- });
623
- ```
624
-
625
- Further reading: [`docs/guides/insights-overview.md`](docs/guides/insights-overview.md), [`docs/guides/insights-ranking.md`](docs/guides/insights-ranking.md), [`docs/guides/insights-comparison-and-trends.md`](docs/guides/insights-comparison-and-trends.md), [`docs/guides/insights-anomalies-and-rules.md`](docs/guides/insights-anomalies-and-rules.md), [`docs/guides/insights-forecasting-and-budget.md`](docs/guides/insights-forecasting-and-budget.md), [`docs/guides/insights-advisors-and-optimization.md`](docs/guides/insights-advisors-and-optimization.md). Deeper topic notes also live under `docs/INSIGHTS*.md` for maintainers.
626
-
627
- CLI (after `npm run build` / install):
628
-
629
- ```bash
630
- npx countex insights:top ./path/to/insight-context.mjs --scope org_123 --counter my_counter
631
- # TypeScript configs: node --import=tsx node_modules/@x12i/countex/dist/cli.js insights:pareto ./ctx.ts
632
- ```
633
-
634
- ---
635
-
636
- ## Events
637
-
638
- The Countex client is an `EventEmitter`. Subscribe to observe what's happening:
639
-
640
- ```ts
641
- countex.on("flush", ({ cells, durationMs }) => {});
642
- countex.on("flush_error", ({ error }) => {});
643
- countex.on("cardinality_warning", ({ kind, count, limit })=> {});
644
- countex.on("overflow", ({ kind, sourceKey }) => {});
645
- countex.on("dropped", ({ reason }) => {});
646
- countex.on("duplicate", ({ scopeId, eventId }) => {});
647
- ```
648
-
649
- All event payloads are typed via `CountexEvents`.
650
-
651
- ---
652
-
653
- ## Operational notes
654
-
655
- ### Buffering and flushing
656
-
657
- Each Countex instance keeps only a delta buffer in memory:
658
-
659
- ```
660
- Map<cellKey, delta>
661
- ```
662
-
663
- Flush triggers (any of):
664
-
665
- - `flush.everyMs` timer (default 60s)
666
- - `flush.maxBufferedCells` reached (default 100k)
667
- - `flush.maxBufferedEvents` reached (default 50k)
668
- - `shutdown()` if `flushOnShutdown` is true (default)
669
- - explicit `flush()`
670
-
671
- Flush is **two-phase**: durable first, cache second. If durable fails, the snapshot is merged back into the buffer — counts are not lost. If cache fails after durable succeeded, it's logged and continues; the next read warms cache from durable.
672
-
673
- At most one flush runs at a time. Concurrent callers receive the same in-flight promise.
674
-
675
- ### Cardinality protection
676
-
677
- A bad mapping (e.g. a free-form URL as a dimension) can balloon the dictionary. Countex tracks per-`(scope, kind, qualifier)` how many distinct keys it has minted, and applies a policy when limits hit:
678
-
679
- - `overflow` (default) — rewrite the source key to `__OTHER__`. Emits `overflow`.
680
- - `drop` — silently drop the increment. Emits `dropped`.
681
- - `error` — throw `CardinalityLimitError`.
682
-
683
- Defaults are conservative; tune them per workload. A warning fires at 80% of the limit (`cardinality_warning`).
684
-
685
- ### Idempotency
686
-
687
- Optional and per-map. Requires a cache adapter that implements `markEventSeen` (the included `RedisCache` and `MemoryCache` both do).
688
-
689
- ```ts
690
- idempotency: {
691
- enabled: true,
692
- eventIdPath: "$.event.id",
693
- ttlSeconds: 86_400,
694
- }
695
- ```
696
-
697
- Dedupe keys are namespaced by `counterId`, so different maps with their own idempotency don't collide on shared event ids.
698
-
699
- ### Late events
700
-
701
- Bucket selection always uses the **event time**, never the ingestion time. Late-arriving events go into the correct historical bucket — `$inc` doesn't care about ordering.
702
-
703
- ### Concurrency safety
704
-
705
- All writes are atomic per cell:
706
-
707
- - Mongo: `bulkWrite` of `updateOne` with `$inc` and `upsert: true`
708
- - Redis: `HINCRBY`
709
-
710
- 20+ Countex workers can safely run in parallel against the same scope. Reads are eventually consistent across the two stores; if Redis is behind durable, the planner will detect partial coverage and read durable.
711
-
712
- ### Backup / verify / restore / rebuild
713
-
714
- Top-level free functions and a `countex.backup` namespace bound to your engine:
715
-
716
- ```ts
717
- import {
718
- exportCells,
719
- computeCellsFingerprint,
720
- createBackupManifest,
721
- verifyBackupAgainstManifest,
722
- restoreSnapshotCells,
723
- rebuildCache,
724
- COUNTEX_CELL_SCHEMA_VERSION,
725
- } from "@x12i/countex";
726
-
727
- const filter = {
728
- scope: "org_123",
729
- counterId: "access_by_subject_metric",
730
- timeframe: { type: "day", from: "2026-02-01", to: "2026-05-08" },
731
- };
732
-
733
- // Stream all cells in a window — bounded async iteration.
734
- for await (const cell of exportCells(durable, filter)) {
735
- // ndjson, S3, whatever you want
736
- }
737
-
738
- // Fingerprint + manifest for cold-storage verification.
739
- const fp = await countex.backup.fingerprint(filter); // { cellCount, contentSha256, countSum }
740
- const manifest = countex.backup.createManifest(filter, fp); // { manifestVersion, schemaVersion, filter, … }
741
-
742
- // Verify a manifest against the live durable adapter.
743
- const ok = await countex.backup.verify(manifest);
744
- if (!ok.ok) throw new Error(ok.detail);
745
-
746
- // Restore. Modes: "validate-only" | "replace-window" (default) | "merge-deltas" (reserved).
747
- const { restored, mode } = await countex.backup.restore(
748
- countex.backup.export(filter),
749
- { mode: "replace-window" },
750
- );
751
-
752
- // Re-warm cache from durable. Each warmed bucket's :cov flips to status=complete.
753
- await countex.backup.rebuildCache(filter);
754
- ```
755
-
756
- See [`docs/guides/backup-export.md`](docs/guides/backup-export.md) for the full recipe and manifest layout.
757
-
758
- ---
759
-
760
- ## Docker integration testing
761
-
762
- Countex ships a Docker-based stack (MongoDB, Redis, MinIO) and integration tests under `tests/integration/docker/` that exercise real adapters: dictionary allocation under concurrency, Mongo **`lookupIds` / `reverseLookupIds`**, an end-to-end **`query()`** with **source-key filters** and **`resolveLabels`**, multi-worker ingestion, Redis cache coverage and durable fallback, `rebuildCache`, simulation fingerprints against a memory reference, and backup / manifest / verify / restore to S3-compatible storage (via the AWS SDK in devDependencies).
763
-
764
- **One-shot local run** (starts services, runs tests with default localhost URLs, tears volumes down):
765
-
766
- ```bash
767
- npm run test:e2e
768
- ```
769
-
770
- **Host-runner** (services in Docker, tests on the host):
771
-
772
- ```bash
773
- npm run docker:up
774
- export COUNTEX_MONGO_URI='mongodb://countex:countex@127.0.0.1:27017/admin'
775
- export COUNTEX_REDIS_URL='redis://127.0.0.1:6379'
776
- export COUNTEX_S3_ENDPOINT='http://127.0.0.1:9000'
777
- export COUNTEX_S3_REGION='us-east-1'
778
- export COUNTEX_S3_ACCESS_KEY_ID='countex'
779
- export COUNTEX_S3_SECRET_ACCESS_KEY='countex-password'
780
- export COUNTEX_S3_BUCKET='countex-backups'
781
- export COUNTEX_INTEGRATION=1
782
- npm run test:integration
783
- npm run docker:down
784
- ```
785
-
786
- Optional tuning: `COUNTEX_TEST_PROFILE`, `COUNTEX_TEST_EVENTS`, `COUNTEX_TEST_WORKERS`. Stress-style full pipeline (larger simulation + S3 round-trip): set `COUNTEX_TEST_FULL=1` (requires S3 env as above).
787
-
788
- **Container-runner** (CI parity): `npm run test:integration:docker` builds `docker/test-runner.Dockerfile` and runs `npm run test:integration:exec` with service hostnames injected by Compose.
789
-
790
- Reports are written to `artifacts/integration/report.json` and `artifacts/integration/report.md` after `npm run test:integration`.
791
-
792
- ---
793
-
794
- ## Errors
795
-
796
- All errors thrown by Countex extend `CountexError`:
797
-
798
- | Class | When |
799
- |--------------------------|------|
800
- | `MappingError` | Invalid mapping config (caught at engine startup). |
801
- | `ExtractionError` | Malformed path or value during extraction. |
802
- | `CardinalityLimitError` | Cardinality limit hit when policy is `error`. |
803
- | `StorageError` | Wrapping a storage-adapter failure. |
804
-
805
- ---
806
-
807
- ## Versioning & scope
808
-
809
- > **Additive counts are not distinct counts.** If `user_42` appears in 100 events, `count = 100`, not 1 unique user. Counter cells store sums of deltas, so a query result of `count = 100` from a subject-grouped view means "100 increments hit this subject", not "100 distinct subjects". For exact distinct subject counts, group by `["subjectType", "subject"]` and use `rows.length` after retrieving every matching cell — see [`docs/guides/querying.md`](docs/guides/querying.md) §"Distinct subjects / metrics as row cardinality". Approximate distinct counts (HLL/sets) are a planned opt-in extension, not core.
810
-
811
- `v0.x` is for stabilizing the public API. Once it's hardened:
812
-
813
- - exact unique counts (HLL / sets) are an opt-in extension, not core
814
- - complex expression deltas are not in core
815
- - distributed locks / exactly-once: not Countex's job; durable storage is the source of truth and `$inc` makes worker-count irrelevant
816
-
817
- ---
818
-
819
- ## Repository layout
820
-
821
- | Path | Role |
822
- |------|------|
823
- | `src/index.ts` | Public exports (`createCountex`, types, errors). |
824
- | `src/core/` | Engine: ingest pipeline, buffer, extraction, mapping validation, time buckets, cardinality. |
825
- | `src/query/` | Query planner (`planQuery` + `executeQuery`) and result formatting. |
826
- | `src/storage/` | Adapters: `memory` (in-process), `mongo` (durable), `redis` (cache); shared coverage helpers. |
827
- | `src/utils/` | JSONPath subset, hashing, logging. |
828
- | `src/backup/` | Export, fingerprint, manifest, verify, restore, cache rebuild helpers. |
829
- | `src/simulation/` | Simulation engine, profiles, expected aggregates, resource reports. |
830
- | `src/insights/` | Ranking, comparison, anomaly detection, advisors, recommendations. |
831
- | `src/cli.ts` | CLI commands for simulation, insights, export, and operational helpers. |
832
- | `examples/` | Runnable demos (`npx tsx examples/basic.ts`). |
833
- | `docs/` | Deployment guide, tutorial, guides, internals, and use cases (published in the npm package). |
834
- | `tests/unit/` | Fast Node test runner tests (`npm test`). |
835
- | `tests/integration/` | Opt-in live tests against real services (`npm run test:integration`). |
836
-
837
- ---
838
-
839
- ## Publishing (npm)
840
-
841
- This package is scoped as **`@x12i/countex`** and is configured for **public** registry access (`publishConfig.access` in `package.json`).
842
-
843
- 1. Ensure you are logged in and allowed to publish under the **`@x12i`** scope on [npm](https://www.npmjs.com/).
844
- 2. Put auth tokens only in **`~/.npmrc`** — do not commit project `.npmrc` with credentials (it is listed in `.gitignore`).
845
- 3. From the repo root:
846
-
847
- ```bash
848
- npm test
849
- npm publish
850
- ```
851
-
852
- `prepublishOnly` runs a clean build so the tarball always contains a fresh `dist/`.
853
-
854
- The **`docs/`** directory (`DEPLOYMENT.md`, `END_TO_END_EXAMPLE.md`) is part of the published package (`files` in `package.json`).
855
-
856
- ---
857
-
858
- ## License
859
-
860
- MIT — see `LICENSE`.
1
+ # @x12i/countex
2
+
3
+ > Generic sparse counter matrices for JSON events.
4
+ > Dynamic counting, aggregation, and retrieval from mapped JSON inputs.
5
+
6
+ `@x12i/countex` is a **generic mapped-event counter**: a source-agnostic counting engine. You give it any JSON event, a declarative mapping config, and storage adapters; it projects events into sparse counter cells, batches deltas locally, and persists them atomically to a durable store (MongoDB or PostgreSQL) and a hot cache (Redis).
7
+
8
+ **Package policy:** Countex is generic. It never references a specific host product, client name, schema, or storage layout. Host-specific behavior belongs in the host application, not in this package.
9
+
10
+ It does **not** know about users, organizations, documents, labels, or events. It only knows generic primitives: **scope, subject, metric, dimensions, timeframe, delta**. Whatever your domain calls those things — Countex doesn't care.
11
+
12
+ **Terminology:** `metric` is the generic Countex name for the thing being counted or categorized. In your domain that may be a label, status, event name, operation, outcome, class, model result, or business category — the `metrics` array in a map simply lists JSON paths that resolve to those categories. Storage and APIs refer to dictionary **`sourceKey`** (the string extracted from the event); avoid confusing that with application-level UUIDs unless your map uses UUIDs as keys.
13
+
14
+ **Process memory vs retained state:** Each worker keeps **unflushed deltas** in an in-memory buffer. After flush, authoritative aggregates live in **durable** storage (MongoDB or PostgreSQL); **Redis** holds a hot cache. Restarting a worker loses only buffered deltas that were not flushed yet — tune flush intervals accordingly.
15
+
16
+ ```
17
+ event JSON ──► map ──► raw increments ──► buffer ──► flush ──► Mongo or Postgres (durable)
18
+ └───► Redis (hot cache)
19
+
20
+ ◄── query ◄── Redis (full coverage) or ◄── durable fallback ──◄
21
+ ```
22
+
23
+ ---
24
+
25
+ ## Install
26
+
27
+ ```bash
28
+ npm install @x12i/countex
29
+ # Bring your own clients — peer deps, both optional:
30
+ npm install mongodb ioredis
31
+ # or, for Postgres durable storage instead of Mongo:
32
+ npm install pg ioredis
33
+ ```
34
+
35
+ Node 18+. Zero runtime dependencies in the core package.
36
+
37
+ ### Documentation shipped with the package
38
+
39
+ These guides are included in the published tarball under `docs/` and exposed as stable import subpaths (resolve to `.md` files for editors and tooling):
40
+
41
+ | Topic | Package subpath | File |
42
+ |-------|-----------------|------|
43
+ | Release & production deployment | `@x12i/countex/docs/deployment` | `docs/DEPLOYMENT.md` |
44
+ | Tutorial — install → map → ingest → query | `@x12i/countex/docs/end-to-end-example` | `docs/END_TO_END_EXAMPLE.md` |
45
+ | Feature guides index | `@x12i/countex/docs/guides` | `docs/guides/README.md` |
46
+ | Counter maps | `@x12i/countex/docs/guides/counter-maps` | `docs/guides/counter-maps.md` |
47
+ | Preview a map (dry-run) | `@x12i/countex/docs/guides/preview` | `docs/guides/preview.md` |
48
+ | Ingestion & flush | `@x12i/countex/docs/guides/ingestion-and-flush` | `docs/guides/ingestion-and-flush.md` |
49
+ | Timeframes | `@x12i/countex/docs/guides/timeframes-and-bucketing` | `docs/guides/timeframes-and-bucketing.md` |
50
+ | Measures | `@x12i/countex/docs/guides/measures` | `docs/guides/measures.md` |
51
+ | Querying | `@x12i/countex/docs/guides/querying` | `docs/guides/querying.md` |
52
+ | Memory adapters | `@x12i/countex/docs/guides/adapter-memory` | `docs/guides/adapter-memory.md` |
53
+ | Mongo durable | `@x12i/countex/docs/guides/adapter-mongo` | `docs/guides/adapter-mongo.md` |
54
+ | Postgres durable | `@x12i/countex/docs/guides/adapter-postgres` | `docs/guides/adapter-postgres.md` |
55
+ | Mongo to Postgres | `@x12i/countex/docs/guides/migrating-mongo-to-postgres` | `docs/guides/migrating-mongo-to-postgres.md` |
56
+ | Redis cache | `@x12i/countex/docs/guides/adapter-redis` | `docs/guides/adapter-redis.md` |
57
+ | Cardinality | `@x12i/countex/docs/guides/cardinality` | `docs/guides/cardinality.md` |
58
+ | Idempotency | `@x12i/countex/docs/guides/idempotency` | `docs/guides/idempotency.md` |
59
+ | Managed commit (Postgres) | `@x12i/countex/docs/guides/managed-commit` | `docs/guides/managed-commit.md` |
60
+ | Backup export | `@x12i/countex/docs/guides/backup-export` | `docs/guides/backup-export.md` |
61
+ | Client events | `@x12i/countex/docs/guides/client-events` | `docs/guides/client-events.md` |
62
+ | Insights overview | `@x12i/countex/docs/guides/insights-overview` | `docs/guides/insights-overview.md` |
63
+ | Insights ranking | `@x12i/countex/docs/guides/insights-ranking` | `docs/guides/insights-ranking.md` |
64
+ | Insights comparison & trends | `@x12i/countex/docs/guides/insights-comparison-and-trends` | `docs/guides/insights-comparison-and-trends.md` |
65
+ | Insights anomalies & rules | `@x12i/countex/docs/guides/insights-anomalies-and-rules` | `docs/guides/insights-anomalies-and-rules.md` |
66
+ | Insights forecasting & budget | `@x12i/countex/docs/guides/insights-forecasting-and-budget` | `docs/guides/insights-forecasting-and-budget.md` |
67
+ | Insights advisors | `@x12i/countex/docs/guides/insights-advisors-and-optimization` | `docs/guides/insights-advisors-and-optimization.md` |
68
+ | CLI | `@x12i/countex/docs/guides/cli` | `docs/guides/cli.md` |
69
+ | Simulation runtime (JS API) | `@x12i/countex/simulation` | `dist/simulation/` |
70
+ | Simulation overview | `@x12i/countex/docs/guides/simulation` | `docs/guides/simulation.md` |
71
+ | Simulation profiles | `@x12i/countex/docs/guides/simulation-profiles` | `docs/guides/simulation-profiles.md` |
72
+ | Simulation resource estimation | `@x12i/countex/docs/guides/simulation-resource-estimation` | `docs/guides/simulation-resource-estimation.md` |
73
+
74
+ **Use cases** (technical playbooks: contracts, fan-out math, exact `QuerySpec`s, production wiring, verification — see [`docs/use-cases/README.md`](docs/use-cases/README.md)):
75
+
76
+ | Topic | Package subpath | File |
77
+ |-------|-----------------|------|
78
+ | Use cases index | `@x12i/countex/docs/use-cases` | `docs/use-cases/README.md` |
79
+ | Audit & security logs (volume, situations, distinct entities, logical event totals) | `@x12i/countex/docs/use-cases/audit-security-log-counts` | `docs/use-cases/audit-security-log-counts.md` |
80
+ | Product & feature activity | `@x12i/countex/docs/use-cases/product-feature-usage` | `docs/use-cases/product-feature-usage.md` |
81
+ | LLM cost & token metering | `@x12i/countex/docs/use-cases/llm-cost-and-token-metering` | `docs/use-cases/llm-cost-and-token-metering.md` |
82
+ | Multi-tenant isolation | `@x12i/countex/docs/use-cases/multi-tenant-isolation` | `docs/use-cases/multi-tenant-isolation.md` |
83
+ | Idempotent webhooks & queues | `@x12i/countex/docs/use-cases/webhook-idempotent-ingestion` | `docs/use-cases/webhook-idempotent-ingestion.md` |
84
+ | Ops dashboards & rollups | `@x12i/countex/docs/use-cases/ops-dashboard-daily-rollups` | `docs/use-cases/ops-dashboard-daily-rollups.md` |
85
+ | Processing & workflow activity | `@x12i/countex/docs/use-cases/processing-workflow-activity` | `docs/use-cases/processing-workflow-activity.md` |
86
+
87
+ **Internals** (how the engine works — diagrams and traces) live under `docs/internals/` in the tarball; browse [`docs/internals/README.md`](docs/internals/README.md).
88
+
89
+ After install: `node_modules/@x12i/countex/docs/`.
90
+
91
+ ---
92
+
93
+ ## The mental model
94
+
95
+ Countex turns mapped JSON events into sparse aggregate cells.
96
+
97
+ A counter cell is uniquely identified by:
98
+
99
+ ```
100
+ (scope, counterId, timeframeType, timeframeKey, subjectType, subjectId, metricId, dimensionsHash)
101
+ ```
102
+
103
+ Each event can produce one or many cell increments. A counter increment is defined by:
104
+
105
+ - **scope** — your tenant / org / account boundary
106
+ - **counterId** — which counter family this is (e.g. `access_by_subject_metric`)
107
+ - **timeframe** — `minute` / `hour` / `day` / `week` / `month` / `quarter` / `year` / `all`
108
+ - **subject** — the thing being counted against (e.g. an actor or a target)
109
+ - **metric** — the count category / label
110
+ - **dimensions** — arbitrary additional breakdowns
111
+ - **delta** — `+1`, `-1`, or any integer
112
+
113
+ Countex buffers increments locally, flushes atomic deltas to durable + cache, and serves queries from cache with durable fallback.
114
+
115
+ ### Measures and rates
116
+
117
+ Maps may declare optional **`measures`**: named numeric fields per event (cost, tokens, latency, …) with aggregations `sum`, `count`, `min`, and `max`. **`avg`** is computed at query time when both `sum` and `count` exist on the measure. Use **`scale`** on a measure for fixed-point storage (e.g. cents). Extracted values attach to each fan-out increment the same way as **`delta`** (see RFC-style docs for fan-out semantics).
118
+
119
+ Queries accept **`measures`** (explicit list or `"*"`), **`orderBy`** on a measure aggregation, and optional **`rate`** (count or a measure’s sum divided by elapsed time in the queried window; for `format: "timeseries"`, per bucket). **`rate`** is invalid for timeframe **`all`**.
120
+
121
+ **Redis cache** stores **sum** and **count** per measure only; **min** / **max** are durable-backed. If a query needs min/max (including `orderBy` on them, or requesting measures that maintain min/max when using `"*"`), the planner reads from **durable** first so cache cannot drift.
122
+
123
+ **`bucketRange`** caps at **10,000** buckets; at **`minute`** resolution that is about **6.94 days** — keep queries within that bound or narrow the range.
124
+
125
+ ### Percentiles & histograms
126
+
127
+ Core cells store **additive** measure aggregates (`sum / count / min / max`) per sparse cell. They do **not** store distributions, so percentile-style answers (p50/p90/p95/p99) can only come from min/max approximation or from histogram/sketch storage you add yourself. Insights advisors that mention percentile-shaped recommendations are heuristic; see [`docs/guides/insights-overview.md`](docs/guides/insights-overview.md) §"Distributions" and [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) "Architecture decisions".
128
+
129
+ ---
130
+
131
+ ## What kinds of events Countex is for
132
+
133
+ Countex is not tied to one event source or domain. It is a **generic mapped-event counter**: any JSON-like input stream where useful aggregates can be produced from paths in the event.
134
+
135
+ Typical sources include:
136
+
137
+ | Source family | Examples | Common Countex projection |
138
+ |---|---|---|
139
+ | Audit and security logs | Microsoft 365 audit logs, Google Workspace logs, Okta logs, cloud audit logs | actor/resource/action/situation/severity |
140
+ | Product and user activity | Woopra, Segment, PostHog-style events, application activity logs | user/account/feature/event/page/funnel |
141
+ | Processing and workflow activity | ETL runs, background jobs, agent graphs, workflow engines, queue processors | job/task/node/status/error/runtime |
142
+ | LLM and AI service usage | Model gateway logs, OpenAI/Anthropic calls, internal LLM services | app/user/model/provider/tokens/cost/latency/result |
143
+ | Business/domain events | CRM, support tickets, payments, commerce, lifecycle events | customer/object/state/category/outcome |
144
+
145
+ Countex only needs a map. The map tells Countex where to find:
146
+
147
+ - the scope boundary
148
+ - the event time
149
+ - the subjects
150
+ - the metrics
151
+ - the dimensions
152
+ - the delta/count
153
+ - optional numeric measures such as cost, tokens, latency, or duration
154
+
155
+ The original event can come from a log, webhook, queue message, analytics event, system activity, processing trace, or generated test fixture. Countex does not care about the event source as long as the map can project it into generic counter primitives.
156
+
157
+ ### Audit-log style event
158
+
159
+ ```json
160
+ {
161
+ "tenant": { "id": "org_123" },
162
+ "event": {
163
+ "id": "audit_001",
164
+ "createdAt": "2026-05-08T14:33:00Z",
165
+ "operation": "FileAccessed",
166
+ "situation": "external"
167
+ },
168
+ "actor": { "id": "user_42", "type": "user" },
169
+ "resource": { "id": "file_99", "type": "document" },
170
+ "labels": ["PII", "Financial"]
171
+ }
172
+ ```
173
+
174
+ Counts:
175
+
176
+ ```text
177
+ actor/resource × label × operation/situation/resourceType × timeframe
178
+ ```
179
+
180
+ ### Product-activity style event
181
+
182
+ ```json
183
+ {
184
+ "account": { "id": "acct_123" },
185
+ "user": { "id": "user_42" },
186
+ "event": {
187
+ "name": "dashboard_viewed",
188
+ "createdAt": "2026-05-08T14:33:00Z"
189
+ },
190
+ "product": {
191
+ "feature": "risk_dashboard",
192
+ "plan": "enterprise"
193
+ }
194
+ }
195
+ ```
196
+
197
+ Counts:
198
+
199
+ ```text
200
+ user/account × activity metric × feature/plan × timeframe
201
+ ```
202
+
203
+ ### LLM-processing style event
204
+
205
+ ```json
206
+ {
207
+ "scope": { "id": "org_123" },
208
+ "request": {
209
+ "id": "llm_req_001",
210
+ "createdAt": "2026-05-08T14:33:00Z",
211
+ "status": "success"
212
+ },
213
+ "app": { "id": "app-1" },
214
+ "model": {
215
+ "provider": "openai",
216
+ "name": "gpt-4.1"
217
+ },
218
+ "usage": {
219
+ "inputTokens": 1200,
220
+ "outputTokens": 350,
221
+ "costCents": 4,
222
+ "latencyMs": 1800
223
+ }
224
+ }
225
+ ```
226
+
227
+ Counts/measures:
228
+
229
+ ```text
230
+ app/model/provider/status × timeframe
231
+ sum inputTokens
232
+ sum outputTokens
233
+ sum costCents
234
+ avg latencyMs
235
+ count requests
236
+ ```
237
+
238
+ ---
239
+
240
+ ## Quick start
241
+
242
+ ```ts
243
+ import { createCountex } from "@x12i/countex";
244
+ import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory";
245
+
246
+ const accessMap = {
247
+ counterId: "access_by_subject_metric",
248
+
249
+ scope: { path: "$.tenant.id" },
250
+ eventTime: { path: "$.event.createdAt" },
251
+
252
+ subjects: [
253
+ { type: "actor", path: "$.actor.id", required: true },
254
+ { type: "target", path: "$.target.id", required: false },
255
+ ],
256
+
257
+ metrics: [
258
+ { path: "$.labels[*]", required: true },
259
+ ],
260
+
261
+ dimensions: [
262
+ { name: "eventType", path: "$.event.type" },
263
+ { name: "situation", path: "$.event.situation" },
264
+ { name: "targetType", path: "$.target.type" },
265
+ ],
266
+
267
+ delta: { default: 1 },
268
+ timeframes: ["day", "week", "month", "quarter", "year"],
269
+ } as const;
270
+
271
+ const countex = createCountex({
272
+ maps: [accessMap],
273
+ durable: new MemoryDurable(),
274
+ cache: new MemoryCache(),
275
+ });
276
+
277
+ await countex.ingest({
278
+ tenant: { id: "org_123" },
279
+ event: { type: "share", situation: "external", createdAt: "2026-05-08T14:33:00Z" },
280
+ actor: { id: "user_42" },
281
+ target: { id: "doc_99", type: "document" },
282
+ labels: ["PCI", "PII"],
283
+ });
284
+
285
+ await countex.flush();
286
+
287
+ const result = await countex.query({
288
+ scope: "org_123",
289
+ counterId: "access_by_subject_metric",
290
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
291
+ groupBy: ["subject", "metric", "dim:eventType"],
292
+ format: "table",
293
+ orderBy: "-count",
294
+ limit: 100,
295
+ });
296
+
297
+ console.log(result.rows);
298
+
299
+ await countex.shutdown();
300
+ ```
301
+
302
+ That single ingest produces:
303
+
304
+ - 1 actor + 1 target = **2 subjects**
305
+ - 2 labels = **2 metrics**
306
+ - 5 timeframes
307
+ - → **2 × 2 × 5 = 20 cell increments**
308
+
309
+ Each cell uses an atomic `$inc` on durable and `HINCRBY` on the cache, so 20 ingest workers can safely run in parallel.
310
+
311
+ ### Preview a map (dry-run)
312
+
313
+ Use `countex.preview(event, { counterId? })` to inspect what one event would produce without any I/O — no dictionary allocation, no buffering, no writes. Returns one `MapPreviewResult` per matching map, each with `cells: PreviewIncrementCell[]` (subjects, metrics, dimensions, dimensionsHash, timeframe keys, deltas, measures), `extractionWarnings`, and `droppedAfterCardinality`. See [docs/guides/preview.md](docs/guides/preview.md).
314
+
315
+ ---
316
+
317
+ ## Production setup (Mongo + Redis)
318
+
319
+ Do **not** derive Mongo database names directly from raw tenant strings (length, `$`, `/`, and Unicode can break DB naming rules or leak structure). Hash or normalize first:
320
+
321
+ ```ts
322
+ import { createHash } from "node:crypto";
323
+ import { MongoClient } from "mongodb";
324
+ import IORedis from "ioredis";
325
+ import { createCountex } from "@x12i/countex";
326
+ import { MongoDurable } from "@x12i/countex/adapters/mongo";
327
+ import { RedisCache } from "@x12i/countex/adapters/redis";
328
+
329
+ function hashScopeId(scope: string): string {
330
+ return createHash("sha256").update(scope, "utf8").digest("hex").slice(0, 16);
331
+ }
332
+
333
+ const mongo = new MongoClient(process.env.MONGO_URI!);
334
+ await mongo.connect();
335
+ const redis = new IORedis(process.env.REDIS_URL!);
336
+
337
+ const countex = createCountex({
338
+ maps: [accessMap],
339
+ durable: new MongoDurable({
340
+ client: mongo,
341
+ dbResolver: ({ scope }) => `countex_${hashScopeId(scope)}`,
342
+ }),
343
+ cache: new RedisCache({
344
+ client: redis,
345
+ keyPrefix: "countex",
346
+ // Optional adapter default TTLs; when `retention.redis` is set below, the
347
+ // engine resolves those strings to seconds and forwards them via
348
+ // `setRetentionPolicy`, overriding adapter defaults for matching timeframes.
349
+ }),
350
+ retention: {
351
+ redis: {
352
+ minute: "48h",
353
+ hour: "14d",
354
+ day: "92d",
355
+ week: "52w",
356
+ month: "24mo",
357
+ quarter: "24mo",
358
+ year: "forever",
359
+ all: "forever",
360
+ },
361
+ // Durable TTL / prune is not enforced by the engine. Operators apply Mongo
362
+ // TTL indexes, Postgres partition drops, or scheduled jobs.
363
+ // `retention.mongo` and `retention.postgres` are optional per-timeframe
364
+ // documentation (same duration grammar as redis); see types / DEPLOYMENT.
365
+ },
366
+ flush: {
367
+ everyMs: 60_000,
368
+ maxBufferedCells: 100_000,
369
+ maxBufferedEvents: 50_000,
370
+ flushOnShutdown: true,
371
+ },
372
+ cardinality: {
373
+ maxDimensionValuesPerName: 10_000,
374
+ onLimit: "overflow",
375
+ overflowValue: "__OTHER__",
376
+ },
377
+ });
378
+
379
+ await countex.init();
380
+
381
+ process.on("SIGTERM", async () => { await countex.shutdown(); });
382
+ ```
383
+
384
+ ---
385
+
386
+ ## Mapping reference
387
+
388
+ A `CounterMap` is the heart of the package. Everything Countex knows about your data is in here.
389
+
390
+ | Field | Type | Notes |
391
+ |---------------|----------------------------|-------|
392
+ | `counterId` | `string` | Stable identifier for this counter family. |
393
+ | `scope` | `{ path }` | Path into the event for the tenant/scope id. Required. |
394
+ | `eventTime` | `{ path }` | Path to a Date / ISO string / epoch ms. Drives bucket selection. |
395
+ | `subjects` | `SubjectMapping[]` | One or many. Each has `type` and `path`; multi-value paths fan out. |
396
+ | `metrics` | `MetricMapping[]` | Same fanout semantics. `$.labels[*]` is common. |
397
+ | `measures` | `MeasureMapping[]` | Optional numeric fields per event (`sum` / `count` / `min` / `max`); see [Measures](./docs/guides/measures.md). |
398
+ | `dimensions` | `DimensionMapping[]` | Named extra breakdowns. Optional by default. |
399
+ | `delta` | `{ path?, default? }` | Numeric value at runtime. Default `1`. Negative deltas are fine. |
400
+ | `timeframes` | `TimeframeType[]` | Subset of `minute` / `hour` / `day` / `week` / `month` / `quarter` / `year` / `all`. |
401
+ | `idempotency` | `IdempotencyPolicy` | Optional; needs a cache adapter that implements `markEventSeen`. |
402
+ | `cardinality` | `CardinalityPolicy` | Optional; per-map override of engine-level defaults. |
403
+ | `options` | `MappingOptions` | `skipMissingOptionalDimensions` (default true), `normalizeValues` (default true). |
404
+
405
+ ### Path syntax
406
+
407
+ A small JSONPath subset:
408
+
409
+ | Syntax | Meaning |
410
+ |-------------------|--------------------------------------------------|
411
+ | `$` | Root. |
412
+ | `.name` | Property. |
413
+ | `['name']` | Property (bracket form, allows arbitrary chars). |
414
+ | `[N]` | Numeric array index (negatives allowed). |
415
+ | `[*]` | Wildcard — fan out across array elems / object values. |
416
+
417
+ Examples:
418
+
419
+ ```
420
+ $.tenant.id
421
+ $.actor.id
422
+ $.labels[*]
423
+ $.items[*].id
424
+ $['weird.key.with.dots']
425
+ ```
426
+
427
+ ### Fanout
428
+
429
+ A single event with 2 subjects and 3 labels (metrics) and 5 selected timeframes produces **2 × 3 × 5 = 30** atomic increments. This is intentional — read your mapping carefully. Countex emits a `dropped` event when cardinality protection bites, so you'll know.
430
+
431
+ ---
432
+
433
+ ## Storage adapters
434
+
435
+ Countex defines two contracts and ships three adapters.
436
+
437
+ ```ts
438
+ interface DurableStorage {
439
+ applyDeltas(deltas: PendingDelta[]): Promise<void>;
440
+ readCells(spec: QuerySpec): Promise<CounterCell[]>;
441
+ resolveIds(lookups: DictionaryLookup[]): Promise<Map<string, number>>;
442
+ streamCells(filter): AsyncIterable<CounterCell>;
443
+ init?(): Promise<void>;
444
+ close?(): Promise<void>;
445
+ }
446
+
447
+ interface CacheStorage {
448
+ applyDeltas(deltas: PendingDelta[]): Promise<void>;
449
+ readCells(spec: QuerySpec): Promise<{ cells: CounterCell[]; coverage: "full" | "partial" | "miss" }>;
450
+ warm(cells: CounterCell[]): Promise<void>;
451
+ markEventSeen?(scopeId, eventId, ttlSeconds): Promise<boolean>;
452
+ init?(): Promise<void>;
453
+ close?(): Promise<void>;
454
+ }
455
+ ```
456
+
457
+ ### Built-in adapters
458
+
459
+ | Module | Purpose |
460
+ |-----------------------------------|--------------------------------------|
461
+ | `@x12i/countex/adapters/memory` | In-process durable + cache; tests / local dev. `MemoryCache` + Mongo is single-process only (`init` warns). |
462
+ | `@x12i/countex/adapters/mongo` | Durable. Requires `mongodb` peer. |
463
+ | `@x12i/countex/adapters/postgres` | Durable. Requires `pg` peer. PostgreSQL 15+. |
464
+ | `@x12i/countex/adapters/redis` | Hot cache. Requires `ioredis` peer. |
465
+ | `@x12i/countex/adapters/durable-cache` | No Redis: each query reads durable once. Low volume / one machine. |
466
+
467
+ Bring your own clients — Countex doesn't open or close them. That keeps lifecycle in the host app's hands.
468
+
469
+ ### Mongo schema
470
+
471
+ `countex_cells`:
472
+
473
+ ```json
474
+ {
475
+ "scopeId": "org_123",
476
+ "counterId": "access_by_subject_metric",
477
+ "timeframeType": "day",
478
+ "timeframeKey": "2026-05-08",
479
+ "subjectType": "actor",
480
+ "subjectId": 10001,
481
+ "metricId": 41,
482
+ "dimensionsHash": "a8f21c0b1d2e",
483
+ "dimensions": { "eventType": "share", "situation": "external" },
484
+ "count": 42,
485
+ "createdAt": "...",
486
+ "updatedAt": "..."
487
+ }
488
+ ```
489
+
490
+ Indexes (created automatically on first write):
491
+
492
+ - `uniq_cell_identity` (unique, on the full identity tuple)
493
+ - `by_timeframe` — for range scans
494
+ - `by_subject` — for subject filters
495
+ - `by_metric` — for metric filters
496
+
497
+ `countex_dict` — the per-scope subject/metric dictionary, with monotonic ids allocated via `countex_seq`. Allocation is **concurrency-safe by construction**: read existing → atomic sequence `$inc` → `insertMany({ ordered: false })` → on duplicate-key, re-read. See [`docs/guides/adapter-mongo.md`](docs/guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency" and [`docs/internals/internal-storage-mongo.md`](docs/internals/internal-storage-mongo.md).
498
+
499
+ `dimensionsHash` is canonicalized so any two workers produce the same hash for the same logical dimension tuple: keys sorted lexicographically, values type-tagged (`n: / b: / i: / s:`), serialized, hashed (SHA-1, truncated to 12 hex). See [`docs/internals/internal-dimension-hash.md`](docs/internals/internal-dimension-hash.md).
500
+
501
+ ### Redis layout
502
+
503
+ ```
504
+ countex:{scope}:{counterId}:{timeframeType}:{timeframeKey} → HASH
505
+ field = "{subjectType}|{subjectId}|{metricId}|{dimensionsHash}"
506
+ value = count
507
+
508
+ countex:{scope}:{counterId}:{timeframeType}:{timeframeKey}:cov → HASH (coverage meta)
509
+ status = complete | partial | rebuilding (see BucketCoverageMeta; reads may infer missing)
510
+ lastWarmAt = epoch ms (string)
511
+ mapVersion = optional opaque integer (string)
512
+ schemaVersion = cache layout version (string)
513
+ cellCount = cells last warmed into this bucket (string, when known)
514
+
515
+ countex:{scope}:{counterId}:...:{timeframeKey}:meta:{field} → JSON of dimensions
516
+
517
+ countex:{scope}:dedupe:{eventId} → "1" with TTL
518
+ ```
519
+
520
+ One hash per `(scope, counterId, bucket)` keeps Redis key counts small and per-bucket reads to a single `HGETALL`. **Do not** treat “cell hash is non-empty” as full cache coverage — use the **`:cov`** hash (`status === complete` after `warm`, `partial` after incremental `applyDeltas`, `rebuilding` during cache rebuild). See [`docs/guides/adapter-redis.md`](docs/guides/adapter-redis.md) and [`docs/internals/internal-storage-redis.md`](docs/internals/internal-storage-redis.md).
521
+
522
+ ---
523
+
524
+ ## Query API
525
+
526
+ ```ts
527
+ const result = await countex.query({
528
+ scope: "org_123",
529
+ counterId: "access_by_subject_metric",
530
+
531
+ timeframe: {
532
+ type: "day",
533
+ from: "2026-05-01",
534
+ to: "2026-05-08",
535
+ },
536
+
537
+ // Filters — all optional (subject.ids / metrics: internal numbers and/or dictionary source strings)
538
+ subject: { type: "actor", ids: [10001, 10002, "user_uuid_a"] },
539
+ metrics: [41, 42, "PCI"],
540
+ dimensions: { eventType: ["share", "download"], situation: ["external"] },
541
+
542
+ // Shape
543
+ groupBy: ["subject", "metric", "dim:eventType"],
544
+ format: "table", // "cells" | "table" | "matrix" | "timeseries"
545
+ orderBy: "-count",
546
+ limit: 100,
547
+
548
+ // Optional: use dictionary source keys in grouped keys / cells instead of numeric ids only
549
+ resolveLabels: true,
550
+
551
+ // Diagnostics
552
+ bypassCache: false,
553
+ signal: AbortSignal.timeout(5_000),
554
+ });
555
+ ```
556
+
557
+ **Filters:** `subject.ids` and `metrics` may mix **internal numeric ids** (from ingest) and **string source keys** (UUIDs, labels, and so on). String subject keys require `subject.type`. Set **`resolveLabels: true`** to return source keys in grouped `keys.subject` / `keys.metric`, and `subjectSourceKey` / `metricSourceKey` on **`format: "cells"`** rows. See [`docs/guides/querying.md`](docs/guides/querying.md).
558
+
559
+ `explainQuery` runs the planner **without** materializing cells. It returns a **`QueryPlan`**: expanded `bucketsRequested`, `cacheEligible` / `cacheReason`, predicted `durableFallbackPredicted`, `cacheBucketsExpectedHit` (when the cache adapter supports `probeBucketCoverage`), heuristic `plannedCellsEstimate`, and cost hints (`expensive`, `groupByCost`). Use it to preflight dashboards and expensive `groupBy` shapes.
560
+
561
+ ```ts
562
+ const plan = await countex.explainQuery({
563
+ scope: "org_123",
564
+ counterId: "access_by_subject_metric",
565
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
566
+ groupBy: ["subject", "metric"],
567
+ format: "table",
568
+ });
569
+ ```
570
+
571
+ Result:
572
+
573
+ ```ts
574
+ {
575
+ format: "table",
576
+ rows: [
577
+ { keys: { subject: 10001, metric: 41, "dim:eventType": "share" }, count: 142 },
578
+ // With resolveLabels: true (and known dictionary rows), subject/metric keys are strings:
579
+ // { keys: { subject: "user_42", metric: "PCI", "dim:eventType": "share" }, count: 142 },
580
+ ...
581
+ ],
582
+ total: 1234,
583
+ source: "cache" | "durable" | "merged",
584
+ }
585
+ ```
586
+
587
+ String filters are resolved to internal ids before reads; unknown source keys are omitted from the filter. With **`resolveLabels: true`**, **`format: "cells"`** rows include **`subjectSourceKey`** and **`metricSourceKey`** in addition to numeric ids.
588
+
589
+ `groupBy` accepts:
590
+ - `"subject"`, `"subjectType"`, `"metric"`, `"timeframeKey"`
591
+ - `"dim:NAME"` — group by a dimension value
592
+
593
+ ---
594
+
595
+ ## Insights and advisors
596
+
597
+ Countex can optionally analyze stored counters and measures to produce higher-level insights. The core engine answers “what happened?” by maintaining sparse aggregate cells. The insights layer answers “what matters?” by ranking, comparing periods, detecting anomalies, forecasting usage, and emitting evidence-backed recommendations.
598
+
599
+ Typical uses:
600
+
601
+ - Top / bottom N subjects, metrics, or dimensions; top/bottom percent of groups; Pareto and contribution views
602
+ - Period-over-period comparison and simple trends on timeseries
603
+ - Explainable anomaly detection (robust z-score, thresholds, missing activity)
604
+ - Rule evaluation with normalized recommendations
605
+ - Cost / budget / max-token style advisors (generic measures only — no domain lock-in)
606
+ - Map and cardinality health when you pass the counter map into `createCountexInsights`
607
+ - Resource projections from observed cells via `estimateResourcesFromActuals` (requires the durable adapter reference)
608
+
609
+ **Safety:** Insights honor `maxBuckets` (aligned with the core 10,000 bucket cap) and `maxGroups`. Grouped queries load all matching cells for the timeframe before aggregation, so narrow filters when possible.
610
+
611
+ **Sparse counters:** Bottom-N style rankings omit zero rows by default; Countex only stores cells that received increments.
612
+
613
+ ```ts
614
+ import { createCountexInsights } from "@x12i/countex/insights";
615
+
616
+ const insights = createCountexInsights({
617
+ countex,
618
+ durable, // optional; required for estimateResourcesFromActuals → streamCells
619
+ mapsByCounterId: new Map(maps.map((m) => [m.counterId, m])),
620
+ });
621
+
622
+ const topCostDrivers = await insights.topPercent({
623
+ scope: "org_123",
624
+ counterId: "llm_usage",
625
+ timeframe: { type: "month", from: "2026-05", to: "2026-05" },
626
+ groupBy: ["subject"],
627
+ measure: "costMicros.sum",
628
+ percent: 20,
629
+ });
630
+ ```
631
+
632
+ Further reading: [`docs/guides/insights-overview.md`](docs/guides/insights-overview.md), [`docs/guides/insights-ranking.md`](docs/guides/insights-ranking.md), [`docs/guides/insights-comparison-and-trends.md`](docs/guides/insights-comparison-and-trends.md), [`docs/guides/insights-anomalies-and-rules.md`](docs/guides/insights-anomalies-and-rules.md), [`docs/guides/insights-forecasting-and-budget.md`](docs/guides/insights-forecasting-and-budget.md), [`docs/guides/insights-advisors-and-optimization.md`](docs/guides/insights-advisors-and-optimization.md). Deeper topic notes also live under `docs/INSIGHTS*.md` for maintainers.
633
+
634
+ CLI (after `npm run build` / install):
635
+
636
+ ```bash
637
+ npx countex insights:top ./path/to/insight-context.mjs --scope org_123 --counter my_counter
638
+ # TypeScript configs: node --import=tsx node_modules/@x12i/countex/dist/cli.js insights:pareto ./ctx.ts
639
+ ```
640
+
641
+ ---
642
+
643
+ ## Events
644
+
645
+ The Countex client is an `EventEmitter`. Subscribe to observe what's happening:
646
+
647
+ ```ts
648
+ countex.on("flush", ({ cells, durationMs }) => {});
649
+ countex.on("flush_error", ({ error }) => {});
650
+ countex.on("cardinality_warning", ({ kind, count, limit })=> {});
651
+ countex.on("overflow", ({ kind, sourceKey }) => {});
652
+ countex.on("dropped", ({ reason }) => {});
653
+ countex.on("duplicate", ({ scopeId, eventId }) => {});
654
+ ```
655
+
656
+ All event payloads are typed via `CountexEvents`.
657
+
658
+ ---
659
+
660
+ ## Operational notes
661
+
662
+ ### Buffering and flushing
663
+
664
+ Each Countex instance keeps only a delta buffer in memory:
665
+
666
+ ```
667
+ Map<cellKey, delta>
668
+ ```
669
+
670
+ Flush triggers (any of):
671
+
672
+ - `flush.everyMs` timer (default 60s)
673
+ - `flush.maxBufferedCells` reached (default 100k)
674
+ - `flush.maxBufferedEvents` reached (default 50k)
675
+ - `shutdown()` if `flushOnShutdown` is true (default)
676
+ - explicit `flush()`
677
+
678
+ Flush is **two-phase**: durable first, cache second. If durable fails, the snapshot is merged back into the buffer — counts are not lost. If cache fails after durable succeeded, it's logged and continues; the next read warms cache from durable.
679
+
680
+ At most one flush runs at a time. Concurrent callers receive the same in-flight promise.
681
+
682
+ ### Cardinality protection
683
+
684
+ A bad mapping (e.g. a free-form URL as a dimension) can balloon the dictionary. Countex tracks per-`(scope, kind, qualifier)` how many distinct keys it has minted, and applies a policy when limits hit:
685
+
686
+ - `overflow` (default) — rewrite the source key to `__OTHER__`. Emits `overflow`.
687
+ - `drop` — silently drop the increment. Emits `dropped`.
688
+ - `error` — throw `CardinalityLimitError`.
689
+
690
+ Defaults are conservative; tune them per workload. A warning fires at 80% of the limit (`cardinality_warning`).
691
+
692
+ ### Idempotency
693
+
694
+ Optional and per-map. Requires a cache adapter that implements `markEventSeen` (the included `RedisCache` and `MemoryCache` both do).
695
+
696
+ ```ts
697
+ idempotency: {
698
+ enabled: true,
699
+ eventIdPath: "$.event.id",
700
+ ttlSeconds: 86_400,
701
+ }
702
+ ```
703
+
704
+ Dedupe keys are namespaced by `counterId`, so different maps with their own idempotency don't collide on shared event ids.
705
+
706
+ ### Late events
707
+
708
+ Bucket selection always uses the **event time**, never the ingestion time. Late-arriving events go into the correct historical bucket — `$inc` doesn't care about ordering.
709
+
710
+ ### Concurrency safety
711
+
712
+ All writes are atomic per cell:
713
+
714
+ - Mongo: `bulkWrite` of `updateOne` with `$inc` and `upsert: true`
715
+ - Redis: `HINCRBY`
716
+
717
+ 20+ Countex workers can safely run in parallel against the same scope. Reads are eventually consistent across the two stores; if Redis is behind durable, the planner will detect partial coverage and read durable.
718
+
719
+ ### Backup / verify / restore / rebuild
720
+
721
+ Top-level free functions and a `countex.backup` namespace bound to your engine:
722
+
723
+ ```ts
724
+ import {
725
+ exportCells,
726
+ computeCellsFingerprint,
727
+ createBackupManifest,
728
+ verifyBackupAgainstManifest,
729
+ restoreSnapshotCells,
730
+ rebuildCache,
731
+ COUNTEX_CELL_SCHEMA_VERSION,
732
+ } from "@x12i/countex";
733
+
734
+ const filter = {
735
+ scope: "org_123",
736
+ counterId: "access_by_subject_metric",
737
+ timeframe: { type: "day", from: "2026-02-01", to: "2026-05-08" },
738
+ };
739
+
740
+ // Stream all cells in a window — bounded async iteration.
741
+ for await (const cell of exportCells(durable, filter)) {
742
+ // ndjson, S3, whatever you want
743
+ }
744
+
745
+ // Fingerprint + manifest for cold-storage verification.
746
+ const fp = await countex.backup.fingerprint(filter); // { cellCount, contentSha256, countSum }
747
+ const manifest = countex.backup.createManifest(filter, fp); // { manifestVersion, schemaVersion, filter, … }
748
+
749
+ // Verify a manifest against the live durable adapter.
750
+ const ok = await countex.backup.verify(manifest);
751
+ if (!ok.ok) throw new Error(ok.detail);
752
+
753
+ // Restore. Modes: "validate-only" | "replace-window" (default) | "merge-deltas" (reserved).
754
+ const { restored, mode } = await countex.backup.restore(
755
+ countex.backup.export(filter),
756
+ { mode: "replace-window" },
757
+ );
758
+
759
+ // Re-warm cache from durable. Each warmed bucket's :cov flips to status=complete.
760
+ await countex.backup.rebuildCache(filter);
761
+ ```
762
+
763
+ See [`docs/guides/backup-export.md`](docs/guides/backup-export.md) for the full recipe and manifest layout.
764
+
765
+ ---
766
+
767
+ ## Docker integration testing
768
+
769
+ Countex ships a Docker-based stack (MongoDB, Redis, MinIO) and integration tests under `tests/integration/docker/` that exercise real adapters: dictionary allocation under concurrency, Mongo **`lookupIds` / `reverseLookupIds`**, an end-to-end **`query()`** with **source-key filters** and **`resolveLabels`**, multi-worker ingestion, Redis cache coverage and durable fallback, `rebuildCache`, simulation fingerprints against a memory reference, and backup / manifest / verify / restore to S3-compatible storage (via the AWS SDK in devDependencies).
770
+
771
+ **One-shot local run** (starts services, runs tests with default localhost URLs, tears volumes down):
772
+
773
+ ```bash
774
+ npm run test:e2e
775
+ ```
776
+
777
+ **Host-runner** (services in Docker, tests on the host):
778
+
779
+ ```bash
780
+ npm run docker:up
781
+ export COUNTEX_MONGO_URI='mongodb://countex:countex@127.0.0.1:27017/admin'
782
+ export COUNTEX_REDIS_URL='redis://127.0.0.1:6379'
783
+ export COUNTEX_S3_ENDPOINT='http://127.0.0.1:9000'
784
+ export COUNTEX_S3_REGION='us-east-1'
785
+ export COUNTEX_S3_ACCESS_KEY_ID='countex'
786
+ export COUNTEX_S3_SECRET_ACCESS_KEY='countex-password'
787
+ export COUNTEX_S3_BUCKET='countex-backups'
788
+ export COUNTEX_INTEGRATION=1
789
+ npm run test:integration
790
+ npm run docker:down
791
+ ```
792
+
793
+ Optional tuning: `COUNTEX_TEST_PROFILE`, `COUNTEX_TEST_EVENTS`, `COUNTEX_TEST_WORKERS`. Stress-style full pipeline (larger simulation + S3 round-trip): set `COUNTEX_TEST_FULL=1` (requires S3 env as above).
794
+
795
+ **Container-runner** (CI parity): `npm run test:integration:docker` builds `docker/test-runner.Dockerfile` and runs `npm run test:integration:exec` with service hostnames injected by Compose.
796
+
797
+ Reports are written to `artifacts/integration/report.json` and `artifacts/integration/report.md` after `npm run test:integration`.
798
+
799
+ ---
800
+
801
+ ## Errors
802
+
803
+ All errors thrown by Countex extend `CountexError`:
804
+
805
+ | Class | When |
806
+ |--------------------------|------|
807
+ | `MappingError` | Invalid mapping config (caught at engine startup). |
808
+ | `ExtractionError` | Malformed path or value during extraction. |
809
+ | `CardinalityLimitError` | Cardinality limit hit when policy is `error`. |
810
+ | `StorageError` | Wrapping a storage-adapter failure. |
811
+
812
+ ---
813
+
814
+ ## Versioning & scope
815
+
816
+ > **Additive counts are not distinct counts.** If `user_42` appears in 100 events, `count = 100`, not 1 unique user. Counter cells store sums of deltas, so a query result of `count = 100` from a subject-grouped view means "100 increments hit this subject", not "100 distinct subjects". For exact distinct subject counts, group by `["subjectType", "subject"]` and use `rows.length` after retrieving every matching cell — see [`docs/guides/querying.md`](docs/guides/querying.md) §"Distinct subjects / metrics as row cardinality". Approximate distinct counts (HLL/sets) are a planned opt-in extension, not core.
817
+
818
+ `v0.x` is for stabilizing the public API. Once it's hardened:
819
+
820
+ - exact unique counts (HLL / sets) are an opt-in extension, not core
821
+ - complex expression deltas are not in core
822
+ - distributed locks / exactly-once: not Countex's job; durable storage is the source of truth and `$inc` makes worker-count irrelevant
823
+
824
+ ---
825
+
826
+ ## Repository layout
827
+
828
+ | Path | Role |
829
+ |------|------|
830
+ | `src/index.ts` | Public exports (`createCountex`, types, errors). |
831
+ | `src/core/` | Engine: ingest pipeline, buffer, extraction, mapping validation, time buckets, cardinality. |
832
+ | `src/query/` | Query planner (`planQuery` + `executeQuery`) and result formatting. |
833
+ | `src/storage/` | Adapters: `memory` (in-process), `mongo` (durable), `redis` (cache); shared coverage helpers. |
834
+ | `src/utils/` | JSONPath subset, hashing, logging. |
835
+ | `src/backup/` | Export, fingerprint, manifest, verify, restore, cache rebuild helpers. |
836
+ | `src/simulation/` | Simulation engine, profiles, expected aggregates, resource reports. |
837
+ | `src/insights/` | Ranking, comparison, anomaly detection, advisors, recommendations. |
838
+ | `src/cli.ts` | CLI commands for simulation, insights, export, and operational helpers. |
839
+ | `examples/` | Runnable demos (`npx tsx examples/basic.ts`). |
840
+ | `docs/` | Deployment guide, tutorial, guides, internals, and use cases (published in the npm package). |
841
+ | `tests/unit/` | Fast Node test runner tests (`npm test`). |
842
+ | `tests/integration/` | Opt-in live tests against real services (`npm run test:integration`). |
843
+
844
+ ---
845
+
846
+ ## Publishing (npm)
847
+
848
+ This package is scoped as **`@x12i/countex`** and is configured for **public** registry access (`publishConfig.access` in `package.json`).
849
+
850
+ 1. Ensure you are logged in and allowed to publish under the **`@x12i`** scope on [npm](https://www.npmjs.com/).
851
+ 2. Put auth tokens only in **`~/.npmrc`** — do not commit project `.npmrc` with credentials (it is listed in `.gitignore`).
852
+ 3. From the repo root:
853
+
854
+ ```bash
855
+ npm test
856
+ npm publish
857
+ ```
858
+
859
+ `prepublishOnly` runs a clean build so the tarball always contains a fresh `dist/`.
860
+
861
+ The **`docs/`** directory (`DEPLOYMENT.md`, `END_TO_END_EXAMPLE.md`) is part of the published package (`files` in `package.json`).
862
+
863
+ ---
864
+
865
+ ## License
866
+
867
+ MIT — see `LICENSE`.