@x12i/countex 1.8.3 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +866 -858
  3. package/dist/cli.js +6 -6
  4. package/dist/index.d.ts +1 -0
  5. package/dist/index.d.ts.map +1 -1
  6. package/dist/index.js +1 -0
  7. package/dist/index.js.map +1 -1
  8. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  9. package/dist/insights/advisors/resource-advisor.js +5 -1
  10. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  11. package/dist/insights/types.d.ts +4 -0
  12. package/dist/insights/types.d.ts.map +1 -1
  13. package/dist/storage/postgres.d.ts +104 -0
  14. package/dist/storage/postgres.d.ts.map +1 -0
  15. package/dist/storage/postgres.js +1023 -0
  16. package/dist/storage/postgres.js.map +1 -0
  17. package/dist/types/index.d.ts +11 -5
  18. package/dist/types/index.d.ts.map +1 -1
  19. package/docs/DEPLOYMENT.md +399 -375
  20. package/docs/END_TO_END_EXAMPLE.md +316 -316
  21. package/docs/INSIGHTS.md +59 -59
  22. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  23. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  24. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  25. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  26. package/docs/guides/README.md +36 -34
  27. package/docs/guides/adapter-memory.md +60 -60
  28. package/docs/guides/adapter-mongo.md +64 -62
  29. package/docs/guides/adapter-postgres.md +90 -0
  30. package/docs/guides/adapter-redis.md +89 -89
  31. package/docs/guides/backup-export.md +135 -135
  32. package/docs/guides/cardinality.md +50 -50
  33. package/docs/guides/cli.md +53 -53
  34. package/docs/guides/client-events.md +65 -65
  35. package/docs/guides/counter-maps.md +160 -160
  36. package/docs/guides/idempotency.md +39 -39
  37. package/docs/guides/ingestion-and-flush.md +67 -67
  38. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  39. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  40. package/docs/guides/insights-comparison-and-trends.md +47 -47
  41. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  42. package/docs/guides/insights-overview.md +82 -82
  43. package/docs/guides/insights-ranking.md +81 -81
  44. package/docs/guides/measures.md +71 -71
  45. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  46. package/docs/guides/preview.md +92 -92
  47. package/docs/guides/querying.md +178 -178
  48. package/docs/guides/simulation-profiles.md +33 -33
  49. package/docs/guides/simulation-resource-estimation.md +31 -30
  50. package/docs/guides/simulation.md +36 -36
  51. package/docs/guides/timeframes-and-bucketing.md +55 -55
  52. package/docs/internals/README.md +21 -21
  53. package/docs/internals/internal-architecture-overview.md +65 -65
  54. package/docs/internals/internal-backup-stream.md +23 -23
  55. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  56. package/docs/internals/internal-cardinality.md +30 -30
  57. package/docs/internals/internal-dimension-hash.md +34 -34
  58. package/docs/internals/internal-idempotency.md +36 -36
  59. package/docs/internals/internal-ingest-pipeline.md +52 -52
  60. package/docs/internals/internal-insights-layer.md +36 -36
  61. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  62. package/docs/internals/internal-measures-and-scales.md +35 -35
  63. package/docs/internals/internal-query-path.md +46 -46
  64. package/docs/internals/internal-storage-memory.md +13 -13
  65. package/docs/internals/internal-storage-mongo.md +44 -44
  66. package/docs/internals/internal-storage-postgres.md +32 -0
  67. package/docs/internals/internal-storage-redis.md +63 -63
  68. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  69. package/docs/use-cases/README.md +56 -56
  70. package/docs/use-cases/audit-security-log-counts.md +550 -550
  71. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  72. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  73. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  74. package/docs/use-cases/processing-workflow-activity.md +165 -165
  75. package/docs/use-cases/product-feature-usage.md +240 -240
  76. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  77. package/package.json +184 -169
package/README.md CHANGED
@@ -1,858 +1,866 @@
1
- # @x12i/countex
2
-
3
- > Generic sparse counter matrices for JSON events.
4
- > Dynamic counting, aggregation, and retrieval from mapped JSON inputs.
5
-
6
- `@x12i/countex` is a **generic mapped-event counter**: a source-agnostic counting engine. You give it any JSON event, a declarative mapping config, and storage adapters; it projects events into sparse counter cells, batches deltas locally, and persists them atomically to a durable store (MongoDB) and a hot cache (Redis).
7
-
8
- It does **not** know about users, organizations, documents, labels, or events. It only knows generic primitives: **scope, subject, metric, dimensions, timeframe, delta**. Whatever your domain calls those things — Countex doesn't care.
9
-
10
- **Terminology:** `metric` is the generic Countex name for the thing being counted or categorized. In your domain that may be a label, status, event name, operation, outcome, class, model result, or business category — the `metrics` array in a map simply lists JSON paths that resolve to those categories. Storage and APIs refer to dictionary **`sourceKey`** (the string extracted from the event); avoid confusing that with application-level UUIDs unless your map uses UUIDs as keys.
11
-
12
- **Process memory vs retained state:** Each worker keeps **unflushed deltas** in an in-memory buffer. After flush, authoritative aggregates live in **durable** storage (MongoDB); **Redis** holds a hot cache. Restarting a worker loses only buffered deltas that were not flushed yet — tune flush intervals accordingly.
13
-
14
- ```
15
- event JSON ──► map ──► raw increments ──► buffer ──► flush ──► Mongo (durable)
16
- └───► Redis (hot cache)
17
-
18
- ◄── query ◄── Redis (full coverage) or ◄── Mongo (fallback) ──◄
19
- ```
20
-
21
- ---
22
-
23
- ## Install
24
-
25
- ```bash
26
- npm install @x12i/countex
27
- # Bring your own clients — peer deps, both optional:
28
- npm install mongodb ioredis
29
- ```
30
-
31
- Node 18+. Zero runtime dependencies in the core package.
32
-
33
- ### Documentation shipped with the package
34
-
35
- These guides are included in the published tarball under `docs/` and exposed as stable import subpaths (resolve to `.md` files for editors and tooling):
36
-
37
- | Topic | Package subpath | File |
38
- |-------|-----------------|------|
39
- | Release & production deployment | `@x12i/countex/docs/deployment` | `docs/DEPLOYMENT.md` |
40
- | Tutorial — install → map → ingest → query | `@x12i/countex/docs/end-to-end-example` | `docs/END_TO_END_EXAMPLE.md` |
41
- | Feature guides index | `@x12i/countex/docs/guides` | `docs/guides/README.md` |
42
- | Counter maps | `@x12i/countex/docs/guides/counter-maps` | `docs/guides/counter-maps.md` |
43
- | Preview a map (dry-run) | `@x12i/countex/docs/guides/preview` | `docs/guides/preview.md` |
44
- | Ingestion & flush | `@x12i/countex/docs/guides/ingestion-and-flush` | `docs/guides/ingestion-and-flush.md` |
45
- | Timeframes | `@x12i/countex/docs/guides/timeframes-and-bucketing` | `docs/guides/timeframes-and-bucketing.md` |
46
- | Measures | `@x12i/countex/docs/guides/measures` | `docs/guides/measures.md` |
47
- | Querying | `@x12i/countex/docs/guides/querying` | `docs/guides/querying.md` |
48
- | Memory adapters | `@x12i/countex/docs/guides/adapter-memory` | `docs/guides/adapter-memory.md` |
49
- | Mongo durable | `@x12i/countex/docs/guides/adapter-mongo` | `docs/guides/adapter-mongo.md` |
50
- | Redis cache | `@x12i/countex/docs/guides/adapter-redis` | `docs/guides/adapter-redis.md` |
51
- | Cardinality | `@x12i/countex/docs/guides/cardinality` | `docs/guides/cardinality.md` |
52
- | Idempotency | `@x12i/countex/docs/guides/idempotency` | `docs/guides/idempotency.md` |
53
- | Backup export | `@x12i/countex/docs/guides/backup-export` | `docs/guides/backup-export.md` |
54
- | Client events | `@x12i/countex/docs/guides/client-events` | `docs/guides/client-events.md` |
55
- | Insights overview | `@x12i/countex/docs/guides/insights-overview` | `docs/guides/insights-overview.md` |
56
- | Insights ranking | `@x12i/countex/docs/guides/insights-ranking` | `docs/guides/insights-ranking.md` |
57
- | Insights comparison & trends | `@x12i/countex/docs/guides/insights-comparison-and-trends` | `docs/guides/insights-comparison-and-trends.md` |
58
- | Insights anomalies & rules | `@x12i/countex/docs/guides/insights-anomalies-and-rules` | `docs/guides/insights-anomalies-and-rules.md` |
59
- | Insights forecasting & budget | `@x12i/countex/docs/guides/insights-forecasting-and-budget` | `docs/guides/insights-forecasting-and-budget.md` |
60
- | Insights advisors | `@x12i/countex/docs/guides/insights-advisors-and-optimization` | `docs/guides/insights-advisors-and-optimization.md` |
61
- | CLI | `@x12i/countex/docs/guides/cli` | `docs/guides/cli.md` |
62
- | Simulation runtime (JS API) | `@x12i/countex/simulation` | `dist/simulation/` |
63
- | Simulation overview | `@x12i/countex/docs/guides/simulation` | `docs/guides/simulation.md` |
64
- | Simulation profiles | `@x12i/countex/docs/guides/simulation-profiles` | `docs/guides/simulation-profiles.md` |
65
- | Simulation resource estimation | `@x12i/countex/docs/guides/simulation-resource-estimation` | `docs/guides/simulation-resource-estimation.md` |
66
-
67
- **Use cases** (technical playbooks: contracts, fan-out math, exact `QuerySpec`s, production wiring, verification — see [`docs/use-cases/README.md`](docs/use-cases/README.md)):
68
-
69
- | Topic | Package subpath | File |
70
- |-------|-----------------|------|
71
- | Use cases index | `@x12i/countex/docs/use-cases` | `docs/use-cases/README.md` |
72
- | Audit & security logs (volume, situations, distinct entities, logical event totals) | `@x12i/countex/docs/use-cases/audit-security-log-counts` | `docs/use-cases/audit-security-log-counts.md` |
73
- | Product & feature activity | `@x12i/countex/docs/use-cases/product-feature-usage` | `docs/use-cases/product-feature-usage.md` |
74
- | LLM cost & token metering | `@x12i/countex/docs/use-cases/llm-cost-and-token-metering` | `docs/use-cases/llm-cost-and-token-metering.md` |
75
- | Multi-tenant isolation | `@x12i/countex/docs/use-cases/multi-tenant-isolation` | `docs/use-cases/multi-tenant-isolation.md` |
76
- | Idempotent webhooks & queues | `@x12i/countex/docs/use-cases/webhook-idempotent-ingestion` | `docs/use-cases/webhook-idempotent-ingestion.md` |
77
- | Ops dashboards & rollups | `@x12i/countex/docs/use-cases/ops-dashboard-daily-rollups` | `docs/use-cases/ops-dashboard-daily-rollups.md` |
78
- | Processing & workflow activity | `@x12i/countex/docs/use-cases/processing-workflow-activity` | `docs/use-cases/processing-workflow-activity.md` |
79
-
80
- **Internals** (how the engine works — diagrams and traces) live under `docs/internals/` in the tarball; browse [`docs/internals/README.md`](docs/internals/README.md).
81
-
82
- After install: `node_modules/@x12i/countex/docs/`.
83
-
84
- ---
85
-
86
- ## The mental model
87
-
88
- Countex turns mapped JSON events into sparse aggregate cells.
89
-
90
- A counter cell is uniquely identified by:
91
-
92
- ```
93
- (scope, counterId, timeframeType, timeframeKey, subjectType, subjectId, metricId, dimensionsHash)
94
- ```
95
-
96
- Each event can produce one or many cell increments. A counter increment is defined by:
97
-
98
- - **scope** — your tenant / org / account boundary
99
- - **counterId** — which counter family this is (e.g. `access_by_subject_metric`)
100
- - **timeframe** — `minute` / `hour` / `day` / `week` / `month` / `quarter` / `year` / `all`
101
- - **subject** — the thing being counted against (e.g. an actor or a target)
102
- - **metric** — the count category / label
103
- - **dimensions** — arbitrary additional breakdowns
104
- - **delta** — `+1`, `-1`, or any integer
105
-
106
- Countex buffers increments locally, flushes atomic deltas to durable + cache, and serves queries from cache with durable fallback.
107
-
108
- ### Measures and rates
109
-
110
- Maps may declare optional **`measures`**: named numeric fields per event (cost, tokens, latency, …) with aggregations `sum`, `count`, `min`, and `max`. **`avg`** is computed at query time when both `sum` and `count` exist on the measure. Use **`scale`** on a measure for fixed-point storage (e.g. cents). Extracted values attach to each fan-out increment the same way as **`delta`** (see RFC-style docs for fan-out semantics).
111
-
112
- Queries accept **`measures`** (explicit list or `"*"`), **`orderBy`** on a measure aggregation, and optional **`rate`** (count or a measure’s sum divided by elapsed time in the queried window; for `format: "timeseries"`, per bucket). **`rate`** is invalid for timeframe **`all`**.
113
-
114
- **Redis cache** stores **sum** and **count** per measure only; **min** / **max** are durable-backed. If a query needs min/max (including `orderBy` on them, or requesting measures that maintain min/max when using `"*"`), the planner reads from **durable** first so cache cannot drift.
115
-
116
- **`bucketRange`** caps at **10,000** buckets; at **`minute`** resolution that is about **6.94 days** — keep queries within that bound or narrow the range.
117
-
118
- ### Percentiles & histograms
119
-
120
- Core cells store **additive** measure aggregates (`sum / count / min / max`) per sparse cell. They do **not** store distributions, so percentile-style answers (p50/p90/p95/p99) can only come from min/max approximation or from histogram/sketch storage you add yourself. Insights advisors that mention percentile-shaped recommendations are heuristic; see [`docs/guides/insights-overview.md`](docs/guides/insights-overview.md) §"Distributions" and [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) "Architecture decisions".
121
-
122
- ---
123
-
124
- ## What kinds of events Countex is for
125
-
126
- Countex is not tied to one event source or domain. It is a **generic mapped-event counter**: any JSON-like input stream where useful aggregates can be produced from paths in the event.
127
-
128
- Typical sources include:
129
-
130
- | Source family | Examples | Common Countex projection |
131
- |---|---|---|
132
- | Audit and security logs | Microsoft 365 audit logs, Google Workspace logs, Okta logs, cloud audit logs | actor/resource/action/situation/severity |
133
- | Product and user activity | Woopra, Segment, PostHog-style events, application activity logs | user/account/feature/event/page/funnel |
134
- | Processing and workflow activity | ETL runs, background jobs, agent graphs, workflow engines, queue processors | job/task/node/status/error/runtime |
135
- | LLM and AI service usage | Model gateway logs, OpenAI/Anthropic calls, internal LLM services | app/user/model/provider/tokens/cost/latency/result |
136
- | Business/domain events | CRM, support tickets, payments, commerce, lifecycle events | customer/object/state/category/outcome |
137
-
138
- Countex only needs a map. The map tells Countex where to find:
139
-
140
- - the scope boundary
141
- - the event time
142
- - the subjects
143
- - the metrics
144
- - the dimensions
145
- - the delta/count
146
- - optional numeric measures such as cost, tokens, latency, or duration
147
-
148
- The original event can come from a log, webhook, queue message, analytics event, system activity, processing trace, or generated test fixture. Countex does not care about the event source as long as the map can project it into generic counter primitives.
149
-
150
- ### Audit-log style event
151
-
152
- ```json
153
- {
154
- "tenant": { "id": "org_123" },
155
- "event": {
156
- "id": "audit_001",
157
- "createdAt": "2026-05-08T14:33:00Z",
158
- "operation": "FileAccessed",
159
- "situation": "external"
160
- },
161
- "actor": { "id": "user_42", "type": "user" },
162
- "resource": { "id": "file_99", "type": "document" },
163
- "labels": ["PII", "Financial"]
164
- }
165
- ```
166
-
167
- Counts:
168
-
169
- ```text
170
- actor/resource × label × operation/situation/resourceType × timeframe
171
- ```
172
-
173
- ### Product-activity style event
174
-
175
- ```json
176
- {
177
- "account": { "id": "acct_123" },
178
- "user": { "id": "user_42" },
179
- "event": {
180
- "name": "dashboard_viewed",
181
- "createdAt": "2026-05-08T14:33:00Z"
182
- },
183
- "product": {
184
- "feature": "risk_dashboard",
185
- "plan": "enterprise"
186
- }
187
- }
188
- ```
189
-
190
- Counts:
191
-
192
- ```text
193
- user/account × activity metric × feature/plan × timeframe
194
- ```
195
-
196
- ### LLM-processing style event
197
-
198
- ```json
199
- {
200
- "scope": { "id": "org_123" },
201
- "request": {
202
- "id": "llm_req_001",
203
- "createdAt": "2026-05-08T14:33:00Z",
204
- "status": "success"
205
- },
206
- "app": { "id": "graphs-studio" },
207
- "model": {
208
- "provider": "openai",
209
- "name": "gpt-4.1"
210
- },
211
- "usage": {
212
- "inputTokens": 1200,
213
- "outputTokens": 350,
214
- "costCents": 4,
215
- "latencyMs": 1800
216
- }
217
- }
218
- ```
219
-
220
- Counts/measures:
221
-
222
- ```text
223
- app/model/provider/status × timeframe
224
- sum inputTokens
225
- sum outputTokens
226
- sum costCents
227
- avg latencyMs
228
- count requests
229
- ```
230
-
231
- ---
232
-
233
- ## Quick start
234
-
235
- ```ts
236
- import { createCountex } from "@x12i/countex";
237
- import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory";
238
-
239
- const accessMap = {
240
- counterId: "access_by_subject_metric",
241
-
242
- scope: { path: "$.tenant.id" },
243
- eventTime: { path: "$.event.createdAt" },
244
-
245
- subjects: [
246
- { type: "actor", path: "$.actor.id", required: true },
247
- { type: "target", path: "$.target.id", required: false },
248
- ],
249
-
250
- metrics: [
251
- { path: "$.labels[*]", required: true },
252
- ],
253
-
254
- dimensions: [
255
- { name: "eventType", path: "$.event.type" },
256
- { name: "situation", path: "$.event.situation" },
257
- { name: "targetType", path: "$.target.type" },
258
- ],
259
-
260
- delta: { default: 1 },
261
- timeframes: ["day", "week", "month", "quarter", "year"],
262
- } as const;
263
-
264
- const countex = createCountex({
265
- maps: [accessMap],
266
- durable: new MemoryDurable(),
267
- cache: new MemoryCache(),
268
- });
269
-
270
- await countex.ingest({
271
- tenant: { id: "org_123" },
272
- event: { type: "share", situation: "external", createdAt: "2026-05-08T14:33:00Z" },
273
- actor: { id: "user_42" },
274
- target: { id: "doc_99", type: "document" },
275
- labels: ["PCI", "PII"],
276
- });
277
-
278
- await countex.flush();
279
-
280
- const result = await countex.query({
281
- scope: "org_123",
282
- counterId: "access_by_subject_metric",
283
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
284
- groupBy: ["subject", "metric", "dim:eventType"],
285
- format: "table",
286
- orderBy: "-count",
287
- limit: 100,
288
- });
289
-
290
- console.log(result.rows);
291
-
292
- await countex.shutdown();
293
- ```
294
-
295
- That single ingest produces:
296
-
297
- - 1 actor + 1 target = **2 subjects**
298
- - 2 labels = **2 metrics**
299
- - 5 timeframes
300
- - → **2 × 2 × 5 = 20 cell increments**
301
-
302
- Each cell uses an atomic `$inc` on durable and `HINCRBY` on the cache, so 20 ingest workers can safely run in parallel.
303
-
304
- ### Preview a map (dry-run)
305
-
306
- Use `countex.preview(event, { counterId? })` to inspect what one event would produce without any I/O — no dictionary allocation, no buffering, no writes. Returns one `MapPreviewResult` per matching map, each with `cells: PreviewIncrementCell[]` (subjects, metrics, dimensions, dimensionsHash, timeframe keys, deltas, measures), `extractionWarnings`, and `droppedAfterCardinality`. See [docs/guides/preview.md](docs/guides/preview.md).
307
-
308
- ---
309
-
310
- ## Production setup (Mongo + Redis)
311
-
312
- Do **not** derive Mongo database names directly from raw tenant strings (length, `$`, `/`, and Unicode can break DB naming rules or leak structure). Hash or normalize first:
313
-
314
- ```ts
315
- import { createHash } from "node:crypto";
316
- import { MongoClient } from "mongodb";
317
- import IORedis from "ioredis";
318
- import { createCountex } from "@x12i/countex";
319
- import { MongoDurable } from "@x12i/countex/adapters/mongo";
320
- import { RedisCache } from "@x12i/countex/adapters/redis";
321
-
322
- function hashScopeId(scope: string): string {
323
- return createHash("sha256").update(scope, "utf8").digest("hex").slice(0, 16);
324
- }
325
-
326
- const mongo = new MongoClient(process.env.MONGO_URI!);
327
- await mongo.connect();
328
- const redis = new IORedis(process.env.REDIS_URL!);
329
-
330
- const countex = createCountex({
331
- maps: [accessMap],
332
- durable: new MongoDurable({
333
- client: mongo,
334
- dbResolver: ({ scope }) => `countex_${hashScopeId(scope)}`,
335
- }),
336
- cache: new RedisCache({
337
- client: redis,
338
- keyPrefix: "countex",
339
- // Optional adapter default TTLs; when `retention.redis` is set below, the
340
- // engine resolves those strings to seconds and forwards them via
341
- // `setRetentionPolicy`, overriding adapter defaults for matching timeframes.
342
- }),
343
- retention: {
344
- redis: {
345
- minute: "48h",
346
- hour: "14d",
347
- day: "92d",
348
- week: "52w",
349
- month: "24mo",
350
- quarter: "24mo",
351
- year: "forever",
352
- all: "forever",
353
- },
354
- // Durable TTL / prune is not enforced by the engine — operators apply Mongo
355
- // TTL indexes or scheduled jobs. `retention.mongo` is optional per-timeframe
356
- // documentation (same duration grammar as redis); see types / DEPLOYMENT.
357
- },
358
- flush: {
359
- everyMs: 60_000,
360
- maxBufferedCells: 100_000,
361
- maxBufferedEvents: 50_000,
362
- flushOnShutdown: true,
363
- },
364
- cardinality: {
365
- maxDimensionValuesPerName: 10_000,
366
- onLimit: "overflow",
367
- overflowValue: "__OTHER__",
368
- },
369
- });
370
-
371
- await countex.init();
372
-
373
- process.on("SIGTERM", async () => { await countex.shutdown(); });
374
- ```
375
-
376
- ---
377
-
378
- ## Mapping reference
379
-
380
- A `CounterMap` is the heart of the package. Everything Countex knows about your data is in here.
381
-
382
- | Field | Type | Notes |
383
- |---------------|----------------------------|-------|
384
- | `counterId` | `string` | Stable identifier for this counter family. |
385
- | `scope` | `{ path }` | Path into the event for the tenant/scope id. Required. |
386
- | `eventTime` | `{ path }` | Path to a Date / ISO string / epoch ms. Drives bucket selection. |
387
- | `subjects` | `SubjectMapping[]` | One or many. Each has `type` and `path`; multi-value paths fan out. |
388
- | `metrics` | `MetricMapping[]` | Same fanout semantics. `$.labels[*]` is common. |
389
- | `measures` | `MeasureMapping[]` | Optional numeric fields per event (`sum` / `count` / `min` / `max`); see [Measures](./docs/guides/measures.md). |
390
- | `dimensions` | `DimensionMapping[]` | Named extra breakdowns. Optional by default. |
391
- | `delta` | `{ path?, default? }` | Numeric value at runtime. Default `1`. Negative deltas are fine. |
392
- | `timeframes` | `TimeframeType[]` | Subset of `minute` / `hour` / `day` / `week` / `month` / `quarter` / `year` / `all`. |
393
- | `idempotency` | `IdempotencyPolicy` | Optional; needs a cache adapter that implements `markEventSeen`. |
394
- | `cardinality` | `CardinalityPolicy` | Optional; per-map override of engine-level defaults. |
395
- | `options` | `MappingOptions` | `skipMissingOptionalDimensions` (default true), `normalizeValues` (default true). |
396
-
397
- ### Path syntax
398
-
399
- A small JSONPath subset:
400
-
401
- | Syntax | Meaning |
402
- |-------------------|--------------------------------------------------|
403
- | `$` | Root. |
404
- | `.name` | Property. |
405
- | `['name']` | Property (bracket form, allows arbitrary chars). |
406
- | `[N]` | Numeric array index (negatives allowed). |
407
- | `[*]` | Wildcard — fan out across array elems / object values. |
408
-
409
- Examples:
410
-
411
- ```
412
- $.tenant.id
413
- $.actor.id
414
- $.labels[*]
415
- $.items[*].id
416
- $['weird.key.with.dots']
417
- ```
418
-
419
- ### Fanout
420
-
421
- A single event with 2 subjects and 3 labels (metrics) and 5 selected timeframes produces **2 × 3 × 5 = 30** atomic increments. This is intentional — read your mapping carefully. Countex emits a `dropped` event when cardinality protection bites, so you'll know.
422
-
423
- ---
424
-
425
- ## Storage adapters
426
-
427
- Countex defines two contracts and ships three adapters.
428
-
429
- ```ts
430
- interface DurableStorage {
431
- applyDeltas(deltas: PendingDelta[]): Promise<void>;
432
- readCells(spec: QuerySpec): Promise<CounterCell[]>;
433
- resolveIds(lookups: DictionaryLookup[]): Promise<Map<string, number>>;
434
- streamCells(filter): AsyncIterable<CounterCell>;
435
- init?(): Promise<void>;
436
- close?(): Promise<void>;
437
- }
438
-
439
- interface CacheStorage {
440
- applyDeltas(deltas: PendingDelta[]): Promise<void>;
441
- readCells(spec: QuerySpec): Promise<{ cells: CounterCell[]; coverage: "full" | "partial" | "miss" }>;
442
- warm(cells: CounterCell[]): Promise<void>;
443
- markEventSeen?(scopeId, eventId, ttlSeconds): Promise<boolean>;
444
- init?(): Promise<void>;
445
- close?(): Promise<void>;
446
- }
447
- ```
448
-
449
- ### Built-in adapters
450
-
451
- | Module | Purpose |
452
- |-----------------------------------|--------------------------------------|
453
- | `@x12i/countex/adapters/memory` | In-process durable + cache; tests / local dev. `MemoryCache` + Mongo is single-process only (`init` warns). |
454
- | `@x12i/countex/adapters/mongo` | Durable. Requires `mongodb` peer. |
455
- | `@x12i/countex/adapters/redis` | Hot cache. Requires `ioredis` peer. |
456
- | `@x12i/countex/adapters/durable-cache` | No Redis: each query reads durable once. Low volume / one machine. |
457
-
458
- Bring your own clients — Countex doesn't open or close them. That keeps lifecycle in the host app's hands.
459
-
460
- ### Mongo schema
461
-
462
- `countex_cells`:
463
-
464
- ```json
465
- {
466
- "scopeId": "org_123",
467
- "counterId": "access_by_subject_metric",
468
- "timeframeType": "day",
469
- "timeframeKey": "2026-05-08",
470
- "subjectType": "actor",
471
- "subjectId": 10001,
472
- "metricId": 41,
473
- "dimensionsHash": "a8f21c0b1d2e",
474
- "dimensions": { "eventType": "share", "situation": "external" },
475
- "count": 42,
476
- "createdAt": "...",
477
- "updatedAt": "..."
478
- }
479
- ```
480
-
481
- Indexes (created automatically on first write):
482
-
483
- - `uniq_cell_identity` (unique, on the full identity tuple)
484
- - `by_timeframe` — for range scans
485
- - `by_subject` — for subject filters
486
- - `by_metric` — for metric filters
487
-
488
- `countex_dict` — the per-scope subject/metric dictionary, with monotonic ids allocated via `countex_seq`. Allocation is **concurrency-safe by construction**: read existing → atomic sequence `$inc` → `insertMany({ ordered: false })` → on duplicate-key, re-read. See [`docs/guides/adapter-mongo.md`](docs/guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency" and [`docs/internals/internal-storage-mongo.md`](docs/internals/internal-storage-mongo.md).
489
-
490
- `dimensionsHash` is canonicalized so any two workers produce the same hash for the same logical dimension tuple: keys sorted lexicographically, values type-tagged (`n: / b: / i: / s:`), serialized, hashed (SHA-1, truncated to 12 hex). See [`docs/internals/internal-dimension-hash.md`](docs/internals/internal-dimension-hash.md).
491
-
492
- ### Redis layout
493
-
494
- ```
495
- countex:{scope}:{counterId}:{timeframeType}:{timeframeKey} → HASH
496
- field = "{subjectType}|{subjectId}|{metricId}|{dimensionsHash}"
497
- value = count
498
-
499
- countex:{scope}:{counterId}:{timeframeType}:{timeframeKey}:cov → HASH (coverage meta)
500
- status = complete | partial | rebuilding (see BucketCoverageMeta; reads may infer missing)
501
- lastWarmAt = epoch ms (string)
502
- mapVersion = optional opaque integer (string)
503
- schemaVersion = cache layout version (string)
504
- cellCount = cells last warmed into this bucket (string, when known)
505
-
506
- countex:{scope}:{counterId}:...:{timeframeKey}:meta:{field} → JSON of dimensions
507
-
508
- countex:{scope}:dedupe:{eventId} → "1" with TTL
509
- ```
510
-
511
- One hash per `(scope, counterId, bucket)` keeps Redis key counts small and per-bucket reads to a single `HGETALL`. **Do not** treat “cell hash is non-empty” as full cache coverage — use the **`:cov`** hash (`status === complete` after `warm`, `partial` after incremental `applyDeltas`, `rebuilding` during cache rebuild). See [`docs/guides/adapter-redis.md`](docs/guides/adapter-redis.md) and [`docs/internals/internal-storage-redis.md`](docs/internals/internal-storage-redis.md).
512
-
513
- ---
514
-
515
- ## Query API
516
-
517
- ```ts
518
- const result = await countex.query({
519
- scope: "org_123",
520
- counterId: "access_by_subject_metric",
521
-
522
- timeframe: {
523
- type: "day",
524
- from: "2026-05-01",
525
- to: "2026-05-08",
526
- },
527
-
528
- // Filters — all optional (subject.ids / metrics: internal numbers and/or dictionary source strings)
529
- subject: { type: "actor", ids: [10001, 10002, "user_uuid_a"] },
530
- metrics: [41, 42, "PCI"],
531
- dimensions: { eventType: ["share", "download"], situation: ["external"] },
532
-
533
- // Shape
534
- groupBy: ["subject", "metric", "dim:eventType"],
535
- format: "table", // "cells" | "table" | "matrix" | "timeseries"
536
- orderBy: "-count",
537
- limit: 100,
538
-
539
- // Optional: use dictionary source keys in grouped keys / cells instead of numeric ids only
540
- resolveLabels: true,
541
-
542
- // Diagnostics
543
- bypassCache: false,
544
- signal: AbortSignal.timeout(5_000),
545
- });
546
- ```
547
-
548
- **Filters:** `subject.ids` and `metrics` may mix **internal numeric ids** (from ingest) and **string source keys** (UUIDs, labels, and so on). String subject keys require `subject.type`. Set **`resolveLabels: true`** to return source keys in grouped `keys.subject` / `keys.metric`, and `subjectSourceKey` / `metricSourceKey` on **`format: "cells"`** rows. See [`docs/guides/querying.md`](docs/guides/querying.md).
549
-
550
- `explainQuery` runs the planner **without** materializing cells. It returns a **`QueryPlan`**: expanded `bucketsRequested`, `cacheEligible` / `cacheReason`, predicted `durableFallbackPredicted`, `cacheBucketsExpectedHit` (when the cache adapter supports `probeBucketCoverage`), heuristic `plannedCellsEstimate`, and cost hints (`expensive`, `groupByCost`). Use it to preflight dashboards and expensive `groupBy` shapes.
551
-
552
- ```ts
553
- const plan = await countex.explainQuery({
554
- scope: "org_123",
555
- counterId: "access_by_subject_metric",
556
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
557
- groupBy: ["subject", "metric"],
558
- format: "table",
559
- });
560
- ```
561
-
562
- Result:
563
-
564
- ```ts
565
- {
566
- format: "table",
567
- rows: [
568
- { keys: { subject: 10001, metric: 41, "dim:eventType": "share" }, count: 142 },
569
- // With resolveLabels: true (and known dictionary rows), subject/metric keys are strings:
570
- // { keys: { subject: "user_42", metric: "PCI", "dim:eventType": "share" }, count: 142 },
571
- ...
572
- ],
573
- total: 1234,
574
- source: "cache" | "durable" | "merged",
575
- }
576
- ```
577
-
578
- String filters are resolved to internal ids before reads; unknown source keys are omitted from the filter. With **`resolveLabels: true`**, **`format: "cells"`** rows include **`subjectSourceKey`** and **`metricSourceKey`** in addition to numeric ids.
579
-
580
- `groupBy` accepts:
581
- - `"subject"`, `"subjectType"`, `"metric"`, `"timeframeKey"`
582
- - `"dim:NAME"` — group by a dimension value
583
-
584
- ---
585
-
586
- ## Insights and advisors
587
-
588
- Countex can optionally analyze stored counters and measures to produce higher-level insights. The core engine answers “what happened?” by maintaining sparse aggregate cells. The insights layer answers “what matters?” by ranking, comparing periods, detecting anomalies, forecasting usage, and emitting evidence-backed recommendations.
589
-
590
- Typical uses:
591
-
592
- - Top / bottom N subjects, metrics, or dimensions; top/bottom percent of groups; Pareto and contribution views
593
- - Period-over-period comparison and simple trends on timeseries
594
- - Explainable anomaly detection (robust z-score, thresholds, missing activity)
595
- - Rule evaluation with normalized recommendations
596
- - Cost / budget / max-token style advisors (generic measures only — no domain lock-in)
597
- - Map and cardinality health when you pass the counter map into `createCountexInsights`
598
- - Resource projections from observed cells via `estimateResourcesFromActuals` (requires the durable adapter reference)
599
-
600
- **Safety:** Insights honor `maxBuckets` (aligned with the core 10,000 bucket cap) and `maxGroups`. Grouped queries load all matching cells for the timeframe before aggregation, so narrow filters when possible.
601
-
602
- **Sparse counters:** Bottom-N style rankings omit zero rows by default; Countex only stores cells that received increments.
603
-
604
- ```ts
605
- import { createCountexInsights } from "@x12i/countex/insights";
606
-
607
- const insights = createCountexInsights({
608
- countex,
609
- durable, // optional; required for estimateResourcesFromActuals → streamCells
610
- mapsByCounterId: new Map(maps.map((m) => [m.counterId, m])),
611
- });
612
-
613
- const topCostDrivers = await insights.topPercent({
614
- scope: "org_123",
615
- counterId: "llm_usage",
616
- timeframe: { type: "month", from: "2026-05", to: "2026-05" },
617
- groupBy: ["subject"],
618
- measure: "costMicros.sum",
619
- percent: 20,
620
- });
621
- ```
622
-
623
- Further reading: [`docs/guides/insights-overview.md`](docs/guides/insights-overview.md), [`docs/guides/insights-ranking.md`](docs/guides/insights-ranking.md), [`docs/guides/insights-comparison-and-trends.md`](docs/guides/insights-comparison-and-trends.md), [`docs/guides/insights-anomalies-and-rules.md`](docs/guides/insights-anomalies-and-rules.md), [`docs/guides/insights-forecasting-and-budget.md`](docs/guides/insights-forecasting-and-budget.md), [`docs/guides/insights-advisors-and-optimization.md`](docs/guides/insights-advisors-and-optimization.md). Deeper topic notes also live under `docs/INSIGHTS*.md` for maintainers.
624
-
625
- CLI (after `npm run build` / install):
626
-
627
- ```bash
628
- npx countex insights:top ./path/to/insight-context.mjs --scope org_123 --counter my_counter
629
- # TypeScript configs: node --import=tsx node_modules/@x12i/countex/dist/cli.js insights:pareto ./ctx.ts
630
- ```
631
-
632
- ---
633
-
634
- ## Events
635
-
636
- The Countex client is an `EventEmitter`. Subscribe to observe what's happening:
637
-
638
- ```ts
639
- countex.on("flush", ({ cells, durationMs }) => {});
640
- countex.on("flush_error", ({ error }) => {});
641
- countex.on("cardinality_warning", ({ kind, count, limit })=> {});
642
- countex.on("overflow", ({ kind, sourceKey }) => {});
643
- countex.on("dropped", ({ reason }) => {});
644
- countex.on("duplicate", ({ scopeId, eventId }) => {});
645
- ```
646
-
647
- All event payloads are typed via `CountexEvents`.
648
-
649
- ---
650
-
651
- ## Operational notes
652
-
653
- ### Buffering and flushing
654
-
655
- Each Countex instance keeps only a delta buffer in memory:
656
-
657
- ```
658
- Map<cellKey, delta>
659
- ```
660
-
661
- Flush triggers (any of):
662
-
663
- - `flush.everyMs` timer (default 60s)
664
- - `flush.maxBufferedCells` reached (default 100k)
665
- - `flush.maxBufferedEvents` reached (default 50k)
666
- - `shutdown()` if `flushOnShutdown` is true (default)
667
- - explicit `flush()`
668
-
669
- Flush is **two-phase**: durable first, cache second. If durable fails, the snapshot is merged back into the buffer — counts are not lost. If cache fails after durable succeeded, it's logged and continues; the next read warms cache from durable.
670
-
671
- At most one flush runs at a time. Concurrent callers receive the same in-flight promise.
672
-
673
- ### Cardinality protection
674
-
675
- A bad mapping (e.g. a free-form URL as a dimension) can balloon the dictionary. Countex tracks per-`(scope, kind, qualifier)` how many distinct keys it has minted, and applies a policy when limits hit:
676
-
677
- - `overflow` (default) — rewrite the source key to `__OTHER__`. Emits `overflow`.
678
- - `drop` — silently drop the increment. Emits `dropped`.
679
- - `error` — throw `CardinalityLimitError`.
680
-
681
- Defaults are conservative; tune them per workload. A warning fires at 80% of the limit (`cardinality_warning`).
682
-
683
- ### Idempotency
684
-
685
- Optional and per-map. Requires a cache adapter that implements `markEventSeen` (the included `RedisCache` and `MemoryCache` both do).
686
-
687
- ```ts
688
- idempotency: {
689
- enabled: true,
690
- eventIdPath: "$.event.id",
691
- ttlSeconds: 86_400,
692
- }
693
- ```
694
-
695
- Dedupe keys are namespaced by `counterId`, so different maps with their own idempotency don't collide on shared event ids.
696
-
697
- ### Late events
698
-
699
- Bucket selection always uses the **event time**, never the ingestion time. Late-arriving events go into the correct historical bucket — `$inc` doesn't care about ordering.
700
-
701
- ### Concurrency safety
702
-
703
- All writes are atomic per cell:
704
-
705
- - Mongo: `bulkWrite` of `updateOne` with `$inc` and `upsert: true`
706
- - Redis: `HINCRBY`
707
-
708
- 20+ Countex workers can safely run in parallel against the same scope. Reads are eventually consistent across the two stores; if Redis is behind durable, the planner will detect partial coverage and read durable.
709
-
710
- ### Backup / verify / restore / rebuild
711
-
712
- Top-level free functions and a `countex.backup` namespace bound to your engine:
713
-
714
- ```ts
715
- import {
716
- exportCells,
717
- computeCellsFingerprint,
718
- createBackupManifest,
719
- verifyBackupAgainstManifest,
720
- restoreSnapshotCells,
721
- rebuildCache,
722
- COUNTEX_CELL_SCHEMA_VERSION,
723
- } from "@x12i/countex";
724
-
725
- const filter = {
726
- scope: "org_123",
727
- counterId: "access_by_subject_metric",
728
- timeframe: { type: "day", from: "2026-02-01", to: "2026-05-08" },
729
- };
730
-
731
- // Stream all cells in a window — bounded async iteration.
732
- for await (const cell of exportCells(durable, filter)) {
733
- // ndjson, S3, whatever you want
734
- }
735
-
736
- // Fingerprint + manifest for cold-storage verification.
737
- const fp = await countex.backup.fingerprint(filter); // { cellCount, contentSha256, countSum }
738
- const manifest = countex.backup.createManifest(filter, fp); // { manifestVersion, schemaVersion, filter, … }
739
-
740
- // Verify a manifest against the live durable adapter.
741
- const ok = await countex.backup.verify(manifest);
742
- if (!ok.ok) throw new Error(ok.detail);
743
-
744
- // Restore. Modes: "validate-only" | "replace-window" (default) | "merge-deltas" (reserved).
745
- const { restored, mode } = await countex.backup.restore(
746
- countex.backup.export(filter),
747
- { mode: "replace-window" },
748
- );
749
-
750
- // Re-warm cache from durable. Each warmed bucket's :cov flips to status=complete.
751
- await countex.backup.rebuildCache(filter);
752
- ```
753
-
754
- See [`docs/guides/backup-export.md`](docs/guides/backup-export.md) for the full recipe and manifest layout.
755
-
756
- ---
757
-
758
- ## Docker integration testing
759
-
760
- Countex ships a Docker-based stack (MongoDB, Redis, MinIO) and integration tests under `tests/integration/docker/` that exercise real adapters: dictionary allocation under concurrency, Mongo **`lookupIds` / `reverseLookupIds`**, an end-to-end **`query()`** with **source-key filters** and **`resolveLabels`**, multi-worker ingestion, Redis cache coverage and durable fallback, `rebuildCache`, simulation fingerprints against a memory reference, and backup / manifest / verify / restore to S3-compatible storage (via the AWS SDK in devDependencies).
761
-
762
- **One-shot local run** (starts services, runs tests with default localhost URLs, tears volumes down):
763
-
764
- ```bash
765
- npm run test:e2e
766
- ```
767
-
768
- **Host-runner** (services in Docker, tests on the host):
769
-
770
- ```bash
771
- npm run docker:up
772
- export COUNTEX_MONGO_URI='mongodb://countex:countex@127.0.0.1:27017/admin'
773
- export COUNTEX_REDIS_URL='redis://127.0.0.1:6379'
774
- export COUNTEX_S3_ENDPOINT='http://127.0.0.1:9000'
775
- export COUNTEX_S3_REGION='us-east-1'
776
- export COUNTEX_S3_ACCESS_KEY_ID='countex'
777
- export COUNTEX_S3_SECRET_ACCESS_KEY='countex-password'
778
- export COUNTEX_S3_BUCKET='countex-backups'
779
- export COUNTEX_INTEGRATION=1
780
- npm run test:integration
781
- npm run docker:down
782
- ```
783
-
784
- Optional tuning: `COUNTEX_TEST_PROFILE`, `COUNTEX_TEST_EVENTS`, `COUNTEX_TEST_WORKERS`. Stress-style full pipeline (larger simulation + S3 round-trip): set `COUNTEX_TEST_FULL=1` (requires S3 env as above).
785
-
786
- **Container-runner** (CI parity): `npm run test:integration:docker` builds `docker/test-runner.Dockerfile` and runs `npm run test:integration:exec` with service hostnames injected by Compose.
787
-
788
- Reports are written to `artifacts/integration/report.json` and `artifacts/integration/report.md` after `npm run test:integration`.
789
-
790
- ---
791
-
792
- ## Errors
793
-
794
- All errors thrown by Countex extend `CountexError`:
795
-
796
- | Class | When |
797
- |--------------------------|------|
798
- | `MappingError` | Invalid mapping config (caught at engine startup). |
799
- | `ExtractionError` | Malformed path or value during extraction. |
800
- | `CardinalityLimitError` | Cardinality limit hit when policy is `error`. |
801
- | `StorageError` | Wrapping a storage-adapter failure. |
802
-
803
- ---
804
-
805
- ## Versioning & scope
806
-
807
- > **Additive counts are not distinct counts.** If `user_42` appears in 100 events, `count = 100`, not 1 unique user. Counter cells store sums of deltas, so a query result of `count = 100` from a subject-grouped view means "100 increments hit this subject", not "100 distinct subjects". For exact distinct subject counts, group by `["subjectType", "subject"]` and use `rows.length` after retrieving every matching cell — see [`docs/guides/querying.md`](docs/guides/querying.md) §"Distinct subjects / metrics as row cardinality". Approximate distinct counts (HLL/sets) are a planned opt-in extension, not core.
808
-
809
- `v0.x` is for stabilizing the public API. Once it's hardened:
810
-
811
- - exact unique counts (HLL / sets) are an opt-in extension, not core
812
- - complex expression deltas are not in core
813
- - distributed locks / exactly-once: not Countex's job; durable storage is the source of truth and `$inc` makes worker-count irrelevant
814
-
815
- ---
816
-
817
- ## Repository layout
818
-
819
- | Path | Role |
820
- |------|------|
821
- | `src/index.ts` | Public exports (`createCountex`, types, errors). |
822
- | `src/core/` | Engine: ingest pipeline, buffer, extraction, mapping validation, time buckets, cardinality. |
823
- | `src/query/` | Query planner (`planQuery` + `executeQuery`) and result formatting. |
824
- | `src/storage/` | Adapters: `memory` (in-process), `mongo` (durable), `redis` (cache); shared coverage helpers. |
825
- | `src/utils/` | JSONPath subset, hashing, logging. |
826
- | `src/backup/` | Export, fingerprint, manifest, verify, restore, cache rebuild helpers. |
827
- | `src/simulation/` | Simulation engine, profiles, expected aggregates, resource reports. |
828
- | `src/insights/` | Ranking, comparison, anomaly detection, advisors, recommendations. |
829
- | `src/cli.ts` | CLI commands for simulation, insights, export, and operational helpers. |
830
- | `examples/` | Runnable demos (`npx tsx examples/basic.ts`). |
831
- | `docs/` | Deployment guide, tutorial, guides, internals, and use cases (published in the npm package). |
832
- | `tests/unit/` | Fast Node test runner tests (`npm test`). |
833
- | `tests/integration/` | Opt-in live tests against real services (`npm run test:integration`). |
834
-
835
- ---
836
-
837
- ## Publishing (npm)
838
-
839
- This package is scoped as **`@x12i/countex`** and is configured for **public** registry access (`publishConfig.access` in `package.json`).
840
-
841
- 1. Ensure you are logged in and allowed to publish under the **`@x12i`** scope on [npm](https://www.npmjs.com/).
842
- 2. Put auth tokens only in **`~/.npmrc`** — do not commit project `.npmrc` with credentials (it is listed in `.gitignore`).
843
- 3. From the repo root:
844
-
845
- ```bash
846
- npm test
847
- npm publish
848
- ```
849
-
850
- `prepublishOnly` runs a clean build so the tarball always contains a fresh `dist/`.
851
-
852
- The **`docs/`** directory (`DEPLOYMENT.md`, `END_TO_END_EXAMPLE.md`) is part of the published package (`files` in `package.json`).
853
-
854
- ---
855
-
856
- ## License
857
-
858
- MIT — see `LICENSE`.
1
+ # @x12i/countex
2
+
3
+ > Generic sparse counter matrices for JSON events.
4
+ > Dynamic counting, aggregation, and retrieval from mapped JSON inputs.
5
+
6
+ `@x12i/countex` is a **generic mapped-event counter**: a source-agnostic counting engine. You give it any JSON event, a declarative mapping config, and storage adapters; it projects events into sparse counter cells, batches deltas locally, and persists them atomically to a durable store (MongoDB or PostgreSQL) and a hot cache (Redis).
7
+
8
+ **Package policy:** Countex is generic. It never references a specific host product, client name, schema, or storage layout. Host-specific behavior belongs in the host application, not in this package.
9
+
10
+ It does **not** know about users, organizations, documents, labels, or events. It only knows generic primitives: **scope, subject, metric, dimensions, timeframe, delta**. Whatever your domain calls those things — Countex doesn't care.
11
+
12
+ **Terminology:** `metric` is the generic Countex name for the thing being counted or categorized. In your domain that may be a label, status, event name, operation, outcome, class, model result, or business category — the `metrics` array in a map simply lists JSON paths that resolve to those categories. Storage and APIs refer to dictionary **`sourceKey`** (the string extracted from the event); avoid confusing that with application-level UUIDs unless your map uses UUIDs as keys.
13
+
14
+ **Process memory vs retained state:** Each worker keeps **unflushed deltas** in an in-memory buffer. After flush, authoritative aggregates live in **durable** storage (MongoDB or PostgreSQL); **Redis** holds a hot cache. Restarting a worker loses only buffered deltas that were not flushed yet — tune flush intervals accordingly.
15
+
16
+ ```
17
+ event JSON ──► map ──► raw increments ──► buffer ──► flush ──► Mongo or Postgres (durable)
18
+ └───► Redis (hot cache)
19
+
20
+ ◄── query ◄── Redis (full coverage) or ◄── durable fallback ──◄
21
+ ```
22
+
23
+ ---
24
+
25
+ ## Install
26
+
27
+ ```bash
28
+ npm install @x12i/countex
29
+ # Bring your own clients — peer deps, both optional:
30
+ npm install mongodb ioredis
31
+ # or, for Postgres durable storage instead of Mongo:
32
+ npm install pg ioredis
33
+ ```
34
+
35
+ Node 18+. Zero runtime dependencies in the core package.
36
+
37
+ ### Documentation shipped with the package
38
+
39
+ These guides are included in the published tarball under `docs/` and exposed as stable import subpaths (resolve to `.md` files for editors and tooling):
40
+
41
+ | Topic | Package subpath | File |
42
+ |-------|-----------------|------|
43
+ | Release & production deployment | `@x12i/countex/docs/deployment` | `docs/DEPLOYMENT.md` |
44
+ | Tutorial — install → map → ingest → query | `@x12i/countex/docs/end-to-end-example` | `docs/END_TO_END_EXAMPLE.md` |
45
+ | Feature guides index | `@x12i/countex/docs/guides` | `docs/guides/README.md` |
46
+ | Counter maps | `@x12i/countex/docs/guides/counter-maps` | `docs/guides/counter-maps.md` |
47
+ | Preview a map (dry-run) | `@x12i/countex/docs/guides/preview` | `docs/guides/preview.md` |
48
+ | Ingestion & flush | `@x12i/countex/docs/guides/ingestion-and-flush` | `docs/guides/ingestion-and-flush.md` |
49
+ | Timeframes | `@x12i/countex/docs/guides/timeframes-and-bucketing` | `docs/guides/timeframes-and-bucketing.md` |
50
+ | Measures | `@x12i/countex/docs/guides/measures` | `docs/guides/measures.md` |
51
+ | Querying | `@x12i/countex/docs/guides/querying` | `docs/guides/querying.md` |
52
+ | Memory adapters | `@x12i/countex/docs/guides/adapter-memory` | `docs/guides/adapter-memory.md` |
53
+ | Mongo durable | `@x12i/countex/docs/guides/adapter-mongo` | `docs/guides/adapter-mongo.md` |
54
+ | Postgres durable | `@x12i/countex/docs/guides/adapter-postgres` | `docs/guides/adapter-postgres.md` |
55
+ | Mongo to Postgres | `@x12i/countex/docs/guides/migrating-mongo-to-postgres` | `docs/guides/migrating-mongo-to-postgres.md` |
56
+ | Redis cache | `@x12i/countex/docs/guides/adapter-redis` | `docs/guides/adapter-redis.md` |
57
+ | Cardinality | `@x12i/countex/docs/guides/cardinality` | `docs/guides/cardinality.md` |
58
+ | Idempotency | `@x12i/countex/docs/guides/idempotency` | `docs/guides/idempotency.md` |
59
+ | Backup export | `@x12i/countex/docs/guides/backup-export` | `docs/guides/backup-export.md` |
60
+ | Client events | `@x12i/countex/docs/guides/client-events` | `docs/guides/client-events.md` |
61
+ | Insights overview | `@x12i/countex/docs/guides/insights-overview` | `docs/guides/insights-overview.md` |
62
+ | Insights ranking | `@x12i/countex/docs/guides/insights-ranking` | `docs/guides/insights-ranking.md` |
63
+ | Insights comparison & trends | `@x12i/countex/docs/guides/insights-comparison-and-trends` | `docs/guides/insights-comparison-and-trends.md` |
64
+ | Insights anomalies & rules | `@x12i/countex/docs/guides/insights-anomalies-and-rules` | `docs/guides/insights-anomalies-and-rules.md` |
65
+ | Insights forecasting & budget | `@x12i/countex/docs/guides/insights-forecasting-and-budget` | `docs/guides/insights-forecasting-and-budget.md` |
66
+ | Insights advisors | `@x12i/countex/docs/guides/insights-advisors-and-optimization` | `docs/guides/insights-advisors-and-optimization.md` |
67
+ | CLI | `@x12i/countex/docs/guides/cli` | `docs/guides/cli.md` |
68
+ | Simulation runtime (JS API) | `@x12i/countex/simulation` | `dist/simulation/` |
69
+ | Simulation overview | `@x12i/countex/docs/guides/simulation` | `docs/guides/simulation.md` |
70
+ | Simulation profiles | `@x12i/countex/docs/guides/simulation-profiles` | `docs/guides/simulation-profiles.md` |
71
+ | Simulation resource estimation | `@x12i/countex/docs/guides/simulation-resource-estimation` | `docs/guides/simulation-resource-estimation.md` |
72
+
73
+ **Use cases** (technical playbooks: contracts, fan-out math, exact `QuerySpec`s, production wiring, verification — see [`docs/use-cases/README.md`](docs/use-cases/README.md)):
74
+
75
+ | Topic | Package subpath | File |
76
+ |-------|-----------------|------|
77
+ | Use cases index | `@x12i/countex/docs/use-cases` | `docs/use-cases/README.md` |
78
+ | Audit & security logs (volume, situations, distinct entities, logical event totals) | `@x12i/countex/docs/use-cases/audit-security-log-counts` | `docs/use-cases/audit-security-log-counts.md` |
79
+ | Product & feature activity | `@x12i/countex/docs/use-cases/product-feature-usage` | `docs/use-cases/product-feature-usage.md` |
80
+ | LLM cost & token metering | `@x12i/countex/docs/use-cases/llm-cost-and-token-metering` | `docs/use-cases/llm-cost-and-token-metering.md` |
81
+ | Multi-tenant isolation | `@x12i/countex/docs/use-cases/multi-tenant-isolation` | `docs/use-cases/multi-tenant-isolation.md` |
82
+ | Idempotent webhooks & queues | `@x12i/countex/docs/use-cases/webhook-idempotent-ingestion` | `docs/use-cases/webhook-idempotent-ingestion.md` |
83
+ | Ops dashboards & rollups | `@x12i/countex/docs/use-cases/ops-dashboard-daily-rollups` | `docs/use-cases/ops-dashboard-daily-rollups.md` |
84
+ | Processing & workflow activity | `@x12i/countex/docs/use-cases/processing-workflow-activity` | `docs/use-cases/processing-workflow-activity.md` |
85
+
86
+ **Internals** (how the engine works — diagrams and traces) live under `docs/internals/` in the tarball; browse [`docs/internals/README.md`](docs/internals/README.md).
87
+
88
+ After install: `node_modules/@x12i/countex/docs/`.
89
+
90
+ ---
91
+
92
+ ## The mental model
93
+
94
+ Countex turns mapped JSON events into sparse aggregate cells.
95
+
96
+ A counter cell is uniquely identified by:
97
+
98
+ ```
99
+ (scope, counterId, timeframeType, timeframeKey, subjectType, subjectId, metricId, dimensionsHash)
100
+ ```
101
+
102
+ Each event can produce one or many cell increments. A counter increment is defined by:
103
+
104
+ - **scope** — your tenant / org / account boundary
105
+ - **counterId** — which counter family this is (e.g. `access_by_subject_metric`)
106
+ - **timeframe** — `minute` / `hour` / `day` / `week` / `month` / `quarter` / `year` / `all`
107
+ - **subject** — the thing being counted against (e.g. an actor or a target)
108
+ - **metric** — the count category / label
109
+ - **dimensions** — arbitrary additional breakdowns
110
+ - **delta** — `+1`, `-1`, or any integer
111
+
112
+ Countex buffers increments locally, flushes atomic deltas to durable + cache, and serves queries from cache with durable fallback.
113
+
114
+ ### Measures and rates
115
+
116
+ Maps may declare optional **`measures`**: named numeric fields per event (cost, tokens, latency, …) with aggregations `sum`, `count`, `min`, and `max`. **`avg`** is computed at query time when both `sum` and `count` exist on the measure. Use **`scale`** on a measure for fixed-point storage (e.g. cents). Extracted values attach to each fan-out increment the same way as **`delta`** (see RFC-style docs for fan-out semantics).
117
+
118
+ Queries accept **`measures`** (explicit list or `"*"`), **`orderBy`** on a measure aggregation, and optional **`rate`** (count or a measure’s sum divided by elapsed time in the queried window; for `format: "timeseries"`, per bucket). **`rate`** is invalid for timeframe **`all`**.
119
+
120
+ **Redis cache** stores **sum** and **count** per measure only; **min** / **max** are durable-backed. If a query needs min/max (including `orderBy` on them, or requesting measures that maintain min/max when using `"*"`), the planner reads from **durable** first so cache cannot drift.
121
+
122
+ **`bucketRange`** caps at **10,000** buckets; at **`minute`** resolution that is about **6.94 days** — keep queries within that bound or narrow the range.
123
+
124
+ ### Percentiles & histograms
125
+
126
+ Core cells store **additive** measure aggregates (`sum / count / min / max`) per sparse cell. They do **not** store distributions, so percentile-style answers (p50/p90/p95/p99) can only come from min/max approximation or from histogram/sketch storage you add yourself. Insights advisors that mention percentile-shaped recommendations are heuristic; see [`docs/guides/insights-overview.md`](docs/guides/insights-overview.md) §"Distributions" and [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md) "Architecture decisions".
127
+
128
+ ---
129
+
130
+ ## What kinds of events Countex is for
131
+
132
+ Countex is not tied to one event source or domain. It is a **generic mapped-event counter**: any JSON-like input stream where useful aggregates can be produced from paths in the event.
133
+
134
+ Typical sources include:
135
+
136
+ | Source family | Examples | Common Countex projection |
137
+ |---|---|---|
138
+ | Audit and security logs | Microsoft 365 audit logs, Google Workspace logs, Okta logs, cloud audit logs | actor/resource/action/situation/severity |
139
+ | Product and user activity | Woopra, Segment, PostHog-style events, application activity logs | user/account/feature/event/page/funnel |
140
+ | Processing and workflow activity | ETL runs, background jobs, agent graphs, workflow engines, queue processors | job/task/node/status/error/runtime |
141
+ | LLM and AI service usage | Model gateway logs, OpenAI/Anthropic calls, internal LLM services | app/user/model/provider/tokens/cost/latency/result |
142
+ | Business/domain events | CRM, support tickets, payments, commerce, lifecycle events | customer/object/state/category/outcome |
143
+
144
+ Countex only needs a map. The map tells Countex where to find:
145
+
146
+ - the scope boundary
147
+ - the event time
148
+ - the subjects
149
+ - the metrics
150
+ - the dimensions
151
+ - the delta/count
152
+ - optional numeric measures such as cost, tokens, latency, or duration
153
+
154
+ The original event can come from a log, webhook, queue message, analytics event, system activity, processing trace, or generated test fixture. Countex does not care about the event source as long as the map can project it into generic counter primitives.
155
+
156
+ ### Audit-log style event
157
+
158
+ ```json
159
+ {
160
+ "tenant": { "id": "org_123" },
161
+ "event": {
162
+ "id": "audit_001",
163
+ "createdAt": "2026-05-08T14:33:00Z",
164
+ "operation": "FileAccessed",
165
+ "situation": "external"
166
+ },
167
+ "actor": { "id": "user_42", "type": "user" },
168
+ "resource": { "id": "file_99", "type": "document" },
169
+ "labels": ["PII", "Financial"]
170
+ }
171
+ ```
172
+
173
+ Counts:
174
+
175
+ ```text
176
+ actor/resource × label × operation/situation/resourceType × timeframe
177
+ ```
178
+
179
+ ### Product-activity style event
180
+
181
+ ```json
182
+ {
183
+ "account": { "id": "acct_123" },
184
+ "user": { "id": "user_42" },
185
+ "event": {
186
+ "name": "dashboard_viewed",
187
+ "createdAt": "2026-05-08T14:33:00Z"
188
+ },
189
+ "product": {
190
+ "feature": "risk_dashboard",
191
+ "plan": "enterprise"
192
+ }
193
+ }
194
+ ```
195
+
196
+ Counts:
197
+
198
+ ```text
199
+ user/account × activity metric × feature/plan × timeframe
200
+ ```
201
+
202
+ ### LLM-processing style event
203
+
204
+ ```json
205
+ {
206
+ "scope": { "id": "org_123" },
207
+ "request": {
208
+ "id": "llm_req_001",
209
+ "createdAt": "2026-05-08T14:33:00Z",
210
+ "status": "success"
211
+ },
212
+ "app": { "id": "app-1" },
213
+ "model": {
214
+ "provider": "openai",
215
+ "name": "gpt-4.1"
216
+ },
217
+ "usage": {
218
+ "inputTokens": 1200,
219
+ "outputTokens": 350,
220
+ "costCents": 4,
221
+ "latencyMs": 1800
222
+ }
223
+ }
224
+ ```
225
+
226
+ Counts/measures:
227
+
228
+ ```text
229
+ app/model/provider/status × timeframe
230
+ sum inputTokens
231
+ sum outputTokens
232
+ sum costCents
233
+ avg latencyMs
234
+ count requests
235
+ ```
236
+
237
+ ---
238
+
239
+ ## Quick start
240
+
241
+ ```ts
242
+ import { createCountex } from "@x12i/countex";
243
+ import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory";
244
+
245
+ const accessMap = {
246
+ counterId: "access_by_subject_metric",
247
+
248
+ scope: { path: "$.tenant.id" },
249
+ eventTime: { path: "$.event.createdAt" },
250
+
251
+ subjects: [
252
+ { type: "actor", path: "$.actor.id", required: true },
253
+ { type: "target", path: "$.target.id", required: false },
254
+ ],
255
+
256
+ metrics: [
257
+ { path: "$.labels[*]", required: true },
258
+ ],
259
+
260
+ dimensions: [
261
+ { name: "eventType", path: "$.event.type" },
262
+ { name: "situation", path: "$.event.situation" },
263
+ { name: "targetType", path: "$.target.type" },
264
+ ],
265
+
266
+ delta: { default: 1 },
267
+ timeframes: ["day", "week", "month", "quarter", "year"],
268
+ } as const;
269
+
270
+ const countex = createCountex({
271
+ maps: [accessMap],
272
+ durable: new MemoryDurable(),
273
+ cache: new MemoryCache(),
274
+ });
275
+
276
+ await countex.ingest({
277
+ tenant: { id: "org_123" },
278
+ event: { type: "share", situation: "external", createdAt: "2026-05-08T14:33:00Z" },
279
+ actor: { id: "user_42" },
280
+ target: { id: "doc_99", type: "document" },
281
+ labels: ["PCI", "PII"],
282
+ });
283
+
284
+ await countex.flush();
285
+
286
+ const result = await countex.query({
287
+ scope: "org_123",
288
+ counterId: "access_by_subject_metric",
289
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
290
+ groupBy: ["subject", "metric", "dim:eventType"],
291
+ format: "table",
292
+ orderBy: "-count",
293
+ limit: 100,
294
+ });
295
+
296
+ console.log(result.rows);
297
+
298
+ await countex.shutdown();
299
+ ```
300
+
301
+ That single ingest produces:
302
+
303
+ - 1 actor + 1 target = **2 subjects**
304
+ - 2 labels = **2 metrics**
305
+ - 5 timeframes
306
+ - → **2 × 2 × 5 = 20 cell increments**
307
+
308
+ Each cell uses an atomic `$inc` on durable and `HINCRBY` on the cache, so 20 ingest workers can safely run in parallel.
309
+
310
+ ### Preview a map (dry-run)
311
+
312
+ Use `countex.preview(event, { counterId? })` to inspect what one event would produce without any I/O — no dictionary allocation, no buffering, no writes. Returns one `MapPreviewResult` per matching map, each with `cells: PreviewIncrementCell[]` (subjects, metrics, dimensions, dimensionsHash, timeframe keys, deltas, measures), `extractionWarnings`, and `droppedAfterCardinality`. See [docs/guides/preview.md](docs/guides/preview.md).
313
+
314
+ ---
315
+
316
+ ## Production setup (Mongo + Redis)
317
+
318
+ Do **not** derive Mongo database names directly from raw tenant strings (length, `$`, `/`, and Unicode can break DB naming rules or leak structure). Hash or normalize first:
319
+
320
+ ```ts
321
+ import { createHash } from "node:crypto";
322
+ import { MongoClient } from "mongodb";
323
+ import IORedis from "ioredis";
324
+ import { createCountex } from "@x12i/countex";
325
+ import { MongoDurable } from "@x12i/countex/adapters/mongo";
326
+ import { RedisCache } from "@x12i/countex/adapters/redis";
327
+
328
+ function hashScopeId(scope: string): string {
329
+ return createHash("sha256").update(scope, "utf8").digest("hex").slice(0, 16);
330
+ }
331
+
332
+ const mongo = new MongoClient(process.env.MONGO_URI!);
333
+ await mongo.connect();
334
+ const redis = new IORedis(process.env.REDIS_URL!);
335
+
336
+ const countex = createCountex({
337
+ maps: [accessMap],
338
+ durable: new MongoDurable({
339
+ client: mongo,
340
+ dbResolver: ({ scope }) => `countex_${hashScopeId(scope)}`,
341
+ }),
342
+ cache: new RedisCache({
343
+ client: redis,
344
+ keyPrefix: "countex",
345
+ // Optional adapter default TTLs; when `retention.redis` is set below, the
346
+ // engine resolves those strings to seconds and forwards them via
347
+ // `setRetentionPolicy`, overriding adapter defaults for matching timeframes.
348
+ }),
349
+ retention: {
350
+ redis: {
351
+ minute: "48h",
352
+ hour: "14d",
353
+ day: "92d",
354
+ week: "52w",
355
+ month: "24mo",
356
+ quarter: "24mo",
357
+ year: "forever",
358
+ all: "forever",
359
+ },
360
+ // Durable TTL / prune is not enforced by the engine. Operators apply Mongo
361
+ // TTL indexes, Postgres partition drops, or scheduled jobs.
362
+ // `retention.mongo` and `retention.postgres` are optional per-timeframe
363
+ // documentation (same duration grammar as redis); see types / DEPLOYMENT.
364
+ },
365
+ flush: {
366
+ everyMs: 60_000,
367
+ maxBufferedCells: 100_000,
368
+ maxBufferedEvents: 50_000,
369
+ flushOnShutdown: true,
370
+ },
371
+ cardinality: {
372
+ maxDimensionValuesPerName: 10_000,
373
+ onLimit: "overflow",
374
+ overflowValue: "__OTHER__",
375
+ },
376
+ });
377
+
378
+ await countex.init();
379
+
380
+ process.on("SIGTERM", async () => { await countex.shutdown(); });
381
+ ```
382
+
383
+ ---
384
+
385
+ ## Mapping reference
386
+
387
+ A `CounterMap` is the heart of the package. Everything Countex knows about your data is in here.
388
+
389
+ | Field | Type | Notes |
390
+ |---------------|----------------------------|-------|
391
+ | `counterId` | `string` | Stable identifier for this counter family. |
392
+ | `scope` | `{ path }` | Path into the event for the tenant/scope id. Required. |
393
+ | `eventTime` | `{ path }` | Path to a Date / ISO string / epoch ms. Drives bucket selection. |
394
+ | `subjects` | `SubjectMapping[]` | One or many. Each has `type` and `path`; multi-value paths fan out. |
395
+ | `metrics` | `MetricMapping[]` | Same fanout semantics. `$.labels[*]` is common. |
396
+ | `measures` | `MeasureMapping[]` | Optional numeric fields per event (`sum` / `count` / `min` / `max`); see [Measures](./docs/guides/measures.md). |
397
+ | `dimensions` | `DimensionMapping[]` | Named extra breakdowns. Optional by default. |
398
+ | `delta` | `{ path?, default? }` | Numeric value at runtime. Default `1`. Negative deltas are fine. |
399
+ | `timeframes` | `TimeframeType[]` | Subset of `minute` / `hour` / `day` / `week` / `month` / `quarter` / `year` / `all`. |
400
+ | `idempotency` | `IdempotencyPolicy` | Optional; needs a cache adapter that implements `markEventSeen`. |
401
+ | `cardinality` | `CardinalityPolicy` | Optional; per-map override of engine-level defaults. |
402
+ | `options` | `MappingOptions` | `skipMissingOptionalDimensions` (default true), `normalizeValues` (default true). |
403
+
404
+ ### Path syntax
405
+
406
+ A small JSONPath subset:
407
+
408
+ | Syntax | Meaning |
409
+ |-------------------|--------------------------------------------------|
410
+ | `$` | Root. |
411
+ | `.name` | Property. |
412
+ | `['name']` | Property (bracket form, allows arbitrary chars). |
413
+ | `[N]` | Numeric array index (negatives allowed). |
414
+ | `[*]` | Wildcard — fan out across array elems / object values. |
415
+
416
+ Examples:
417
+
418
+ ```
419
+ $.tenant.id
420
+ $.actor.id
421
+ $.labels[*]
422
+ $.items[*].id
423
+ $['weird.key.with.dots']
424
+ ```
425
+
426
+ ### Fanout
427
+
428
+ A single event with 2 subjects and 3 labels (metrics) and 5 selected timeframes produces **2 × 3 × 5 = 30** atomic increments. This is intentional — read your mapping carefully. Countex emits a `dropped` event when cardinality protection bites, so you'll know.
429
+
430
+ ---
431
+
432
+ ## Storage adapters
433
+
434
+ Countex defines two contracts and ships three adapters.
435
+
436
+ ```ts
437
+ interface DurableStorage {
438
+ applyDeltas(deltas: PendingDelta[]): Promise<void>;
439
+ readCells(spec: QuerySpec): Promise<CounterCell[]>;
440
+ resolveIds(lookups: DictionaryLookup[]): Promise<Map<string, number>>;
441
+ streamCells(filter): AsyncIterable<CounterCell>;
442
+ init?(): Promise<void>;
443
+ close?(): Promise<void>;
444
+ }
445
+
446
+ interface CacheStorage {
447
+ applyDeltas(deltas: PendingDelta[]): Promise<void>;
448
+ readCells(spec: QuerySpec): Promise<{ cells: CounterCell[]; coverage: "full" | "partial" | "miss" }>;
449
+ warm(cells: CounterCell[]): Promise<void>;
450
+ markEventSeen?(scopeId, eventId, ttlSeconds): Promise<boolean>;
451
+ init?(): Promise<void>;
452
+ close?(): Promise<void>;
453
+ }
454
+ ```
455
+
456
+ ### Built-in adapters
457
+
458
+ | Module | Purpose |
459
+ |-----------------------------------|--------------------------------------|
460
+ | `@x12i/countex/adapters/memory` | In-process durable + cache; tests / local dev. `MemoryCache` + Mongo is single-process only (`init` warns). |
461
+ | `@x12i/countex/adapters/mongo` | Durable. Requires `mongodb` peer. |
462
+ | `@x12i/countex/adapters/postgres` | Durable. Requires `pg` peer. PostgreSQL 15+. |
463
+ | `@x12i/countex/adapters/redis` | Hot cache. Requires `ioredis` peer. |
464
+ | `@x12i/countex/adapters/durable-cache` | No Redis: each query reads durable once. Low volume / one machine. |
465
+
466
+ Bring your own clients — Countex doesn't open or close them. That keeps lifecycle in the host app's hands.
467
+
468
+ ### Mongo schema
469
+
470
+ `countex_cells`:
471
+
472
+ ```json
473
+ {
474
+ "scopeId": "org_123",
475
+ "counterId": "access_by_subject_metric",
476
+ "timeframeType": "day",
477
+ "timeframeKey": "2026-05-08",
478
+ "subjectType": "actor",
479
+ "subjectId": 10001,
480
+ "metricId": 41,
481
+ "dimensionsHash": "a8f21c0b1d2e",
482
+ "dimensions": { "eventType": "share", "situation": "external" },
483
+ "count": 42,
484
+ "createdAt": "...",
485
+ "updatedAt": "..."
486
+ }
487
+ ```
488
+
489
+ Indexes (created automatically on first write):
490
+
491
+ - `uniq_cell_identity` (unique, on the full identity tuple)
492
+ - `by_timeframe` — for range scans
493
+ - `by_subject` — for subject filters
494
+ - `by_metric` — for metric filters
495
+
496
+ `countex_dict` — the per-scope subject/metric dictionary, with monotonic ids allocated via `countex_seq`. Allocation is **concurrency-safe by construction**: read existing → atomic sequence `$inc` → `insertMany({ ordered: false })` → on duplicate-key, re-read. See [`docs/guides/adapter-mongo.md`](docs/guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency" and [`docs/internals/internal-storage-mongo.md`](docs/internals/internal-storage-mongo.md).
497
+
498
+ `dimensionsHash` is canonicalized so any two workers produce the same hash for the same logical dimension tuple: keys sorted lexicographically, values type-tagged (`n: / b: / i: / s:`), serialized, hashed (SHA-1, truncated to 12 hex). See [`docs/internals/internal-dimension-hash.md`](docs/internals/internal-dimension-hash.md).
499
+
500
+ ### Redis layout
501
+
502
+ ```
503
+ countex:{scope}:{counterId}:{timeframeType}:{timeframeKey} → HASH
504
+ field = "{subjectType}|{subjectId}|{metricId}|{dimensionsHash}"
505
+ value = count
506
+
507
+ countex:{scope}:{counterId}:{timeframeType}:{timeframeKey}:cov → HASH (coverage meta)
508
+ status = complete | partial | rebuilding (see BucketCoverageMeta; reads may infer missing)
509
+ lastWarmAt = epoch ms (string)
510
+ mapVersion = optional opaque integer (string)
511
+ schemaVersion = cache layout version (string)
512
+ cellCount = cells last warmed into this bucket (string, when known)
513
+
514
+ countex:{scope}:{counterId}:...:{timeframeKey}:meta:{field} → JSON of dimensions
515
+
516
+ countex:{scope}:dedupe:{eventId} → "1" with TTL
517
+ ```
518
+
519
+ One hash per `(scope, counterId, bucket)` keeps Redis key counts small and per-bucket reads to a single `HGETALL`. **Do not** treat “cell hash is non-empty” as full cache coverage — use the **`:cov`** hash (`status === complete` after `warm`, `partial` after incremental `applyDeltas`, `rebuilding` during cache rebuild). See [`docs/guides/adapter-redis.md`](docs/guides/adapter-redis.md) and [`docs/internals/internal-storage-redis.md`](docs/internals/internal-storage-redis.md).
520
+
521
+ ---
522
+
523
+ ## Query API
524
+
525
+ ```ts
526
+ const result = await countex.query({
527
+ scope: "org_123",
528
+ counterId: "access_by_subject_metric",
529
+
530
+ timeframe: {
531
+ type: "day",
532
+ from: "2026-05-01",
533
+ to: "2026-05-08",
534
+ },
535
+
536
+ // Filters — all optional (subject.ids / metrics: internal numbers and/or dictionary source strings)
537
+ subject: { type: "actor", ids: [10001, 10002, "user_uuid_a"] },
538
+ metrics: [41, 42, "PCI"],
539
+ dimensions: { eventType: ["share", "download"], situation: ["external"] },
540
+
541
+ // Shape
542
+ groupBy: ["subject", "metric", "dim:eventType"],
543
+ format: "table", // "cells" | "table" | "matrix" | "timeseries"
544
+ orderBy: "-count",
545
+ limit: 100,
546
+
547
+ // Optional: use dictionary source keys in grouped keys / cells instead of numeric ids only
548
+ resolveLabels: true,
549
+
550
+ // Diagnostics
551
+ bypassCache: false,
552
+ signal: AbortSignal.timeout(5_000),
553
+ });
554
+ ```
555
+
556
+ **Filters:** `subject.ids` and `metrics` may mix **internal numeric ids** (from ingest) and **string source keys** (UUIDs, labels, and so on). String subject keys require `subject.type`. Set **`resolveLabels: true`** to return source keys in grouped `keys.subject` / `keys.metric`, and `subjectSourceKey` / `metricSourceKey` on **`format: "cells"`** rows. See [`docs/guides/querying.md`](docs/guides/querying.md).
557
+
558
+ `explainQuery` runs the planner **without** materializing cells. It returns a **`QueryPlan`**: expanded `bucketsRequested`, `cacheEligible` / `cacheReason`, predicted `durableFallbackPredicted`, `cacheBucketsExpectedHit` (when the cache adapter supports `probeBucketCoverage`), heuristic `plannedCellsEstimate`, and cost hints (`expensive`, `groupByCost`). Use it to preflight dashboards and expensive `groupBy` shapes.
559
+
560
+ ```ts
561
+ const plan = await countex.explainQuery({
562
+ scope: "org_123",
563
+ counterId: "access_by_subject_metric",
564
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
565
+ groupBy: ["subject", "metric"],
566
+ format: "table",
567
+ });
568
+ ```
569
+
570
+ Result:
571
+
572
+ ```ts
573
+ {
574
+ format: "table",
575
+ rows: [
576
+ { keys: { subject: 10001, metric: 41, "dim:eventType": "share" }, count: 142 },
577
+ // With resolveLabels: true (and known dictionary rows), subject/metric keys are strings:
578
+ // { keys: { subject: "user_42", metric: "PCI", "dim:eventType": "share" }, count: 142 },
579
+ ...
580
+ ],
581
+ total: 1234,
582
+ source: "cache" | "durable" | "merged",
583
+ }
584
+ ```
585
+
586
+ String filters are resolved to internal ids before reads; unknown source keys are omitted from the filter. With **`resolveLabels: true`**, **`format: "cells"`** rows include **`subjectSourceKey`** and **`metricSourceKey`** in addition to numeric ids.
587
+
588
+ `groupBy` accepts:
589
+ - `"subject"`, `"subjectType"`, `"metric"`, `"timeframeKey"`
590
+ - `"dim:NAME"` — group by a dimension value
591
+
592
+ ---
593
+
594
+ ## Insights and advisors
595
+
596
+ Countex can optionally analyze stored counters and measures to produce higher-level insights. The core engine answers “what happened?” by maintaining sparse aggregate cells. The insights layer answers “what matters?” by ranking, comparing periods, detecting anomalies, forecasting usage, and emitting evidence-backed recommendations.
597
+
598
+ Typical uses:
599
+
600
+ - Top / bottom N subjects, metrics, or dimensions; top/bottom percent of groups; Pareto and contribution views
601
+ - Period-over-period comparison and simple trends on timeseries
602
+ - Explainable anomaly detection (robust z-score, thresholds, missing activity)
603
+ - Rule evaluation with normalized recommendations
604
+ - Cost / budget / max-token style advisors (generic measures only — no domain lock-in)
605
+ - Map and cardinality health when you pass the counter map into `createCountexInsights`
606
+ - Resource projections from observed cells via `estimateResourcesFromActuals` (requires the durable adapter reference)
607
+
608
+ **Safety:** Insights honor `maxBuckets` (aligned with the core 10,000 bucket cap) and `maxGroups`. Grouped queries load all matching cells for the timeframe before aggregation, so narrow filters when possible.
609
+
610
+ **Sparse counters:** Bottom-N style rankings omit zero rows by default; Countex only stores cells that received increments.
611
+
612
+ ```ts
613
+ import { createCountexInsights } from "@x12i/countex/insights";
614
+
615
+ const insights = createCountexInsights({
616
+ countex,
617
+ durable, // optional; required for estimateResourcesFromActuals → streamCells
618
+ mapsByCounterId: new Map(maps.map((m) => [m.counterId, m])),
619
+ });
620
+
621
+ const topCostDrivers = await insights.topPercent({
622
+ scope: "org_123",
623
+ counterId: "llm_usage",
624
+ timeframe: { type: "month", from: "2026-05", to: "2026-05" },
625
+ groupBy: ["subject"],
626
+ measure: "costMicros.sum",
627
+ percent: 20,
628
+ });
629
+ ```
630
+
631
+ Further reading: [`docs/guides/insights-overview.md`](docs/guides/insights-overview.md), [`docs/guides/insights-ranking.md`](docs/guides/insights-ranking.md), [`docs/guides/insights-comparison-and-trends.md`](docs/guides/insights-comparison-and-trends.md), [`docs/guides/insights-anomalies-and-rules.md`](docs/guides/insights-anomalies-and-rules.md), [`docs/guides/insights-forecasting-and-budget.md`](docs/guides/insights-forecasting-and-budget.md), [`docs/guides/insights-advisors-and-optimization.md`](docs/guides/insights-advisors-and-optimization.md). Deeper topic notes also live under `docs/INSIGHTS*.md` for maintainers.
632
+
633
+ CLI (after `npm run build` / install):
634
+
635
+ ```bash
636
+ npx countex insights:top ./path/to/insight-context.mjs --scope org_123 --counter my_counter
637
+ # TypeScript configs: node --import=tsx node_modules/@x12i/countex/dist/cli.js insights:pareto ./ctx.ts
638
+ ```
639
+
640
+ ---
641
+
642
+ ## Events
643
+
644
+ The Countex client is an `EventEmitter`. Subscribe to observe what's happening:
645
+
646
+ ```ts
647
+ countex.on("flush", ({ cells, durationMs }) => {});
648
+ countex.on("flush_error", ({ error }) => {});
649
+ countex.on("cardinality_warning", ({ kind, count, limit })=> {});
650
+ countex.on("overflow", ({ kind, sourceKey }) => {});
651
+ countex.on("dropped", ({ reason }) => {});
652
+ countex.on("duplicate", ({ scopeId, eventId }) => {});
653
+ ```
654
+
655
+ All event payloads are typed via `CountexEvents`.
656
+
657
+ ---
658
+
659
+ ## Operational notes
660
+
661
+ ### Buffering and flushing
662
+
663
+ Each Countex instance keeps only a delta buffer in memory:
664
+
665
+ ```
666
+ Map<cellKey, delta>
667
+ ```
668
+
669
+ Flush triggers (any of):
670
+
671
+ - `flush.everyMs` timer (default 60s)
672
+ - `flush.maxBufferedCells` reached (default 100k)
673
+ - `flush.maxBufferedEvents` reached (default 50k)
674
+ - `shutdown()` if `flushOnShutdown` is true (default)
675
+ - explicit `flush()`
676
+
677
+ Flush is **two-phase**: durable first, cache second. If durable fails, the snapshot is merged back into the buffer — counts are not lost. If cache fails after durable succeeded, it's logged and continues; the next read warms cache from durable.
678
+
679
+ At most one flush runs at a time. Concurrent callers receive the same in-flight promise.
680
+
681
+ ### Cardinality protection
682
+
683
+ A bad mapping (e.g. a free-form URL as a dimension) can balloon the dictionary. Countex tracks per-`(scope, kind, qualifier)` how many distinct keys it has minted, and applies a policy when limits hit:
684
+
685
+ - `overflow` (default) — rewrite the source key to `__OTHER__`. Emits `overflow`.
686
+ - `drop` — silently drop the increment. Emits `dropped`.
687
+ - `error` — throw `CardinalityLimitError`.
688
+
689
+ Defaults are conservative; tune them per workload. A warning fires at 80% of the limit (`cardinality_warning`).
690
+
691
+ ### Idempotency
692
+
693
+ Optional and per-map. Requires a cache adapter that implements `markEventSeen` (the included `RedisCache` and `MemoryCache` both do).
694
+
695
+ ```ts
696
+ idempotency: {
697
+ enabled: true,
698
+ eventIdPath: "$.event.id",
699
+ ttlSeconds: 86_400,
700
+ }
701
+ ```
702
+
703
+ Dedupe keys are namespaced by `counterId`, so different maps with their own idempotency don't collide on shared event ids.
704
+
705
+ ### Late events
706
+
707
+ Bucket selection always uses the **event time**, never the ingestion time. Late-arriving events go into the correct historical bucket — `$inc` doesn't care about ordering.
708
+
709
+ ### Concurrency safety
710
+
711
+ All writes are atomic per cell:
712
+
713
+ - Mongo: `bulkWrite` of `updateOne` with `$inc` and `upsert: true`
714
+ - Redis: `HINCRBY`
715
+
716
+ 20+ Countex workers can safely run in parallel against the same scope. Reads are eventually consistent across the two stores; if Redis is behind durable, the planner will detect partial coverage and read durable.
717
+
718
+ ### Backup / verify / restore / rebuild
719
+
720
+ Top-level free functions and a `countex.backup` namespace bound to your engine:
721
+
722
+ ```ts
723
+ import {
724
+ exportCells,
725
+ computeCellsFingerprint,
726
+ createBackupManifest,
727
+ verifyBackupAgainstManifest,
728
+ restoreSnapshotCells,
729
+ rebuildCache,
730
+ COUNTEX_CELL_SCHEMA_VERSION,
731
+ } from "@x12i/countex";
732
+
733
+ const filter = {
734
+ scope: "org_123",
735
+ counterId: "access_by_subject_metric",
736
+ timeframe: { type: "day", from: "2026-02-01", to: "2026-05-08" },
737
+ };
738
+
739
+ // Stream all cells in a window — bounded async iteration.
740
+ for await (const cell of exportCells(durable, filter)) {
741
+ // ndjson, S3, whatever you want
742
+ }
743
+
744
+ // Fingerprint + manifest for cold-storage verification.
745
+ const fp = await countex.backup.fingerprint(filter); // { cellCount, contentSha256, countSum }
746
+ const manifest = countex.backup.createManifest(filter, fp); // { manifestVersion, schemaVersion, filter, … }
747
+
748
+ // Verify a manifest against the live durable adapter.
749
+ const ok = await countex.backup.verify(manifest);
750
+ if (!ok.ok) throw new Error(ok.detail);
751
+
752
+ // Restore. Modes: "validate-only" | "replace-window" (default) | "merge-deltas" (reserved).
753
+ const { restored, mode } = await countex.backup.restore(
754
+ countex.backup.export(filter),
755
+ { mode: "replace-window" },
756
+ );
757
+
758
+ // Re-warm cache from durable. Each warmed bucket's :cov flips to status=complete.
759
+ await countex.backup.rebuildCache(filter);
760
+ ```
761
+
762
+ See [`docs/guides/backup-export.md`](docs/guides/backup-export.md) for the full recipe and manifest layout.
763
+
764
+ ---
765
+
766
+ ## Docker integration testing
767
+
768
+ Countex ships a Docker-based stack (MongoDB, Redis, MinIO) and integration tests under `tests/integration/docker/` that exercise real adapters: dictionary allocation under concurrency, Mongo **`lookupIds` / `reverseLookupIds`**, an end-to-end **`query()`** with **source-key filters** and **`resolveLabels`**, multi-worker ingestion, Redis cache coverage and durable fallback, `rebuildCache`, simulation fingerprints against a memory reference, and backup / manifest / verify / restore to S3-compatible storage (via the AWS SDK in devDependencies).
769
+
770
+ **One-shot local run** (starts services, runs tests with default localhost URLs, tears volumes down):
771
+
772
+ ```bash
773
+ npm run test:e2e
774
+ ```
775
+
776
+ **Host-runner** (services in Docker, tests on the host):
777
+
778
+ ```bash
779
+ npm run docker:up
780
+ export COUNTEX_MONGO_URI='mongodb://countex:countex@127.0.0.1:27017/admin'
781
+ export COUNTEX_REDIS_URL='redis://127.0.0.1:6379'
782
+ export COUNTEX_S3_ENDPOINT='http://127.0.0.1:9000'
783
+ export COUNTEX_S3_REGION='us-east-1'
784
+ export COUNTEX_S3_ACCESS_KEY_ID='countex'
785
+ export COUNTEX_S3_SECRET_ACCESS_KEY='countex-password'
786
+ export COUNTEX_S3_BUCKET='countex-backups'
787
+ export COUNTEX_INTEGRATION=1
788
+ npm run test:integration
789
+ npm run docker:down
790
+ ```
791
+
792
+ Optional tuning: `COUNTEX_TEST_PROFILE`, `COUNTEX_TEST_EVENTS`, `COUNTEX_TEST_WORKERS`. Stress-style full pipeline (larger simulation + S3 round-trip): set `COUNTEX_TEST_FULL=1` (requires S3 env as above).
793
+
794
+ **Container-runner** (CI parity): `npm run test:integration:docker` builds `docker/test-runner.Dockerfile` and runs `npm run test:integration:exec` with service hostnames injected by Compose.
795
+
796
+ Reports are written to `artifacts/integration/report.json` and `artifacts/integration/report.md` after `npm run test:integration`.
797
+
798
+ ---
799
+
800
+ ## Errors
801
+
802
+ All errors thrown by Countex extend `CountexError`:
803
+
804
+ | Class | When |
805
+ |--------------------------|------|
806
+ | `MappingError` | Invalid mapping config (caught at engine startup). |
807
+ | `ExtractionError` | Malformed path or value during extraction. |
808
+ | `CardinalityLimitError` | Cardinality limit hit when policy is `error`. |
809
+ | `StorageError` | Wrapping a storage-adapter failure. |
810
+
811
+ ---
812
+
813
+ ## Versioning & scope
814
+
815
+ > **Additive counts are not distinct counts.** If `user_42` appears in 100 events, `count = 100`, not 1 unique user. Counter cells store sums of deltas, so a query result of `count = 100` from a subject-grouped view means "100 increments hit this subject", not "100 distinct subjects". For exact distinct subject counts, group by `["subjectType", "subject"]` and use `rows.length` after retrieving every matching cell — see [`docs/guides/querying.md`](docs/guides/querying.md) §"Distinct subjects / metrics as row cardinality". Approximate distinct counts (HLL/sets) are a planned opt-in extension, not core.
816
+
817
+ `v0.x` is for stabilizing the public API. Once it's hardened:
818
+
819
+ - exact unique counts (HLL / sets) are an opt-in extension, not core
820
+ - complex expression deltas are not in core
821
+ - distributed locks / exactly-once: not Countex's job; durable storage is the source of truth and `$inc` makes worker-count irrelevant
822
+
823
+ ---
824
+
825
+ ## Repository layout
826
+
827
+ | Path | Role |
828
+ |------|------|
829
+ | `src/index.ts` | Public exports (`createCountex`, types, errors). |
830
+ | `src/core/` | Engine: ingest pipeline, buffer, extraction, mapping validation, time buckets, cardinality. |
831
+ | `src/query/` | Query planner (`planQuery` + `executeQuery`) and result formatting. |
832
+ | `src/storage/` | Adapters: `memory` (in-process), `mongo` (durable), `redis` (cache); shared coverage helpers. |
833
+ | `src/utils/` | JSONPath subset, hashing, logging. |
834
+ | `src/backup/` | Export, fingerprint, manifest, verify, restore, cache rebuild helpers. |
835
+ | `src/simulation/` | Simulation engine, profiles, expected aggregates, resource reports. |
836
+ | `src/insights/` | Ranking, comparison, anomaly detection, advisors, recommendations. |
837
+ | `src/cli.ts` | CLI commands for simulation, insights, export, and operational helpers. |
838
+ | `examples/` | Runnable demos (`npx tsx examples/basic.ts`). |
839
+ | `docs/` | Deployment guide, tutorial, guides, internals, and use cases (published in the npm package). |
840
+ | `tests/unit/` | Fast Node test runner tests (`npm test`). |
841
+ | `tests/integration/` | Opt-in live tests against real services (`npm run test:integration`). |
842
+
843
+ ---
844
+
845
+ ## Publishing (npm)
846
+
847
+ This package is scoped as **`@x12i/countex`** and is configured for **public** registry access (`publishConfig.access` in `package.json`).
848
+
849
+ 1. Ensure you are logged in and allowed to publish under the **`@x12i`** scope on [npm](https://www.npmjs.com/).
850
+ 2. Put auth tokens only in **`~/.npmrc`** — do not commit project `.npmrc` with credentials (it is listed in `.gitignore`).
851
+ 3. From the repo root:
852
+
853
+ ```bash
854
+ npm test
855
+ npm publish
856
+ ```
857
+
858
+ `prepublishOnly` runs a clean build so the tarball always contains a fresh `dist/`.
859
+
860
+ The **`docs/`** directory (`DEPLOYMENT.md`, `END_TO_END_EXAMPLE.md`) is part of the published package (`files` in `package.json`).
861
+
862
+ ---
863
+
864
+ ## License
865
+
866
+ MIT — see `LICENSE`.