@x12i/countex 1.8.3 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +866 -858
  3. package/dist/cli.js +6 -6
  4. package/dist/index.d.ts +1 -0
  5. package/dist/index.d.ts.map +1 -1
  6. package/dist/index.js +1 -0
  7. package/dist/index.js.map +1 -1
  8. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  9. package/dist/insights/advisors/resource-advisor.js +5 -1
  10. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  11. package/dist/insights/types.d.ts +4 -0
  12. package/dist/insights/types.d.ts.map +1 -1
  13. package/dist/storage/postgres.d.ts +104 -0
  14. package/dist/storage/postgres.d.ts.map +1 -0
  15. package/dist/storage/postgres.js +1023 -0
  16. package/dist/storage/postgres.js.map +1 -0
  17. package/dist/types/index.d.ts +11 -5
  18. package/dist/types/index.d.ts.map +1 -1
  19. package/docs/DEPLOYMENT.md +399 -375
  20. package/docs/END_TO_END_EXAMPLE.md +316 -316
  21. package/docs/INSIGHTS.md +59 -59
  22. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  23. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  24. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  25. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  26. package/docs/guides/README.md +36 -34
  27. package/docs/guides/adapter-memory.md +60 -60
  28. package/docs/guides/adapter-mongo.md +64 -62
  29. package/docs/guides/adapter-postgres.md +90 -0
  30. package/docs/guides/adapter-redis.md +89 -89
  31. package/docs/guides/backup-export.md +135 -135
  32. package/docs/guides/cardinality.md +50 -50
  33. package/docs/guides/cli.md +53 -53
  34. package/docs/guides/client-events.md +65 -65
  35. package/docs/guides/counter-maps.md +160 -160
  36. package/docs/guides/idempotency.md +39 -39
  37. package/docs/guides/ingestion-and-flush.md +67 -67
  38. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  39. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  40. package/docs/guides/insights-comparison-and-trends.md +47 -47
  41. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  42. package/docs/guides/insights-overview.md +82 -82
  43. package/docs/guides/insights-ranking.md +81 -81
  44. package/docs/guides/measures.md +71 -71
  45. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  46. package/docs/guides/preview.md +92 -92
  47. package/docs/guides/querying.md +178 -178
  48. package/docs/guides/simulation-profiles.md +33 -33
  49. package/docs/guides/simulation-resource-estimation.md +31 -30
  50. package/docs/guides/simulation.md +36 -36
  51. package/docs/guides/timeframes-and-bucketing.md +55 -55
  52. package/docs/internals/README.md +21 -21
  53. package/docs/internals/internal-architecture-overview.md +65 -65
  54. package/docs/internals/internal-backup-stream.md +23 -23
  55. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  56. package/docs/internals/internal-cardinality.md +30 -30
  57. package/docs/internals/internal-dimension-hash.md +34 -34
  58. package/docs/internals/internal-idempotency.md +36 -36
  59. package/docs/internals/internal-ingest-pipeline.md +52 -52
  60. package/docs/internals/internal-insights-layer.md +36 -36
  61. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  62. package/docs/internals/internal-measures-and-scales.md +35 -35
  63. package/docs/internals/internal-query-path.md +46 -46
  64. package/docs/internals/internal-storage-memory.md +13 -13
  65. package/docs/internals/internal-storage-mongo.md +44 -44
  66. package/docs/internals/internal-storage-postgres.md +32 -0
  67. package/docs/internals/internal-storage-redis.md +63 -63
  68. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  69. package/docs/use-cases/README.md +56 -56
  70. package/docs/use-cases/audit-security-log-counts.md +550 -550
  71. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  72. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  73. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  74. package/docs/use-cases/processing-workflow-activity.md +165 -165
  75. package/docs/use-cases/product-feature-usage.md +240 -240
  76. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  77. package/package.json +184 -169
@@ -1,55 +1,55 @@
1
- # Guide: Timeframes and bucketing
2
-
3
- Timeframes control **how event time is bucketed** when Countex writes cells. Queries later ask for a **range of bucket keys** (`from` / `to`).
4
-
5
- ## Step 1: Pick timeframe types on the map
6
-
7
- ```ts
8
- const map = {
9
- counterId: "events",
10
- // ...
11
- timeframes: ["minute", "hour", "day", "week", "month", "quarter", "year", "all"],
12
- } as const;
13
- ```
14
-
15
- Only listed timeframes receive writes for each matching event.
16
-
17
- ## Step 2: Understand bucket keys (UTC)
18
-
19
- | Type | Example key |
20
- |------|----------------|
21
- | minute | `2026-05-08T12:30` |
22
- | hour | `2026-05-08T12` |
23
- | day | `2026-05-08` |
24
- | week | `2026-W19` (ISO week) |
25
- | month | `2026-05` |
26
- | quarter | `2026-Q2` |
27
- | year | `2026` |
28
- | all | `all` |
29
-
30
- Buckets are derived from **`eventTime`**, not ingestion time, so late data lands in the correct historical bucket.
31
-
32
- ## Step 3: Query a range
33
-
34
- ```ts
35
- const result = await countex.query({
36
- scope: "org_1",
37
- counterId: "events",
38
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
39
- format: "table",
40
- });
41
- ```
42
-
43
- ## Step 4: Respect the bucket cap
44
-
45
- Range expansion is capped at **10,000** buckets. Narrow the window or coarsen the timeframe if you hit limits.
46
-
47
- ## Step 5: Rates
48
-
49
- `rate` in queries divides counts or measure sums by elapsed seconds in the queried window. **`rate` is invalid when timeframe type is `all`** (no meaningful duration).
50
-
51
- ## See also
52
-
53
- - [Querying](./querying.md)
54
- - [Measures](./measures.md)
55
- - Internal: [Timeframe bucketing](../internals/internal-timeframe-bucketing.md)
1
+ # Guide: Timeframes and bucketing
2
+
3
+ Timeframes control **how event time is bucketed** when Countex writes cells. Queries later ask for a **range of bucket keys** (`from` / `to`).
4
+
5
+ ## Step 1: Pick timeframe types on the map
6
+
7
+ ```ts
8
+ const map = {
9
+ counterId: "events",
10
+ // ...
11
+ timeframes: ["minute", "hour", "day", "week", "month", "quarter", "year", "all"],
12
+ } as const;
13
+ ```
14
+
15
+ Only listed timeframes receive writes for each matching event.
16
+
17
+ ## Step 2: Understand bucket keys (UTC)
18
+
19
+ | Type | Example key |
20
+ |------|----------------|
21
+ | minute | `2026-05-08T12:30` |
22
+ | hour | `2026-05-08T12` |
23
+ | day | `2026-05-08` |
24
+ | week | `2026-W19` (ISO week) |
25
+ | month | `2026-05` |
26
+ | quarter | `2026-Q2` |
27
+ | year | `2026` |
28
+ | all | `all` |
29
+
30
+ Buckets are derived from **`eventTime`**, not ingestion time, so late data lands in the correct historical bucket.
31
+
32
+ ## Step 3: Query a range
33
+
34
+ ```ts
35
+ const result = await countex.query({
36
+ scope: "org_1",
37
+ counterId: "events",
38
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
39
+ format: "table",
40
+ });
41
+ ```
42
+
43
+ ## Step 4: Respect the bucket cap
44
+
45
+ Range expansion is capped at **10,000** buckets. Narrow the window or coarsen the timeframe if you hit limits.
46
+
47
+ ## Step 5: Rates
48
+
49
+ `rate` in queries divides counts or measure sums by elapsed seconds in the queried window. **`rate` is invalid when timeframe type is `all`** (no meaningful duration).
50
+
51
+ ## See also
52
+
53
+ - [Querying](./querying.md)
54
+ - [Measures](./measures.md)
55
+ - Internal: [Timeframe bucketing](../internals/internal-timeframe-bucketing.md)
@@ -1,21 +1,21 @@
1
- # Countex internals
2
-
3
- Deep dives into how the engine executes ingest, flush, storage IO, and insights post-processing. These documents are **not** imported as package subpaths; they ship in the npm tarball under `docs/internals/` for repository readers and advanced operators.
4
-
5
- Start here:
6
-
7
- | Topic | Document |
8
- |-------|----------|
9
- | End-to-end architecture | [Architecture overview](./internal-architecture-overview.md) |
10
- | Map startup validation & JSON extraction | [Mapping validation & extraction](./internal-mapping-validation-and-extraction.md) |
11
- | `ingestBatch` phases | [Ingest pipeline](./internal-ingest-pipeline.md) |
12
- | In-memory merge | [Buffer merge & flush](./internal-buffer-merge-and-flush.md) |
13
- | Reads | [Query path](./internal-query-path.md) |
14
- | Bucket keys | [Timeframe bucketing](./internal-timeframe-bucketing.md) |
15
- | Dimension hashing | [Dimension canonicalization & hash](./internal-dimension-hash.md) |
16
- | Measures | [Measures & scales](./internal-measures-and-scales.md) |
17
- | Cardinality | [Cardinality guard](./internal-cardinality.md) |
18
- | Dedupe | [Idempotency](./internal-idempotency.md) |
19
- | Adapters | [Mongo](./internal-storage-mongo.md), [Redis](./internal-storage-redis.md), [Memory](./internal-storage-memory.md) |
20
- | Ops helpers | [Backup streaming](./internal-backup-stream.md) |
21
- | Insights | [Insights layer](./internal-insights-layer.md) |
1
+ # Countex internals
2
+
3
+ Deep dives into how the engine executes ingest, flush, storage IO, and insights post-processing. These documents are **not** imported as package subpaths; they ship in the npm tarball under `docs/internals/` for repository readers and advanced operators.
4
+
5
+ Start here:
6
+
7
+ | Topic | Document |
8
+ |-------|----------|
9
+ | End-to-end architecture | [Architecture overview](./internal-architecture-overview.md) |
10
+ | Map startup validation & JSON extraction | [Mapping validation & extraction](./internal-mapping-validation-and-extraction.md) |
11
+ | `ingestBatch` phases | [Ingest pipeline](./internal-ingest-pipeline.md) |
12
+ | In-memory merge | [Buffer merge & flush](./internal-buffer-merge-and-flush.md) |
13
+ | Reads | [Query path](./internal-query-path.md) |
14
+ | Bucket keys | [Timeframe bucketing](./internal-timeframe-bucketing.md) |
15
+ | Dimension hashing | [Dimension canonicalization & hash](./internal-dimension-hash.md) |
16
+ | Measures | [Measures & scales](./internal-measures-and-scales.md) |
17
+ | Cardinality | [Cardinality guard](./internal-cardinality.md) |
18
+ | Dedupe | [Idempotency](./internal-idempotency.md) |
19
+ | Adapters | [Mongo](./internal-storage-mongo.md), [Postgres](./internal-storage-postgres.md), [Redis](./internal-storage-redis.md), [Memory](./internal-storage-memory.md) |
20
+ | Ops helpers | [Backup streaming](./internal-backup-stream.md) |
21
+ | Insights | [Insights layer](./internal-insights-layer.md) |
@@ -1,65 +1,65 @@
1
- # Internal: Architecture overview
2
-
3
- Countex separates **projection** (events → sparse cells), **durability** (authoritative totals), **cache** (hot reads + dedupe), and **query shaping**.
4
-
5
- ## Component diagram
6
-
7
- ```mermaid
8
- flowchart LR
9
- subgraph app [Host application]
10
- CE[Countex client]
11
- end
12
- subgraph core [Core engine]
13
- MAP[CounterMap registry]
14
- EXT[extractIncrements]
15
- CARD[CardinalityGuard]
16
- BUF[DeltaBuffer]
17
- PLAN[executeQuery]
18
- FMT[formatResult]
19
- end
20
- subgraph io [Storage]
21
- DUR[DurableStorage]
22
- CACHE[CacheStorage]
23
- end
24
- CE --> MAP
25
- CE --> EXT
26
- EXT --> CARD
27
- CARD --> DUR
28
- CARD --> BUF
29
- BUF --> DUR
30
- BUF --> CACHE
31
- CE --> PLAN
32
- PLAN --> CACHE
33
- PLAN --> DUR
34
- PLAN --> FMT
35
- ```
36
-
37
- ## Write trace (conceptual)
38
-
39
- 1. **Validate maps** at `createCountex` construction (`validateMaps`).
40
- 2. **Extract** raw increments per map/event (`extractIncrements`).
41
- 3. **Dedupe** optional (`cache.markEventSeen`).
42
- 4. **Cardinality** rewrite or drop (`CardinalityGuard.check`).
43
- 5. **Dictionary** allocate numeric ids (`durable.resolveIds`).
44
- 6. **Buffer** merge by hashed cell identity (`DeltaBuffer.add`).
45
- 7. **Flush** snapshot → `durable.applyDeltas` → best-effort `cache.applyDeltas`.
46
-
47
- ## Read trace (conceptual)
48
-
49
- 1. **Planner** tries cache when allowed (`cacheCanServe`).
50
- 2. On partial/miss, **durable.readCells**.
51
- 3. Optional **cache.warm** with durable rows.
52
- 4. **Formatter** applies `groupBy`, measures, rates, limits.
53
-
54
- ## Source anchors
55
-
56
- - Client orchestration: `src/core/countex.ts`
57
- - Extraction: `src/core/extraction.ts`
58
- - Buffer: `src/core/buffer.ts`
59
- - Planner: `src/query/planner.ts`
60
- - Formatter: `src/query/formatter.ts`
61
-
62
- ## See also
63
-
64
- - [Ingest pipeline](./internal-ingest-pipeline.md)
65
- - [Query path](./internal-query-path.md)
1
+ # Internal: Architecture overview
2
+
3
+ Countex separates **projection** (events → sparse cells), **durability** (authoritative totals), **cache** (hot reads + dedupe), and **query shaping**.
4
+
5
+ ## Component diagram
6
+
7
+ ```mermaid
8
+ flowchart LR
9
+ subgraph app [Host application]
10
+ CE[Countex client]
11
+ end
12
+ subgraph core [Core engine]
13
+ MAP[CounterMap registry]
14
+ EXT[extractIncrements]
15
+ CARD[CardinalityGuard]
16
+ BUF[DeltaBuffer]
17
+ PLAN[executeQuery]
18
+ FMT[formatResult]
19
+ end
20
+ subgraph io [Storage]
21
+ DUR[DurableStorage]
22
+ CACHE[CacheStorage]
23
+ end
24
+ CE --> MAP
25
+ CE --> EXT
26
+ EXT --> CARD
27
+ CARD --> DUR
28
+ CARD --> BUF
29
+ BUF --> DUR
30
+ BUF --> CACHE
31
+ CE --> PLAN
32
+ PLAN --> CACHE
33
+ PLAN --> DUR
34
+ PLAN --> FMT
35
+ ```
36
+
37
+ ## Write trace (conceptual)
38
+
39
+ 1. **Validate maps** at `createCountex` construction (`validateMaps`).
40
+ 2. **Extract** raw increments per map/event (`extractIncrements`).
41
+ 3. **Dedupe** optional (`cache.markEventSeen`).
42
+ 4. **Cardinality** rewrite or drop (`CardinalityGuard.check`).
43
+ 5. **Dictionary** allocate numeric ids (`durable.resolveIds`).
44
+ 6. **Buffer** merge by hashed cell identity (`DeltaBuffer.add`).
45
+ 7. **Flush** snapshot → `durable.applyDeltas` → best-effort `cache.applyDeltas`.
46
+
47
+ ## Read trace (conceptual)
48
+
49
+ 1. **Planner** tries cache when allowed (`cacheCanServe`).
50
+ 2. On partial/miss, **durable.readCells**.
51
+ 3. Optional **cache.warm** with durable rows.
52
+ 4. **Formatter** applies `groupBy`, measures, rates, limits.
53
+
54
+ ## Source anchors
55
+
56
+ - Client orchestration: `src/core/countex.ts`
57
+ - Extraction: `src/core/extraction.ts`
58
+ - Buffer: `src/core/buffer.ts`
59
+ - Planner: `src/query/planner.ts`
60
+ - Formatter: `src/query/formatter.ts`
61
+
62
+ ## See also
63
+
64
+ - [Ingest pipeline](./internal-ingest-pipeline.md)
65
+ - [Query path](./internal-query-path.md)
@@ -1,23 +1,23 @@
1
- # Internal: Backup streaming
2
-
3
- `exportCells` and `rebuildCache` (`src/backup/export.ts`) are thin wrappers over `durable.streamCells`:
4
-
5
- ```mermaid
6
- flowchart LR
7
- DUR[durable.streamCells filter] --> EXPORT[AsyncIterable CounterCell]
8
- DUR --> BATCH[batch warm loop]
9
- BATCH --> CACHE[cache.warm batches]
10
- ```
11
-
12
- ## Contract expectations
13
-
14
- Implementations must honor:
15
-
16
- - Scope/counter/timeframe bounding
17
- - Optional `AbortSignal` propagation
18
-
19
- `rebuildCache` batches to limit Redis/Mongo round trips during warm operations.
20
-
21
- ## See also
22
-
23
- - [Mongo durable](./internal-storage-mongo.md)
1
+ # Internal: Backup streaming
2
+
3
+ `exportCells` and `rebuildCache` (`src/backup/export.ts`) are thin wrappers over `durable.streamCells`:
4
+
5
+ ```mermaid
6
+ flowchart LR
7
+ DUR[durable.streamCells filter] --> EXPORT[AsyncIterable CounterCell]
8
+ DUR --> BATCH[batch warm loop]
9
+ BATCH --> CACHE[cache.warm batches]
10
+ ```
11
+
12
+ ## Contract expectations
13
+
14
+ Implementations must honor:
15
+
16
+ - Scope/counter/timeframe bounding
17
+ - Optional `AbortSignal` propagation
18
+
19
+ `rebuildCache` batches to limit Redis/Mongo round trips during warm operations.
20
+
21
+ ## See also
22
+
23
+ - [Mongo durable](./internal-storage-mongo.md)
@@ -1,51 +1,51 @@
1
- # Internal: Buffer merge & flush
2
-
3
- ## DeltaBuffer structure
4
-
5
- `DeltaBuffer` (`src/core/buffer.ts`) stores:
6
-
7
- - Map key: string from `cellKey(identity)`
8
- - Value: `{ identity, delta, measures?, measureScales? }`
9
-
10
- ## Merge semantics
11
-
12
- `add` merges:
13
-
14
- - **Counts** — integer delta summed.
15
- - **Measures** — per aggregation field:
16
- - `sum`, `count` additive
17
- - `min` / `max` rolling extrema
18
-
19
- `measureScales` merge retains first-seen scale per measure name (consistent with map declaration).
20
-
21
- ## Flush lifecycle
22
-
23
- ```mermaid
24
- stateDiagram-v2
25
- [*] --> Idle
26
- Idle --> Draining: flush() & buffer non-empty
27
- Draining --> DurableWrite: snapshot taken
28
- DurableWrite --> CacheWrite: durable OK
29
- DurableWrite --> MergeBack: durable error
30
- MergeBack --> Idle
31
- CacheWrite --> Idle: cache success or logged warn
32
- ```
33
-
34
- `flush()` returns the **same promise** while in-flight (`flushing` lock) so concurrent callers coalesce.
35
-
36
- ### Durable failure
37
-
38
- Snapshot is **`mergeBack`** into the buffer so counts are not lost; `flush_error` emits.
39
-
40
- ### Cache failure
41
-
42
- Logged as non-fatal — durable already committed.
43
-
44
- ## Event accounting
45
-
46
- `noteEvent()` increments an event counter used solely for flush thresholds (reset implicitly when draining — payload uses duration only).
47
-
48
- ## See also
49
-
50
- - [Ingest pipeline](./internal-ingest-pipeline.md)
51
- - [Measures & scales](./internal-measures-and-scales.md)
1
+ # Internal: Buffer merge & flush
2
+
3
+ ## DeltaBuffer structure
4
+
5
+ `DeltaBuffer` (`src/core/buffer.ts`) stores:
6
+
7
+ - Map key: string from `cellKey(identity)`
8
+ - Value: `{ identity, delta, measures?, measureScales? }`
9
+
10
+ ## Merge semantics
11
+
12
+ `add` merges:
13
+
14
+ - **Counts** — integer delta summed.
15
+ - **Measures** — per aggregation field:
16
+ - `sum`, `count` additive
17
+ - `min` / `max` rolling extrema
18
+
19
+ `measureScales` merge retains first-seen scale per measure name (consistent with map declaration).
20
+
21
+ ## Flush lifecycle
22
+
23
+ ```mermaid
24
+ stateDiagram-v2
25
+ [*] --> Idle
26
+ Idle --> Draining: flush() & buffer non-empty
27
+ Draining --> DurableWrite: snapshot taken
28
+ DurableWrite --> CacheWrite: durable OK
29
+ DurableWrite --> MergeBack: durable error
30
+ MergeBack --> Idle
31
+ CacheWrite --> Idle: cache success or logged warn
32
+ ```
33
+
34
+ `flush()` returns the **same promise** while in-flight (`flushing` lock) so concurrent callers coalesce.
35
+
36
+ ### Durable failure
37
+
38
+ Snapshot is **`mergeBack`** into the buffer so counts are not lost; `flush_error` emits.
39
+
40
+ ### Cache failure
41
+
42
+ Logged as non-fatal — durable already committed.
43
+
44
+ ## Event accounting
45
+
46
+ `noteEvent()` increments an event counter used solely for flush thresholds (reset implicitly when draining — payload uses duration only).
47
+
48
+ ## See also
49
+
50
+ - [Ingest pipeline](./internal-ingest-pipeline.md)
51
+ - [Measures & scales](./internal-measures-and-scales.md)
@@ -1,30 +1,30 @@
1
- # Internal: Cardinality guard
2
-
3
- `CardinalityGuard` (`src/core/cardinality.ts`) tracks distinct source keys per `(scopeId, kind, qualifier)` using in-memory `Set`s.
4
-
5
- ## Check algorithm
6
-
7
- ```mermaid
8
- flowchart TD
9
- IN[sourceKey] --> KEY[keyFor scope/kind/qualifier]
10
- KEY --> SET{Set contains key?}
11
- SET -->|yes| OK[return original or existing overflow mapping]
12
- SET -->|no| LIM{count < limit?}
13
- LIM -->|yes| ADD[add to Set -> return key]
14
- LIM -->|no| POL{onLimit}
15
- POL -->|overflow| OVF[emit overflow event -> return sentinel]
16
- POL -->|drop| DROP[return null]
17
- POL -->|error| ERR[throw CardinalityLimitError]
18
- ```
19
-
20
- ## Dictionary kinds
21
-
22
- Kinds align with subject/metric/dimension value namespaces; dimension checks include the dimension **name** as qualifier.
23
-
24
- ## Approximation warning
25
-
26
- Commentary in source notes multi-worker deployments see approximate counts — acceptable because the guard targets runaway growth, not strict quotas.
27
-
28
- ## See also
29
-
30
- - [Ingest pipeline](./internal-ingest-pipeline.md)
1
+ # Internal: Cardinality guard
2
+
3
+ `CardinalityGuard` (`src/core/cardinality.ts`) tracks distinct source keys per `(scopeId, kind, qualifier)` using in-memory `Set`s.
4
+
5
+ ## Check algorithm
6
+
7
+ ```mermaid
8
+ flowchart TD
9
+ IN[sourceKey] --> KEY[keyFor scope/kind/qualifier]
10
+ KEY --> SET{Set contains key?}
11
+ SET -->|yes| OK[return original or existing overflow mapping]
12
+ SET -->|no| LIM{count < limit?}
13
+ LIM -->|yes| ADD[add to Set -> return key]
14
+ LIM -->|no| POL{onLimit}
15
+ POL -->|overflow| OVF[emit overflow event -> return sentinel]
16
+ POL -->|drop| DROP[return null]
17
+ POL -->|error| ERR[throw CardinalityLimitError]
18
+ ```
19
+
20
+ ## Dictionary kinds
21
+
22
+ Kinds align with subject/metric/dimension value namespaces; dimension checks include the dimension **name** as qualifier.
23
+
24
+ ## Approximation warning
25
+
26
+ Commentary in source notes multi-worker deployments see approximate counts — acceptable because the guard targets runaway growth, not strict quotas.
27
+
28
+ ## See also
29
+
30
+ - [Ingest pipeline](./internal-ingest-pipeline.md)
@@ -1,34 +1,34 @@
1
- # Internal: Dimension canonicalization and hashing
2
-
3
- Each counter cell identity includes a **`dimensionsHash`** derived from the dimension key/value object for that increment.
4
-
5
- ## Normalization
6
-
7
- Before hashing, dimension values pass through `normalizeDimensionValue` in `src/core/normalization.ts`:
8
-
9
- - `null` / missing optional dimensions → `null`.
10
- - Booleans and finite numbers preserved as typed values.
11
- - Strings: optionally trimmed only (`normalizeValues` on the map); types are **not** coerced (`"1"` and `1` stay distinct).
12
-
13
- Dictionary keys for subjects/metrics use **`normalizeKey`** (trim + lowercase strings) — that is separate from dimension hashing.
14
-
15
- ## Canonical string form
16
-
17
- `canonicalizeDimensions` in `src/utils/hash.ts`:
18
-
19
- 1. Sort dimension **names** lexicographically.
20
- 2. Serialize as `key=value` pairs joined with `&`.
21
- 3. Escape `\`, `=`, and `&` in keys and string values with backslash.
22
- 4. Encode values with type prefixes: `n:` (null), `b:0|1`, `i:<number>`, `s:<escaped>`.
23
-
24
- ## Hash
25
-
26
- `dimensionsHash` is **SHA-1** of the canonical string, truncated to **12 hex characters** (48 bits). Empty dimensions → literal `"0"`.
27
-
28
- Collisions are statistically negligible at intended cardinalities; cells are further scoped by `(scopeId, counterId, timeframe, subject, metric)`.
29
-
30
- ## See also
31
-
32
- - `src/utils/hash.ts`
33
- - `src/core/normalization.ts`
34
- - [Cardinality](../guides/cardinality.md)
1
+ # Internal: Dimension canonicalization and hashing
2
+
3
+ Each counter cell identity includes a **`dimensionsHash`** derived from the dimension key/value object for that increment.
4
+
5
+ ## Normalization
6
+
7
+ Before hashing, dimension values pass through `normalizeDimensionValue` in `src/core/normalization.ts`:
8
+
9
+ - `null` / missing optional dimensions → `null`.
10
+ - Booleans and finite numbers preserved as typed values.
11
+ - Strings: optionally trimmed only (`normalizeValues` on the map); types are **not** coerced (`"1"` and `1` stay distinct).
12
+
13
+ Dictionary keys for subjects/metrics use **`normalizeKey`** (trim + lowercase strings) — that is separate from dimension hashing.
14
+
15
+ ## Canonical string form
16
+
17
+ `canonicalizeDimensions` in `src/utils/hash.ts`:
18
+
19
+ 1. Sort dimension **names** lexicographically.
20
+ 2. Serialize as `key=value` pairs joined with `&`.
21
+ 3. Escape `\`, `=`, and `&` in keys and string values with backslash.
22
+ 4. Encode values with type prefixes: `n:` (null), `b:0|1`, `i:<number>`, `s:<escaped>`.
23
+
24
+ ## Hash
25
+
26
+ `dimensionsHash` is **SHA-1** of the canonical string, truncated to **12 hex characters** (48 bits). Empty dimensions → literal `"0"`.
27
+
28
+ Collisions are statistically negligible at intended cardinalities; cells are further scoped by `(scopeId, counterId, timeframe, subject, metric)`.
29
+
30
+ ## See also
31
+
32
+ - `src/utils/hash.ts`
33
+ - `src/core/normalization.ts`
34
+ - [Cardinality](../guides/cardinality.md)
@@ -1,36 +1,36 @@
1
- # Internal: Idempotency
2
-
3
- Idempotency spans three layers:
4
-
5
- 1. **Map config** — `idempotency.enabled` + `eventIdPath`.
6
- 2. **Extraction** — resolves optional `eventId` string (`src/core/extraction.ts`).
7
- 3. **Ingest gate** — parallel `cache.markEventSeen` before buffering (`src/core/countex.ts`).
8
-
9
- ## Sequence
10
-
11
- ```mermaid
12
- sequenceDiagram
13
- participant CX as Countex
14
- participant EXT as extractIncrements
15
- participant CACHE as CacheStorage
16
- participant BUF as DeltaBuffer
17
- CX->>EXT: increments + eventId
18
- CX->>CACHE: markEventSeen(scope, counterId:eventId, ttl)
19
- CACHE-->>CX: isNew boolean
20
- alt duplicate
21
- CX->>CX: emit duplicate
22
- CX->>CX: zero increments for this map slice
23
- else new
24
- CX->>BUF: add deltas
25
- end
26
- ```
27
-
28
- Namespace **`counterId:`** prefix prevents collisions across maps sharing physical cache.
29
-
30
- ## Failure modes
31
-
32
- Missing event ids produce warnings during extraction but do not automatically dedupe unless your logic skips ingest.
33
-
34
- ## See also
35
-
36
- - [Ingest pipeline](./internal-ingest-pipeline.md)
1
+ # Internal: Idempotency
2
+
3
+ Idempotency spans three layers:
4
+
5
+ 1. **Map config** — `idempotency.enabled` + `eventIdPath`.
6
+ 2. **Extraction** — resolves optional `eventId` string (`src/core/extraction.ts`).
7
+ 3. **Ingest gate** — parallel `cache.markEventSeen` before buffering (`src/core/countex.ts`).
8
+
9
+ ## Sequence
10
+
11
+ ```mermaid
12
+ sequenceDiagram
13
+ participant CX as Countex
14
+ participant EXT as extractIncrements
15
+ participant CACHE as CacheStorage
16
+ participant BUF as DeltaBuffer
17
+ CX->>EXT: increments + eventId
18
+ CX->>CACHE: markEventSeen(scope, counterId:eventId, ttl)
19
+ CACHE-->>CX: isNew boolean
20
+ alt duplicate
21
+ CX->>CX: emit duplicate
22
+ CX->>CX: zero increments for this map slice
23
+ else new
24
+ CX->>BUF: add deltas
25
+ end
26
+ ```
27
+
28
+ Namespace **`counterId:`** prefix prevents collisions across maps sharing physical cache.
29
+
30
+ ## Failure modes
31
+
32
+ Missing event ids produce warnings during extraction but do not automatically dedupe unless your logic skips ingest.
33
+
34
+ ## See also
35
+
36
+ - [Ingest pipeline](./internal-ingest-pipeline.md)