@x12i/countex 1.8.5 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +866 -860
  3. package/dist/cli.js +6 -6
  4. package/dist/index.d.ts +1 -0
  5. package/dist/index.d.ts.map +1 -1
  6. package/dist/index.js +1 -0
  7. package/dist/index.js.map +1 -1
  8. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  9. package/dist/insights/advisors/resource-advisor.js +5 -1
  10. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  11. package/dist/insights/types.d.ts +4 -0
  12. package/dist/insights/types.d.ts.map +1 -1
  13. package/dist/storage/postgres.d.ts +104 -0
  14. package/dist/storage/postgres.d.ts.map +1 -0
  15. package/dist/storage/postgres.js +1023 -0
  16. package/dist/storage/postgres.js.map +1 -0
  17. package/dist/types/index.d.ts +11 -5
  18. package/dist/types/index.d.ts.map +1 -1
  19. package/docs/DEPLOYMENT.md +399 -375
  20. package/docs/END_TO_END_EXAMPLE.md +316 -316
  21. package/docs/INSIGHTS.md +59 -59
  22. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  23. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  24. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  25. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  26. package/docs/guides/README.md +36 -34
  27. package/docs/guides/adapter-memory.md +60 -60
  28. package/docs/guides/adapter-mongo.md +64 -62
  29. package/docs/guides/adapter-postgres.md +90 -0
  30. package/docs/guides/adapter-redis.md +89 -89
  31. package/docs/guides/backup-export.md +135 -135
  32. package/docs/guides/cardinality.md +50 -50
  33. package/docs/guides/cli.md +53 -53
  34. package/docs/guides/client-events.md +65 -65
  35. package/docs/guides/counter-maps.md +160 -160
  36. package/docs/guides/idempotency.md +39 -39
  37. package/docs/guides/ingestion-and-flush.md +67 -67
  38. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  39. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  40. package/docs/guides/insights-comparison-and-trends.md +47 -47
  41. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  42. package/docs/guides/insights-overview.md +82 -82
  43. package/docs/guides/insights-ranking.md +81 -81
  44. package/docs/guides/measures.md +71 -71
  45. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  46. package/docs/guides/preview.md +92 -92
  47. package/docs/guides/querying.md +178 -178
  48. package/docs/guides/simulation-profiles.md +33 -33
  49. package/docs/guides/simulation-resource-estimation.md +31 -30
  50. package/docs/guides/simulation.md +36 -36
  51. package/docs/guides/timeframes-and-bucketing.md +55 -55
  52. package/docs/internals/README.md +21 -21
  53. package/docs/internals/internal-architecture-overview.md +65 -65
  54. package/docs/internals/internal-backup-stream.md +23 -23
  55. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  56. package/docs/internals/internal-cardinality.md +30 -30
  57. package/docs/internals/internal-dimension-hash.md +34 -34
  58. package/docs/internals/internal-idempotency.md +36 -36
  59. package/docs/internals/internal-ingest-pipeline.md +52 -52
  60. package/docs/internals/internal-insights-layer.md +36 -36
  61. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  62. package/docs/internals/internal-measures-and-scales.md +35 -35
  63. package/docs/internals/internal-query-path.md +46 -46
  64. package/docs/internals/internal-storage-memory.md +13 -13
  65. package/docs/internals/internal-storage-mongo.md +44 -44
  66. package/docs/internals/internal-storage-postgres.md +32 -0
  67. package/docs/internals/internal-storage-redis.md +63 -63
  68. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  69. package/docs/use-cases/README.md +56 -56
  70. package/docs/use-cases/audit-security-log-counts.md +550 -550
  71. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  72. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  73. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  74. package/docs/use-cases/processing-workflow-activity.md +165 -165
  75. package/docs/use-cases/product-feature-usage.md +240 -240
  76. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  77. package/package.json +184 -170
@@ -1,52 +1,52 @@
1
- # Internal: Ingest pipeline (`ingestBatch`)
2
-
3
- `Countex.ingest` delegates to `ingestBatch` (`src/core/countex.ts`). The implementation is intentionally phased so CPU work stays synchronous while IO stays bounded.
4
-
5
- ## Sequence diagram
6
-
7
- ```mermaid
8
- sequenceDiagram
9
- participant App
10
- participant CX as Countex
11
- participant EXT as extractIncrements
12
- participant CACHE as CacheStorage
13
- participant CARD as CardinalityGuard
14
- participant DUR as DurableStorage
15
- participant BUF as DeltaBuffer
16
- App->>CX: ingestBatch(events)
17
- loop each event × map
18
- CX->>EXT: extractIncrements(map, ev)
19
- EXT-->>CX: RawIncrement[]
20
- end
21
- CX->>CACHE: markEventSeen (optional, parallel)
22
- CACHE-->>CX: boolean flags
23
- CX->>CARD: check subject/metric/dimensions
24
- CARD-->>CX: rewritten keys or drop
25
- CX->>DUR: resolveIds(lookups)
26
- DUR-->>CX: Map lookup → numeric id
27
- loop each RawIncrement × timeframe
28
- CX->>BUF: add(identity, delta, measures)
29
- end
30
- CX->>CX: maybe flush() on thresholds
31
- ```
32
-
33
- ## Phase notes
34
-
35
- 1. **Extract** — warnings for measures surface as `measure_invalid` events when meta is present.
36
- 2. **Dedupe** — `markEventSeen` receives `${counterId}:${eventId}` to isolate maps.
37
- 3. **Cardinality** — dimension iteration may drop with `dropped` reason `cardinality:dimension`.
38
- 4. **Dictionary** — only **subjects** and **metrics** resolve via `resolveIds`; dimension values remain literal on cells.
39
- 5. **Buffer key** — identity includes `dimensionsHash` (`src/utils/hash.ts`).
40
- 6. **Timeframe expansion** — `bucketKey(tf, eventTime)` invoked per configured timeframe.
41
-
42
- ## Flush coupling
43
-
44
- Threshold triggers call `flush()`:
45
-
46
- - `onOverflow: "block"` awaits flush (back-pressure).
47
- - `"drop"` schedules flush without awaiting.
48
-
49
- ## See also
50
-
51
- - [Buffer merge & flush](./internal-buffer-merge-and-flush.md)
52
- - [Cardinality](./internal-cardinality.md)
1
+ # Internal: Ingest pipeline (`ingestBatch`)
2
+
3
+ `Countex.ingest` delegates to `ingestBatch` (`src/core/countex.ts`). The implementation is intentionally phased so CPU work stays synchronous while IO stays bounded.
4
+
5
+ ## Sequence diagram
6
+
7
+ ```mermaid
8
+ sequenceDiagram
9
+ participant App
10
+ participant CX as Countex
11
+ participant EXT as extractIncrements
12
+ participant CACHE as CacheStorage
13
+ participant CARD as CardinalityGuard
14
+ participant DUR as DurableStorage
15
+ participant BUF as DeltaBuffer
16
+ App->>CX: ingestBatch(events)
17
+ loop each event × map
18
+ CX->>EXT: extractIncrements(map, ev)
19
+ EXT-->>CX: RawIncrement[]
20
+ end
21
+ CX->>CACHE: markEventSeen (optional, parallel)
22
+ CACHE-->>CX: boolean flags
23
+ CX->>CARD: check subject/metric/dimensions
24
+ CARD-->>CX: rewritten keys or drop
25
+ CX->>DUR: resolveIds(lookups)
26
+ DUR-->>CX: Map lookup → numeric id
27
+ loop each RawIncrement × timeframe
28
+ CX->>BUF: add(identity, delta, measures)
29
+ end
30
+ CX->>CX: maybe flush() on thresholds
31
+ ```
32
+
33
+ ## Phase notes
34
+
35
+ 1. **Extract** — warnings for measures surface as `measure_invalid` events when meta is present.
36
+ 2. **Dedupe** — `markEventSeen` receives `${counterId}:${eventId}` to isolate maps.
37
+ 3. **Cardinality** — dimension iteration may drop with `dropped` reason `cardinality:dimension`.
38
+ 4. **Dictionary** — only **subjects** and **metrics** resolve via `resolveIds`; dimension values remain literal on cells.
39
+ 5. **Buffer key** — identity includes `dimensionsHash` (`src/utils/hash.ts`).
40
+ 6. **Timeframe expansion** — `bucketKey(tf, eventTime)` invoked per configured timeframe.
41
+
42
+ ## Flush coupling
43
+
44
+ Threshold triggers call `flush()`:
45
+
46
+ - `onOverflow: "block"` awaits flush (back-pressure).
47
+ - `"drop"` schedules flush without awaiting.
48
+
49
+ ## See also
50
+
51
+ - [Buffer merge & flush](./internal-buffer-merge-and-flush.md)
52
+ - [Cardinality](./internal-cardinality.md)
@@ -1,36 +1,36 @@
1
- # Internal: Insights layer
2
-
3
- `CountexInsights` (`src/insights/insights-client.ts`) wraps an existing `Countex` instance.
4
-
5
- ## Pattern
6
-
7
- ```mermaid
8
- flowchart LR
9
- IN[Insight input] --> QB[baseToQuerySpec +\nparseMeasureSelector]
10
- QB --> Q[countex.query]
11
- Q --> POST[ranking / anomaly /\nforecast helpers]
12
- POST --> OUT[typed result]
13
- ```
14
-
15
- Shared utilities:
16
-
17
- - `utils/query-builder.ts` translates insight inputs → `QuerySpec`.
18
- - Ranking helpers sort rows, compute totals, enforce `maxGroups`.
19
- - Anomaly pipeline fetches **eval** + **baseline** timeseries concurrently (`detectAnomalies`).
20
- - Forecast/budget/token modules emit recommendations and bubble events.
21
-
22
- ## Event bridge
23
-
24
- Methods emit:
25
-
26
- - `insight_query` — performance telemetry
27
- - `anomaly`, `recommendation`, `insufficient_data` — operator-facing signals
28
-
29
- ## Resource advisor
30
-
31
- `estimateResourcesFromActuals` bypasses normal query formatting and streams cells via `durable.streamCells`, requiring the durable reference injected at construction.
32
-
33
- ## See also
34
-
35
- - [Query path](./internal-query-path.md)
36
- - `docs/INSIGHTS.md` for product-level narrative
1
+ # Internal: Insights layer
2
+
3
+ `CountexInsights` (`src/insights/insights-client.ts`) wraps an existing `Countex` instance.
4
+
5
+ ## Pattern
6
+
7
+ ```mermaid
8
+ flowchart LR
9
+ IN[Insight input] --> QB[baseToQuerySpec +\nparseMeasureSelector]
10
+ QB --> Q[countex.query]
11
+ Q --> POST[ranking / anomaly /\nforecast helpers]
12
+ POST --> OUT[typed result]
13
+ ```
14
+
15
+ Shared utilities:
16
+
17
+ - `utils/query-builder.ts` translates insight inputs → `QuerySpec`.
18
+ - Ranking helpers sort rows, compute totals, enforce `maxGroups`.
19
+ - Anomaly pipeline fetches **eval** + **baseline** timeseries concurrently (`detectAnomalies`).
20
+ - Forecast/budget/token modules emit recommendations and bubble events.
21
+
22
+ ## Event bridge
23
+
24
+ Methods emit:
25
+
26
+ - `insight_query` — performance telemetry
27
+ - `anomaly`, `recommendation`, `insufficient_data` — operator-facing signals
28
+
29
+ ## Resource advisor
30
+
31
+ `estimateResourcesFromActuals` bypasses normal query formatting and streams cells via `durable.streamCells`, requiring the durable reference injected at construction.
32
+
33
+ ## See also
34
+
35
+ - [Query path](./internal-query-path.md)
36
+ - `docs/INSIGHTS.md` for product-level narrative
@@ -1,59 +1,59 @@
1
- # Internal: Mapping validation & extraction
2
-
3
- ## Startup validation
4
-
5
- `validateMaps` runs once inside the `Countex` constructor (`src/core/mapping.ts`):
6
-
7
- - Ensures unique `counterId` values.
8
- - Parses every configured JSONPath with `parsePath`.
9
- - Verifies required structural rules (non-empty subjects/metrics/timeframes, valid timeframe enums, measure aggregation enums, idempotency consistency).
10
-
11
- Failures throw **`MappingError`** immediately.
12
-
13
- ## Extraction pipeline
14
-
15
- `extractIncrements(map, event)` (`src/core/extraction.ts`) executes in order:
16
-
17
- ```mermaid
18
- flowchart TD
19
- A[extract scope] -->|missing| SKIP[skipped event]
20
- A --> B[extract eventTime]
21
- B -->|invalid| SKIP
22
- B --> C[optional eventId for idempotency]
23
- C --> D[resolve delta]
24
- D -->|zero| SKIP
25
- D --> E[fan-out subjects]
26
- E -->|none| SKIP
27
- E --> F[fan-out metrics]
28
- F -->|none| SKIP
29
- F --> G[dimensions]
30
- G -->|required missing| SKIP
31
- G --> H[measures]
32
- H -->|drop policy| SKIP
33
- H --> I[cross product increments]
34
- ```
35
-
36
- ### Fan-out
37
-
38
- Nested loops build `RawIncrement` rows:
39
-
40
- ```
41
- for subject in subjectFanout:
42
- for metricKey in metricFanout:
43
- push RawIncrement(... timeframes: map.timeframes ...)
44
- ```
45
-
46
- Each increment carries **copies** of dimensions and measures for that Cartesian tuple.
47
-
48
- ### Measures
49
-
50
- `extractMeasuresForEvent` coerces numbers, applies `scale` truncation rules, and maps declared aggregations into pending contributions via `rawMeasuresToPendingDelta` during buffering.
51
-
52
- ### JSON paths
53
-
54
- `extract` (`src/utils/jsonpath.ts`) evaluates the supported subset: properties, indices, and `[*]` expansion.
55
-
56
- ## See also
57
-
58
- - [Ingest pipeline](./internal-ingest-pipeline.md)
59
- - [Measures & scales](./internal-measures-and-scales.md)
1
+ # Internal: Mapping validation & extraction
2
+
3
+ ## Startup validation
4
+
5
+ `validateMaps` runs once inside the `Countex` constructor (`src/core/mapping.ts`):
6
+
7
+ - Ensures unique `counterId` values.
8
+ - Parses every configured JSONPath with `parsePath`.
9
+ - Verifies required structural rules (non-empty subjects/metrics/timeframes, valid timeframe enums, measure aggregation enums, idempotency consistency).
10
+
11
+ Failures throw **`MappingError`** immediately.
12
+
13
+ ## Extraction pipeline
14
+
15
+ `extractIncrements(map, event)` (`src/core/extraction.ts`) executes in order:
16
+
17
+ ```mermaid
18
+ flowchart TD
19
+ A[extract scope] -->|missing| SKIP[skipped event]
20
+ A --> B[extract eventTime]
21
+ B -->|invalid| SKIP
22
+ B --> C[optional eventId for idempotency]
23
+ C --> D[resolve delta]
24
+ D -->|zero| SKIP
25
+ D --> E[fan-out subjects]
26
+ E -->|none| SKIP
27
+ E --> F[fan-out metrics]
28
+ F -->|none| SKIP
29
+ F --> G[dimensions]
30
+ G -->|required missing| SKIP
31
+ G --> H[measures]
32
+ H -->|drop policy| SKIP
33
+ H --> I[cross product increments]
34
+ ```
35
+
36
+ ### Fan-out
37
+
38
+ Nested loops build `RawIncrement` rows:
39
+
40
+ ```
41
+ for subject in subjectFanout:
42
+ for metricKey in metricFanout:
43
+ push RawIncrement(... timeframes: map.timeframes ...)
44
+ ```
45
+
46
+ Each increment carries **copies** of dimensions and measures for that Cartesian tuple.
47
+
48
+ ### Measures
49
+
50
+ `extractMeasuresForEvent` coerces numbers, applies `scale` truncation rules, and maps declared aggregations into pending contributions via `rawMeasuresToPendingDelta` during buffering.
51
+
52
+ ### JSON paths
53
+
54
+ `extract` (`src/utils/jsonpath.ts`) evaluates the supported subset: properties, indices, and `[*]` expansion.
55
+
56
+ ## See also
57
+
58
+ - [Ingest pipeline](./internal-ingest-pipeline.md)
59
+ - [Measures & scales](./internal-measures-and-scales.md)
@@ -1,35 +1,35 @@
1
- # Internal: Measures & scales
2
-
3
- ## Effective aggregations
4
-
5
- `effectiveMeasureAggregations` (`src/core/extraction.ts`) ensures **`count`** exists whenever **`sum`**, **`min`**, or **`max`** are present so averages remain computable later.
6
-
7
- ## Pending deltas
8
-
9
- `rawMeasuresToPendingDelta` converts each extracted measure into:
10
-
11
- - `sum` contribution
12
- - `count` increment (`1` when enabled)
13
- - `min` / `max` seeds
14
-
15
- These mirror what storage adapters must merge atomically.
16
-
17
- ## Buffer merging
18
-
19
- `mergeMeasureDelta` (`src/core/buffer.ts`) applies associative merges identical to eventual durable semantics.
20
-
21
- ## Scale metadata
22
-
23
- Maps may declare `scale` per measure. During ingest, numeric extraction truncates after multiply; `measureScales` propagate into buffer entries so adapters interpret stored integers consistently.
24
-
25
- ## Query-time averages
26
-
27
- Formatter / row builders derive **`avg = sum / count`** when both components exist.
28
-
29
- ## Cache limitation
30
-
31
- Planner denies cache for queries needing min/max fidelity — see `queryNeedsMinMax` in `src/query/planner.ts`.
32
-
33
- ## See also
34
-
35
- - [Query path](./internal-query-path.md)
1
+ # Internal: Measures & scales
2
+
3
+ ## Effective aggregations
4
+
5
+ `effectiveMeasureAggregations` (`src/core/extraction.ts`) ensures **`count`** exists whenever **`sum`**, **`min`**, or **`max`** are present so averages remain computable later.
6
+
7
+ ## Pending deltas
8
+
9
+ `rawMeasuresToPendingDelta` converts each extracted measure into:
10
+
11
+ - `sum` contribution
12
+ - `count` increment (`1` when enabled)
13
+ - `min` / `max` seeds
14
+
15
+ These mirror what storage adapters must merge atomically.
16
+
17
+ ## Buffer merging
18
+
19
+ `mergeMeasureDelta` (`src/core/buffer.ts`) applies associative merges identical to eventual durable semantics.
20
+
21
+ ## Scale metadata
22
+
23
+ Maps may declare `scale` per measure. During ingest, numeric extraction truncates after multiply; `measureScales` propagate into buffer entries so adapters interpret stored integers consistently.
24
+
25
+ ## Query-time averages
26
+
27
+ Formatter / row builders derive **`avg = sum / count`** when both components exist.
28
+
29
+ ## Cache limitation
30
+
31
+ Planner denies cache for queries needing min/max fidelity — see `queryNeedsMinMax` in `src/query/planner.ts`.
32
+
33
+ ## See also
34
+
35
+ - [Query path](./internal-query-path.md)
@@ -1,46 +1,46 @@
1
- # Internal: Query path
2
-
3
- `Countex.query` delegates to `executeQuery` (`src/query/planner.ts`) with optional measure metadata from `buildMeasureAggIndex(map)`.
4
-
5
- ## Decision flow
6
-
7
- ```mermaid
8
- flowchart TD
9
- Q[QuerySpec] --> C{cache configured\nand not bypassed\nand cacheCanServe?}
10
- C -->|yes| R1[cache.readCells]
11
- R1 -->|coverage full| F[formatResult source=cache]
12
- R1 -->|partial/miss| D[durable.readCells]
13
- C -->|no| D
14
- D --> W{cache && rows && not bypass?}
15
- W -->|yes| WARM[cache.warm best-effort]
16
- W -->|no| F2[formatResult source=merged or durable]
17
- WARM --> F2
18
- ```
19
-
20
- ## Cache eligibility (`cacheCanServe`)
21
-
22
- When the counter map is known, queries that require **min/max** measure accuracy skip cache:
23
-
24
- - `orderBy` targeting min/max aggregation.
25
- - Explicit measure list referencing min/max-maintained measures.
26
- - `measures: "*"` while any declared measure tracks min/max.
27
-
28
- This avoids stale extrema because Redis stores sum/count only for measures.
29
-
30
- ## Formatter responsibilities (`formatResult`)
31
-
32
- `src/query/formatter.ts`:
33
-
34
- - Groups cells by `groupBy` keys.
35
- - Builds `table` / `matrix` / `timeseries` projections.
36
- - Applies **`rate`** using `bucketDurationSeconds` except timeframe `all`.
37
- - Sorts and truncates via `orderBy` / `limit`.
38
-
39
- ## Bucket expansion
40
-
41
- Range queries call `bucketRange` (`src/core/timeframe.ts`) capped at **10,000** buckets.
42
-
43
- ## See also
44
-
45
- - [Timeframe bucketing](./internal-timeframe-bucketing.md)
46
- - [Measures & scales](./internal-measures-and-scales.md)
1
+ # Internal: Query path
2
+
3
+ `Countex.query` delegates to `executeQuery` (`src/query/planner.ts`) with optional measure metadata from `buildMeasureAggIndex(map)`.
4
+
5
+ ## Decision flow
6
+
7
+ ```mermaid
8
+ flowchart TD
9
+ Q[QuerySpec] --> C{cache configured\nand not bypassed\nand cacheCanServe?}
10
+ C -->|yes| R1[cache.readCells]
11
+ R1 -->|coverage full| F[formatResult source=cache]
12
+ R1 -->|partial/miss| D[durable.readCells]
13
+ C -->|no| D
14
+ D --> W{cache && rows && not bypass?}
15
+ W -->|yes| WARM[cache.warm best-effort]
16
+ W -->|no| F2[formatResult source=merged or durable]
17
+ WARM --> F2
18
+ ```
19
+
20
+ ## Cache eligibility (`cacheCanServe`)
21
+
22
+ When the counter map is known, queries that require **min/max** measure accuracy skip cache:
23
+
24
+ - `orderBy` targeting min/max aggregation.
25
+ - Explicit measure list referencing min/max-maintained measures.
26
+ - `measures: "*"` while any declared measure tracks min/max.
27
+
28
+ This avoids stale extrema because Redis stores sum/count only for measures.
29
+
30
+ ## Formatter responsibilities (`formatResult`)
31
+
32
+ `src/query/formatter.ts`:
33
+
34
+ - Groups cells by `groupBy` keys.
35
+ - Builds `table` / `matrix` / `timeseries` projections.
36
+ - Applies **`rate`** using `bucketDurationSeconds` except timeframe `all`.
37
+ - Sorts and truncates via `orderBy` / `limit`.
38
+
39
+ ## Bucket expansion
40
+
41
+ Range queries call `bucketRange` (`src/core/timeframe.ts`) capped at **10,000** buckets.
42
+
43
+ ## See also
44
+
45
+ - [Timeframe bucketing](./internal-timeframe-bucketing.md)
46
+ - [Measures & scales](./internal-measures-and-scales.md)
@@ -1,13 +1,13 @@
1
- # Internal: Memory adapters
2
-
3
- `MemoryDurable` and `MemoryCache` (`src/storage/memory.ts`) mirror production contracts using `Map` structures:
4
-
5
- - Dictionary resolution mimics monotonic ids.
6
- - Cells keyed similarly to durable identity tuples.
7
- - Cache implements simplified coverage semantics for tests.
8
-
9
- Used heavily by unit tests (`tests/unit/*.test.ts`) because they require no external services.
10
-
11
- ## See also
12
-
13
- - [Architecture overview](./internal-architecture-overview.md)
1
+ # Internal: Memory adapters
2
+
3
+ `MemoryDurable` and `MemoryCache` (`src/storage/memory.ts`) mirror production contracts using `Map` structures:
4
+
5
+ - Dictionary resolution mimics monotonic ids.
6
+ - Cells keyed similarly to durable identity tuples.
7
+ - Cache implements simplified coverage semantics for tests.
8
+
9
+ Used heavily by unit tests (`tests/unit/*.test.ts`) because they require no external services.
10
+
11
+ ## See also
12
+
13
+ - [Architecture overview](./internal-architecture-overview.md)
@@ -1,44 +1,44 @@
1
- # Internal: Mongo durable adapter
2
-
3
- `MongoDurable` (`src/storage/mongo.ts`) implements:
4
-
5
- - `applyDeltas` — upserts with `$inc` style operations for counts and compatible measure fields.
6
- - `readCells` — satisfies `QuerySpec` filters using compound indexes.
7
- - `resolveIds` — concurrency-safe allocation of dictionary ids (`countex_dict`, `countex_seq`).
8
- - `streamCells` — bounded iteration for backup/export.
9
- - `importSnapshotCells` — replace-window restore target (called by `restoreSnapshotCells`).
10
-
11
- ## Write path interaction
12
-
13
- Flush snapshots arrive as arrays of `PendingDelta`; Mongo applies them idempotently per cell identity tuple via `bulkWrite` of `updateOne` operations with `$inc` and `upsert: true`.
14
-
15
- ## Indexes
16
-
17
- Created lazily on first write paths:
18
-
19
- - `countex_cells` — `uniq_cell_identity` (unique on the full identity tuple), `by_timeframe`, `by_subject`, `by_metric`.
20
- - `countex_dict` — `uniq_dict_key` (unique on `scopeId, kind, qualifier, sourceKey`).
21
-
22
- ## Dictionary allocation under concurrency
23
-
24
- `resolveIds` follows a strict five-step recipe so 20+ workers seeing the same new `sourceKey` at once still converge on a single internal id:
25
-
26
- 1. **Read** existing dict rows for the requested `(scopeId, kind, qualifier, sourceKey)` batch in one round-trip.
27
- 2. **Allocate** a contiguous block of ids from the per-scope sequence document via `findOneAndUpdate({ _id }, { $inc: { seq: count } })` — atomic per sequence document.
28
- 3. **Insert** the new dict rows with `insertMany({ ordered: false })`. The `uniq_dict_key` index forbids duplicates.
29
- 4. On **duplicate-key error** (another worker won the race), **re-read** the dict for the same identities — the winning row is now visible.
30
- 5. **Cache** the resolved id in the local LRU and (optionally) Redis.
31
-
32
- Bad outcome avoided: two workers each minting `user_42 → 10001` and `user_42 → 10002`. With this protocol one insert wins, the loser re-reads the winner's id, and counts never fragment across two ids.
33
-
34
- The user-facing recipe lives in [`docs/guides/adapter-mongo.md`](../guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency".
35
-
36
- ## Dictionary vs dimensions
37
-
38
- Subjects/metrics map to integers via `countex_dict`. Dimension VALUES are stored as raw values on cells (not via the dictionary), keeping cells self-describing and avoiding dictionary churn for low-cardinality dimensions.
39
-
40
- ## See also
41
-
42
- - [Backup streaming](./internal-backup-stream.md)
43
- - [Dimension hash](./internal-dimension-hash.md) — how the `dimensionsHash` part of the cell identity is computed.
44
- - [Mongo adapter guide](../guides/adapter-mongo.md)
1
+ # Internal: Mongo durable adapter
2
+
3
+ `MongoDurable` (`src/storage/mongo.ts`) implements:
4
+
5
+ - `applyDeltas` — upserts with `$inc` style operations for counts and compatible measure fields.
6
+ - `readCells` — satisfies `QuerySpec` filters using compound indexes.
7
+ - `resolveIds` — concurrency-safe allocation of dictionary ids (`countex_dict`, `countex_seq`).
8
+ - `streamCells` — bounded iteration for backup/export.
9
+ - `importSnapshotCells` — replace-window restore target (called by `restoreSnapshotCells`).
10
+
11
+ ## Write path interaction
12
+
13
+ Flush snapshots arrive as arrays of `PendingDelta`; Mongo applies them idempotently per cell identity tuple via `bulkWrite` of `updateOne` operations with `$inc` and `upsert: true`.
14
+
15
+ ## Indexes
16
+
17
+ Created lazily on first write paths:
18
+
19
+ - `countex_cells` — `uniq_cell_identity` (unique on the full identity tuple), `by_timeframe`, `by_subject`, `by_metric`.
20
+ - `countex_dict` — `uniq_dict_key` (unique on `scopeId, kind, qualifier, sourceKey`).
21
+
22
+ ## Dictionary allocation under concurrency
23
+
24
+ `resolveIds` follows a strict five-step recipe so 20+ workers seeing the same new `sourceKey` at once still converge on a single internal id:
25
+
26
+ 1. **Read** existing dict rows for the requested `(scopeId, kind, qualifier, sourceKey)` batch in one round-trip.
27
+ 2. **Allocate** a contiguous block of ids from the per-scope sequence document via `findOneAndUpdate({ _id }, { $inc: { seq: count } })` — atomic per sequence document.
28
+ 3. **Insert** the new dict rows with `insertMany({ ordered: false })`. The `uniq_dict_key` index forbids duplicates.
29
+ 4. On **duplicate-key error** (another worker won the race), **re-read** the dict for the same identities — the winning row is now visible.
30
+ 5. **Cache** the resolved id in the local LRU and (optionally) Redis.
31
+
32
+ Bad outcome avoided: two workers each minting `user_42 → 10001` and `user_42 → 10002`. With this protocol one insert wins, the loser re-reads the winner's id, and counts never fragment across two ids.
33
+
34
+ The user-facing recipe lives in [`docs/guides/adapter-mongo.md`](../guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency".
35
+
36
+ ## Dictionary vs dimensions
37
+
38
+ Subjects/metrics map to integers via `countex_dict`. Dimension VALUES are stored as raw values on cells (not via the dictionary), keeping cells self-describing and avoiding dictionary churn for low-cardinality dimensions.
39
+
40
+ ## See also
41
+
42
+ - [Backup streaming](./internal-backup-stream.md)
43
+ - [Dimension hash](./internal-dimension-hash.md) — how the `dimensionsHash` part of the cell identity is computed.
44
+ - [Mongo adapter guide](../guides/adapter-mongo.md)
@@ -0,0 +1,32 @@
1
+ # Internal: Postgres durable adapter
2
+
3
+ `PostgresDurable` (`src/storage/postgres.ts`) implements `DurableStorage`:
4
+
5
+ - `applyDeltas` — one transaction of batched `INSERT … ON CONFLICT DO UPDATE`. `merge_measures` adds sum and count, takes least/greatest for min/max, and keeps scale from the existing row on conflict.
6
+ - `readCells` — same filters as Mongo: scope, counter, timeframe buckets, optional subject and metric ids, jsonb dimension equality.
7
+ - `resolveIds` — read, allocate from `seq`, `INSERT … ON CONFLICT DO NOTHING`, re-read.
8
+ - `streamCells` — keyset pages of 500 rows for backup and the resource advisor.
9
+ - `importSnapshotCells` — absolute replace of count and measures (restore).
10
+ - `appendEvidence` — insert, then delete the oldest rows past `maxSamplesAtCapture`.
11
+
12
+ Duplicate identities inside one `applyDeltas` call are collapsed with the same sum/min/max rules as the in-memory buffer before the statement is sent. Postgres rejects an `INSERT` that would update the same conflict target twice.
13
+
14
+ ## Indexes and partitions
15
+
16
+ Created on first use of a schema unless `skipIndexInit` is set:
17
+
18
+ - `cells` — unique identity constraint (also serves timeframe lookups), `cells_by_subject`, `cells_by_metric`. List partitions: `cells_minute`, `cells_hour`, `cells_default`.
19
+ - `dict` — unique `(scope_id, kind, qualifier, source_key)`, plus `(scope_id, kind, qualifier, internal_id)`.
20
+ - `evidence` — `(identity…, captured_at)`.
21
+
22
+ `updated_at` is not indexed. Identity columns are not modified by an increment, which is what makes HOT updates possible. Do not add the changing columns to the unique index.
23
+
24
+ ## Dictionary vs dimensions
25
+
26
+ Subjects and metrics map to integers via `dict`. Dimension values stay on the cell as jsonb, same as Mongo, so cells stay self-describing.
27
+
28
+ ## See also
29
+
30
+ - [Mongo durable](./internal-storage-mongo.md)
31
+ - [Postgres adapter guide](../guides/adapter-postgres.md)
32
+ - [Migrating MongoDB to PostgreSQL](../guides/migrating-mongo-to-postgres.md)