@x12i/countex 1.8.5 → 1.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +867 -860
- package/dist/cli.js +6 -6
- package/dist/core/countex.d.ts +61 -1
- package/dist/core/countex.d.ts.map +1 -1
- package/dist/core/countex.js +241 -0
- package/dist/core/countex.js.map +1 -1
- package/dist/core/managed.d.ts +200 -0
- package/dist/core/managed.d.ts.map +1 -0
- package/dist/core/managed.js +431 -0
- package/dist/core/managed.js.map +1 -0
- package/dist/core/timeframe.d.ts +5 -0
- package/dist/core/timeframe.d.ts.map +1 -1
- package/dist/core/timeframe.js +36 -0
- package/dist/core/timeframe.js.map +1 -1
- package/dist/index.d.ts +6 -2
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +9 -1
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
- package/dist/insights/advisors/resource-advisor.js +5 -1
- package/dist/insights/advisors/resource-advisor.js.map +1 -1
- package/dist/insights/types.d.ts +4 -0
- package/dist/insights/types.d.ts.map +1 -1
- package/dist/storage/managed-memory.d.ts +33 -0
- package/dist/storage/managed-memory.d.ts.map +1 -0
- package/dist/storage/managed-memory.js +211 -0
- package/dist/storage/managed-memory.js.map +1 -0
- package/dist/storage/memory.d.ts +7 -4
- package/dist/storage/memory.d.ts.map +1 -1
- package/dist/storage/memory.js.map +1 -1
- package/dist/storage/postgres.d.ts +123 -0
- package/dist/storage/postgres.d.ts.map +1 -0
- package/dist/storage/postgres.js +1325 -0
- package/dist/storage/postgres.js.map +1 -0
- package/dist/storage/redis.d.ts +6 -0
- package/dist/storage/redis.d.ts.map +1 -1
- package/dist/storage/redis.js +21 -0
- package/dist/storage/redis.js.map +1 -1
- package/dist/types/index.d.ts +63 -5
- package/dist/types/index.d.ts.map +1 -1
- package/dist/utils/errors.d.ts +12 -0
- package/dist/utils/errors.d.ts.map +1 -1
- package/dist/utils/errors.js +22 -1
- package/dist/utils/errors.js.map +1 -1
- package/docs/DEPLOYMENT.md +399 -375
- package/docs/END_TO_END_EXAMPLE.md +316 -316
- package/docs/INSIGHTS.md +59 -59
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
- package/docs/guides/README.md +37 -34
- package/docs/guides/adapter-memory.md +60 -60
- package/docs/guides/adapter-mongo.md +64 -62
- package/docs/guides/adapter-postgres.md +92 -0
- package/docs/guides/adapter-redis.md +89 -89
- package/docs/guides/backup-export.md +135 -135
- package/docs/guides/cardinality.md +50 -50
- package/docs/guides/cli.md +53 -53
- package/docs/guides/client-events.md +65 -65
- package/docs/guides/counter-maps.md +160 -160
- package/docs/guides/idempotency.md +39 -39
- package/docs/guides/ingestion-and-flush.md +67 -67
- package/docs/guides/insights-advisors-and-optimization.md +39 -39
- package/docs/guides/insights-anomalies-and-rules.md +48 -48
- package/docs/guides/insights-comparison-and-trends.md +47 -47
- package/docs/guides/insights-forecasting-and-budget.md +61 -61
- package/docs/guides/insights-overview.md +82 -82
- package/docs/guides/insights-ranking.md +81 -81
- package/docs/guides/managed-commit.md +119 -0
- package/docs/guides/measures.md +72 -71
- package/docs/guides/migrating-mongo-to-postgres.md +127 -0
- package/docs/guides/preview.md +92 -92
- package/docs/guides/querying.md +178 -178
- package/docs/guides/simulation-profiles.md +33 -33
- package/docs/guides/simulation-resource-estimation.md +31 -30
- package/docs/guides/simulation.md +36 -36
- package/docs/guides/timeframes-and-bucketing.md +55 -55
- package/docs/internals/README.md +21 -21
- package/docs/internals/internal-architecture-overview.md +65 -65
- package/docs/internals/internal-backup-stream.md +23 -23
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
- package/docs/internals/internal-cardinality.md +30 -30
- package/docs/internals/internal-dimension-hash.md +34 -34
- package/docs/internals/internal-idempotency.md +36 -36
- package/docs/internals/internal-ingest-pipeline.md +52 -52
- package/docs/internals/internal-insights-layer.md +36 -36
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
- package/docs/internals/internal-measures-and-scales.md +35 -35
- package/docs/internals/internal-query-path.md +46 -46
- package/docs/internals/internal-storage-memory.md +13 -13
- package/docs/internals/internal-storage-mongo.md +44 -44
- package/docs/internals/internal-storage-postgres.md +32 -0
- package/docs/internals/internal-storage-redis.md +63 -63
- package/docs/internals/internal-timeframe-bucketing.md +33 -33
- package/docs/use-cases/README.md +56 -56
- package/docs/use-cases/audit-security-log-counts.md +550 -550
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
- package/docs/use-cases/multi-tenant-isolation.md +203 -203
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
- package/docs/use-cases/processing-workflow-activity.md +165 -165
- package/docs/use-cases/product-feature-usage.md +240 -240
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
- package/package.json +190 -170
|
@@ -1,52 +1,52 @@
|
|
|
1
|
-
# Internal: Ingest pipeline (`ingestBatch`)
|
|
2
|
-
|
|
3
|
-
`Countex.ingest` delegates to `ingestBatch` (`src/core/countex.ts`). The implementation is intentionally phased so CPU work stays synchronous while IO stays bounded.
|
|
4
|
-
|
|
5
|
-
## Sequence diagram
|
|
6
|
-
|
|
7
|
-
```mermaid
|
|
8
|
-
sequenceDiagram
|
|
9
|
-
participant App
|
|
10
|
-
participant CX as Countex
|
|
11
|
-
participant EXT as extractIncrements
|
|
12
|
-
participant CACHE as CacheStorage
|
|
13
|
-
participant CARD as CardinalityGuard
|
|
14
|
-
participant DUR as DurableStorage
|
|
15
|
-
participant BUF as DeltaBuffer
|
|
16
|
-
App->>CX: ingestBatch(events)
|
|
17
|
-
loop each event × map
|
|
18
|
-
CX->>EXT: extractIncrements(map, ev)
|
|
19
|
-
EXT-->>CX: RawIncrement[]
|
|
20
|
-
end
|
|
21
|
-
CX->>CACHE: markEventSeen (optional, parallel)
|
|
22
|
-
CACHE-->>CX: boolean flags
|
|
23
|
-
CX->>CARD: check subject/metric/dimensions
|
|
24
|
-
CARD-->>CX: rewritten keys or drop
|
|
25
|
-
CX->>DUR: resolveIds(lookups)
|
|
26
|
-
DUR-->>CX: Map lookup → numeric id
|
|
27
|
-
loop each RawIncrement × timeframe
|
|
28
|
-
CX->>BUF: add(identity, delta, measures)
|
|
29
|
-
end
|
|
30
|
-
CX->>CX: maybe flush() on thresholds
|
|
31
|
-
```
|
|
32
|
-
|
|
33
|
-
## Phase notes
|
|
34
|
-
|
|
35
|
-
1. **Extract** — warnings for measures surface as `measure_invalid` events when meta is present.
|
|
36
|
-
2. **Dedupe** — `markEventSeen` receives `${counterId}:${eventId}` to isolate maps.
|
|
37
|
-
3. **Cardinality** — dimension iteration may drop with `dropped` reason `cardinality:dimension`.
|
|
38
|
-
4. **Dictionary** — only **subjects** and **metrics** resolve via `resolveIds`; dimension values remain literal on cells.
|
|
39
|
-
5. **Buffer key** — identity includes `dimensionsHash` (`src/utils/hash.ts`).
|
|
40
|
-
6. **Timeframe expansion** — `bucketKey(tf, eventTime)` invoked per configured timeframe.
|
|
41
|
-
|
|
42
|
-
## Flush coupling
|
|
43
|
-
|
|
44
|
-
Threshold triggers call `flush()`:
|
|
45
|
-
|
|
46
|
-
- `onOverflow: "block"` awaits flush (back-pressure).
|
|
47
|
-
- `"drop"` schedules flush without awaiting.
|
|
48
|
-
|
|
49
|
-
## See also
|
|
50
|
-
|
|
51
|
-
- [Buffer merge & flush](./internal-buffer-merge-and-flush.md)
|
|
52
|
-
- [Cardinality](./internal-cardinality.md)
|
|
1
|
+
# Internal: Ingest pipeline (`ingestBatch`)
|
|
2
|
+
|
|
3
|
+
`Countex.ingest` delegates to `ingestBatch` (`src/core/countex.ts`). The implementation is intentionally phased so CPU work stays synchronous while IO stays bounded.
|
|
4
|
+
|
|
5
|
+
## Sequence diagram
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
sequenceDiagram
|
|
9
|
+
participant App
|
|
10
|
+
participant CX as Countex
|
|
11
|
+
participant EXT as extractIncrements
|
|
12
|
+
participant CACHE as CacheStorage
|
|
13
|
+
participant CARD as CardinalityGuard
|
|
14
|
+
participant DUR as DurableStorage
|
|
15
|
+
participant BUF as DeltaBuffer
|
|
16
|
+
App->>CX: ingestBatch(events)
|
|
17
|
+
loop each event × map
|
|
18
|
+
CX->>EXT: extractIncrements(map, ev)
|
|
19
|
+
EXT-->>CX: RawIncrement[]
|
|
20
|
+
end
|
|
21
|
+
CX->>CACHE: markEventSeen (optional, parallel)
|
|
22
|
+
CACHE-->>CX: boolean flags
|
|
23
|
+
CX->>CARD: check subject/metric/dimensions
|
|
24
|
+
CARD-->>CX: rewritten keys or drop
|
|
25
|
+
CX->>DUR: resolveIds(lookups)
|
|
26
|
+
DUR-->>CX: Map lookup → numeric id
|
|
27
|
+
loop each RawIncrement × timeframe
|
|
28
|
+
CX->>BUF: add(identity, delta, measures)
|
|
29
|
+
end
|
|
30
|
+
CX->>CX: maybe flush() on thresholds
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## Phase notes
|
|
34
|
+
|
|
35
|
+
1. **Extract** — warnings for measures surface as `measure_invalid` events when meta is present.
|
|
36
|
+
2. **Dedupe** — `markEventSeen` receives `${counterId}:${eventId}` to isolate maps.
|
|
37
|
+
3. **Cardinality** — dimension iteration may drop with `dropped` reason `cardinality:dimension`.
|
|
38
|
+
4. **Dictionary** — only **subjects** and **metrics** resolve via `resolveIds`; dimension values remain literal on cells.
|
|
39
|
+
5. **Buffer key** — identity includes `dimensionsHash` (`src/utils/hash.ts`).
|
|
40
|
+
6. **Timeframe expansion** — `bucketKey(tf, eventTime)` invoked per configured timeframe.
|
|
41
|
+
|
|
42
|
+
## Flush coupling
|
|
43
|
+
|
|
44
|
+
Threshold triggers call `flush()`:
|
|
45
|
+
|
|
46
|
+
- `onOverflow: "block"` awaits flush (back-pressure).
|
|
47
|
+
- `"drop"` schedules flush without awaiting.
|
|
48
|
+
|
|
49
|
+
## See also
|
|
50
|
+
|
|
51
|
+
- [Buffer merge & flush](./internal-buffer-merge-and-flush.md)
|
|
52
|
+
- [Cardinality](./internal-cardinality.md)
|
|
@@ -1,36 +1,36 @@
|
|
|
1
|
-
# Internal: Insights layer
|
|
2
|
-
|
|
3
|
-
`CountexInsights` (`src/insights/insights-client.ts`) wraps an existing `Countex` instance.
|
|
4
|
-
|
|
5
|
-
## Pattern
|
|
6
|
-
|
|
7
|
-
```mermaid
|
|
8
|
-
flowchart LR
|
|
9
|
-
IN[Insight input] --> QB[baseToQuerySpec +\nparseMeasureSelector]
|
|
10
|
-
QB --> Q[countex.query]
|
|
11
|
-
Q --> POST[ranking / anomaly /\nforecast helpers]
|
|
12
|
-
POST --> OUT[typed result]
|
|
13
|
-
```
|
|
14
|
-
|
|
15
|
-
Shared utilities:
|
|
16
|
-
|
|
17
|
-
- `utils/query-builder.ts` translates insight inputs → `QuerySpec`.
|
|
18
|
-
- Ranking helpers sort rows, compute totals, enforce `maxGroups`.
|
|
19
|
-
- Anomaly pipeline fetches **eval** + **baseline** timeseries concurrently (`detectAnomalies`).
|
|
20
|
-
- Forecast/budget/token modules emit recommendations and bubble events.
|
|
21
|
-
|
|
22
|
-
## Event bridge
|
|
23
|
-
|
|
24
|
-
Methods emit:
|
|
25
|
-
|
|
26
|
-
- `insight_query` — performance telemetry
|
|
27
|
-
- `anomaly`, `recommendation`, `insufficient_data` — operator-facing signals
|
|
28
|
-
|
|
29
|
-
## Resource advisor
|
|
30
|
-
|
|
31
|
-
`estimateResourcesFromActuals` bypasses normal query formatting and streams cells via `durable.streamCells`, requiring the durable reference injected at construction.
|
|
32
|
-
|
|
33
|
-
## See also
|
|
34
|
-
|
|
35
|
-
- [Query path](./internal-query-path.md)
|
|
36
|
-
- `docs/INSIGHTS.md` for product-level narrative
|
|
1
|
+
# Internal: Insights layer
|
|
2
|
+
|
|
3
|
+
`CountexInsights` (`src/insights/insights-client.ts`) wraps an existing `Countex` instance.
|
|
4
|
+
|
|
5
|
+
## Pattern
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
flowchart LR
|
|
9
|
+
IN[Insight input] --> QB[baseToQuerySpec +\nparseMeasureSelector]
|
|
10
|
+
QB --> Q[countex.query]
|
|
11
|
+
Q --> POST[ranking / anomaly /\nforecast helpers]
|
|
12
|
+
POST --> OUT[typed result]
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Shared utilities:
|
|
16
|
+
|
|
17
|
+
- `utils/query-builder.ts` translates insight inputs → `QuerySpec`.
|
|
18
|
+
- Ranking helpers sort rows, compute totals, enforce `maxGroups`.
|
|
19
|
+
- Anomaly pipeline fetches **eval** + **baseline** timeseries concurrently (`detectAnomalies`).
|
|
20
|
+
- Forecast/budget/token modules emit recommendations and bubble events.
|
|
21
|
+
|
|
22
|
+
## Event bridge
|
|
23
|
+
|
|
24
|
+
Methods emit:
|
|
25
|
+
|
|
26
|
+
- `insight_query` — performance telemetry
|
|
27
|
+
- `anomaly`, `recommendation`, `insufficient_data` — operator-facing signals
|
|
28
|
+
|
|
29
|
+
## Resource advisor
|
|
30
|
+
|
|
31
|
+
`estimateResourcesFromActuals` bypasses normal query formatting and streams cells via `durable.streamCells`, requiring the durable reference injected at construction.
|
|
32
|
+
|
|
33
|
+
## See also
|
|
34
|
+
|
|
35
|
+
- [Query path](./internal-query-path.md)
|
|
36
|
+
- `docs/INSIGHTS.md` for product-level narrative
|
|
@@ -1,59 +1,59 @@
|
|
|
1
|
-
# Internal: Mapping validation & extraction
|
|
2
|
-
|
|
3
|
-
## Startup validation
|
|
4
|
-
|
|
5
|
-
`validateMaps` runs once inside the `Countex` constructor (`src/core/mapping.ts`):
|
|
6
|
-
|
|
7
|
-
- Ensures unique `counterId` values.
|
|
8
|
-
- Parses every configured JSONPath with `parsePath`.
|
|
9
|
-
- Verifies required structural rules (non-empty subjects/metrics/timeframes, valid timeframe enums, measure aggregation enums, idempotency consistency).
|
|
10
|
-
|
|
11
|
-
Failures throw **`MappingError`** immediately.
|
|
12
|
-
|
|
13
|
-
## Extraction pipeline
|
|
14
|
-
|
|
15
|
-
`extractIncrements(map, event)` (`src/core/extraction.ts`) executes in order:
|
|
16
|
-
|
|
17
|
-
```mermaid
|
|
18
|
-
flowchart TD
|
|
19
|
-
A[extract scope] -->|missing| SKIP[skipped event]
|
|
20
|
-
A --> B[extract eventTime]
|
|
21
|
-
B -->|invalid| SKIP
|
|
22
|
-
B --> C[optional eventId for idempotency]
|
|
23
|
-
C --> D[resolve delta]
|
|
24
|
-
D -->|zero| SKIP
|
|
25
|
-
D --> E[fan-out subjects]
|
|
26
|
-
E -->|none| SKIP
|
|
27
|
-
E --> F[fan-out metrics]
|
|
28
|
-
F -->|none| SKIP
|
|
29
|
-
F --> G[dimensions]
|
|
30
|
-
G -->|required missing| SKIP
|
|
31
|
-
G --> H[measures]
|
|
32
|
-
H -->|drop policy| SKIP
|
|
33
|
-
H --> I[cross product increments]
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
### Fan-out
|
|
37
|
-
|
|
38
|
-
Nested loops build `RawIncrement` rows:
|
|
39
|
-
|
|
40
|
-
```
|
|
41
|
-
for subject in subjectFanout:
|
|
42
|
-
for metricKey in metricFanout:
|
|
43
|
-
push RawIncrement(... timeframes: map.timeframes ...)
|
|
44
|
-
```
|
|
45
|
-
|
|
46
|
-
Each increment carries **copies** of dimensions and measures for that Cartesian tuple.
|
|
47
|
-
|
|
48
|
-
### Measures
|
|
49
|
-
|
|
50
|
-
`extractMeasuresForEvent` coerces numbers, applies `scale` truncation rules, and maps declared aggregations into pending contributions via `rawMeasuresToPendingDelta` during buffering.
|
|
51
|
-
|
|
52
|
-
### JSON paths
|
|
53
|
-
|
|
54
|
-
`extract` (`src/utils/jsonpath.ts`) evaluates the supported subset: properties, indices, and `[*]` expansion.
|
|
55
|
-
|
|
56
|
-
## See also
|
|
57
|
-
|
|
58
|
-
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
59
|
-
- [Measures & scales](./internal-measures-and-scales.md)
|
|
1
|
+
# Internal: Mapping validation & extraction
|
|
2
|
+
|
|
3
|
+
## Startup validation
|
|
4
|
+
|
|
5
|
+
`validateMaps` runs once inside the `Countex` constructor (`src/core/mapping.ts`):
|
|
6
|
+
|
|
7
|
+
- Ensures unique `counterId` values.
|
|
8
|
+
- Parses every configured JSONPath with `parsePath`.
|
|
9
|
+
- Verifies required structural rules (non-empty subjects/metrics/timeframes, valid timeframe enums, measure aggregation enums, idempotency consistency).
|
|
10
|
+
|
|
11
|
+
Failures throw **`MappingError`** immediately.
|
|
12
|
+
|
|
13
|
+
## Extraction pipeline
|
|
14
|
+
|
|
15
|
+
`extractIncrements(map, event)` (`src/core/extraction.ts`) executes in order:
|
|
16
|
+
|
|
17
|
+
```mermaid
|
|
18
|
+
flowchart TD
|
|
19
|
+
A[extract scope] -->|missing| SKIP[skipped event]
|
|
20
|
+
A --> B[extract eventTime]
|
|
21
|
+
B -->|invalid| SKIP
|
|
22
|
+
B --> C[optional eventId for idempotency]
|
|
23
|
+
C --> D[resolve delta]
|
|
24
|
+
D -->|zero| SKIP
|
|
25
|
+
D --> E[fan-out subjects]
|
|
26
|
+
E -->|none| SKIP
|
|
27
|
+
E --> F[fan-out metrics]
|
|
28
|
+
F -->|none| SKIP
|
|
29
|
+
F --> G[dimensions]
|
|
30
|
+
G -->|required missing| SKIP
|
|
31
|
+
G --> H[measures]
|
|
32
|
+
H -->|drop policy| SKIP
|
|
33
|
+
H --> I[cross product increments]
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
### Fan-out
|
|
37
|
+
|
|
38
|
+
Nested loops build `RawIncrement` rows:
|
|
39
|
+
|
|
40
|
+
```
|
|
41
|
+
for subject in subjectFanout:
|
|
42
|
+
for metricKey in metricFanout:
|
|
43
|
+
push RawIncrement(... timeframes: map.timeframes ...)
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Each increment carries **copies** of dimensions and measures for that Cartesian tuple.
|
|
47
|
+
|
|
48
|
+
### Measures
|
|
49
|
+
|
|
50
|
+
`extractMeasuresForEvent` coerces numbers, applies `scale` truncation rules, and maps declared aggregations into pending contributions via `rawMeasuresToPendingDelta` during buffering.
|
|
51
|
+
|
|
52
|
+
### JSON paths
|
|
53
|
+
|
|
54
|
+
`extract` (`src/utils/jsonpath.ts`) evaluates the supported subset: properties, indices, and `[*]` expansion.
|
|
55
|
+
|
|
56
|
+
## See also
|
|
57
|
+
|
|
58
|
+
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
59
|
+
- [Measures & scales](./internal-measures-and-scales.md)
|
|
@@ -1,35 +1,35 @@
|
|
|
1
|
-
# Internal: Measures & scales
|
|
2
|
-
|
|
3
|
-
## Effective aggregations
|
|
4
|
-
|
|
5
|
-
`effectiveMeasureAggregations` (`src/core/extraction.ts`) ensures **`count`** exists whenever **`sum`**, **`min`**, or **`max`** are present so averages remain computable later.
|
|
6
|
-
|
|
7
|
-
## Pending deltas
|
|
8
|
-
|
|
9
|
-
`rawMeasuresToPendingDelta` converts each extracted measure into:
|
|
10
|
-
|
|
11
|
-
- `sum` contribution
|
|
12
|
-
- `count` increment (`1` when enabled)
|
|
13
|
-
- `min` / `max` seeds
|
|
14
|
-
|
|
15
|
-
These mirror what storage adapters must merge atomically.
|
|
16
|
-
|
|
17
|
-
## Buffer merging
|
|
18
|
-
|
|
19
|
-
`mergeMeasureDelta` (`src/core/buffer.ts`) applies associative merges identical to eventual durable semantics.
|
|
20
|
-
|
|
21
|
-
## Scale metadata
|
|
22
|
-
|
|
23
|
-
Maps may declare `scale` per measure. During ingest, numeric extraction truncates after multiply; `measureScales` propagate into buffer entries so adapters interpret stored integers consistently.
|
|
24
|
-
|
|
25
|
-
## Query-time averages
|
|
26
|
-
|
|
27
|
-
Formatter / row builders derive **`avg = sum / count`** when both components exist.
|
|
28
|
-
|
|
29
|
-
## Cache limitation
|
|
30
|
-
|
|
31
|
-
Planner denies cache for queries needing min/max fidelity — see `queryNeedsMinMax` in `src/query/planner.ts`.
|
|
32
|
-
|
|
33
|
-
## See also
|
|
34
|
-
|
|
35
|
-
- [Query path](./internal-query-path.md)
|
|
1
|
+
# Internal: Measures & scales
|
|
2
|
+
|
|
3
|
+
## Effective aggregations
|
|
4
|
+
|
|
5
|
+
`effectiveMeasureAggregations` (`src/core/extraction.ts`) ensures **`count`** exists whenever **`sum`**, **`min`**, or **`max`** are present so averages remain computable later.
|
|
6
|
+
|
|
7
|
+
## Pending deltas
|
|
8
|
+
|
|
9
|
+
`rawMeasuresToPendingDelta` converts each extracted measure into:
|
|
10
|
+
|
|
11
|
+
- `sum` contribution
|
|
12
|
+
- `count` increment (`1` when enabled)
|
|
13
|
+
- `min` / `max` seeds
|
|
14
|
+
|
|
15
|
+
These mirror what storage adapters must merge atomically.
|
|
16
|
+
|
|
17
|
+
## Buffer merging
|
|
18
|
+
|
|
19
|
+
`mergeMeasureDelta` (`src/core/buffer.ts`) applies associative merges identical to eventual durable semantics.
|
|
20
|
+
|
|
21
|
+
## Scale metadata
|
|
22
|
+
|
|
23
|
+
Maps may declare `scale` per measure. During ingest, numeric extraction truncates after multiply; `measureScales` propagate into buffer entries so adapters interpret stored integers consistently.
|
|
24
|
+
|
|
25
|
+
## Query-time averages
|
|
26
|
+
|
|
27
|
+
Formatter / row builders derive **`avg = sum / count`** when both components exist.
|
|
28
|
+
|
|
29
|
+
## Cache limitation
|
|
30
|
+
|
|
31
|
+
Planner denies cache for queries needing min/max fidelity — see `queryNeedsMinMax` in `src/query/planner.ts`.
|
|
32
|
+
|
|
33
|
+
## See also
|
|
34
|
+
|
|
35
|
+
- [Query path](./internal-query-path.md)
|
|
@@ -1,46 +1,46 @@
|
|
|
1
|
-
# Internal: Query path
|
|
2
|
-
|
|
3
|
-
`Countex.query` delegates to `executeQuery` (`src/query/planner.ts`) with optional measure metadata from `buildMeasureAggIndex(map)`.
|
|
4
|
-
|
|
5
|
-
## Decision flow
|
|
6
|
-
|
|
7
|
-
```mermaid
|
|
8
|
-
flowchart TD
|
|
9
|
-
Q[QuerySpec] --> C{cache configured\nand not bypassed\nand cacheCanServe?}
|
|
10
|
-
C -->|yes| R1[cache.readCells]
|
|
11
|
-
R1 -->|coverage full| F[formatResult source=cache]
|
|
12
|
-
R1 -->|partial/miss| D[durable.readCells]
|
|
13
|
-
C -->|no| D
|
|
14
|
-
D --> W{cache && rows && not bypass?}
|
|
15
|
-
W -->|yes| WARM[cache.warm best-effort]
|
|
16
|
-
W -->|no| F2[formatResult source=merged or durable]
|
|
17
|
-
WARM --> F2
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
## Cache eligibility (`cacheCanServe`)
|
|
21
|
-
|
|
22
|
-
When the counter map is known, queries that require **min/max** measure accuracy skip cache:
|
|
23
|
-
|
|
24
|
-
- `orderBy` targeting min/max aggregation.
|
|
25
|
-
- Explicit measure list referencing min/max-maintained measures.
|
|
26
|
-
- `measures: "*"` while any declared measure tracks min/max.
|
|
27
|
-
|
|
28
|
-
This avoids stale extrema because Redis stores sum/count only for measures.
|
|
29
|
-
|
|
30
|
-
## Formatter responsibilities (`formatResult`)
|
|
31
|
-
|
|
32
|
-
`src/query/formatter.ts`:
|
|
33
|
-
|
|
34
|
-
- Groups cells by `groupBy` keys.
|
|
35
|
-
- Builds `table` / `matrix` / `timeseries` projections.
|
|
36
|
-
- Applies **`rate`** using `bucketDurationSeconds` except timeframe `all`.
|
|
37
|
-
- Sorts and truncates via `orderBy` / `limit`.
|
|
38
|
-
|
|
39
|
-
## Bucket expansion
|
|
40
|
-
|
|
41
|
-
Range queries call `bucketRange` (`src/core/timeframe.ts`) capped at **10,000** buckets.
|
|
42
|
-
|
|
43
|
-
## See also
|
|
44
|
-
|
|
45
|
-
- [Timeframe bucketing](./internal-timeframe-bucketing.md)
|
|
46
|
-
- [Measures & scales](./internal-measures-and-scales.md)
|
|
1
|
+
# Internal: Query path
|
|
2
|
+
|
|
3
|
+
`Countex.query` delegates to `executeQuery` (`src/query/planner.ts`) with optional measure metadata from `buildMeasureAggIndex(map)`.
|
|
4
|
+
|
|
5
|
+
## Decision flow
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
flowchart TD
|
|
9
|
+
Q[QuerySpec] --> C{cache configured\nand not bypassed\nand cacheCanServe?}
|
|
10
|
+
C -->|yes| R1[cache.readCells]
|
|
11
|
+
R1 -->|coverage full| F[formatResult source=cache]
|
|
12
|
+
R1 -->|partial/miss| D[durable.readCells]
|
|
13
|
+
C -->|no| D
|
|
14
|
+
D --> W{cache && rows && not bypass?}
|
|
15
|
+
W -->|yes| WARM[cache.warm best-effort]
|
|
16
|
+
W -->|no| F2[formatResult source=merged or durable]
|
|
17
|
+
WARM --> F2
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Cache eligibility (`cacheCanServe`)
|
|
21
|
+
|
|
22
|
+
When the counter map is known, queries that require **min/max** measure accuracy skip cache:
|
|
23
|
+
|
|
24
|
+
- `orderBy` targeting min/max aggregation.
|
|
25
|
+
- Explicit measure list referencing min/max-maintained measures.
|
|
26
|
+
- `measures: "*"` while any declared measure tracks min/max.
|
|
27
|
+
|
|
28
|
+
This avoids stale extrema because Redis stores sum/count only for measures.
|
|
29
|
+
|
|
30
|
+
## Formatter responsibilities (`formatResult`)
|
|
31
|
+
|
|
32
|
+
`src/query/formatter.ts`:
|
|
33
|
+
|
|
34
|
+
- Groups cells by `groupBy` keys.
|
|
35
|
+
- Builds `table` / `matrix` / `timeseries` projections.
|
|
36
|
+
- Applies **`rate`** using `bucketDurationSeconds` except timeframe `all`.
|
|
37
|
+
- Sorts and truncates via `orderBy` / `limit`.
|
|
38
|
+
|
|
39
|
+
## Bucket expansion
|
|
40
|
+
|
|
41
|
+
Range queries call `bucketRange` (`src/core/timeframe.ts`) capped at **10,000** buckets.
|
|
42
|
+
|
|
43
|
+
## See also
|
|
44
|
+
|
|
45
|
+
- [Timeframe bucketing](./internal-timeframe-bucketing.md)
|
|
46
|
+
- [Measures & scales](./internal-measures-and-scales.md)
|
|
@@ -1,13 +1,13 @@
|
|
|
1
|
-
# Internal: Memory adapters
|
|
2
|
-
|
|
3
|
-
`MemoryDurable` and `MemoryCache` (`src/storage/memory.ts`) mirror production contracts using `Map` structures:
|
|
4
|
-
|
|
5
|
-
- Dictionary resolution mimics monotonic ids.
|
|
6
|
-
- Cells keyed similarly to durable identity tuples.
|
|
7
|
-
- Cache implements simplified coverage semantics for tests.
|
|
8
|
-
|
|
9
|
-
Used heavily by unit tests (`tests/unit/*.test.ts`) because they require no external services.
|
|
10
|
-
|
|
11
|
-
## See also
|
|
12
|
-
|
|
13
|
-
- [Architecture overview](./internal-architecture-overview.md)
|
|
1
|
+
# Internal: Memory adapters
|
|
2
|
+
|
|
3
|
+
`MemoryDurable` and `MemoryCache` (`src/storage/memory.ts`) mirror production contracts using `Map` structures:
|
|
4
|
+
|
|
5
|
+
- Dictionary resolution mimics monotonic ids.
|
|
6
|
+
- Cells keyed similarly to durable identity tuples.
|
|
7
|
+
- Cache implements simplified coverage semantics for tests.
|
|
8
|
+
|
|
9
|
+
Used heavily by unit tests (`tests/unit/*.test.ts`) because they require no external services.
|
|
10
|
+
|
|
11
|
+
## See also
|
|
12
|
+
|
|
13
|
+
- [Architecture overview](./internal-architecture-overview.md)
|
|
@@ -1,44 +1,44 @@
|
|
|
1
|
-
# Internal: Mongo durable adapter
|
|
2
|
-
|
|
3
|
-
`MongoDurable` (`src/storage/mongo.ts`) implements:
|
|
4
|
-
|
|
5
|
-
- `applyDeltas` — upserts with `$inc` style operations for counts and compatible measure fields.
|
|
6
|
-
- `readCells` — satisfies `QuerySpec` filters using compound indexes.
|
|
7
|
-
- `resolveIds` — concurrency-safe allocation of dictionary ids (`countex_dict`, `countex_seq`).
|
|
8
|
-
- `streamCells` — bounded iteration for backup/export.
|
|
9
|
-
- `importSnapshotCells` — replace-window restore target (called by `restoreSnapshotCells`).
|
|
10
|
-
|
|
11
|
-
## Write path interaction
|
|
12
|
-
|
|
13
|
-
Flush snapshots arrive as arrays of `PendingDelta`; Mongo applies them idempotently per cell identity tuple via `bulkWrite` of `updateOne` operations with `$inc` and `upsert: true`.
|
|
14
|
-
|
|
15
|
-
## Indexes
|
|
16
|
-
|
|
17
|
-
Created lazily on first write paths:
|
|
18
|
-
|
|
19
|
-
- `countex_cells` — `uniq_cell_identity` (unique on the full identity tuple), `by_timeframe`, `by_subject`, `by_metric`.
|
|
20
|
-
- `countex_dict` — `uniq_dict_key` (unique on `scopeId, kind, qualifier, sourceKey`).
|
|
21
|
-
|
|
22
|
-
## Dictionary allocation under concurrency
|
|
23
|
-
|
|
24
|
-
`resolveIds` follows a strict five-step recipe so 20+ workers seeing the same new `sourceKey` at once still converge on a single internal id:
|
|
25
|
-
|
|
26
|
-
1. **Read** existing dict rows for the requested `(scopeId, kind, qualifier, sourceKey)` batch in one round-trip.
|
|
27
|
-
2. **Allocate** a contiguous block of ids from the per-scope sequence document via `findOneAndUpdate({ _id }, { $inc: { seq: count } })` — atomic per sequence document.
|
|
28
|
-
3. **Insert** the new dict rows with `insertMany({ ordered: false })`. The `uniq_dict_key` index forbids duplicates.
|
|
29
|
-
4. On **duplicate-key error** (another worker won the race), **re-read** the dict for the same identities — the winning row is now visible.
|
|
30
|
-
5. **Cache** the resolved id in the local LRU and (optionally) Redis.
|
|
31
|
-
|
|
32
|
-
Bad outcome avoided: two workers each minting `user_42 → 10001` and `user_42 → 10002`. With this protocol one insert wins, the loser re-reads the winner's id, and counts never fragment across two ids.
|
|
33
|
-
|
|
34
|
-
The user-facing recipe lives in [`docs/guides/adapter-mongo.md`](../guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency".
|
|
35
|
-
|
|
36
|
-
## Dictionary vs dimensions
|
|
37
|
-
|
|
38
|
-
Subjects/metrics map to integers via `countex_dict`. Dimension VALUES are stored as raw values on cells (not via the dictionary), keeping cells self-describing and avoiding dictionary churn for low-cardinality dimensions.
|
|
39
|
-
|
|
40
|
-
## See also
|
|
41
|
-
|
|
42
|
-
- [Backup streaming](./internal-backup-stream.md)
|
|
43
|
-
- [Dimension hash](./internal-dimension-hash.md) — how the `dimensionsHash` part of the cell identity is computed.
|
|
44
|
-
- [Mongo adapter guide](../guides/adapter-mongo.md)
|
|
1
|
+
# Internal: Mongo durable adapter
|
|
2
|
+
|
|
3
|
+
`MongoDurable` (`src/storage/mongo.ts`) implements:
|
|
4
|
+
|
|
5
|
+
- `applyDeltas` — upserts with `$inc` style operations for counts and compatible measure fields.
|
|
6
|
+
- `readCells` — satisfies `QuerySpec` filters using compound indexes.
|
|
7
|
+
- `resolveIds` — concurrency-safe allocation of dictionary ids (`countex_dict`, `countex_seq`).
|
|
8
|
+
- `streamCells` — bounded iteration for backup/export.
|
|
9
|
+
- `importSnapshotCells` — replace-window restore target (called by `restoreSnapshotCells`).
|
|
10
|
+
|
|
11
|
+
## Write path interaction
|
|
12
|
+
|
|
13
|
+
Flush snapshots arrive as arrays of `PendingDelta`; Mongo applies them idempotently per cell identity tuple via `bulkWrite` of `updateOne` operations with `$inc` and `upsert: true`.
|
|
14
|
+
|
|
15
|
+
## Indexes
|
|
16
|
+
|
|
17
|
+
Created lazily on first write paths:
|
|
18
|
+
|
|
19
|
+
- `countex_cells` — `uniq_cell_identity` (unique on the full identity tuple), `by_timeframe`, `by_subject`, `by_metric`.
|
|
20
|
+
- `countex_dict` — `uniq_dict_key` (unique on `scopeId, kind, qualifier, sourceKey`).
|
|
21
|
+
|
|
22
|
+
## Dictionary allocation under concurrency
|
|
23
|
+
|
|
24
|
+
`resolveIds` follows a strict five-step recipe so 20+ workers seeing the same new `sourceKey` at once still converge on a single internal id:
|
|
25
|
+
|
|
26
|
+
1. **Read** existing dict rows for the requested `(scopeId, kind, qualifier, sourceKey)` batch in one round-trip.
|
|
27
|
+
2. **Allocate** a contiguous block of ids from the per-scope sequence document via `findOneAndUpdate({ _id }, { $inc: { seq: count } })` — atomic per sequence document.
|
|
28
|
+
3. **Insert** the new dict rows with `insertMany({ ordered: false })`. The `uniq_dict_key` index forbids duplicates.
|
|
29
|
+
4. On **duplicate-key error** (another worker won the race), **re-read** the dict for the same identities — the winning row is now visible.
|
|
30
|
+
5. **Cache** the resolved id in the local LRU and (optionally) Redis.
|
|
31
|
+
|
|
32
|
+
Bad outcome avoided: two workers each minting `user_42 → 10001` and `user_42 → 10002`. With this protocol one insert wins, the loser re-reads the winner's id, and counts never fragment across two ids.
|
|
33
|
+
|
|
34
|
+
The user-facing recipe lives in [`docs/guides/adapter-mongo.md`](../guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency".
|
|
35
|
+
|
|
36
|
+
## Dictionary vs dimensions
|
|
37
|
+
|
|
38
|
+
Subjects/metrics map to integers via `countex_dict`. Dimension VALUES are stored as raw values on cells (not via the dictionary), keeping cells self-describing and avoiding dictionary churn for low-cardinality dimensions.
|
|
39
|
+
|
|
40
|
+
## See also
|
|
41
|
+
|
|
42
|
+
- [Backup streaming](./internal-backup-stream.md)
|
|
43
|
+
- [Dimension hash](./internal-dimension-hash.md) — how the `dimensionsHash` part of the cell identity is computed.
|
|
44
|
+
- [Mongo adapter guide](../guides/adapter-mongo.md)
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Internal: Postgres durable adapter
|
|
2
|
+
|
|
3
|
+
`PostgresDurable` (`src/storage/postgres.ts`) implements `DurableStorage`:
|
|
4
|
+
|
|
5
|
+
- `applyDeltas` — one transaction of batched `INSERT … ON CONFLICT DO UPDATE`. `merge_measures` adds sum and count, takes least/greatest for min/max, and keeps scale from the existing row on conflict.
|
|
6
|
+
- `readCells` — same filters as Mongo: scope, counter, timeframe buckets, optional subject and metric ids, jsonb dimension equality.
|
|
7
|
+
- `resolveIds` — read, allocate from `seq`, `INSERT … ON CONFLICT DO NOTHING`, re-read.
|
|
8
|
+
- `streamCells` — keyset pages of 500 rows for backup and the resource advisor.
|
|
9
|
+
- `importSnapshotCells` — absolute replace of count and measures (restore).
|
|
10
|
+
- `appendEvidence` — insert, then delete the oldest rows past `maxSamplesAtCapture`.
|
|
11
|
+
|
|
12
|
+
Duplicate identities inside one `applyDeltas` call are collapsed with the same sum/min/max rules as the in-memory buffer before the statement is sent. Postgres rejects an `INSERT` that would update the same conflict target twice.
|
|
13
|
+
|
|
14
|
+
## Indexes and partitions
|
|
15
|
+
|
|
16
|
+
Created on first use of a schema unless `skipIndexInit` is set:
|
|
17
|
+
|
|
18
|
+
- `cells` — unique identity constraint (also serves timeframe lookups), `cells_by_subject`, `cells_by_metric`. List partitions: `cells_minute`, `cells_hour`, `cells_default`.
|
|
19
|
+
- `dict` — unique `(scope_id, kind, qualifier, source_key)`, plus `(scope_id, kind, qualifier, internal_id)`.
|
|
20
|
+
- `evidence` — `(identity…, captured_at)`.
|
|
21
|
+
|
|
22
|
+
`updated_at` is not indexed. Identity columns are not modified by an increment, which is what makes HOT updates possible. Do not add the changing columns to the unique index.
|
|
23
|
+
|
|
24
|
+
## Dictionary vs dimensions
|
|
25
|
+
|
|
26
|
+
Subjects and metrics map to integers via `dict`. Dimension values stay on the cell as jsonb, same as Mongo, so cells stay self-describing.
|
|
27
|
+
|
|
28
|
+
## See also
|
|
29
|
+
|
|
30
|
+
- [Mongo durable](./internal-storage-mongo.md)
|
|
31
|
+
- [Postgres adapter guide](../guides/adapter-postgres.md)
|
|
32
|
+
- [Migrating MongoDB to PostgreSQL](../guides/migrating-mongo-to-postgres.md)
|