@x12i/countex 1.8.3 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +866 -858
- package/dist/cli.js +6 -6
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
- package/dist/insights/advisors/resource-advisor.js +5 -1
- package/dist/insights/advisors/resource-advisor.js.map +1 -1
- package/dist/insights/types.d.ts +4 -0
- package/dist/insights/types.d.ts.map +1 -1
- package/dist/storage/postgres.d.ts +104 -0
- package/dist/storage/postgres.d.ts.map +1 -0
- package/dist/storage/postgres.js +1023 -0
- package/dist/storage/postgres.js.map +1 -0
- package/dist/types/index.d.ts +11 -5
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +399 -375
- package/docs/END_TO_END_EXAMPLE.md +316 -316
- package/docs/INSIGHTS.md +59 -59
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
- package/docs/guides/README.md +36 -34
- package/docs/guides/adapter-memory.md +60 -60
- package/docs/guides/adapter-mongo.md +64 -62
- package/docs/guides/adapter-postgres.md +90 -0
- package/docs/guides/adapter-redis.md +89 -89
- package/docs/guides/backup-export.md +135 -135
- package/docs/guides/cardinality.md +50 -50
- package/docs/guides/cli.md +53 -53
- package/docs/guides/client-events.md +65 -65
- package/docs/guides/counter-maps.md +160 -160
- package/docs/guides/idempotency.md +39 -39
- package/docs/guides/ingestion-and-flush.md +67 -67
- package/docs/guides/insights-advisors-and-optimization.md +39 -39
- package/docs/guides/insights-anomalies-and-rules.md +48 -48
- package/docs/guides/insights-comparison-and-trends.md +47 -47
- package/docs/guides/insights-forecasting-and-budget.md +61 -61
- package/docs/guides/insights-overview.md +82 -82
- package/docs/guides/insights-ranking.md +81 -81
- package/docs/guides/measures.md +71 -71
- package/docs/guides/migrating-mongo-to-postgres.md +127 -0
- package/docs/guides/preview.md +92 -92
- package/docs/guides/querying.md +178 -178
- package/docs/guides/simulation-profiles.md +33 -33
- package/docs/guides/simulation-resource-estimation.md +31 -30
- package/docs/guides/simulation.md +36 -36
- package/docs/guides/timeframes-and-bucketing.md +55 -55
- package/docs/internals/README.md +21 -21
- package/docs/internals/internal-architecture-overview.md +65 -65
- package/docs/internals/internal-backup-stream.md +23 -23
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
- package/docs/internals/internal-cardinality.md +30 -30
- package/docs/internals/internal-dimension-hash.md +34 -34
- package/docs/internals/internal-idempotency.md +36 -36
- package/docs/internals/internal-ingest-pipeline.md +52 -52
- package/docs/internals/internal-insights-layer.md +36 -36
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
- package/docs/internals/internal-measures-and-scales.md +35 -35
- package/docs/internals/internal-query-path.md +46 -46
- package/docs/internals/internal-storage-memory.md +13 -13
- package/docs/internals/internal-storage-mongo.md +44 -44
- package/docs/internals/internal-storage-postgres.md +32 -0
- package/docs/internals/internal-storage-redis.md +63 -63
- package/docs/internals/internal-timeframe-bucketing.md +33 -33
- package/docs/use-cases/README.md +56 -56
- package/docs/use-cases/audit-security-log-counts.md +550 -550
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
- package/docs/use-cases/multi-tenant-isolation.md +203 -203
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
- package/docs/use-cases/processing-workflow-activity.md +165 -165
- package/docs/use-cases/product-feature-usage.md +240 -240
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
- package/package.json +184 -169
|
@@ -1,55 +1,55 @@
|
|
|
1
|
-
# Guide: Timeframes and bucketing
|
|
2
|
-
|
|
3
|
-
Timeframes control **how event time is bucketed** when Countex writes cells. Queries later ask for a **range of bucket keys** (`from` / `to`).
|
|
4
|
-
|
|
5
|
-
## Step 1: Pick timeframe types on the map
|
|
6
|
-
|
|
7
|
-
```ts
|
|
8
|
-
const map = {
|
|
9
|
-
counterId: "events",
|
|
10
|
-
// ...
|
|
11
|
-
timeframes: ["minute", "hour", "day", "week", "month", "quarter", "year", "all"],
|
|
12
|
-
} as const;
|
|
13
|
-
```
|
|
14
|
-
|
|
15
|
-
Only listed timeframes receive writes for each matching event.
|
|
16
|
-
|
|
17
|
-
## Step 2: Understand bucket keys (UTC)
|
|
18
|
-
|
|
19
|
-
| Type | Example key |
|
|
20
|
-
|------|----------------|
|
|
21
|
-
| minute | `2026-05-08T12:30` |
|
|
22
|
-
| hour | `2026-05-08T12` |
|
|
23
|
-
| day | `2026-05-08` |
|
|
24
|
-
| week | `2026-W19` (ISO week) |
|
|
25
|
-
| month | `2026-05` |
|
|
26
|
-
| quarter | `2026-Q2` |
|
|
27
|
-
| year | `2026` |
|
|
28
|
-
| all | `all` |
|
|
29
|
-
|
|
30
|
-
Buckets are derived from **`eventTime`**, not ingestion time, so late data lands in the correct historical bucket.
|
|
31
|
-
|
|
32
|
-
## Step 3: Query a range
|
|
33
|
-
|
|
34
|
-
```ts
|
|
35
|
-
const result = await countex.query({
|
|
36
|
-
scope: "org_1",
|
|
37
|
-
counterId: "events",
|
|
38
|
-
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
|
|
39
|
-
format: "table",
|
|
40
|
-
});
|
|
41
|
-
```
|
|
42
|
-
|
|
43
|
-
## Step 4: Respect the bucket cap
|
|
44
|
-
|
|
45
|
-
Range expansion is capped at **10,000** buckets. Narrow the window or coarsen the timeframe if you hit limits.
|
|
46
|
-
|
|
47
|
-
## Step 5: Rates
|
|
48
|
-
|
|
49
|
-
`rate` in queries divides counts or measure sums by elapsed seconds in the queried window. **`rate` is invalid when timeframe type is `all`** (no meaningful duration).
|
|
50
|
-
|
|
51
|
-
## See also
|
|
52
|
-
|
|
53
|
-
- [Querying](./querying.md)
|
|
54
|
-
- [Measures](./measures.md)
|
|
55
|
-
- Internal: [Timeframe bucketing](../internals/internal-timeframe-bucketing.md)
|
|
1
|
+
# Guide: Timeframes and bucketing
|
|
2
|
+
|
|
3
|
+
Timeframes control **how event time is bucketed** when Countex writes cells. Queries later ask for a **range of bucket keys** (`from` / `to`).
|
|
4
|
+
|
|
5
|
+
## Step 1: Pick timeframe types on the map
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const map = {
|
|
9
|
+
counterId: "events",
|
|
10
|
+
// ...
|
|
11
|
+
timeframes: ["minute", "hour", "day", "week", "month", "quarter", "year", "all"],
|
|
12
|
+
} as const;
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Only listed timeframes receive writes for each matching event.
|
|
16
|
+
|
|
17
|
+
## Step 2: Understand bucket keys (UTC)
|
|
18
|
+
|
|
19
|
+
| Type | Example key |
|
|
20
|
+
|------|----------------|
|
|
21
|
+
| minute | `2026-05-08T12:30` |
|
|
22
|
+
| hour | `2026-05-08T12` |
|
|
23
|
+
| day | `2026-05-08` |
|
|
24
|
+
| week | `2026-W19` (ISO week) |
|
|
25
|
+
| month | `2026-05` |
|
|
26
|
+
| quarter | `2026-Q2` |
|
|
27
|
+
| year | `2026` |
|
|
28
|
+
| all | `all` |
|
|
29
|
+
|
|
30
|
+
Buckets are derived from **`eventTime`**, not ingestion time, so late data lands in the correct historical bucket.
|
|
31
|
+
|
|
32
|
+
## Step 3: Query a range
|
|
33
|
+
|
|
34
|
+
```ts
|
|
35
|
+
const result = await countex.query({
|
|
36
|
+
scope: "org_1",
|
|
37
|
+
counterId: "events",
|
|
38
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
|
|
39
|
+
format: "table",
|
|
40
|
+
});
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Step 4: Respect the bucket cap
|
|
44
|
+
|
|
45
|
+
Range expansion is capped at **10,000** buckets. Narrow the window or coarsen the timeframe if you hit limits.
|
|
46
|
+
|
|
47
|
+
## Step 5: Rates
|
|
48
|
+
|
|
49
|
+
`rate` in queries divides counts or measure sums by elapsed seconds in the queried window. **`rate` is invalid when timeframe type is `all`** (no meaningful duration).
|
|
50
|
+
|
|
51
|
+
## See also
|
|
52
|
+
|
|
53
|
+
- [Querying](./querying.md)
|
|
54
|
+
- [Measures](./measures.md)
|
|
55
|
+
- Internal: [Timeframe bucketing](../internals/internal-timeframe-bucketing.md)
|
package/docs/internals/README.md
CHANGED
|
@@ -1,21 +1,21 @@
|
|
|
1
|
-
# Countex internals
|
|
2
|
-
|
|
3
|
-
Deep dives into how the engine executes ingest, flush, storage IO, and insights post-processing. These documents are **not** imported as package subpaths; they ship in the npm tarball under `docs/internals/` for repository readers and advanced operators.
|
|
4
|
-
|
|
5
|
-
Start here:
|
|
6
|
-
|
|
7
|
-
| Topic | Document |
|
|
8
|
-
|-------|----------|
|
|
9
|
-
| End-to-end architecture | [Architecture overview](./internal-architecture-overview.md) |
|
|
10
|
-
| Map startup validation & JSON extraction | [Mapping validation & extraction](./internal-mapping-validation-and-extraction.md) |
|
|
11
|
-
| `ingestBatch` phases | [Ingest pipeline](./internal-ingest-pipeline.md) |
|
|
12
|
-
| In-memory merge | [Buffer merge & flush](./internal-buffer-merge-and-flush.md) |
|
|
13
|
-
| Reads | [Query path](./internal-query-path.md) |
|
|
14
|
-
| Bucket keys | [Timeframe bucketing](./internal-timeframe-bucketing.md) |
|
|
15
|
-
| Dimension hashing | [Dimension canonicalization & hash](./internal-dimension-hash.md) |
|
|
16
|
-
| Measures | [Measures & scales](./internal-measures-and-scales.md) |
|
|
17
|
-
| Cardinality | [Cardinality guard](./internal-cardinality.md) |
|
|
18
|
-
| Dedupe | [Idempotency](./internal-idempotency.md) |
|
|
19
|
-
| Adapters | [Mongo](./internal-storage-mongo.md), [Redis](./internal-storage-redis.md), [Memory](./internal-storage-memory.md) |
|
|
20
|
-
| Ops helpers | [Backup streaming](./internal-backup-stream.md) |
|
|
21
|
-
| Insights | [Insights layer](./internal-insights-layer.md) |
|
|
1
|
+
# Countex internals
|
|
2
|
+
|
|
3
|
+
Deep dives into how the engine executes ingest, flush, storage IO, and insights post-processing. These documents are **not** imported as package subpaths; they ship in the npm tarball under `docs/internals/` for repository readers and advanced operators.
|
|
4
|
+
|
|
5
|
+
Start here:
|
|
6
|
+
|
|
7
|
+
| Topic | Document |
|
|
8
|
+
|-------|----------|
|
|
9
|
+
| End-to-end architecture | [Architecture overview](./internal-architecture-overview.md) |
|
|
10
|
+
| Map startup validation & JSON extraction | [Mapping validation & extraction](./internal-mapping-validation-and-extraction.md) |
|
|
11
|
+
| `ingestBatch` phases | [Ingest pipeline](./internal-ingest-pipeline.md) |
|
|
12
|
+
| In-memory merge | [Buffer merge & flush](./internal-buffer-merge-and-flush.md) |
|
|
13
|
+
| Reads | [Query path](./internal-query-path.md) |
|
|
14
|
+
| Bucket keys | [Timeframe bucketing](./internal-timeframe-bucketing.md) |
|
|
15
|
+
| Dimension hashing | [Dimension canonicalization & hash](./internal-dimension-hash.md) |
|
|
16
|
+
| Measures | [Measures & scales](./internal-measures-and-scales.md) |
|
|
17
|
+
| Cardinality | [Cardinality guard](./internal-cardinality.md) |
|
|
18
|
+
| Dedupe | [Idempotency](./internal-idempotency.md) |
|
|
19
|
+
| Adapters | [Mongo](./internal-storage-mongo.md), [Postgres](./internal-storage-postgres.md), [Redis](./internal-storage-redis.md), [Memory](./internal-storage-memory.md) |
|
|
20
|
+
| Ops helpers | [Backup streaming](./internal-backup-stream.md) |
|
|
21
|
+
| Insights | [Insights layer](./internal-insights-layer.md) |
|
|
@@ -1,65 +1,65 @@
|
|
|
1
|
-
# Internal: Architecture overview
|
|
2
|
-
|
|
3
|
-
Countex separates **projection** (events → sparse cells), **durability** (authoritative totals), **cache** (hot reads + dedupe), and **query shaping**.
|
|
4
|
-
|
|
5
|
-
## Component diagram
|
|
6
|
-
|
|
7
|
-
```mermaid
|
|
8
|
-
flowchart LR
|
|
9
|
-
subgraph app [Host application]
|
|
10
|
-
CE[Countex client]
|
|
11
|
-
end
|
|
12
|
-
subgraph core [Core engine]
|
|
13
|
-
MAP[CounterMap registry]
|
|
14
|
-
EXT[extractIncrements]
|
|
15
|
-
CARD[CardinalityGuard]
|
|
16
|
-
BUF[DeltaBuffer]
|
|
17
|
-
PLAN[executeQuery]
|
|
18
|
-
FMT[formatResult]
|
|
19
|
-
end
|
|
20
|
-
subgraph io [Storage]
|
|
21
|
-
DUR[DurableStorage]
|
|
22
|
-
CACHE[CacheStorage]
|
|
23
|
-
end
|
|
24
|
-
CE --> MAP
|
|
25
|
-
CE --> EXT
|
|
26
|
-
EXT --> CARD
|
|
27
|
-
CARD --> DUR
|
|
28
|
-
CARD --> BUF
|
|
29
|
-
BUF --> DUR
|
|
30
|
-
BUF --> CACHE
|
|
31
|
-
CE --> PLAN
|
|
32
|
-
PLAN --> CACHE
|
|
33
|
-
PLAN --> DUR
|
|
34
|
-
PLAN --> FMT
|
|
35
|
-
```
|
|
36
|
-
|
|
37
|
-
## Write trace (conceptual)
|
|
38
|
-
|
|
39
|
-
1. **Validate maps** at `createCountex` construction (`validateMaps`).
|
|
40
|
-
2. **Extract** raw increments per map/event (`extractIncrements`).
|
|
41
|
-
3. **Dedupe** optional (`cache.markEventSeen`).
|
|
42
|
-
4. **Cardinality** rewrite or drop (`CardinalityGuard.check`).
|
|
43
|
-
5. **Dictionary** allocate numeric ids (`durable.resolveIds`).
|
|
44
|
-
6. **Buffer** merge by hashed cell identity (`DeltaBuffer.add`).
|
|
45
|
-
7. **Flush** snapshot → `durable.applyDeltas` → best-effort `cache.applyDeltas`.
|
|
46
|
-
|
|
47
|
-
## Read trace (conceptual)
|
|
48
|
-
|
|
49
|
-
1. **Planner** tries cache when allowed (`cacheCanServe`).
|
|
50
|
-
2. On partial/miss, **durable.readCells**.
|
|
51
|
-
3. Optional **cache.warm** with durable rows.
|
|
52
|
-
4. **Formatter** applies `groupBy`, measures, rates, limits.
|
|
53
|
-
|
|
54
|
-
## Source anchors
|
|
55
|
-
|
|
56
|
-
- Client orchestration: `src/core/countex.ts`
|
|
57
|
-
- Extraction: `src/core/extraction.ts`
|
|
58
|
-
- Buffer: `src/core/buffer.ts`
|
|
59
|
-
- Planner: `src/query/planner.ts`
|
|
60
|
-
- Formatter: `src/query/formatter.ts`
|
|
61
|
-
|
|
62
|
-
## See also
|
|
63
|
-
|
|
64
|
-
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
65
|
-
- [Query path](./internal-query-path.md)
|
|
1
|
+
# Internal: Architecture overview
|
|
2
|
+
|
|
3
|
+
Countex separates **projection** (events → sparse cells), **durability** (authoritative totals), **cache** (hot reads + dedupe), and **query shaping**.
|
|
4
|
+
|
|
5
|
+
## Component diagram
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
flowchart LR
|
|
9
|
+
subgraph app [Host application]
|
|
10
|
+
CE[Countex client]
|
|
11
|
+
end
|
|
12
|
+
subgraph core [Core engine]
|
|
13
|
+
MAP[CounterMap registry]
|
|
14
|
+
EXT[extractIncrements]
|
|
15
|
+
CARD[CardinalityGuard]
|
|
16
|
+
BUF[DeltaBuffer]
|
|
17
|
+
PLAN[executeQuery]
|
|
18
|
+
FMT[formatResult]
|
|
19
|
+
end
|
|
20
|
+
subgraph io [Storage]
|
|
21
|
+
DUR[DurableStorage]
|
|
22
|
+
CACHE[CacheStorage]
|
|
23
|
+
end
|
|
24
|
+
CE --> MAP
|
|
25
|
+
CE --> EXT
|
|
26
|
+
EXT --> CARD
|
|
27
|
+
CARD --> DUR
|
|
28
|
+
CARD --> BUF
|
|
29
|
+
BUF --> DUR
|
|
30
|
+
BUF --> CACHE
|
|
31
|
+
CE --> PLAN
|
|
32
|
+
PLAN --> CACHE
|
|
33
|
+
PLAN --> DUR
|
|
34
|
+
PLAN --> FMT
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## Write trace (conceptual)
|
|
38
|
+
|
|
39
|
+
1. **Validate maps** at `createCountex` construction (`validateMaps`).
|
|
40
|
+
2. **Extract** raw increments per map/event (`extractIncrements`).
|
|
41
|
+
3. **Dedupe** optional (`cache.markEventSeen`).
|
|
42
|
+
4. **Cardinality** rewrite or drop (`CardinalityGuard.check`).
|
|
43
|
+
5. **Dictionary** allocate numeric ids (`durable.resolveIds`).
|
|
44
|
+
6. **Buffer** merge by hashed cell identity (`DeltaBuffer.add`).
|
|
45
|
+
7. **Flush** snapshot → `durable.applyDeltas` → best-effort `cache.applyDeltas`.
|
|
46
|
+
|
|
47
|
+
## Read trace (conceptual)
|
|
48
|
+
|
|
49
|
+
1. **Planner** tries cache when allowed (`cacheCanServe`).
|
|
50
|
+
2. On partial/miss, **durable.readCells**.
|
|
51
|
+
3. Optional **cache.warm** with durable rows.
|
|
52
|
+
4. **Formatter** applies `groupBy`, measures, rates, limits.
|
|
53
|
+
|
|
54
|
+
## Source anchors
|
|
55
|
+
|
|
56
|
+
- Client orchestration: `src/core/countex.ts`
|
|
57
|
+
- Extraction: `src/core/extraction.ts`
|
|
58
|
+
- Buffer: `src/core/buffer.ts`
|
|
59
|
+
- Planner: `src/query/planner.ts`
|
|
60
|
+
- Formatter: `src/query/formatter.ts`
|
|
61
|
+
|
|
62
|
+
## See also
|
|
63
|
+
|
|
64
|
+
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
65
|
+
- [Query path](./internal-query-path.md)
|
|
@@ -1,23 +1,23 @@
|
|
|
1
|
-
# Internal: Backup streaming
|
|
2
|
-
|
|
3
|
-
`exportCells` and `rebuildCache` (`src/backup/export.ts`) are thin wrappers over `durable.streamCells`:
|
|
4
|
-
|
|
5
|
-
```mermaid
|
|
6
|
-
flowchart LR
|
|
7
|
-
DUR[durable.streamCells filter] --> EXPORT[AsyncIterable CounterCell]
|
|
8
|
-
DUR --> BATCH[batch warm loop]
|
|
9
|
-
BATCH --> CACHE[cache.warm batches]
|
|
10
|
-
```
|
|
11
|
-
|
|
12
|
-
## Contract expectations
|
|
13
|
-
|
|
14
|
-
Implementations must honor:
|
|
15
|
-
|
|
16
|
-
- Scope/counter/timeframe bounding
|
|
17
|
-
- Optional `AbortSignal` propagation
|
|
18
|
-
|
|
19
|
-
`rebuildCache` batches to limit Redis/Mongo round trips during warm operations.
|
|
20
|
-
|
|
21
|
-
## See also
|
|
22
|
-
|
|
23
|
-
- [Mongo durable](./internal-storage-mongo.md)
|
|
1
|
+
# Internal: Backup streaming
|
|
2
|
+
|
|
3
|
+
`exportCells` and `rebuildCache` (`src/backup/export.ts`) are thin wrappers over `durable.streamCells`:
|
|
4
|
+
|
|
5
|
+
```mermaid
|
|
6
|
+
flowchart LR
|
|
7
|
+
DUR[durable.streamCells filter] --> EXPORT[AsyncIterable CounterCell]
|
|
8
|
+
DUR --> BATCH[batch warm loop]
|
|
9
|
+
BATCH --> CACHE[cache.warm batches]
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
## Contract expectations
|
|
13
|
+
|
|
14
|
+
Implementations must honor:
|
|
15
|
+
|
|
16
|
+
- Scope/counter/timeframe bounding
|
|
17
|
+
- Optional `AbortSignal` propagation
|
|
18
|
+
|
|
19
|
+
`rebuildCache` batches to limit Redis/Mongo round trips during warm operations.
|
|
20
|
+
|
|
21
|
+
## See also
|
|
22
|
+
|
|
23
|
+
- [Mongo durable](./internal-storage-mongo.md)
|
|
@@ -1,51 +1,51 @@
|
|
|
1
|
-
# Internal: Buffer merge & flush
|
|
2
|
-
|
|
3
|
-
## DeltaBuffer structure
|
|
4
|
-
|
|
5
|
-
`DeltaBuffer` (`src/core/buffer.ts`) stores:
|
|
6
|
-
|
|
7
|
-
- Map key: string from `cellKey(identity)`
|
|
8
|
-
- Value: `{ identity, delta, measures?, measureScales? }`
|
|
9
|
-
|
|
10
|
-
## Merge semantics
|
|
11
|
-
|
|
12
|
-
`add` merges:
|
|
13
|
-
|
|
14
|
-
- **Counts** — integer delta summed.
|
|
15
|
-
- **Measures** — per aggregation field:
|
|
16
|
-
- `sum`, `count` additive
|
|
17
|
-
- `min` / `max` rolling extrema
|
|
18
|
-
|
|
19
|
-
`measureScales` merge retains first-seen scale per measure name (consistent with map declaration).
|
|
20
|
-
|
|
21
|
-
## Flush lifecycle
|
|
22
|
-
|
|
23
|
-
```mermaid
|
|
24
|
-
stateDiagram-v2
|
|
25
|
-
[*] --> Idle
|
|
26
|
-
Idle --> Draining: flush() & buffer non-empty
|
|
27
|
-
Draining --> DurableWrite: snapshot taken
|
|
28
|
-
DurableWrite --> CacheWrite: durable OK
|
|
29
|
-
DurableWrite --> MergeBack: durable error
|
|
30
|
-
MergeBack --> Idle
|
|
31
|
-
CacheWrite --> Idle: cache success or logged warn
|
|
32
|
-
```
|
|
33
|
-
|
|
34
|
-
`flush()` returns the **same promise** while in-flight (`flushing` lock) so concurrent callers coalesce.
|
|
35
|
-
|
|
36
|
-
### Durable failure
|
|
37
|
-
|
|
38
|
-
Snapshot is **`mergeBack`** into the buffer so counts are not lost; `flush_error` emits.
|
|
39
|
-
|
|
40
|
-
### Cache failure
|
|
41
|
-
|
|
42
|
-
Logged as non-fatal — durable already committed.
|
|
43
|
-
|
|
44
|
-
## Event accounting
|
|
45
|
-
|
|
46
|
-
`noteEvent()` increments an event counter used solely for flush thresholds (reset implicitly when draining — payload uses duration only).
|
|
47
|
-
|
|
48
|
-
## See also
|
|
49
|
-
|
|
50
|
-
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
51
|
-
- [Measures & scales](./internal-measures-and-scales.md)
|
|
1
|
+
# Internal: Buffer merge & flush
|
|
2
|
+
|
|
3
|
+
## DeltaBuffer structure
|
|
4
|
+
|
|
5
|
+
`DeltaBuffer` (`src/core/buffer.ts`) stores:
|
|
6
|
+
|
|
7
|
+
- Map key: string from `cellKey(identity)`
|
|
8
|
+
- Value: `{ identity, delta, measures?, measureScales? }`
|
|
9
|
+
|
|
10
|
+
## Merge semantics
|
|
11
|
+
|
|
12
|
+
`add` merges:
|
|
13
|
+
|
|
14
|
+
- **Counts** — integer delta summed.
|
|
15
|
+
- **Measures** — per aggregation field:
|
|
16
|
+
- `sum`, `count` additive
|
|
17
|
+
- `min` / `max` rolling extrema
|
|
18
|
+
|
|
19
|
+
`measureScales` merge retains first-seen scale per measure name (consistent with map declaration).
|
|
20
|
+
|
|
21
|
+
## Flush lifecycle
|
|
22
|
+
|
|
23
|
+
```mermaid
|
|
24
|
+
stateDiagram-v2
|
|
25
|
+
[*] --> Idle
|
|
26
|
+
Idle --> Draining: flush() & buffer non-empty
|
|
27
|
+
Draining --> DurableWrite: snapshot taken
|
|
28
|
+
DurableWrite --> CacheWrite: durable OK
|
|
29
|
+
DurableWrite --> MergeBack: durable error
|
|
30
|
+
MergeBack --> Idle
|
|
31
|
+
CacheWrite --> Idle: cache success or logged warn
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
`flush()` returns the **same promise** while in-flight (`flushing` lock) so concurrent callers coalesce.
|
|
35
|
+
|
|
36
|
+
### Durable failure
|
|
37
|
+
|
|
38
|
+
Snapshot is **`mergeBack`** into the buffer so counts are not lost; `flush_error` emits.
|
|
39
|
+
|
|
40
|
+
### Cache failure
|
|
41
|
+
|
|
42
|
+
Logged as non-fatal — durable already committed.
|
|
43
|
+
|
|
44
|
+
## Event accounting
|
|
45
|
+
|
|
46
|
+
`noteEvent()` increments an event counter used solely for flush thresholds (reset implicitly when draining — payload uses duration only).
|
|
47
|
+
|
|
48
|
+
## See also
|
|
49
|
+
|
|
50
|
+
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
51
|
+
- [Measures & scales](./internal-measures-and-scales.md)
|
|
@@ -1,30 +1,30 @@
|
|
|
1
|
-
# Internal: Cardinality guard
|
|
2
|
-
|
|
3
|
-
`CardinalityGuard` (`src/core/cardinality.ts`) tracks distinct source keys per `(scopeId, kind, qualifier)` using in-memory `Set`s.
|
|
4
|
-
|
|
5
|
-
## Check algorithm
|
|
6
|
-
|
|
7
|
-
```mermaid
|
|
8
|
-
flowchart TD
|
|
9
|
-
IN[sourceKey] --> KEY[keyFor scope/kind/qualifier]
|
|
10
|
-
KEY --> SET{Set contains key?}
|
|
11
|
-
SET -->|yes| OK[return original or existing overflow mapping]
|
|
12
|
-
SET -->|no| LIM{count < limit?}
|
|
13
|
-
LIM -->|yes| ADD[add to Set -> return key]
|
|
14
|
-
LIM -->|no| POL{onLimit}
|
|
15
|
-
POL -->|overflow| OVF[emit overflow event -> return sentinel]
|
|
16
|
-
POL -->|drop| DROP[return null]
|
|
17
|
-
POL -->|error| ERR[throw CardinalityLimitError]
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
## Dictionary kinds
|
|
21
|
-
|
|
22
|
-
Kinds align with subject/metric/dimension value namespaces; dimension checks include the dimension **name** as qualifier.
|
|
23
|
-
|
|
24
|
-
## Approximation warning
|
|
25
|
-
|
|
26
|
-
Commentary in source notes multi-worker deployments see approximate counts — acceptable because the guard targets runaway growth, not strict quotas.
|
|
27
|
-
|
|
28
|
-
## See also
|
|
29
|
-
|
|
30
|
-
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
1
|
+
# Internal: Cardinality guard
|
|
2
|
+
|
|
3
|
+
`CardinalityGuard` (`src/core/cardinality.ts`) tracks distinct source keys per `(scopeId, kind, qualifier)` using in-memory `Set`s.
|
|
4
|
+
|
|
5
|
+
## Check algorithm
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
flowchart TD
|
|
9
|
+
IN[sourceKey] --> KEY[keyFor scope/kind/qualifier]
|
|
10
|
+
KEY --> SET{Set contains key?}
|
|
11
|
+
SET -->|yes| OK[return original or existing overflow mapping]
|
|
12
|
+
SET -->|no| LIM{count < limit?}
|
|
13
|
+
LIM -->|yes| ADD[add to Set -> return key]
|
|
14
|
+
LIM -->|no| POL{onLimit}
|
|
15
|
+
POL -->|overflow| OVF[emit overflow event -> return sentinel]
|
|
16
|
+
POL -->|drop| DROP[return null]
|
|
17
|
+
POL -->|error| ERR[throw CardinalityLimitError]
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Dictionary kinds
|
|
21
|
+
|
|
22
|
+
Kinds align with subject/metric/dimension value namespaces; dimension checks include the dimension **name** as qualifier.
|
|
23
|
+
|
|
24
|
+
## Approximation warning
|
|
25
|
+
|
|
26
|
+
Commentary in source notes multi-worker deployments see approximate counts — acceptable because the guard targets runaway growth, not strict quotas.
|
|
27
|
+
|
|
28
|
+
## See also
|
|
29
|
+
|
|
30
|
+
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
@@ -1,34 +1,34 @@
|
|
|
1
|
-
# Internal: Dimension canonicalization and hashing
|
|
2
|
-
|
|
3
|
-
Each counter cell identity includes a **`dimensionsHash`** derived from the dimension key/value object for that increment.
|
|
4
|
-
|
|
5
|
-
## Normalization
|
|
6
|
-
|
|
7
|
-
Before hashing, dimension values pass through `normalizeDimensionValue` in `src/core/normalization.ts`:
|
|
8
|
-
|
|
9
|
-
- `null` / missing optional dimensions → `null`.
|
|
10
|
-
- Booleans and finite numbers preserved as typed values.
|
|
11
|
-
- Strings: optionally trimmed only (`normalizeValues` on the map); types are **not** coerced (`"1"` and `1` stay distinct).
|
|
12
|
-
|
|
13
|
-
Dictionary keys for subjects/metrics use **`normalizeKey`** (trim + lowercase strings) — that is separate from dimension hashing.
|
|
14
|
-
|
|
15
|
-
## Canonical string form
|
|
16
|
-
|
|
17
|
-
`canonicalizeDimensions` in `src/utils/hash.ts`:
|
|
18
|
-
|
|
19
|
-
1. Sort dimension **names** lexicographically.
|
|
20
|
-
2. Serialize as `key=value` pairs joined with `&`.
|
|
21
|
-
3. Escape `\`, `=`, and `&` in keys and string values with backslash.
|
|
22
|
-
4. Encode values with type prefixes: `n:` (null), `b:0|1`, `i:<number>`, `s:<escaped>`.
|
|
23
|
-
|
|
24
|
-
## Hash
|
|
25
|
-
|
|
26
|
-
`dimensionsHash` is **SHA-1** of the canonical string, truncated to **12 hex characters** (48 bits). Empty dimensions → literal `"0"`.
|
|
27
|
-
|
|
28
|
-
Collisions are statistically negligible at intended cardinalities; cells are further scoped by `(scopeId, counterId, timeframe, subject, metric)`.
|
|
29
|
-
|
|
30
|
-
## See also
|
|
31
|
-
|
|
32
|
-
- `src/utils/hash.ts`
|
|
33
|
-
- `src/core/normalization.ts`
|
|
34
|
-
- [Cardinality](../guides/cardinality.md)
|
|
1
|
+
# Internal: Dimension canonicalization and hashing
|
|
2
|
+
|
|
3
|
+
Each counter cell identity includes a **`dimensionsHash`** derived from the dimension key/value object for that increment.
|
|
4
|
+
|
|
5
|
+
## Normalization
|
|
6
|
+
|
|
7
|
+
Before hashing, dimension values pass through `normalizeDimensionValue` in `src/core/normalization.ts`:
|
|
8
|
+
|
|
9
|
+
- `null` / missing optional dimensions → `null`.
|
|
10
|
+
- Booleans and finite numbers preserved as typed values.
|
|
11
|
+
- Strings: optionally trimmed only (`normalizeValues` on the map); types are **not** coerced (`"1"` and `1` stay distinct).
|
|
12
|
+
|
|
13
|
+
Dictionary keys for subjects/metrics use **`normalizeKey`** (trim + lowercase strings) — that is separate from dimension hashing.
|
|
14
|
+
|
|
15
|
+
## Canonical string form
|
|
16
|
+
|
|
17
|
+
`canonicalizeDimensions` in `src/utils/hash.ts`:
|
|
18
|
+
|
|
19
|
+
1. Sort dimension **names** lexicographically.
|
|
20
|
+
2. Serialize as `key=value` pairs joined with `&`.
|
|
21
|
+
3. Escape `\`, `=`, and `&` in keys and string values with backslash.
|
|
22
|
+
4. Encode values with type prefixes: `n:` (null), `b:0|1`, `i:<number>`, `s:<escaped>`.
|
|
23
|
+
|
|
24
|
+
## Hash
|
|
25
|
+
|
|
26
|
+
`dimensionsHash` is **SHA-1** of the canonical string, truncated to **12 hex characters** (48 bits). Empty dimensions → literal `"0"`.
|
|
27
|
+
|
|
28
|
+
Collisions are statistically negligible at intended cardinalities; cells are further scoped by `(scopeId, counterId, timeframe, subject, metric)`.
|
|
29
|
+
|
|
30
|
+
## See also
|
|
31
|
+
|
|
32
|
+
- `src/utils/hash.ts`
|
|
33
|
+
- `src/core/normalization.ts`
|
|
34
|
+
- [Cardinality](../guides/cardinality.md)
|
|
@@ -1,36 +1,36 @@
|
|
|
1
|
-
# Internal: Idempotency
|
|
2
|
-
|
|
3
|
-
Idempotency spans three layers:
|
|
4
|
-
|
|
5
|
-
1. **Map config** — `idempotency.enabled` + `eventIdPath`.
|
|
6
|
-
2. **Extraction** — resolves optional `eventId` string (`src/core/extraction.ts`).
|
|
7
|
-
3. **Ingest gate** — parallel `cache.markEventSeen` before buffering (`src/core/countex.ts`).
|
|
8
|
-
|
|
9
|
-
## Sequence
|
|
10
|
-
|
|
11
|
-
```mermaid
|
|
12
|
-
sequenceDiagram
|
|
13
|
-
participant CX as Countex
|
|
14
|
-
participant EXT as extractIncrements
|
|
15
|
-
participant CACHE as CacheStorage
|
|
16
|
-
participant BUF as DeltaBuffer
|
|
17
|
-
CX->>EXT: increments + eventId
|
|
18
|
-
CX->>CACHE: markEventSeen(scope, counterId:eventId, ttl)
|
|
19
|
-
CACHE-->>CX: isNew boolean
|
|
20
|
-
alt duplicate
|
|
21
|
-
CX->>CX: emit duplicate
|
|
22
|
-
CX->>CX: zero increments for this map slice
|
|
23
|
-
else new
|
|
24
|
-
CX->>BUF: add deltas
|
|
25
|
-
end
|
|
26
|
-
```
|
|
27
|
-
|
|
28
|
-
Namespace **`counterId:`** prefix prevents collisions across maps sharing physical cache.
|
|
29
|
-
|
|
30
|
-
## Failure modes
|
|
31
|
-
|
|
32
|
-
Missing event ids produce warnings during extraction but do not automatically dedupe unless your logic skips ingest.
|
|
33
|
-
|
|
34
|
-
## See also
|
|
35
|
-
|
|
36
|
-
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
1
|
+
# Internal: Idempotency
|
|
2
|
+
|
|
3
|
+
Idempotency spans three layers:
|
|
4
|
+
|
|
5
|
+
1. **Map config** — `idempotency.enabled` + `eventIdPath`.
|
|
6
|
+
2. **Extraction** — resolves optional `eventId` string (`src/core/extraction.ts`).
|
|
7
|
+
3. **Ingest gate** — parallel `cache.markEventSeen` before buffering (`src/core/countex.ts`).
|
|
8
|
+
|
|
9
|
+
## Sequence
|
|
10
|
+
|
|
11
|
+
```mermaid
|
|
12
|
+
sequenceDiagram
|
|
13
|
+
participant CX as Countex
|
|
14
|
+
participant EXT as extractIncrements
|
|
15
|
+
participant CACHE as CacheStorage
|
|
16
|
+
participant BUF as DeltaBuffer
|
|
17
|
+
CX->>EXT: increments + eventId
|
|
18
|
+
CX->>CACHE: markEventSeen(scope, counterId:eventId, ttl)
|
|
19
|
+
CACHE-->>CX: isNew boolean
|
|
20
|
+
alt duplicate
|
|
21
|
+
CX->>CX: emit duplicate
|
|
22
|
+
CX->>CX: zero increments for this map slice
|
|
23
|
+
else new
|
|
24
|
+
CX->>BUF: add deltas
|
|
25
|
+
end
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Namespace **`counterId:`** prefix prevents collisions across maps sharing physical cache.
|
|
29
|
+
|
|
30
|
+
## Failure modes
|
|
31
|
+
|
|
32
|
+
Missing event ids produce warnings during extraction but do not automatically dedupe unless your logic skips ingest.
|
|
33
|
+
|
|
34
|
+
## See also
|
|
35
|
+
|
|
36
|
+
- [Ingest pipeline](./internal-ingest-pipeline.md)
|