@x12i/countex 1.8.5 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +866 -860
- package/dist/cli.js +6 -6
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
- package/dist/insights/advisors/resource-advisor.js +5 -1
- package/dist/insights/advisors/resource-advisor.js.map +1 -1
- package/dist/insights/types.d.ts +4 -0
- package/dist/insights/types.d.ts.map +1 -1
- package/dist/storage/postgres.d.ts +104 -0
- package/dist/storage/postgres.d.ts.map +1 -0
- package/dist/storage/postgres.js +1023 -0
- package/dist/storage/postgres.js.map +1 -0
- package/dist/types/index.d.ts +11 -5
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +399 -375
- package/docs/END_TO_END_EXAMPLE.md +316 -316
- package/docs/INSIGHTS.md +59 -59
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
- package/docs/guides/README.md +36 -34
- package/docs/guides/adapter-memory.md +60 -60
- package/docs/guides/adapter-mongo.md +64 -62
- package/docs/guides/adapter-postgres.md +90 -0
- package/docs/guides/adapter-redis.md +89 -89
- package/docs/guides/backup-export.md +135 -135
- package/docs/guides/cardinality.md +50 -50
- package/docs/guides/cli.md +53 -53
- package/docs/guides/client-events.md +65 -65
- package/docs/guides/counter-maps.md +160 -160
- package/docs/guides/idempotency.md +39 -39
- package/docs/guides/ingestion-and-flush.md +67 -67
- package/docs/guides/insights-advisors-and-optimization.md +39 -39
- package/docs/guides/insights-anomalies-and-rules.md +48 -48
- package/docs/guides/insights-comparison-and-trends.md +47 -47
- package/docs/guides/insights-forecasting-and-budget.md +61 -61
- package/docs/guides/insights-overview.md +82 -82
- package/docs/guides/insights-ranking.md +81 -81
- package/docs/guides/measures.md +71 -71
- package/docs/guides/migrating-mongo-to-postgres.md +127 -0
- package/docs/guides/preview.md +92 -92
- package/docs/guides/querying.md +178 -178
- package/docs/guides/simulation-profiles.md +33 -33
- package/docs/guides/simulation-resource-estimation.md +31 -30
- package/docs/guides/simulation.md +36 -36
- package/docs/guides/timeframes-and-bucketing.md +55 -55
- package/docs/internals/README.md +21 -21
- package/docs/internals/internal-architecture-overview.md +65 -65
- package/docs/internals/internal-backup-stream.md +23 -23
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
- package/docs/internals/internal-cardinality.md +30 -30
- package/docs/internals/internal-dimension-hash.md +34 -34
- package/docs/internals/internal-idempotency.md +36 -36
- package/docs/internals/internal-ingest-pipeline.md +52 -52
- package/docs/internals/internal-insights-layer.md +36 -36
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
- package/docs/internals/internal-measures-and-scales.md +35 -35
- package/docs/internals/internal-query-path.md +46 -46
- package/docs/internals/internal-storage-memory.md +13 -13
- package/docs/internals/internal-storage-mongo.md +44 -44
- package/docs/internals/internal-storage-postgres.md +32 -0
- package/docs/internals/internal-storage-redis.md +63 -63
- package/docs/internals/internal-timeframe-bucketing.md +33 -33
- package/docs/use-cases/README.md +56 -56
- package/docs/use-cases/audit-security-log-counts.md +550 -550
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
- package/docs/use-cases/multi-tenant-isolation.md +203 -203
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
- package/docs/use-cases/processing-workflow-activity.md +165 -165
- package/docs/use-cases/product-feature-usage.md +240 -240
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
- package/package.json +184 -170
|
@@ -1,160 +1,160 @@
|
|
|
1
|
-
# Guide: Counter maps
|
|
2
|
-
|
|
3
|
-
A **counter map** (`CounterMap`) tells Countex how to read one JSON event and turn it into sparse counter updates. This guide walks through defining a map from scratch.
|
|
4
|
-
|
|
5
|
-
## Metrics vs domain “labels”
|
|
6
|
-
|
|
7
|
-
Countex uses **`metrics`** in the map config: paths that resolve to the category or label you want one counter split per value (for example `$.labels[*]` or `$.event.operation`). That field name is generic — your payload might call them labels, statuses, tags, or event types. They all map to Countex **metrics**.
|
|
8
|
-
|
|
9
|
-
Documentation refers to **`sourceKey`** for the string form of a subject or metric before dictionary compaction; that is not necessarily a UUID unless you choose UUID strings as keys.
|
|
10
|
-
|
|
11
|
-
## What you will configure
|
|
12
|
-
|
|
13
|
-
- **scope** — tenant or isolation boundary (string id).
|
|
14
|
-
- **eventTime** — when the event happened (drives timeframe buckets).
|
|
15
|
-
- **subjects** — one or more “things” the count attaches to (fan-out supported).
|
|
16
|
-
- **metrics** — labels or categories being counted (fan-out supported).
|
|
17
|
-
- **dimensions** — optional breakdown fields stored on each cell.
|
|
18
|
-
- **delta** — how much to add (default `1`).
|
|
19
|
-
- **timeframes** — which bucket granularities to write (`minute` … `all`).
|
|
20
|
-
- Optional: **measures**, **idempotency**, **cardinality**, **options**.
|
|
21
|
-
- Optional: **evidence** (exemplar snapshots in Mongo only — omit for **no** evidence / zero exemplars).
|
|
22
|
-
- Optional: **triage** (HTTP gate before counting this map).
|
|
23
|
-
|
|
24
|
-
## Step 1: Choose a stable `counterId`
|
|
25
|
-
|
|
26
|
-
Use a string that identifies this projection family across deployments, for example `access_by_actor_label`.
|
|
27
|
-
|
|
28
|
-
## Step 2: Point at scope and time
|
|
29
|
-
|
|
30
|
-
```ts
|
|
31
|
-
const map = {
|
|
32
|
-
counterId: "access_by_actor_label",
|
|
33
|
-
scope: { path: "$.tenant.id" },
|
|
34
|
-
eventTime: { path: "$.event.createdAt" },
|
|
35
|
-
// ...
|
|
36
|
-
} as const;
|
|
37
|
-
```
|
|
38
|
-
|
|
39
|
-
Rules:
|
|
40
|
-
|
|
41
|
-
- Both paths must resolve to usable values or the event is skipped for this map.
|
|
42
|
-
- `eventTime` accepts `Date`, ISO strings, or epoch numbers.
|
|
43
|
-
|
|
44
|
-
## Step 3: Declare subjects
|
|
45
|
-
|
|
46
|
-
Each subject has a `type` (your label) and a `path`. Multi-value paths fan out.
|
|
47
|
-
|
|
48
|
-
```ts
|
|
49
|
-
subjects: [
|
|
50
|
-
{ type: "actor", path: "$.actor.id", required: true },
|
|
51
|
-
{ type: "resource", path: "$.resource.id", required: false },
|
|
52
|
-
],
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
- **`required: true`** — if nothing matches, the whole event is skipped.
|
|
56
|
-
- **`required: false`** — missing values omit that subject type for this event.
|
|
57
|
-
|
|
58
|
-
## Step 4: Declare metrics
|
|
59
|
-
|
|
60
|
-
```ts
|
|
61
|
-
metrics: [
|
|
62
|
-
{ path: "$.labels[*]", required: true },
|
|
63
|
-
],
|
|
64
|
-
```
|
|
65
|
-
|
|
66
|
-
Wildcards (`[*]`) expand to one increment per array element (or object values).
|
|
67
|
-
|
|
68
|
-
## Step 5: Add dimensions (optional)
|
|
69
|
-
|
|
70
|
-
```ts
|
|
71
|
-
dimensions: [
|
|
72
|
-
{ name: "operation", path: "$.event.operation" },
|
|
73
|
-
{ name: "situation", path: "$.event.situation", required: false },
|
|
74
|
-
],
|
|
75
|
-
```
|
|
76
|
-
|
|
77
|
-
## Step 6: Delta and timeframes
|
|
78
|
-
|
|
79
|
-
```ts
|
|
80
|
-
delta: { default: 1 },
|
|
81
|
-
timeframes: ["day", "week", "month"],
|
|
82
|
-
```
|
|
83
|
-
|
|
84
|
-
Use `delta: { path: "$.weight" }` when counts are not always `1`.
|
|
85
|
-
|
|
86
|
-
## Step 7: Pass maps into `createCountex`
|
|
87
|
-
|
|
88
|
-
```ts
|
|
89
|
-
import { createCountex } from "@x12i/countex";
|
|
90
|
-
import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory";
|
|
91
|
-
|
|
92
|
-
const countex = createCountex({
|
|
93
|
-
maps: [map],
|
|
94
|
-
durable: new MemoryDurable(),
|
|
95
|
-
cache: new MemoryCache(),
|
|
96
|
-
});
|
|
97
|
-
```
|
|
98
|
-
|
|
99
|
-
Maps are validated at construction time; invalid paths or duplicate `counterId` values throw immediately.
|
|
100
|
-
|
|
101
|
-
## Path syntax (subset)
|
|
102
|
-
|
|
103
|
-
| Syntax | Meaning |
|
|
104
|
-
|--------|---------|
|
|
105
|
-
| `$` | Root |
|
|
106
|
-
| `.name` | Property |
|
|
107
|
-
| `['name']` | Bracket property |
|
|
108
|
-
| `[N]` | Array index |
|
|
109
|
-
| `[*]` | Fan-out wildcard |
|
|
110
|
-
|
|
111
|
-
## Fan-out math
|
|
112
|
-
|
|
113
|
-
Total raw increments per event ≈
|
|
114
|
-
|
|
115
|
-
`(subjects) × (metrics) × (number of timeframes)`.
|
|
116
|
-
|
|
117
|
-
Plan capacity accordingly.
|
|
118
|
-
|
|
119
|
-
## Preview before ingesting
|
|
120
|
-
|
|
121
|
-
Run `countex.preview(event, { counterId })` on a sample event to see exactly what cells your map would generate — without any dictionary writes or buffering. See [Preview a map](./preview.md) for the full result shape (`subjectSourceKey`, `metricSourceKey`, `dimensionsHash`, `timeframes`, `delta`, `measures`, optional `evidenceSample` when the map declares evidence, plus `extractionWarnings` and `droppedAfterCardinality`). Preview does **not** call triage endpoints.
|
|
122
|
-
|
|
123
|
-
## Evidence (optional exemplars)
|
|
124
|
-
|
|
125
|
-
By default a map has **no** evidence configuration: Countex only updates counter cells. To retain occasional raw field snapshots for debugging or audit, add **`evidence`**:
|
|
126
|
-
|
|
127
|
-
- **`fields`**: non-empty list of `{ name, path }` — JSONPath values copied into each stored exemplar (missing paths are omitted).
|
|
128
|
-
- **`eventIdPath`**: optional path for a string id stored on the exemplar.
|
|
129
|
-
- **`maxSamples`**: max rows retained **per counter cell** in the evidence collection (default `20`). Older rows are dropped after writes.
|
|
130
|
-
- **`sampleEvery`**: deterministic thinning **per cell** — keep one sample every N increments (default `1` = no thinning).
|
|
131
|
-
|
|
132
|
-
Samples are written only to durable storage (`MongoDurable.appendEvidence`), collection default `countex_evidence`, never to Redis. Use `MongoDurable` (or implement `appendEvidence` yourself); maps that declare evidence without `appendEvidence` log a warning at `init` and drop exemplars on flush.
|
|
133
|
-
|
|
134
|
-
**Tests**: `npm test` runs unit coverage for evidence sampling, flush recovery, triage policies, `fetchTriageInvoke`, and concurrency caps. Mongo exemplar trimming is asserted in [`tests/integration/docker/mongo-adapter.integration.test.ts`](../../tests/integration/docker/mongo-adapter.integration.test.ts) when a Mongo URI is available (for example via `npm run test:integration` / docker integration flows).
|
|
135
|
-
|
|
136
|
-
## Triage API (optional HTTP gate)
|
|
137
|
-
|
|
138
|
-
If **`triage`** is set on a map, Countex invokes `POST` (by default) to **`url`** with the raw JSON event body and `Content-Type: application/json` before extraction for **that map**. Response must be JSON **`{ "count": true | false }`**. If `count` is false, the map does nothing for that event (no increments, no dedupe marker).
|
|
139
|
-
|
|
140
|
-
On failure (network error, timeout, non-2xx, invalid JSON, missing `count`), **`onFailure`** applies: **`skip`** (default, do not count), **`count`** (fail-open), or **`error`** (reject `ingest`). Configure **`timeoutMs`** (default `5000`) and optional static **`headers`**.
|
|
141
|
-
|
|
142
|
-
Override the transport with **`CountexConfig.triage.invoke`** (tests, signing, mTLS). Tune parallelism with **`maxConcurrentTriage`** (default `8`). Default invoke uses global **`fetch`**.
|
|
143
|
-
|
|
144
|
-
| Direction | Details |
|
|
145
|
-
|-----------|---------|
|
|
146
|
-
| Request | Same payload as passed to `ingest` / `ingestBatch`, JSON-serialized. |
|
|
147
|
-
| Success | HTTP 2xx and `{ "count": boolean }`. |
|
|
148
|
-
| Preview | Triage is **not** called. |
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
Dimension keys/values feeding into `dimensionsHash` are canonicalized: keys sorted lexicographically, values type-tagged, hashed (SHA-1 truncated to 12 hex). Any two workers always produce the same hash for the same logical dimension tuple. See [internal-dimension-hash.md](../internals/internal-dimension-hash.md) for the exact recipe.
|
|
152
|
-
|
|
153
|
-
## See also
|
|
154
|
-
|
|
155
|
-
- [Timeframes](./timeframes-and-bucketing.md)
|
|
156
|
-
- [Measures](./measures.md)
|
|
157
|
-
- [Cardinality](./cardinality.md)
|
|
158
|
-
- [Preview a map](./preview.md)
|
|
159
|
-
- Internal: [Mapping validation & extraction](../internals/internal-mapping-validation-and-extraction.md)
|
|
160
|
-
- Internal: [Dimension hash](../internals/internal-dimension-hash.md)
|
|
1
|
+
# Guide: Counter maps
|
|
2
|
+
|
|
3
|
+
A **counter map** (`CounterMap`) tells Countex how to read one JSON event and turn it into sparse counter updates. This guide walks through defining a map from scratch.
|
|
4
|
+
|
|
5
|
+
## Metrics vs domain “labels”
|
|
6
|
+
|
|
7
|
+
Countex uses **`metrics`** in the map config: paths that resolve to the category or label you want one counter split per value (for example `$.labels[*]` or `$.event.operation`). That field name is generic — your payload might call them labels, statuses, tags, or event types. They all map to Countex **metrics**.
|
|
8
|
+
|
|
9
|
+
Documentation refers to **`sourceKey`** for the string form of a subject or metric before dictionary compaction; that is not necessarily a UUID unless you choose UUID strings as keys.
|
|
10
|
+
|
|
11
|
+
## What you will configure
|
|
12
|
+
|
|
13
|
+
- **scope** — tenant or isolation boundary (string id).
|
|
14
|
+
- **eventTime** — when the event happened (drives timeframe buckets).
|
|
15
|
+
- **subjects** — one or more “things” the count attaches to (fan-out supported).
|
|
16
|
+
- **metrics** — labels or categories being counted (fan-out supported).
|
|
17
|
+
- **dimensions** — optional breakdown fields stored on each cell.
|
|
18
|
+
- **delta** — how much to add (default `1`).
|
|
19
|
+
- **timeframes** — which bucket granularities to write (`minute` … `all`).
|
|
20
|
+
- Optional: **measures**, **idempotency**, **cardinality**, **options**.
|
|
21
|
+
- Optional: **evidence** (exemplar snapshots in Mongo only — omit for **no** evidence / zero exemplars).
|
|
22
|
+
- Optional: **triage** (HTTP gate before counting this map).
|
|
23
|
+
|
|
24
|
+
## Step 1: Choose a stable `counterId`
|
|
25
|
+
|
|
26
|
+
Use a string that identifies this projection family across deployments, for example `access_by_actor_label`.
|
|
27
|
+
|
|
28
|
+
## Step 2: Point at scope and time
|
|
29
|
+
|
|
30
|
+
```ts
|
|
31
|
+
const map = {
|
|
32
|
+
counterId: "access_by_actor_label",
|
|
33
|
+
scope: { path: "$.tenant.id" },
|
|
34
|
+
eventTime: { path: "$.event.createdAt" },
|
|
35
|
+
// ...
|
|
36
|
+
} as const;
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Rules:
|
|
40
|
+
|
|
41
|
+
- Both paths must resolve to usable values or the event is skipped for this map.
|
|
42
|
+
- `eventTime` accepts `Date`, ISO strings, or epoch numbers.
|
|
43
|
+
|
|
44
|
+
## Step 3: Declare subjects
|
|
45
|
+
|
|
46
|
+
Each subject has a `type` (your label) and a `path`. Multi-value paths fan out.
|
|
47
|
+
|
|
48
|
+
```ts
|
|
49
|
+
subjects: [
|
|
50
|
+
{ type: "actor", path: "$.actor.id", required: true },
|
|
51
|
+
{ type: "resource", path: "$.resource.id", required: false },
|
|
52
|
+
],
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
- **`required: true`** — if nothing matches, the whole event is skipped.
|
|
56
|
+
- **`required: false`** — missing values omit that subject type for this event.
|
|
57
|
+
|
|
58
|
+
## Step 4: Declare metrics
|
|
59
|
+
|
|
60
|
+
```ts
|
|
61
|
+
metrics: [
|
|
62
|
+
{ path: "$.labels[*]", required: true },
|
|
63
|
+
],
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Wildcards (`[*]`) expand to one increment per array element (or object values).
|
|
67
|
+
|
|
68
|
+
## Step 5: Add dimensions (optional)
|
|
69
|
+
|
|
70
|
+
```ts
|
|
71
|
+
dimensions: [
|
|
72
|
+
{ name: "operation", path: "$.event.operation" },
|
|
73
|
+
{ name: "situation", path: "$.event.situation", required: false },
|
|
74
|
+
],
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## Step 6: Delta and timeframes
|
|
78
|
+
|
|
79
|
+
```ts
|
|
80
|
+
delta: { default: 1 },
|
|
81
|
+
timeframes: ["day", "week", "month"],
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Use `delta: { path: "$.weight" }` when counts are not always `1`.
|
|
85
|
+
|
|
86
|
+
## Step 7: Pass maps into `createCountex`
|
|
87
|
+
|
|
88
|
+
```ts
|
|
89
|
+
import { createCountex } from "@x12i/countex";
|
|
90
|
+
import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory";
|
|
91
|
+
|
|
92
|
+
const countex = createCountex({
|
|
93
|
+
maps: [map],
|
|
94
|
+
durable: new MemoryDurable(),
|
|
95
|
+
cache: new MemoryCache(),
|
|
96
|
+
});
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
Maps are validated at construction time; invalid paths or duplicate `counterId` values throw immediately.
|
|
100
|
+
|
|
101
|
+
## Path syntax (subset)
|
|
102
|
+
|
|
103
|
+
| Syntax | Meaning |
|
|
104
|
+
|--------|---------|
|
|
105
|
+
| `$` | Root |
|
|
106
|
+
| `.name` | Property |
|
|
107
|
+
| `['name']` | Bracket property |
|
|
108
|
+
| `[N]` | Array index |
|
|
109
|
+
| `[*]` | Fan-out wildcard |
|
|
110
|
+
|
|
111
|
+
## Fan-out math
|
|
112
|
+
|
|
113
|
+
Total raw increments per event ≈
|
|
114
|
+
|
|
115
|
+
`(subjects) × (metrics) × (number of timeframes)`.
|
|
116
|
+
|
|
117
|
+
Plan capacity accordingly.
|
|
118
|
+
|
|
119
|
+
## Preview before ingesting
|
|
120
|
+
|
|
121
|
+
Run `countex.preview(event, { counterId })` on a sample event to see exactly what cells your map would generate — without any dictionary writes or buffering. See [Preview a map](./preview.md) for the full result shape (`subjectSourceKey`, `metricSourceKey`, `dimensionsHash`, `timeframes`, `delta`, `measures`, optional `evidenceSample` when the map declares evidence, plus `extractionWarnings` and `droppedAfterCardinality`). Preview does **not** call triage endpoints.
|
|
122
|
+
|
|
123
|
+
## Evidence (optional exemplars)
|
|
124
|
+
|
|
125
|
+
By default a map has **no** evidence configuration: Countex only updates counter cells. To retain occasional raw field snapshots for debugging or audit, add **`evidence`**:
|
|
126
|
+
|
|
127
|
+
- **`fields`**: non-empty list of `{ name, path }` — JSONPath values copied into each stored exemplar (missing paths are omitted).
|
|
128
|
+
- **`eventIdPath`**: optional path for a string id stored on the exemplar.
|
|
129
|
+
- **`maxSamples`**: max rows retained **per counter cell** in the evidence collection (default `20`). Older rows are dropped after writes.
|
|
130
|
+
- **`sampleEvery`**: deterministic thinning **per cell** — keep one sample every N increments (default `1` = no thinning).
|
|
131
|
+
|
|
132
|
+
Samples are written only to durable storage (`MongoDurable.appendEvidence`), collection default `countex_evidence`, never to Redis. Use `MongoDurable` (or implement `appendEvidence` yourself); maps that declare evidence without `appendEvidence` log a warning at `init` and drop exemplars on flush.
|
|
133
|
+
|
|
134
|
+
**Tests**: `npm test` runs unit coverage for evidence sampling, flush recovery, triage policies, `fetchTriageInvoke`, and concurrency caps. Mongo exemplar trimming is asserted in [`tests/integration/docker/mongo-adapter.integration.test.ts`](../../tests/integration/docker/mongo-adapter.integration.test.ts) when a Mongo URI is available (for example via `npm run test:integration` / docker integration flows).
|
|
135
|
+
|
|
136
|
+
## Triage API (optional HTTP gate)
|
|
137
|
+
|
|
138
|
+
If **`triage`** is set on a map, Countex invokes `POST` (by default) to **`url`** with the raw JSON event body and `Content-Type: application/json` before extraction for **that map**. Response must be JSON **`{ "count": true | false }`**. If `count` is false, the map does nothing for that event (no increments, no dedupe marker).
|
|
139
|
+
|
|
140
|
+
On failure (network error, timeout, non-2xx, invalid JSON, missing `count`), **`onFailure`** applies: **`skip`** (default, do not count), **`count`** (fail-open), or **`error`** (reject `ingest`). Configure **`timeoutMs`** (default `5000`) and optional static **`headers`**.
|
|
141
|
+
|
|
142
|
+
Override the transport with **`CountexConfig.triage.invoke`** (tests, signing, mTLS). Tune parallelism with **`maxConcurrentTriage`** (default `8`). Default invoke uses global **`fetch`**.
|
|
143
|
+
|
|
144
|
+
| Direction | Details |
|
|
145
|
+
|-----------|---------|
|
|
146
|
+
| Request | Same payload as passed to `ingest` / `ingestBatch`, JSON-serialized. |
|
|
147
|
+
| Success | HTTP 2xx and `{ "count": boolean }`. |
|
|
148
|
+
| Preview | Triage is **not** called. |
|
|
149
|
+
|
|
150
|
+
|
|
151
|
+
Dimension keys/values feeding into `dimensionsHash` are canonicalized: keys sorted lexicographically, values type-tagged, hashed (SHA-1 truncated to 12 hex). Any two workers always produce the same hash for the same logical dimension tuple. See [internal-dimension-hash.md](../internals/internal-dimension-hash.md) for the exact recipe.
|
|
152
|
+
|
|
153
|
+
## See also
|
|
154
|
+
|
|
155
|
+
- [Timeframes](./timeframes-and-bucketing.md)
|
|
156
|
+
- [Measures](./measures.md)
|
|
157
|
+
- [Cardinality](./cardinality.md)
|
|
158
|
+
- [Preview a map](./preview.md)
|
|
159
|
+
- Internal: [Mapping validation & extraction](../internals/internal-mapping-validation-and-extraction.md)
|
|
160
|
+
- Internal: [Dimension hash](../internals/internal-dimension-hash.md)
|
|
@@ -1,39 +1,39 @@
|
|
|
1
|
-
# Guide: Idempotency
|
|
2
|
-
|
|
3
|
-
Idempotency prevents **duplicate events** from incrementing counters twice when retries or at-least-once delivery replay the same logical event.
|
|
4
|
-
|
|
5
|
-
## Step 1: Enable on the map
|
|
6
|
-
|
|
7
|
-
```ts
|
|
8
|
-
const map = {
|
|
9
|
-
counterId: "orders",
|
|
10
|
-
// ...
|
|
11
|
-
idempotency: {
|
|
12
|
-
enabled: true,
|
|
13
|
-
eventIdPath: "$.event.id",
|
|
14
|
-
ttlSeconds: 86_400,
|
|
15
|
-
},
|
|
16
|
-
} as const;
|
|
17
|
-
```
|
|
18
|
-
|
|
19
|
-
If `enabled` is true, `eventIdPath` is required (validated at startup).
|
|
20
|
-
|
|
21
|
-
## Step 2: Use a cache adapter that supports dedupe
|
|
22
|
-
|
|
23
|
-
Implement **`markEventSeen(scopeId, eventId, ttlSeconds): Promise<boolean>`** on your cache adapter. The built-in **`RedisCache`** provides this using Redis keys with TTL.
|
|
24
|
-
|
|
25
|
-
Countex namespaces ids as **`${counterId}:${eventId}`** per map so different counters do not collide.
|
|
26
|
-
|
|
27
|
-
## Step 3: Ingest normally
|
|
28
|
-
|
|
29
|
-
When `markEventSeen` returns **false** (already seen), increments for that map/event are dropped and a **`duplicate`** event is emitted.
|
|
30
|
-
|
|
31
|
-
## Step 4: Handle missing event ids
|
|
32
|
-
|
|
33
|
-
If the path resolves empty while idempotency is enabled, Countex records a warning and still processes unless your extraction logic skips the event — verify your producer always sets ids.
|
|
34
|
-
|
|
35
|
-
## See also
|
|
36
|
-
|
|
37
|
-
- [Redis adapter](./adapter-redis.md)
|
|
38
|
-
- [Client events](./client-events.md)
|
|
39
|
-
- Internal: [Idempotency](../internals/internal-idempotency.md)
|
|
1
|
+
# Guide: Idempotency
|
|
2
|
+
|
|
3
|
+
Idempotency prevents **duplicate events** from incrementing counters twice when retries or at-least-once delivery replay the same logical event.
|
|
4
|
+
|
|
5
|
+
## Step 1: Enable on the map
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const map = {
|
|
9
|
+
counterId: "orders",
|
|
10
|
+
// ...
|
|
11
|
+
idempotency: {
|
|
12
|
+
enabled: true,
|
|
13
|
+
eventIdPath: "$.event.id",
|
|
14
|
+
ttlSeconds: 86_400,
|
|
15
|
+
},
|
|
16
|
+
} as const;
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
If `enabled` is true, `eventIdPath` is required (validated at startup).
|
|
20
|
+
|
|
21
|
+
## Step 2: Use a cache adapter that supports dedupe
|
|
22
|
+
|
|
23
|
+
Implement **`markEventSeen(scopeId, eventId, ttlSeconds): Promise<boolean>`** on your cache adapter. The built-in **`RedisCache`** provides this using Redis keys with TTL.
|
|
24
|
+
|
|
25
|
+
Countex namespaces ids as **`${counterId}:${eventId}`** per map so different counters do not collide.
|
|
26
|
+
|
|
27
|
+
## Step 3: Ingest normally
|
|
28
|
+
|
|
29
|
+
When `markEventSeen` returns **false** (already seen), increments for that map/event are dropped and a **`duplicate`** event is emitted.
|
|
30
|
+
|
|
31
|
+
## Step 4: Handle missing event ids
|
|
32
|
+
|
|
33
|
+
If the path resolves empty while idempotency is enabled, Countex records a warning and still processes unless your extraction logic skips the event — verify your producer always sets ids.
|
|
34
|
+
|
|
35
|
+
## See also
|
|
36
|
+
|
|
37
|
+
- [Redis adapter](./adapter-redis.md)
|
|
38
|
+
- [Client events](./client-events.md)
|
|
39
|
+
- Internal: [Idempotency](../internals/internal-idempotency.md)
|
|
@@ -1,67 +1,67 @@
|
|
|
1
|
-
# Guide: Ingestion and flush
|
|
2
|
-
|
|
3
|
-
Countex keeps **deltas in memory**, then **flushes** them to durable storage and optionally a cache. This guide covers the write path from your application’s perspective.
|
|
4
|
-
|
|
5
|
-
## Step 1: Ingest events
|
|
6
|
-
|
|
7
|
-
```ts
|
|
8
|
-
await countex.ingest({
|
|
9
|
-
tenant: { id: "org_1" },
|
|
10
|
-
event: { createdAt: "2026-05-08T12:00:00.000Z", operation: "read" },
|
|
11
|
-
actor: { id: "user_a" },
|
|
12
|
-
labels: ["docs"],
|
|
13
|
-
});
|
|
14
|
-
```
|
|
15
|
-
|
|
16
|
-
`ingest` accepts `unknown`. Each configured map runs extraction independently.
|
|
17
|
-
|
|
18
|
-
Use **`ingestBatch(events)`** when you already have an array; dictionary lookups are batched across the batch.
|
|
19
|
-
|
|
20
|
-
## Step 2: Understand buffering
|
|
21
|
-
|
|
22
|
-
After ingestion:
|
|
23
|
-
|
|
24
|
-
- Raw increments are merged into an in-memory **`DeltaBuffer`** keyed by cell identity.
|
|
25
|
-
- Nothing hits Mongo/Redis until flush (except dictionary resolution during ingest, which uses durable `resolveIds`).
|
|
26
|
-
|
|
27
|
-
## Step 3: Flush explicitly or on schedule
|
|
28
|
-
|
|
29
|
-
```ts
|
|
30
|
-
await countex.flush();
|
|
31
|
-
```
|
|
32
|
-
|
|
33
|
-
Flush behavior is controlled by `flush` in config:
|
|
34
|
-
|
|
35
|
-
| Option | Role |
|
|
36
|
-
|--------|------|
|
|
37
|
-
| `everyMs` | Periodic background flush (0 disables timer) |
|
|
38
|
-
| `maxBufferedCells` | Threshold on distinct buffered cells |
|
|
39
|
-
| `maxBufferedEvents` | Threshold on ingested events since last drain |
|
|
40
|
-
| `flushOnShutdown` | Drain on `shutdown()` |
|
|
41
|
-
| `onOverflow` | `"block"` awaits flush at threshold; `"drop"` fires flush without blocking |
|
|
42
|
-
|
|
43
|
-
## Step 4: Shutdown cleanly
|
|
44
|
-
|
|
45
|
-
```ts
|
|
46
|
-
await countex.shutdown();
|
|
47
|
-
```
|
|
48
|
-
|
|
49
|
-
Typically clears the timer, optionally flushes, then closes adapters.
|
|
50
|
-
|
|
51
|
-
## Step 5: Initialize adapters in production
|
|
52
|
-
|
|
53
|
-
```ts
|
|
54
|
-
await countex.init(); // calls durable.init?.() and cache.init?.()
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
Call once after constructing clients.
|
|
58
|
-
|
|
59
|
-
## Observability
|
|
60
|
-
|
|
61
|
-
Subscribe to `flush`, `flush_error`, `dropped`, and related events — see [Client events](./client-events.md).
|
|
62
|
-
|
|
63
|
-
## See also
|
|
64
|
-
|
|
65
|
-
- [Counter maps](./counter-maps.md)
|
|
66
|
-
- [Adapter guides](./adapter-mongo.md)
|
|
67
|
-
- Internal: [Ingest pipeline](../internals/internal-ingest-pipeline.md), [Buffer & flush](../internals/internal-buffer-merge-and-flush.md)
|
|
1
|
+
# Guide: Ingestion and flush
|
|
2
|
+
|
|
3
|
+
Countex keeps **deltas in memory**, then **flushes** them to durable storage and optionally a cache. This guide covers the write path from your application’s perspective.
|
|
4
|
+
|
|
5
|
+
## Step 1: Ingest events
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
await countex.ingest({
|
|
9
|
+
tenant: { id: "org_1" },
|
|
10
|
+
event: { createdAt: "2026-05-08T12:00:00.000Z", operation: "read" },
|
|
11
|
+
actor: { id: "user_a" },
|
|
12
|
+
labels: ["docs"],
|
|
13
|
+
});
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
`ingest` accepts `unknown`. Each configured map runs extraction independently.
|
|
17
|
+
|
|
18
|
+
Use **`ingestBatch(events)`** when you already have an array; dictionary lookups are batched across the batch.
|
|
19
|
+
|
|
20
|
+
## Step 2: Understand buffering
|
|
21
|
+
|
|
22
|
+
After ingestion:
|
|
23
|
+
|
|
24
|
+
- Raw increments are merged into an in-memory **`DeltaBuffer`** keyed by cell identity.
|
|
25
|
+
- Nothing hits Mongo/Redis until flush (except dictionary resolution during ingest, which uses durable `resolveIds`).
|
|
26
|
+
|
|
27
|
+
## Step 3: Flush explicitly or on schedule
|
|
28
|
+
|
|
29
|
+
```ts
|
|
30
|
+
await countex.flush();
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Flush behavior is controlled by `flush` in config:
|
|
34
|
+
|
|
35
|
+
| Option | Role |
|
|
36
|
+
|--------|------|
|
|
37
|
+
| `everyMs` | Periodic background flush (0 disables timer) |
|
|
38
|
+
| `maxBufferedCells` | Threshold on distinct buffered cells |
|
|
39
|
+
| `maxBufferedEvents` | Threshold on ingested events since last drain |
|
|
40
|
+
| `flushOnShutdown` | Drain on `shutdown()` |
|
|
41
|
+
| `onOverflow` | `"block"` awaits flush at threshold; `"drop"` fires flush without blocking |
|
|
42
|
+
|
|
43
|
+
## Step 4: Shutdown cleanly
|
|
44
|
+
|
|
45
|
+
```ts
|
|
46
|
+
await countex.shutdown();
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Typically clears the timer, optionally flushes, then closes adapters.
|
|
50
|
+
|
|
51
|
+
## Step 5: Initialize adapters in production
|
|
52
|
+
|
|
53
|
+
```ts
|
|
54
|
+
await countex.init(); // calls durable.init?.() and cache.init?.()
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Call once after constructing clients.
|
|
58
|
+
|
|
59
|
+
## Observability
|
|
60
|
+
|
|
61
|
+
Subscribe to `flush`, `flush_error`, `dropped`, and related events — see [Client events](./client-events.md).
|
|
62
|
+
|
|
63
|
+
## See also
|
|
64
|
+
|
|
65
|
+
- [Counter maps](./counter-maps.md)
|
|
66
|
+
- [Adapter guides](./adapter-mongo.md)
|
|
67
|
+
- Internal: [Ingest pipeline](../internals/internal-ingest-pipeline.md), [Buffer & flush](../internals/internal-buffer-merge-and-flush.md)
|
|
@@ -1,39 +1,39 @@
|
|
|
1
|
-
# Guide: Insights — advisors and resource estimation
|
|
2
|
-
|
|
3
|
-
Advisors inspect **map shape**, **fan-out health**, and optionally **stream durable cells** to estimate capacity-style metrics.
|
|
4
|
-
|
|
5
|
-
## Step 1: Map health
|
|
6
|
-
|
|
7
|
-
Requires `mapsByCounterId` passed to `createCountexInsights`.
|
|
8
|
-
|
|
9
|
-
```ts
|
|
10
|
-
const health = await insights.analyzeMapHealth({
|
|
11
|
-
scope: "org_1",
|
|
12
|
-
counterId: "usage",
|
|
13
|
-
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
14
|
-
});
|
|
15
|
-
```
|
|
16
|
-
|
|
17
|
-
## Step 2: Resource estimation from actuals
|
|
18
|
-
|
|
19
|
-
Requires the **`durable`** reference in `createCountexInsights`.
|
|
20
|
-
|
|
21
|
-
```ts
|
|
22
|
-
const resources = await insights.estimateResourcesFromActuals({
|
|
23
|
-
scope: "org_1",
|
|
24
|
-
counterId: "usage",
|
|
25
|
-
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
26
|
-
});
|
|
27
|
-
```
|
|
28
|
-
|
|
29
|
-
This path uses `durable.streamCells` — ensure your adapter implements streaming efficiently.
|
|
30
|
-
|
|
31
|
-
## Events
|
|
32
|
-
|
|
33
|
-
Recommendations attach to `recommendation` emissions with severity for downstream routing (pager, ticket, etc.).
|
|
34
|
-
|
|
35
|
-
## See also
|
|
36
|
-
|
|
37
|
-
- `docs/INSIGHTS_RESOURCE_ADVISOR.md`
|
|
38
|
-
- [Insights overview](./insights-overview.md)
|
|
39
|
-
- Internal: [Insights layer](../internals/internal-insights-layer.md)
|
|
1
|
+
# Guide: Insights — advisors and resource estimation
|
|
2
|
+
|
|
3
|
+
Advisors inspect **map shape**, **fan-out health**, and optionally **stream durable cells** to estimate capacity-style metrics.
|
|
4
|
+
|
|
5
|
+
## Step 1: Map health
|
|
6
|
+
|
|
7
|
+
Requires `mapsByCounterId` passed to `createCountexInsights`.
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
const health = await insights.analyzeMapHealth({
|
|
11
|
+
scope: "org_1",
|
|
12
|
+
counterId: "usage",
|
|
13
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
14
|
+
});
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Step 2: Resource estimation from actuals
|
|
18
|
+
|
|
19
|
+
Requires the **`durable`** reference in `createCountexInsights`.
|
|
20
|
+
|
|
21
|
+
```ts
|
|
22
|
+
const resources = await insights.estimateResourcesFromActuals({
|
|
23
|
+
scope: "org_1",
|
|
24
|
+
counterId: "usage",
|
|
25
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
26
|
+
});
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
This path uses `durable.streamCells` — ensure your adapter implements streaming efficiently.
|
|
30
|
+
|
|
31
|
+
## Events
|
|
32
|
+
|
|
33
|
+
Recommendations attach to `recommendation` emissions with severity for downstream routing (pager, ticket, etc.).
|
|
34
|
+
|
|
35
|
+
## See also
|
|
36
|
+
|
|
37
|
+
- `docs/INSIGHTS_RESOURCE_ADVISOR.md`
|
|
38
|
+
- [Insights overview](./insights-overview.md)
|
|
39
|
+
- Internal: [Insights layer](../internals/internal-insights-layer.md)
|