@x12i/countex 1.8.3 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +866 -858
- package/dist/cli.js +6 -6
- package/dist/index.d.ts +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +1 -0
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
- package/dist/insights/advisors/resource-advisor.js +5 -1
- package/dist/insights/advisors/resource-advisor.js.map +1 -1
- package/dist/insights/types.d.ts +4 -0
- package/dist/insights/types.d.ts.map +1 -1
- package/dist/storage/postgres.d.ts +104 -0
- package/dist/storage/postgres.d.ts.map +1 -0
- package/dist/storage/postgres.js +1023 -0
- package/dist/storage/postgres.js.map +1 -0
- package/dist/types/index.d.ts +11 -5
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +399 -375
- package/docs/END_TO_END_EXAMPLE.md +316 -316
- package/docs/INSIGHTS.md +59 -59
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
- package/docs/guides/README.md +36 -34
- package/docs/guides/adapter-memory.md +60 -60
- package/docs/guides/adapter-mongo.md +64 -62
- package/docs/guides/adapter-postgres.md +90 -0
- package/docs/guides/adapter-redis.md +89 -89
- package/docs/guides/backup-export.md +135 -135
- package/docs/guides/cardinality.md +50 -50
- package/docs/guides/cli.md +53 -53
- package/docs/guides/client-events.md +65 -65
- package/docs/guides/counter-maps.md +160 -160
- package/docs/guides/idempotency.md +39 -39
- package/docs/guides/ingestion-and-flush.md +67 -67
- package/docs/guides/insights-advisors-and-optimization.md +39 -39
- package/docs/guides/insights-anomalies-and-rules.md +48 -48
- package/docs/guides/insights-comparison-and-trends.md +47 -47
- package/docs/guides/insights-forecasting-and-budget.md +61 -61
- package/docs/guides/insights-overview.md +82 -82
- package/docs/guides/insights-ranking.md +81 -81
- package/docs/guides/measures.md +71 -71
- package/docs/guides/migrating-mongo-to-postgres.md +127 -0
- package/docs/guides/preview.md +92 -92
- package/docs/guides/querying.md +178 -178
- package/docs/guides/simulation-profiles.md +33 -33
- package/docs/guides/simulation-resource-estimation.md +31 -30
- package/docs/guides/simulation.md +36 -36
- package/docs/guides/timeframes-and-bucketing.md +55 -55
- package/docs/internals/README.md +21 -21
- package/docs/internals/internal-architecture-overview.md +65 -65
- package/docs/internals/internal-backup-stream.md +23 -23
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
- package/docs/internals/internal-cardinality.md +30 -30
- package/docs/internals/internal-dimension-hash.md +34 -34
- package/docs/internals/internal-idempotency.md +36 -36
- package/docs/internals/internal-ingest-pipeline.md +52 -52
- package/docs/internals/internal-insights-layer.md +36 -36
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
- package/docs/internals/internal-measures-and-scales.md +35 -35
- package/docs/internals/internal-query-path.md +46 -46
- package/docs/internals/internal-storage-memory.md +13 -13
- package/docs/internals/internal-storage-mongo.md +44 -44
- package/docs/internals/internal-storage-postgres.md +32 -0
- package/docs/internals/internal-storage-redis.md +63 -63
- package/docs/internals/internal-timeframe-bucketing.md +33 -33
- package/docs/use-cases/README.md +56 -56
- package/docs/use-cases/audit-security-log-counts.md +550 -550
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
- package/docs/use-cases/multi-tenant-isolation.md +203 -203
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
- package/docs/use-cases/processing-workflow-activity.md +165 -165
- package/docs/use-cases/product-feature-usage.md +240 -240
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
- package/package.json +184 -169
package/docs/INSIGHTS.md
CHANGED
|
@@ -1,59 +1,59 @@
|
|
|
1
|
-
# Countex Insights
|
|
2
|
-
|
|
3
|
-
## Mental model
|
|
4
|
-
|
|
5
|
-
Insights sits **on top of** `countex.query()`. It does not bypass storage adapters. Each operation builds a `QuerySpec`, runs one or more queries, then derives rankings, deltas, anomalies, or recommendations from the returned rows.
|
|
6
|
-
|
|
7
|
-
Primitives are generic: `scope`, `counterId`, `timeframe`, filters, `groupBy`, `measure` selectors (`count`, `name.sum`, `name.rate`, …), and optional counter map metadata for advisors.
|
|
8
|
-
|
|
9
|
-
## Ranking and slicing
|
|
10
|
-
|
|
11
|
-
- **topN / bottomN** — Sort grouped rows by `count` or a measure aggregation; bottom omits zero values by default (sparse-safe).
|
|
12
|
-
- **topPercent / bottomPercent** — Take the top or bottom _fraction of groups_ (by rank), then report value share of totals.
|
|
13
|
-
- **pareto** — Smallest prefix of contributors reaching a target cumulative share of total value.
|
|
14
|
-
- **contribution** — Full sorted breakdown with `share` and `cumulativeShare`.
|
|
15
|
-
|
|
16
|
-
## Period comparison
|
|
17
|
-
|
|
18
|
-
**comparePeriods** runs two queries (`current` and `previous` timeframes) with the same `groupBy` and measure. Rows are merged on a **stable serialized key** so grouping is order-independent. Directions include `new`, `missing`, `up`, `down`, and `flat`. Percent change is `null` when the previous value is zero.
|
|
19
|
-
|
|
20
|
-
## Anomalies
|
|
21
|
-
|
|
22
|
-
See [INSIGHTS_ANOMALY_DETECTION.md](./INSIGHTS_ANOMALY_DETECTION.md). High level: baseline window is derived from `lookback` (e.g. `14d`) immediately before the evaluation window; per-series robust z-score or threshold checks produce explainable findings.
|
|
23
|
-
|
|
24
|
-
## Rules
|
|
25
|
-
|
|
26
|
-
**evaluateRules** evaluates absolute measure thresholds or simple ratios (numerator/denominator queries per group). Failures can attach a `CountexRecommendation`.
|
|
27
|
-
|
|
28
|
-
## Recommendations
|
|
29
|
-
|
|
30
|
-
Advisor APIs return **evidence-first** objects: timeframe, numeric values, optional rows/baselines, suggested actions as opaque `{ kind, payload }` pairs.
|
|
31
|
-
|
|
32
|
-
## Examples
|
|
33
|
-
|
|
34
|
-
```ts
|
|
35
|
-
import { createCountexInsights } from "@x12i/countex/insights";
|
|
36
|
-
|
|
37
|
-
const insights = createCountexInsights({
|
|
38
|
-
countex,
|
|
39
|
-
durable,
|
|
40
|
-
mapsByCounterId: new Map([[map.counterId, map]]),
|
|
41
|
-
});
|
|
42
|
-
|
|
43
|
-
await insights.topN({
|
|
44
|
-
scope,
|
|
45
|
-
counterId,
|
|
46
|
-
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
47
|
-
groupBy: ["dim:region"],
|
|
48
|
-
measure: "errors.rate",
|
|
49
|
-
n: 20,
|
|
50
|
-
});
|
|
51
|
-
```
|
|
52
|
-
|
|
53
|
-
## Observability
|
|
54
|
-
|
|
55
|
-
`CountexInsights` is an `EventEmitter`: `insight_query`, `recommendation`, `anomaly`, `insufficient_data`.
|
|
56
|
-
|
|
57
|
-
## Errors
|
|
58
|
-
|
|
59
|
-
Typed errors extend `CountexInsightError` — see `@x12i/countex/insights` exports.
|
|
1
|
+
# Countex Insights
|
|
2
|
+
|
|
3
|
+
## Mental model
|
|
4
|
+
|
|
5
|
+
Insights sits **on top of** `countex.query()`. It does not bypass storage adapters. Each operation builds a `QuerySpec`, runs one or more queries, then derives rankings, deltas, anomalies, or recommendations from the returned rows.
|
|
6
|
+
|
|
7
|
+
Primitives are generic: `scope`, `counterId`, `timeframe`, filters, `groupBy`, `measure` selectors (`count`, `name.sum`, `name.rate`, …), and optional counter map metadata for advisors.
|
|
8
|
+
|
|
9
|
+
## Ranking and slicing
|
|
10
|
+
|
|
11
|
+
- **topN / bottomN** — Sort grouped rows by `count` or a measure aggregation; bottom omits zero values by default (sparse-safe).
|
|
12
|
+
- **topPercent / bottomPercent** — Take the top or bottom _fraction of groups_ (by rank), then report value share of totals.
|
|
13
|
+
- **pareto** — Smallest prefix of contributors reaching a target cumulative share of total value.
|
|
14
|
+
- **contribution** — Full sorted breakdown with `share` and `cumulativeShare`.
|
|
15
|
+
|
|
16
|
+
## Period comparison
|
|
17
|
+
|
|
18
|
+
**comparePeriods** runs two queries (`current` and `previous` timeframes) with the same `groupBy` and measure. Rows are merged on a **stable serialized key** so grouping is order-independent. Directions include `new`, `missing`, `up`, `down`, and `flat`. Percent change is `null` when the previous value is zero.
|
|
19
|
+
|
|
20
|
+
## Anomalies
|
|
21
|
+
|
|
22
|
+
See [INSIGHTS_ANOMALY_DETECTION.md](./INSIGHTS_ANOMALY_DETECTION.md). High level: baseline window is derived from `lookback` (e.g. `14d`) immediately before the evaluation window; per-series robust z-score or threshold checks produce explainable findings.
|
|
23
|
+
|
|
24
|
+
## Rules
|
|
25
|
+
|
|
26
|
+
**evaluateRules** evaluates absolute measure thresholds or simple ratios (numerator/denominator queries per group). Failures can attach a `CountexRecommendation`.
|
|
27
|
+
|
|
28
|
+
## Recommendations
|
|
29
|
+
|
|
30
|
+
Advisor APIs return **evidence-first** objects: timeframe, numeric values, optional rows/baselines, suggested actions as opaque `{ kind, payload }` pairs.
|
|
31
|
+
|
|
32
|
+
## Examples
|
|
33
|
+
|
|
34
|
+
```ts
|
|
35
|
+
import { createCountexInsights } from "@x12i/countex/insights";
|
|
36
|
+
|
|
37
|
+
const insights = createCountexInsights({
|
|
38
|
+
countex,
|
|
39
|
+
durable,
|
|
40
|
+
mapsByCounterId: new Map([[map.counterId, map]]),
|
|
41
|
+
});
|
|
42
|
+
|
|
43
|
+
await insights.topN({
|
|
44
|
+
scope,
|
|
45
|
+
counterId,
|
|
46
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
47
|
+
groupBy: ["dim:region"],
|
|
48
|
+
measure: "errors.rate",
|
|
49
|
+
n: 20,
|
|
50
|
+
});
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Observability
|
|
54
|
+
|
|
55
|
+
`CountexInsights` is an `EventEmitter`: `insight_query`, `recommendation`, `anomaly`, `insufficient_data`.
|
|
56
|
+
|
|
57
|
+
## Errors
|
|
58
|
+
|
|
59
|
+
Typed errors extend `CountexInsightError` — see `@x12i/countex/insights` exports.
|
|
@@ -1,33 +1,33 @@
|
|
|
1
|
-
# Insights: anomaly detection
|
|
2
|
-
|
|
3
|
-
## Baselines
|
|
4
|
-
|
|
5
|
-
`detectAnomalies` evaluates the requested **timeframe** against a **baseline window** placed immediately before it. The default lookback string (`14d`, `24h`, `2w`, …) is converted into a number of buckets matching the query timeframe type.
|
|
6
|
-
|
|
7
|
-
## Methods
|
|
8
|
-
|
|
9
|
-
- **robust-zscore** (default) — Uses median and MAD; `0.6745 * (x - median) / MAD`. When MAD is zero, a simple deviation fallback runs.
|
|
10
|
-
- **zscore** — Classical mean/stddev (more sensitive to outliers).
|
|
11
|
-
- **threshold** — Multiplicative deviation from the baseline median (e.g. spike when `observed > 2× median`).
|
|
12
|
-
- **ewma** — Lightweight deviation check vs median (placeholder-style when full EWMA series is not stored).
|
|
13
|
-
|
|
14
|
-
Sensitivity maps to z thresholds: low ≈ 4.0, medium ≈ 3.5, high ≈ 3.0.
|
|
15
|
-
|
|
16
|
-
## Missing activity
|
|
17
|
-
|
|
18
|
-
When the baseline median exceeds a small floor and the observed window is near zero, a **missing** anomaly is emitted.
|
|
19
|
-
|
|
20
|
-
## Spike / drop
|
|
21
|
-
|
|
22
|
-
Large positive z → spike; large negative z → drop (when enabled in `anomalyTypes`).
|
|
23
|
-
|
|
24
|
-
## Compare-by modes
|
|
25
|
-
|
|
26
|
-
- **rolling** — All baseline buckets for the same group key.
|
|
27
|
-
- **same-hour-of-day** / **same-day-of-week** — Filters baseline buckets to those matching the evaluation bucket’s calendar slot.
|
|
28
|
-
|
|
29
|
-
## Limitations
|
|
30
|
-
|
|
31
|
-
- Requires enough baseline buckets (`minBaselineBuckets`, default from factory defaults).
|
|
32
|
-
- Cardinality and “new dimension value” anomalies need histogram/distribution support in core for best results (future).
|
|
33
|
-
- Very wide windows are rejected by `maxBuckets` / `maxGroups` guards.
|
|
1
|
+
# Insights: anomaly detection
|
|
2
|
+
|
|
3
|
+
## Baselines
|
|
4
|
+
|
|
5
|
+
`detectAnomalies` evaluates the requested **timeframe** against a **baseline window** placed immediately before it. The default lookback string (`14d`, `24h`, `2w`, …) is converted into a number of buckets matching the query timeframe type.
|
|
6
|
+
|
|
7
|
+
## Methods
|
|
8
|
+
|
|
9
|
+
- **robust-zscore** (default) — Uses median and MAD; `0.6745 * (x - median) / MAD`. When MAD is zero, a simple deviation fallback runs.
|
|
10
|
+
- **zscore** — Classical mean/stddev (more sensitive to outliers).
|
|
11
|
+
- **threshold** — Multiplicative deviation from the baseline median (e.g. spike when `observed > 2× median`).
|
|
12
|
+
- **ewma** — Lightweight deviation check vs median (placeholder-style when full EWMA series is not stored).
|
|
13
|
+
|
|
14
|
+
Sensitivity maps to z thresholds: low ≈ 4.0, medium ≈ 3.5, high ≈ 3.0.
|
|
15
|
+
|
|
16
|
+
## Missing activity
|
|
17
|
+
|
|
18
|
+
When the baseline median exceeds a small floor and the observed window is near zero, a **missing** anomaly is emitted.
|
|
19
|
+
|
|
20
|
+
## Spike / drop
|
|
21
|
+
|
|
22
|
+
Large positive z → spike; large negative z → drop (when enabled in `anomalyTypes`).
|
|
23
|
+
|
|
24
|
+
## Compare-by modes
|
|
25
|
+
|
|
26
|
+
- **rolling** — All baseline buckets for the same group key.
|
|
27
|
+
- **same-hour-of-day** / **same-day-of-week** — Filters baseline buckets to those matching the evaluation bucket’s calendar slot.
|
|
28
|
+
|
|
29
|
+
## Limitations
|
|
30
|
+
|
|
31
|
+
- Requires enough baseline buckets (`minBaselineBuckets`, default from factory defaults).
|
|
32
|
+
- Cardinality and “new dimension value” anomalies need histogram/distribution support in core for best results (future).
|
|
33
|
+
- Very wide windows are rejected by `maxBuckets` / `maxGroups` guards.
|
|
@@ -1,26 +1,26 @@
|
|
|
1
|
-
# Insights: LLM cost and tokens
|
|
2
|
-
|
|
3
|
-
All APIs remain **measure-oriented**. Names like `costMicros`, `outputTokens`, or `latencyMs` are conventions chosen by your maps — Countex Insights does not treat “LLM” as a special domain type.
|
|
4
|
-
|
|
5
|
-
## Required measures
|
|
6
|
-
|
|
7
|
-
- **costDrivers / budgetProjection** — Any numeric measure with `.sum` (or `count`).
|
|
8
|
-
- **recommendMaxTokens** — Uses grouped aggregates for the named measure (`outputTokens` by convention). Ideal inputs include `sum`, `count`, `min`, `max`, and derived `avg`.
|
|
9
|
-
|
|
10
|
-
## Cost drivers
|
|
11
|
-
|
|
12
|
-
`costDrivers` ranks contributors by a cost measure and emits generic recommendations when concentration is extreme.
|
|
13
|
-
|
|
14
|
-
## Max tokens
|
|
15
|
-
|
|
16
|
-
Without histograms, percentiles are **approximated** from min/max/avg; confidence is marked **low** or **medium**. When core adds optional histogram storage, recommendations should tighten and confidence can rise.
|
|
17
|
-
|
|
18
|
-
Recommended ceiling (conceptually): `ceil(roundTo, percentile × safetyMultiplier)` — implementation uses aggregate proxies until histogram-backed percentiles exist.
|
|
19
|
-
|
|
20
|
-
## Budget projection
|
|
21
|
-
|
|
22
|
-
Linear extrapolation from the **current window** rate into the **target window** length (bucket-count scaling). Status: `under` | `near` | `over` vs a numeric budget.
|
|
23
|
-
|
|
24
|
-
## Percentiles / distributions
|
|
25
|
-
|
|
26
|
-
Histogram-backed measures are a **core** milestone. Insights degrades gracefully until `measure.pXX` selectors exist on `countex.query()`.
|
|
1
|
+
# Insights: LLM cost and tokens
|
|
2
|
+
|
|
3
|
+
All APIs remain **measure-oriented**. Names like `costMicros`, `outputTokens`, or `latencyMs` are conventions chosen by your maps — Countex Insights does not treat “LLM” as a special domain type.
|
|
4
|
+
|
|
5
|
+
## Required measures
|
|
6
|
+
|
|
7
|
+
- **costDrivers / budgetProjection** — Any numeric measure with `.sum` (or `count`).
|
|
8
|
+
- **recommendMaxTokens** — Uses grouped aggregates for the named measure (`outputTokens` by convention). Ideal inputs include `sum`, `count`, `min`, `max`, and derived `avg`.
|
|
9
|
+
|
|
10
|
+
## Cost drivers
|
|
11
|
+
|
|
12
|
+
`costDrivers` ranks contributors by a cost measure and emits generic recommendations when concentration is extreme.
|
|
13
|
+
|
|
14
|
+
## Max tokens
|
|
15
|
+
|
|
16
|
+
Without histograms, percentiles are **approximated** from min/max/avg; confidence is marked **low** or **medium**. When core adds optional histogram storage, recommendations should tighten and confidence can rise.
|
|
17
|
+
|
|
18
|
+
Recommended ceiling (conceptually): `ceil(roundTo, percentile × safetyMultiplier)` — implementation uses aggregate proxies until histogram-backed percentiles exist.
|
|
19
|
+
|
|
20
|
+
## Budget projection
|
|
21
|
+
|
|
22
|
+
Linear extrapolation from the **current window** rate into the **target window** length (bucket-count scaling). Status: `under` | `near` | `over` vs a numeric budget.
|
|
23
|
+
|
|
24
|
+
## Percentiles / distributions
|
|
25
|
+
|
|
26
|
+
Histogram-backed measures are a **core** milestone. Insights degrades gracefully until `measure.pXX` selectors exist on `countex.query()`.
|
|
@@ -1,31 +1,33 @@
|
|
|
1
|
-
# Insights: resource advisor
|
|
2
|
-
|
|
3
|
-
## Purpose
|
|
4
|
-
|
|
5
|
-
`estimateResourcesFromActuals` projects hot-cache and durable storage needs from **observed cell streams**, using configurable byte constants per cell and per dictionary key.
|
|
6
|
-
|
|
7
|
-
## Requirements
|
|
8
|
-
|
|
9
|
-
Pass the **durable** adapter into `createCountexInsights({ durable })`. The advisor calls `durable.streamCells()` over `sampleWindow`.
|
|
10
|
-
|
|
11
|
-
## Observed metrics
|
|
12
|
-
|
|
13
|
-
- Approximate cells per day (mean and a simple P95 across days seen in the sample).
|
|
14
|
-
- Heuristic dictionary key estimate (defaults to ~10% of streamed cells — tune in code if you allocate differently).
|
|
15
|
-
|
|
16
|
-
## Projections
|
|
17
|
-
|
|
18
|
-
Combines:
|
|
19
|
-
|
|
20
|
-
- **Hot cells** — Based on recent daily churn (P95 × short window).
|
|
21
|
-
- **Durable cells** — Scales daily average by a retention multiplier derived from `targetRetention` (days/weeks/months/quarters/years/all).
|
|
22
|
-
|
|
23
|
-
Outputs Redis
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
1
|
+
# Insights: resource advisor
|
|
2
|
+
|
|
3
|
+
## Purpose
|
|
4
|
+
|
|
5
|
+
`estimateResourcesFromActuals` projects hot-cache and durable storage needs from **observed cell streams**, using configurable byte constants per cell and per dictionary key.
|
|
6
|
+
|
|
7
|
+
## Requirements
|
|
8
|
+
|
|
9
|
+
Pass the **durable** adapter into `createCountexInsights({ durable })`. The advisor calls `durable.streamCells()` over `sampleWindow`.
|
|
10
|
+
|
|
11
|
+
## Observed metrics
|
|
12
|
+
|
|
13
|
+
- Approximate cells per day (mean and a simple P95 across days seen in the sample).
|
|
14
|
+
- Heuristic dictionary key estimate (defaults to ~10% of streamed cells — tune in code if you allocate differently).
|
|
15
|
+
|
|
16
|
+
## Projections
|
|
17
|
+
|
|
18
|
+
Combines:
|
|
19
|
+
|
|
20
|
+
- **Hot cells** — Based on recent daily churn (P95 × short window).
|
|
21
|
+
- **Durable cells** — Scales daily average by a retention multiplier derived from `targetRetention` (days/weeks/months/quarters/years/all).
|
|
22
|
+
|
|
23
|
+
Outputs Redis, Mongo, and Postgres byte estimates and suggested RAM/disk figures (informational).
|
|
24
|
+
|
|
25
|
+
`postgresBytes` uses `constants.postgresBytesPerCell` (default **250**), a packed row next to the Mongo document constant (default **750**). Both numbers are returned so a move can be sized before the adapter is switched. They are estimates, not measurements of your dimension payloads.
|
|
26
|
+
|
|
27
|
+
## Map / cardinality risk
|
|
28
|
+
|
|
29
|
+
See **analyzeMapHealth** on the insights client. Pass `mapsByCounterId` so dimension lists and cardinality policies can be interpreted. High distinct counts per dimension trigger recommendations such as bucketing or removing hazardous paths.
|
|
30
|
+
|
|
31
|
+
## Retention advice
|
|
32
|
+
|
|
33
|
+
When daily cell churn is very high, the resource advisor may emit a generic retention recommendation — tune thresholds for your environment.
|