@x12i/countex 1.8.3 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (77) hide show
  1. package/LICENSE +21 -21
  2. package/README.md +866 -858
  3. package/dist/cli.js +6 -6
  4. package/dist/index.d.ts +1 -0
  5. package/dist/index.d.ts.map +1 -1
  6. package/dist/index.js +1 -0
  7. package/dist/index.js.map +1 -1
  8. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -1
  9. package/dist/insights/advisors/resource-advisor.js +5 -1
  10. package/dist/insights/advisors/resource-advisor.js.map +1 -1
  11. package/dist/insights/types.d.ts +4 -0
  12. package/dist/insights/types.d.ts.map +1 -1
  13. package/dist/storage/postgres.d.ts +104 -0
  14. package/dist/storage/postgres.d.ts.map +1 -0
  15. package/dist/storage/postgres.js +1023 -0
  16. package/dist/storage/postgres.js.map +1 -0
  17. package/dist/types/index.d.ts +11 -5
  18. package/dist/types/index.d.ts.map +1 -1
  19. package/docs/DEPLOYMENT.md +399 -375
  20. package/docs/END_TO_END_EXAMPLE.md +316 -316
  21. package/docs/INSIGHTS.md +59 -59
  22. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -33
  23. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -26
  24. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +33 -31
  25. package/docs/countex_gap_analysis_complete_checklist.md +1340 -1340
  26. package/docs/guides/README.md +36 -34
  27. package/docs/guides/adapter-memory.md +60 -60
  28. package/docs/guides/adapter-mongo.md +64 -62
  29. package/docs/guides/adapter-postgres.md +90 -0
  30. package/docs/guides/adapter-redis.md +89 -89
  31. package/docs/guides/backup-export.md +135 -135
  32. package/docs/guides/cardinality.md +50 -50
  33. package/docs/guides/cli.md +53 -53
  34. package/docs/guides/client-events.md +65 -65
  35. package/docs/guides/counter-maps.md +160 -160
  36. package/docs/guides/idempotency.md +39 -39
  37. package/docs/guides/ingestion-and-flush.md +67 -67
  38. package/docs/guides/insights-advisors-and-optimization.md +39 -39
  39. package/docs/guides/insights-anomalies-and-rules.md +48 -48
  40. package/docs/guides/insights-comparison-and-trends.md +47 -47
  41. package/docs/guides/insights-forecasting-and-budget.md +61 -61
  42. package/docs/guides/insights-overview.md +82 -82
  43. package/docs/guides/insights-ranking.md +81 -81
  44. package/docs/guides/measures.md +71 -71
  45. package/docs/guides/migrating-mongo-to-postgres.md +127 -0
  46. package/docs/guides/preview.md +92 -92
  47. package/docs/guides/querying.md +178 -178
  48. package/docs/guides/simulation-profiles.md +33 -33
  49. package/docs/guides/simulation-resource-estimation.md +31 -30
  50. package/docs/guides/simulation.md +36 -36
  51. package/docs/guides/timeframes-and-bucketing.md +55 -55
  52. package/docs/internals/README.md +21 -21
  53. package/docs/internals/internal-architecture-overview.md +65 -65
  54. package/docs/internals/internal-backup-stream.md +23 -23
  55. package/docs/internals/internal-buffer-merge-and-flush.md +51 -51
  56. package/docs/internals/internal-cardinality.md +30 -30
  57. package/docs/internals/internal-dimension-hash.md +34 -34
  58. package/docs/internals/internal-idempotency.md +36 -36
  59. package/docs/internals/internal-ingest-pipeline.md +52 -52
  60. package/docs/internals/internal-insights-layer.md +36 -36
  61. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -59
  62. package/docs/internals/internal-measures-and-scales.md +35 -35
  63. package/docs/internals/internal-query-path.md +46 -46
  64. package/docs/internals/internal-storage-memory.md +13 -13
  65. package/docs/internals/internal-storage-mongo.md +44 -44
  66. package/docs/internals/internal-storage-postgres.md +32 -0
  67. package/docs/internals/internal-storage-redis.md +63 -63
  68. package/docs/internals/internal-timeframe-bucketing.md +33 -33
  69. package/docs/use-cases/README.md +56 -56
  70. package/docs/use-cases/audit-security-log-counts.md +550 -550
  71. package/docs/use-cases/llm-cost-and-token-metering.md +255 -255
  72. package/docs/use-cases/multi-tenant-isolation.md +203 -203
  73. package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -255
  74. package/docs/use-cases/processing-workflow-activity.md +165 -165
  75. package/docs/use-cases/product-feature-usage.md +240 -240
  76. package/docs/use-cases/webhook-idempotent-ingestion.md +188 -188
  77. package/package.json +184 -169
package/docs/INSIGHTS.md CHANGED
@@ -1,59 +1,59 @@
1
- # Countex Insights
2
-
3
- ## Mental model
4
-
5
- Insights sits **on top of** `countex.query()`. It does not bypass storage adapters. Each operation builds a `QuerySpec`, runs one or more queries, then derives rankings, deltas, anomalies, or recommendations from the returned rows.
6
-
7
- Primitives are generic: `scope`, `counterId`, `timeframe`, filters, `groupBy`, `measure` selectors (`count`, `name.sum`, `name.rate`, …), and optional counter map metadata for advisors.
8
-
9
- ## Ranking and slicing
10
-
11
- - **topN / bottomN** — Sort grouped rows by `count` or a measure aggregation; bottom omits zero values by default (sparse-safe).
12
- - **topPercent / bottomPercent** — Take the top or bottom _fraction of groups_ (by rank), then report value share of totals.
13
- - **pareto** — Smallest prefix of contributors reaching a target cumulative share of total value.
14
- - **contribution** — Full sorted breakdown with `share` and `cumulativeShare`.
15
-
16
- ## Period comparison
17
-
18
- **comparePeriods** runs two queries (`current` and `previous` timeframes) with the same `groupBy` and measure. Rows are merged on a **stable serialized key** so grouping is order-independent. Directions include `new`, `missing`, `up`, `down`, and `flat`. Percent change is `null` when the previous value is zero.
19
-
20
- ## Anomalies
21
-
22
- See [INSIGHTS_ANOMALY_DETECTION.md](./INSIGHTS_ANOMALY_DETECTION.md). High level: baseline window is derived from `lookback` (e.g. `14d`) immediately before the evaluation window; per-series robust z-score or threshold checks produce explainable findings.
23
-
24
- ## Rules
25
-
26
- **evaluateRules** evaluates absolute measure thresholds or simple ratios (numerator/denominator queries per group). Failures can attach a `CountexRecommendation`.
27
-
28
- ## Recommendations
29
-
30
- Advisor APIs return **evidence-first** objects: timeframe, numeric values, optional rows/baselines, suggested actions as opaque `{ kind, payload }` pairs.
31
-
32
- ## Examples
33
-
34
- ```ts
35
- import { createCountexInsights } from "@x12i/countex/insights";
36
-
37
- const insights = createCountexInsights({
38
- countex,
39
- durable,
40
- mapsByCounterId: new Map([[map.counterId, map]]),
41
- });
42
-
43
- await insights.topN({
44
- scope,
45
- counterId,
46
- timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
47
- groupBy: ["dim:region"],
48
- measure: "errors.rate",
49
- n: 20,
50
- });
51
- ```
52
-
53
- ## Observability
54
-
55
- `CountexInsights` is an `EventEmitter`: `insight_query`, `recommendation`, `anomaly`, `insufficient_data`.
56
-
57
- ## Errors
58
-
59
- Typed errors extend `CountexInsightError` — see `@x12i/countex/insights` exports.
1
+ # Countex Insights
2
+
3
+ ## Mental model
4
+
5
+ Insights sits **on top of** `countex.query()`. It does not bypass storage adapters. Each operation builds a `QuerySpec`, runs one or more queries, then derives rankings, deltas, anomalies, or recommendations from the returned rows.
6
+
7
+ Primitives are generic: `scope`, `counterId`, `timeframe`, filters, `groupBy`, `measure` selectors (`count`, `name.sum`, `name.rate`, …), and optional counter map metadata for advisors.
8
+
9
+ ## Ranking and slicing
10
+
11
+ - **topN / bottomN** — Sort grouped rows by `count` or a measure aggregation; bottom omits zero values by default (sparse-safe).
12
+ - **topPercent / bottomPercent** — Take the top or bottom _fraction of groups_ (by rank), then report value share of totals.
13
+ - **pareto** — Smallest prefix of contributors reaching a target cumulative share of total value.
14
+ - **contribution** — Full sorted breakdown with `share` and `cumulativeShare`.
15
+
16
+ ## Period comparison
17
+
18
+ **comparePeriods** runs two queries (`current` and `previous` timeframes) with the same `groupBy` and measure. Rows are merged on a **stable serialized key** so grouping is order-independent. Directions include `new`, `missing`, `up`, `down`, and `flat`. Percent change is `null` when the previous value is zero.
19
+
20
+ ## Anomalies
21
+
22
+ See [INSIGHTS_ANOMALY_DETECTION.md](./INSIGHTS_ANOMALY_DETECTION.md). High level: baseline window is derived from `lookback` (e.g. `14d`) immediately before the evaluation window; per-series robust z-score or threshold checks produce explainable findings.
23
+
24
+ ## Rules
25
+
26
+ **evaluateRules** evaluates absolute measure thresholds or simple ratios (numerator/denominator queries per group). Failures can attach a `CountexRecommendation`.
27
+
28
+ ## Recommendations
29
+
30
+ Advisor APIs return **evidence-first** objects: timeframe, numeric values, optional rows/baselines, suggested actions as opaque `{ kind, payload }` pairs.
31
+
32
+ ## Examples
33
+
34
+ ```ts
35
+ import { createCountexInsights } from "@x12i/countex/insights";
36
+
37
+ const insights = createCountexInsights({
38
+ countex,
39
+ durable,
40
+ mapsByCounterId: new Map([[map.counterId, map]]),
41
+ });
42
+
43
+ await insights.topN({
44
+ scope,
45
+ counterId,
46
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
47
+ groupBy: ["dim:region"],
48
+ measure: "errors.rate",
49
+ n: 20,
50
+ });
51
+ ```
52
+
53
+ ## Observability
54
+
55
+ `CountexInsights` is an `EventEmitter`: `insight_query`, `recommendation`, `anomaly`, `insufficient_data`.
56
+
57
+ ## Errors
58
+
59
+ Typed errors extend `CountexInsightError` — see `@x12i/countex/insights` exports.
@@ -1,33 +1,33 @@
1
- # Insights: anomaly detection
2
-
3
- ## Baselines
4
-
5
- `detectAnomalies` evaluates the requested **timeframe** against a **baseline window** placed immediately before it. The default lookback string (`14d`, `24h`, `2w`, …) is converted into a number of buckets matching the query timeframe type.
6
-
7
- ## Methods
8
-
9
- - **robust-zscore** (default) — Uses median and MAD; `0.6745 * (x - median) / MAD`. When MAD is zero, a simple deviation fallback runs.
10
- - **zscore** — Classical mean/stddev (more sensitive to outliers).
11
- - **threshold** — Multiplicative deviation from the baseline median (e.g. spike when `observed > 2× median`).
12
- - **ewma** — Lightweight deviation check vs median (placeholder-style when full EWMA series is not stored).
13
-
14
- Sensitivity maps to z thresholds: low ≈ 4.0, medium ≈ 3.5, high ≈ 3.0.
15
-
16
- ## Missing activity
17
-
18
- When the baseline median exceeds a small floor and the observed window is near zero, a **missing** anomaly is emitted.
19
-
20
- ## Spike / drop
21
-
22
- Large positive z → spike; large negative z → drop (when enabled in `anomalyTypes`).
23
-
24
- ## Compare-by modes
25
-
26
- - **rolling** — All baseline buckets for the same group key.
27
- - **same-hour-of-day** / **same-day-of-week** — Filters baseline buckets to those matching the evaluation bucket’s calendar slot.
28
-
29
- ## Limitations
30
-
31
- - Requires enough baseline buckets (`minBaselineBuckets`, default from factory defaults).
32
- - Cardinality and “new dimension value” anomalies need histogram/distribution support in core for best results (future).
33
- - Very wide windows are rejected by `maxBuckets` / `maxGroups` guards.
1
+ # Insights: anomaly detection
2
+
3
+ ## Baselines
4
+
5
+ `detectAnomalies` evaluates the requested **timeframe** against a **baseline window** placed immediately before it. The default lookback string (`14d`, `24h`, `2w`, …) is converted into a number of buckets matching the query timeframe type.
6
+
7
+ ## Methods
8
+
9
+ - **robust-zscore** (default) — Uses median and MAD; `0.6745 * (x - median) / MAD`. When MAD is zero, a simple deviation fallback runs.
10
+ - **zscore** — Classical mean/stddev (more sensitive to outliers).
11
+ - **threshold** — Multiplicative deviation from the baseline median (e.g. spike when `observed > 2× median`).
12
+ - **ewma** — Lightweight deviation check vs median (placeholder-style when full EWMA series is not stored).
13
+
14
+ Sensitivity maps to z thresholds: low ≈ 4.0, medium ≈ 3.5, high ≈ 3.0.
15
+
16
+ ## Missing activity
17
+
18
+ When the baseline median exceeds a small floor and the observed window is near zero, a **missing** anomaly is emitted.
19
+
20
+ ## Spike / drop
21
+
22
+ Large positive z → spike; large negative z → drop (when enabled in `anomalyTypes`).
23
+
24
+ ## Compare-by modes
25
+
26
+ - **rolling** — All baseline buckets for the same group key.
27
+ - **same-hour-of-day** / **same-day-of-week** — Filters baseline buckets to those matching the evaluation bucket’s calendar slot.
28
+
29
+ ## Limitations
30
+
31
+ - Requires enough baseline buckets (`minBaselineBuckets`, default from factory defaults).
32
+ - Cardinality and “new dimension value” anomalies need histogram/distribution support in core for best results (future).
33
+ - Very wide windows are rejected by `maxBuckets` / `maxGroups` guards.
@@ -1,26 +1,26 @@
1
- # Insights: LLM cost and tokens
2
-
3
- All APIs remain **measure-oriented**. Names like `costMicros`, `outputTokens`, or `latencyMs` are conventions chosen by your maps — Countex Insights does not treat “LLM” as a special domain type.
4
-
5
- ## Required measures
6
-
7
- - **costDrivers / budgetProjection** — Any numeric measure with `.sum` (or `count`).
8
- - **recommendMaxTokens** — Uses grouped aggregates for the named measure (`outputTokens` by convention). Ideal inputs include `sum`, `count`, `min`, `max`, and derived `avg`.
9
-
10
- ## Cost drivers
11
-
12
- `costDrivers` ranks contributors by a cost measure and emits generic recommendations when concentration is extreme.
13
-
14
- ## Max tokens
15
-
16
- Without histograms, percentiles are **approximated** from min/max/avg; confidence is marked **low** or **medium**. When core adds optional histogram storage, recommendations should tighten and confidence can rise.
17
-
18
- Recommended ceiling (conceptually): `ceil(roundTo, percentile × safetyMultiplier)` — implementation uses aggregate proxies until histogram-backed percentiles exist.
19
-
20
- ## Budget projection
21
-
22
- Linear extrapolation from the **current window** rate into the **target window** length (bucket-count scaling). Status: `under` | `near` | `over` vs a numeric budget.
23
-
24
- ## Percentiles / distributions
25
-
26
- Histogram-backed measures are a **core** milestone. Insights degrades gracefully until `measure.pXX` selectors exist on `countex.query()`.
1
+ # Insights: LLM cost and tokens
2
+
3
+ All APIs remain **measure-oriented**. Names like `costMicros`, `outputTokens`, or `latencyMs` are conventions chosen by your maps — Countex Insights does not treat “LLM” as a special domain type.
4
+
5
+ ## Required measures
6
+
7
+ - **costDrivers / budgetProjection** — Any numeric measure with `.sum` (or `count`).
8
+ - **recommendMaxTokens** — Uses grouped aggregates for the named measure (`outputTokens` by convention). Ideal inputs include `sum`, `count`, `min`, `max`, and derived `avg`.
9
+
10
+ ## Cost drivers
11
+
12
+ `costDrivers` ranks contributors by a cost measure and emits generic recommendations when concentration is extreme.
13
+
14
+ ## Max tokens
15
+
16
+ Without histograms, percentiles are **approximated** from min/max/avg; confidence is marked **low** or **medium**. When core adds optional histogram storage, recommendations should tighten and confidence can rise.
17
+
18
+ Recommended ceiling (conceptually): `ceil(roundTo, percentile × safetyMultiplier)` — implementation uses aggregate proxies until histogram-backed percentiles exist.
19
+
20
+ ## Budget projection
21
+
22
+ Linear extrapolation from the **current window** rate into the **target window** length (bucket-count scaling). Status: `under` | `near` | `over` vs a numeric budget.
23
+
24
+ ## Percentiles / distributions
25
+
26
+ Histogram-backed measures are a **core** milestone. Insights degrades gracefully until `measure.pXX` selectors exist on `countex.query()`.
@@ -1,31 +1,33 @@
1
- # Insights: resource advisor
2
-
3
- ## Purpose
4
-
5
- `estimateResourcesFromActuals` projects hot-cache and durable storage needs from **observed cell streams**, using configurable byte constants per cell and per dictionary key.
6
-
7
- ## Requirements
8
-
9
- Pass the **durable** adapter into `createCountexInsights({ durable })`. The advisor calls `durable.streamCells()` over `sampleWindow`.
10
-
11
- ## Observed metrics
12
-
13
- - Approximate cells per day (mean and a simple P95 across days seen in the sample).
14
- - Heuristic dictionary key estimate (defaults to ~10% of streamed cells — tune in code if you allocate differently).
15
-
16
- ## Projections
17
-
18
- Combines:
19
-
20
- - **Hot cells** — Based on recent daily churn (P95 × short window).
21
- - **Durable cells** — Scales daily average by a retention multiplier derived from `targetRetention` (days/weeks/months/quarters/years/all).
22
-
23
- Outputs Redis/Mongo byte estimates and suggested RAM/disk figures (informational).
24
-
25
- ## Map / cardinality risk
26
-
27
- See **analyzeMapHealth** on the insights client. Pass `mapsByCounterId` so dimension lists and cardinality policies can be interpreted. High distinct counts per dimension trigger recommendations such as bucketing or removing hazardous paths.
28
-
29
- ## Retention advice
30
-
31
- When daily cell churn is very high, the resource advisor may emit a generic retention recommendation — tune thresholds for your environment.
1
+ # Insights: resource advisor
2
+
3
+ ## Purpose
4
+
5
+ `estimateResourcesFromActuals` projects hot-cache and durable storage needs from **observed cell streams**, using configurable byte constants per cell and per dictionary key.
6
+
7
+ ## Requirements
8
+
9
+ Pass the **durable** adapter into `createCountexInsights({ durable })`. The advisor calls `durable.streamCells()` over `sampleWindow`.
10
+
11
+ ## Observed metrics
12
+
13
+ - Approximate cells per day (mean and a simple P95 across days seen in the sample).
14
+ - Heuristic dictionary key estimate (defaults to ~10% of streamed cells — tune in code if you allocate differently).
15
+
16
+ ## Projections
17
+
18
+ Combines:
19
+
20
+ - **Hot cells** — Based on recent daily churn (P95 × short window).
21
+ - **Durable cells** — Scales daily average by a retention multiplier derived from `targetRetention` (days/weeks/months/quarters/years/all).
22
+
23
+ Outputs Redis, Mongo, and Postgres byte estimates and suggested RAM/disk figures (informational).
24
+
25
+ `postgresBytes` uses `constants.postgresBytesPerCell` (default **250**), a packed row next to the Mongo document constant (default **750**). Both numbers are returned so a move can be sized before the adapter is switched. They are estimates, not measurements of your dimension payloads.
26
+
27
+ ## Map / cardinality risk
28
+
29
+ See **analyzeMapHealth** on the insights client. Pass `mapsByCounterId` so dimension lists and cardinality policies can be interpreted. High distinct counts per dimension trigger recommendations such as bucketing or removing hazardous paths.
30
+
31
+ ## Retention advice
32
+
33
+ When daily cell churn is very high, the resource advisor may emit a generic retention recommendation — tune thresholds for your environment.