@x12i/countex 1.0.0 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +215 -4
- package/dist/backup/export.d.ts +2 -0
- package/dist/backup/export.d.ts.map +1 -1
- package/dist/backup/export.js +2 -0
- package/dist/backup/export.js.map +1 -1
- package/dist/backup/manifest.d.ts +49 -0
- package/dist/backup/manifest.d.ts.map +1 -0
- package/dist/backup/manifest.js +103 -0
- package/dist/backup/manifest.js.map +1 -0
- package/dist/backup/snapshot-encoding.d.ts +8 -0
- package/dist/backup/snapshot-encoding.d.ts.map +1 -0
- package/dist/backup/snapshot-encoding.js +27 -0
- package/dist/backup/snapshot-encoding.js.map +1 -0
- package/dist/cli.d.ts +12 -0
- package/dist/cli.d.ts.map +1 -0
- package/dist/cli.js +206 -0
- package/dist/cli.js.map +1 -0
- package/dist/core/buffer.d.ts +4 -3
- package/dist/core/buffer.d.ts.map +1 -1
- package/dist/core/buffer.js +65 -9
- package/dist/core/buffer.js.map +1 -1
- package/dist/core/countex.d.ts +14 -1
- package/dist/core/countex.d.ts.map +1 -1
- package/dist/core/countex.js +171 -48
- package/dist/core/countex.js.map +1 -1
- package/dist/core/extraction.d.ts +14 -6
- package/dist/core/extraction.d.ts.map +1 -1
- package/dist/core/extraction.js +117 -1
- package/dist/core/extraction.js.map +1 -1
- package/dist/core/mapping.d.ts +5 -0
- package/dist/core/mapping.d.ts.map +1 -1
- package/dist/core/mapping.js +71 -1
- package/dist/core/mapping.js.map +1 -1
- package/dist/core/timeframe.d.ts +7 -0
- package/dist/core/timeframe.d.ts.map +1 -1
- package/dist/core/timeframe.js +88 -0
- package/dist/core/timeframe.js.map +1 -1
- package/dist/index.d.ts +4 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +13 -1
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/fanout-health.d.ts +3 -0
- package/dist/insights/advisors/fanout-health.d.ts.map +1 -0
- package/dist/insights/advisors/fanout-health.js +3 -0
- package/dist/insights/advisors/fanout-health.js.map +1 -0
- package/dist/insights/advisors/map-health.d.ts +6 -0
- package/dist/insights/advisors/map-health.d.ts.map +1 -0
- package/dist/insights/advisors/map-health.js +154 -0
- package/dist/insights/advisors/map-health.js.map +1 -0
- package/dist/insights/advisors/resource-advisor.d.ts +4 -0
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -0
- package/dist/insights/advisors/resource-advisor.js +102 -0
- package/dist/insights/advisors/resource-advisor.js.map +1 -0
- package/dist/insights/advisors/retention-advisor.d.ts +2 -0
- package/dist/insights/advisors/retention-advisor.d.ts.map +1 -0
- package/dist/insights/advisors/retention-advisor.js +6 -0
- package/dist/insights/advisors/retention-advisor.js.map +1 -0
- package/dist/insights/advisors-subpath.d.ts +4 -0
- package/dist/insights/advisors-subpath.d.ts.map +1 -0
- package/dist/insights/advisors-subpath.js +10 -0
- package/dist/insights/advisors-subpath.js.map +1 -0
- package/dist/insights/anomaly/baseline.d.ts +4 -0
- package/dist/insights/anomaly/baseline.d.ts.map +1 -0
- package/dist/insights/anomaly/baseline.js +54 -0
- package/dist/insights/anomaly/baseline.js.map +1 -0
- package/dist/insights/anomaly/detect-anomalies.d.ts +6 -0
- package/dist/insights/anomaly/detect-anomalies.d.ts.map +1 -0
- package/dist/insights/anomaly/detect-anomalies.js +245 -0
- package/dist/insights/anomaly/detect-anomalies.js.map +1 -0
- package/dist/insights/anomaly/ewma.d.ts +2 -0
- package/dist/insights/anomaly/ewma.d.ts.map +1 -0
- package/dist/insights/anomaly/ewma.js +15 -0
- package/dist/insights/anomaly/ewma.js.map +1 -0
- package/dist/insights/anomaly/missing-activity.d.ts +3 -0
- package/dist/insights/anomaly/missing-activity.d.ts.map +1 -0
- package/dist/insights/anomaly/missing-activity.js +8 -0
- package/dist/insights/anomaly/missing-activity.js.map +1 -0
- package/dist/insights/anomaly/robust-zscore.d.ts +2 -0
- package/dist/insights/anomaly/robust-zscore.d.ts.map +1 -0
- package/dist/insights/anomaly/robust-zscore.js +9 -0
- package/dist/insights/anomaly/robust-zscore.js.map +1 -0
- package/dist/insights/anomaly-subpath.d.ts +7 -0
- package/dist/insights/anomaly-subpath.d.ts.map +1 -0
- package/dist/insights/anomaly-subpath.js +20 -0
- package/dist/insights/anomaly-subpath.js.map +1 -0
- package/dist/insights/comparison/compare-periods.d.ts +5 -0
- package/dist/insights/comparison/compare-periods.d.ts.map +1 -0
- package/dist/insights/comparison/compare-periods.js +64 -0
- package/dist/insights/comparison/compare-periods.js.map +1 -0
- package/dist/insights/comparison/growth-rate.d.ts +3 -0
- package/dist/insights/comparison/growth-rate.d.ts.map +1 -0
- package/dist/insights/comparison/growth-rate.js +3 -0
- package/dist/insights/comparison/growth-rate.js.map +1 -0
- package/dist/insights/comparison/moving-average.d.ts +2 -0
- package/dist/insights/comparison/moving-average.d.ts.map +1 -0
- package/dist/insights/comparison/moving-average.js +6 -0
- package/dist/insights/comparison/moving-average.js.map +1 -0
- package/dist/insights/comparison/trend.d.ts +5 -0
- package/dist/insights/comparison/trend.d.ts.map +1 -0
- package/dist/insights/comparison/trend.js +84 -0
- package/dist/insights/comparison/trend.js.map +1 -0
- package/dist/insights/errors.d.ts +25 -0
- package/dist/insights/errors.d.ts.map +1 -0
- package/dist/insights/errors.js +54 -0
- package/dist/insights/errors.js.map +1 -0
- package/dist/insights/forecast/budget.d.ts +2 -0
- package/dist/insights/forecast/budget.d.ts.map +1 -0
- package/dist/insights/forecast/budget.js +6 -0
- package/dist/insights/forecast/budget.js.map +1 -0
- package/dist/insights/forecast/capacity.d.ts +2 -0
- package/dist/insights/forecast/capacity.d.ts.map +1 -0
- package/dist/insights/forecast/capacity.js +3 -0
- package/dist/insights/forecast/capacity.js.map +1 -0
- package/dist/insights/forecast/forecast.d.ts +5 -0
- package/dist/insights/forecast/forecast.d.ts.map +1 -0
- package/dist/insights/forecast/forecast.js +103 -0
- package/dist/insights/forecast/forecast.js.map +1 -0
- package/dist/insights/forecast/retention.d.ts +2 -0
- package/dist/insights/forecast/retention.d.ts.map +1 -0
- package/dist/insights/forecast/retention.js +3 -0
- package/dist/insights/forecast/retention.js.map +1 -0
- package/dist/insights/index.d.ts +4 -0
- package/dist/insights/index.d.ts.map +1 -0
- package/dist/insights/index.js +23 -0
- package/dist/insights/index.js.map +1 -0
- package/dist/insights/insights-client.d.ts +33 -0
- package/dist/insights/insights-client.d.ts.map +1 -0
- package/dist/insights/insights-client.js +344 -0
- package/dist/insights/insights-client.js.map +1 -0
- package/dist/insights/llm-subpath.d.ts +4 -0
- package/dist/insights/llm-subpath.d.ts.map +1 -0
- package/dist/insights/llm-subpath.js +10 -0
- package/dist/insights/llm-subpath.js.map +1 -0
- package/dist/insights/optimization/cost-optimizer.d.ts +6 -0
- package/dist/insights/optimization/cost-optimizer.d.ts.map +1 -0
- package/dist/insights/optimization/cost-optimizer.js +98 -0
- package/dist/insights/optimization/cost-optimizer.js.map +1 -0
- package/dist/insights/optimization/latency-optimizer.d.ts +3 -0
- package/dist/insights/optimization/latency-optimizer.d.ts.map +1 -0
- package/dist/insights/optimization/latency-optimizer.js +3 -0
- package/dist/insights/optimization/latency-optimizer.js.map +1 -0
- package/dist/insights/optimization/threshold-advisor.d.ts +2 -0
- package/dist/insights/optimization/threshold-advisor.d.ts.map +1 -0
- package/dist/insights/optimization/threshold-advisor.js +3 -0
- package/dist/insights/optimization/threshold-advisor.js.map +1 -0
- package/dist/insights/optimization/token-optimizer.d.ts +4 -0
- package/dist/insights/optimization/token-optimizer.d.ts.map +1 -0
- package/dist/insights/optimization/token-optimizer.js +133 -0
- package/dist/insights/optimization/token-optimizer.js.map +1 -0
- package/dist/insights/ranking/contribution.d.ts +2 -0
- package/dist/insights/ranking/contribution.d.ts.map +1 -0
- package/dist/insights/ranking/contribution.js +6 -0
- package/dist/insights/ranking/contribution.js.map +1 -0
- package/dist/insights/ranking/helpers.d.ts +22 -0
- package/dist/insights/ranking/helpers.d.ts.map +1 -0
- package/dist/insights/ranking/helpers.js +88 -0
- package/dist/insights/ranking/helpers.js.map +1 -0
- package/dist/insights/ranking/pareto.d.ts +2 -0
- package/dist/insights/ranking/pareto.d.ts.map +1 -0
- package/dist/insights/ranking/pareto.js +6 -0
- package/dist/insights/ranking/pareto.js.map +1 -0
- package/dist/insights/ranking/slices.d.ts +2 -0
- package/dist/insights/ranking/slices.d.ts.map +1 -0
- package/dist/insights/ranking/slices.js +6 -0
- package/dist/insights/ranking/slices.js.map +1 -0
- package/dist/insights/ranking/top-bottom.d.ts +2 -0
- package/dist/insights/ranking/top-bottom.d.ts.map +1 -0
- package/dist/insights/ranking/top-bottom.js +7 -0
- package/dist/insights/ranking/top-bottom.js.map +1 -0
- package/dist/insights/recommendations/evidence.d.ts +2 -0
- package/dist/insights/recommendations/evidence.d.ts.map +1 -0
- package/dist/insights/recommendations/evidence.js +3 -0
- package/dist/insights/recommendations/evidence.js.map +1 -0
- package/dist/insights/recommendations/recommendation.d.ts +6 -0
- package/dist/insights/recommendations/recommendation.d.ts.map +1 -0
- package/dist/insights/recommendations/recommendation.js +25 -0
- package/dist/insights/recommendations/recommendation.js.map +1 -0
- package/dist/insights/recommendations/severity.d.ts +2 -0
- package/dist/insights/recommendations/severity.d.ts.map +1 -0
- package/dist/insights/recommendations/severity.js +3 -0
- package/dist/insights/recommendations/severity.js.map +1 -0
- package/dist/insights/rules/evaluate-rules.d.ts +5 -0
- package/dist/insights/rules/evaluate-rules.d.ts.map +1 -0
- package/dist/insights/rules/evaluate-rules.js +156 -0
- package/dist/insights/rules/evaluate-rules.js.map +1 -0
- package/dist/insights/types.d.ts +405 -0
- package/dist/insights/types.d.ts.map +1 -0
- package/dist/insights/types.js +3 -0
- package/dist/insights/types.js.map +1 -0
- package/dist/insights/utils/formatting.d.ts +4 -0
- package/dist/insights/utils/formatting.d.ts.map +1 -0
- package/dist/insights/utils/formatting.js +21 -0
- package/dist/insights/utils/formatting.js.map +1 -0
- package/dist/insights/utils/grouping.d.ts +2 -0
- package/dist/insights/utils/grouping.d.ts.map +1 -0
- package/dist/insights/utils/grouping.js +8 -0
- package/dist/insights/utils/grouping.js.map +1 -0
- package/dist/insights/utils/query-builder.d.ts +21 -0
- package/dist/insights/utils/query-builder.d.ts.map +1 -0
- package/dist/insights/utils/query-builder.js +107 -0
- package/dist/insights/utils/query-builder.js.map +1 -0
- package/dist/insights/utils/row-value.d.ts +5 -0
- package/dist/insights/utils/row-value.d.ts.map +1 -0
- package/dist/insights/utils/row-value.js +36 -0
- package/dist/insights/utils/row-value.js.map +1 -0
- package/dist/insights/utils/stats.d.ts +28 -0
- package/dist/insights/utils/stats.d.ts.map +1 -0
- package/dist/insights/utils/stats.js +125 -0
- package/dist/insights/utils/stats.js.map +1 -0
- package/dist/query/formatter.d.ts +2 -2
- package/dist/query/formatter.d.ts.map +1 -1
- package/dist/query/formatter.js +211 -30
- package/dist/query/formatter.js.map +1 -1
- package/dist/query/planner.d.ts +10 -2
- package/dist/query/planner.d.ts.map +1 -1
- package/dist/query/planner.js +55 -4
- package/dist/query/planner.js.map +1 -1
- package/dist/simulation/context.d.ts +5 -0
- package/dist/simulation/context.d.ts.map +1 -0
- package/dist/simulation/context.js +56 -0
- package/dist/simulation/context.js.map +1 -0
- package/dist/simulation/events.d.ts +3 -0
- package/dist/simulation/events.d.ts.map +1 -0
- package/dist/simulation/events.js +67 -0
- package/dist/simulation/events.js.map +1 -0
- package/dist/simulation/index.d.ts +7 -0
- package/dist/simulation/index.d.ts.map +1 -0
- package/dist/simulation/index.js +16 -0
- package/dist/simulation/index.js.map +1 -0
- package/dist/simulation/prng.d.ts +4 -0
- package/dist/simulation/prng.d.ts.map +1 -0
- package/dist/simulation/prng.js +18 -0
- package/dist/simulation/prng.js.map +1 -0
- package/dist/simulation/resource-estimate.d.ts +6 -0
- package/dist/simulation/resource-estimate.d.ts.map +1 -0
- package/dist/simulation/resource-estimate.js +11 -0
- package/dist/simulation/resource-estimate.js.map +1 -0
- package/dist/simulation/simulator.d.ts +15 -0
- package/dist/simulation/simulator.d.ts.map +1 -0
- package/dist/simulation/simulator.js +67 -0
- package/dist/simulation/simulator.js.map +1 -0
- package/dist/simulation/tracker.d.ts +8 -0
- package/dist/simulation/tracker.d.ts.map +1 -0
- package/dist/simulation/tracker.js +18 -0
- package/dist/simulation/tracker.js.map +1 -0
- package/dist/simulation/types.d.ts +49 -0
- package/dist/simulation/types.d.ts.map +1 -0
- package/dist/simulation/types.js +21 -0
- package/dist/simulation/types.js.map +1 -0
- package/dist/storage/memory.d.ts +3 -0
- package/dist/storage/memory.d.ts.map +1 -1
- package/dist/storage/memory.js +113 -4
- package/dist/storage/memory.js.map +1 -1
- package/dist/storage/mongo.d.ts +5 -0
- package/dist/storage/mongo.d.ts.map +1 -1
- package/dist/storage/mongo.js +129 -17
- package/dist/storage/mongo.js.map +1 -1
- package/dist/storage/redis.d.ts +12 -9
- package/dist/storage/redis.d.ts.map +1 -1
- package/dist/storage/redis.js +126 -47
- package/dist/storage/redis.js.map +1 -1
- package/dist/types/index.d.ts +139 -4
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +40 -2
- package/docs/INSIGHTS.md +59 -0
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -0
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -0
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +31 -0
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -0
- package/docs/guides/README.md +34 -0
- package/docs/guides/adapter-memory.md +48 -0
- package/docs/guides/adapter-mongo.md +62 -0
- package/docs/guides/adapter-redis.md +53 -0
- package/docs/guides/backup-export.md +83 -0
- package/docs/guides/cardinality.md +50 -0
- package/docs/guides/cli.md +53 -0
- package/docs/guides/client-events.md +49 -0
- package/docs/guides/counter-maps.md +122 -0
- package/docs/guides/idempotency.md +39 -0
- package/docs/guides/ingestion-and-flush.md +67 -0
- package/docs/guides/insights-advisors-and-optimization.md +39 -0
- package/docs/guides/insights-anomalies-and-rules.md +48 -0
- package/docs/guides/insights-comparison-and-trends.md +47 -0
- package/docs/guides/insights-forecasting-and-budget.md +61 -0
- package/docs/guides/insights-overview.md +82 -0
- package/docs/guides/insights-ranking.md +81 -0
- package/docs/guides/measures.md +71 -0
- package/docs/guides/querying.md +119 -0
- package/docs/guides/simulation-profiles.md +33 -0
- package/docs/guides/simulation-resource-estimation.md +30 -0
- package/docs/guides/simulation.md +36 -0
- package/docs/guides/timeframes-and-bucketing.md +55 -0
- package/docs/internals/README.md +21 -0
- package/docs/internals/internal-architecture-overview.md +65 -0
- package/docs/internals/internal-backup-stream.md +23 -0
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -0
- package/docs/internals/internal-cardinality.md +30 -0
- package/docs/internals/internal-dimension-hash.md +34 -0
- package/docs/internals/internal-idempotency.md +36 -0
- package/docs/internals/internal-ingest-pipeline.md +52 -0
- package/docs/internals/internal-insights-layer.md +36 -0
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -0
- package/docs/internals/internal-measures-and-scales.md +35 -0
- package/docs/internals/internal-query-path.md +46 -0
- package/docs/internals/internal-storage-memory.md +13 -0
- package/docs/internals/internal-storage-mongo.md +24 -0
- package/docs/internals/internal-storage-redis.md +31 -0
- package/docs/internals/internal-timeframe-bucketing.md +33 -0
- package/docs/use-cases/README.md +56 -0
- package/docs/use-cases/audit-security-log-counts.md +550 -0
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -0
- package/docs/use-cases/multi-tenant-isolation.md +203 -0
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -0
- package/docs/use-cases/processing-workflow-activity.md +165 -0
- package/docs/use-cases/product-feature-usage.md +240 -0
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -0
- package/package.json +61 -2
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# Guide: Insights — advisors and resource estimation
|
|
2
|
+
|
|
3
|
+
Advisors inspect **map shape**, **fan-out health**, and optionally **stream durable cells** to estimate capacity-style metrics.
|
|
4
|
+
|
|
5
|
+
## Step 1: Map health
|
|
6
|
+
|
|
7
|
+
Requires `mapsByCounterId` passed to `createCountexInsights`.
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
const health = await insights.analyzeMapHealth({
|
|
11
|
+
scope: "org_1",
|
|
12
|
+
counterId: "usage",
|
|
13
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
14
|
+
});
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Step 2: Resource estimation from actuals
|
|
18
|
+
|
|
19
|
+
Requires the **`durable`** reference in `createCountexInsights`.
|
|
20
|
+
|
|
21
|
+
```ts
|
|
22
|
+
const resources = await insights.estimateResourcesFromActuals({
|
|
23
|
+
scope: "org_1",
|
|
24
|
+
counterId: "usage",
|
|
25
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
26
|
+
});
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
This path uses `durable.streamCells` — ensure your adapter implements streaming efficiently.
|
|
30
|
+
|
|
31
|
+
## Events
|
|
32
|
+
|
|
33
|
+
Recommendations attach to `recommendation` emissions with severity for downstream routing (pager, ticket, etc.).
|
|
34
|
+
|
|
35
|
+
## See also
|
|
36
|
+
|
|
37
|
+
- `docs/INSIGHTS_RESOURCE_ADVISOR.md`
|
|
38
|
+
- [Insights overview](./insights-overview.md)
|
|
39
|
+
- Internal: [Insights layer](../internals/internal-insights-layer.md)
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# Guide: Insights — anomalies and rules
|
|
2
|
+
|
|
3
|
+
Detect unusual periods relative to a baseline and evaluate declarative **rules** over stored aggregates.
|
|
4
|
+
|
|
5
|
+
## Step 1: Detect anomalies
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const { anomalies } = await insights.detectAnomalies({
|
|
9
|
+
scope: "org_1",
|
|
10
|
+
counterId: "usage",
|
|
11
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-14" },
|
|
12
|
+
groupBy: [],
|
|
13
|
+
measure: "count",
|
|
14
|
+
baseline: {
|
|
15
|
+
timeframe: { type: "day", from: "2026-04-01", to: "2026-04-30" },
|
|
16
|
+
minBuckets: 12,
|
|
17
|
+
},
|
|
18
|
+
});
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Notes:
|
|
22
|
+
|
|
23
|
+
- **`timeframe.type: "all"`** is not supported; you’ll receive an empty list and an `insufficient_data` event.
|
|
24
|
+
- Emits **`anomaly`** events with severity for each finding.
|
|
25
|
+
|
|
26
|
+
## Step 2: Evaluate rules
|
|
27
|
+
|
|
28
|
+
```ts
|
|
29
|
+
const rules = await insights.evaluateRules({
|
|
30
|
+
scope: "org_1",
|
|
31
|
+
counterId: "usage",
|
|
32
|
+
timeframe: { type: "week", from: "2026-W18", to: "2026-W19" },
|
|
33
|
+
rules: [
|
|
34
|
+
{
|
|
35
|
+
id: "spike_cost",
|
|
36
|
+
when: { measure: "cost.sum", op: "gt", value: 10_000 },
|
|
37
|
+
recommend: { title: "Investigate cost spike" },
|
|
38
|
+
},
|
|
39
|
+
],
|
|
40
|
+
});
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Rule evaluation reuses the insights query helper so timeouts and group caps stay consistent.
|
|
44
|
+
|
|
45
|
+
## See also
|
|
46
|
+
|
|
47
|
+
- `docs/INSIGHTS_ANOMALY_DETECTION.md`
|
|
48
|
+
- Internal: [Insights layer](../internals/internal-insights-layer.md)
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Guide: Insights — comparison and trends
|
|
2
|
+
|
|
3
|
+
Compare two time windows or derive simple **trend** statistics from a timeseries query.
|
|
4
|
+
|
|
5
|
+
## Step 1: Compare periods
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const cmp = await insights.comparePeriods({
|
|
9
|
+
scope: "org_1",
|
|
10
|
+
counterId: "usage",
|
|
11
|
+
current: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
12
|
+
previous: { type: "day", from: "2026-05-07", to: "2026-05-07" },
|
|
13
|
+
groupBy: ["metric"],
|
|
14
|
+
measure: "count",
|
|
15
|
+
minPercentChange: 10,
|
|
16
|
+
minAbsoluteChange: 5,
|
|
17
|
+
});
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
The helper issues **two** queries (current and previous) and merges rows by group keys.
|
|
21
|
+
|
|
22
|
+
## Step 2: Trend from timeseries
|
|
23
|
+
|
|
24
|
+
```ts
|
|
25
|
+
const trend = await insights.trend({
|
|
26
|
+
scope: "org_1",
|
|
27
|
+
counterId: "usage",
|
|
28
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-14" },
|
|
29
|
+
groupBy: [],
|
|
30
|
+
measure: "cost.sum",
|
|
31
|
+
method: "linear",
|
|
32
|
+
movingAverageWindow: 3,
|
|
33
|
+
});
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Underlying query uses `format: "timeseries"`; post-processing computes growth / smoothing per configuration.
|
|
37
|
+
|
|
38
|
+
## Options shared with other insights
|
|
39
|
+
|
|
40
|
+
- `filters` — subject/metric/dimension filters compatible with core queries
|
|
41
|
+
- `bypassCache`, `signal`
|
|
42
|
+
- `maxBuckets`, `maxGroups`
|
|
43
|
+
|
|
44
|
+
## See also
|
|
45
|
+
|
|
46
|
+
- [Timeframes](./timeframes-and-bucketing.md)
|
|
47
|
+
- Internal: [Insights layer](../internals/internal-insights-layer.md)
|
|
@@ -0,0 +1,61 @@
|
|
|
1
|
+
# Guide: Insights — forecasting and budget
|
|
2
|
+
|
|
3
|
+
Forecast simple forward curves and relate measures to **budgets**, **cost drivers**, and **token ceilings**.
|
|
4
|
+
|
|
5
|
+
## Step 1: Forecast
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const fc = await insights.forecast({
|
|
9
|
+
scope: "org_1",
|
|
10
|
+
counterId: "usage",
|
|
11
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-14" },
|
|
12
|
+
measure: "count",
|
|
13
|
+
horizonBuckets: 7,
|
|
14
|
+
});
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Recommendations may emit `recommendation` events on the insights client.
|
|
18
|
+
|
|
19
|
+
## Step 2: Budget projection
|
|
20
|
+
|
|
21
|
+
```ts
|
|
22
|
+
const budget = await insights.budgetProjection({
|
|
23
|
+
scope: "org_1",
|
|
24
|
+
counterId: "usage",
|
|
25
|
+
timeframe: { type: "month", from: "2026-05", to: "2026-05" },
|
|
26
|
+
measure: "cost.sum",
|
|
27
|
+
budget: 50_000,
|
|
28
|
+
});
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
## Step 3: Cost drivers
|
|
32
|
+
|
|
33
|
+
```ts
|
|
34
|
+
const drivers = await insights.costDrivers({
|
|
35
|
+
scope: "org_1",
|
|
36
|
+
counterId: "usage",
|
|
37
|
+
timeframe: { type: "month", from: "2026-05", to: "2026-05" },
|
|
38
|
+
groupBy: ["subject", "dim:model"],
|
|
39
|
+
measure: "cost.sum",
|
|
40
|
+
});
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Step 4: Token recommendations
|
|
44
|
+
|
|
45
|
+
```ts
|
|
46
|
+
const tokens = await insights.recommendMaxTokens({
|
|
47
|
+
scope: "org_1",
|
|
48
|
+
counterId: "usage",
|
|
49
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
50
|
+
groupBy: ["dim:model"],
|
|
51
|
+
outputTokenMeasure: "outputTokens",
|
|
52
|
+
safetyFactor: 1.2,
|
|
53
|
+
});
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
Measure names must exist on your map / cells.
|
|
57
|
+
|
|
58
|
+
## See also
|
|
59
|
+
|
|
60
|
+
- `docs/INSIGHTS_LLM_COST_AND_TOKENS.md`
|
|
61
|
+
- Internal: [Insights layer](../internals/internal-insights-layer.md)
|
|
@@ -0,0 +1,82 @@
|
|
|
1
|
+
# Guide: Insights overview
|
|
2
|
+
|
|
3
|
+
The **`@x12i/countex/insights`** module layers ranking, comparison, anomaly detection, forecasting, and advisory helpers on top of ordinary `countex.query` results.
|
|
4
|
+
|
|
5
|
+
## Step 1: Install / import
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
import { createCountexInsights } from "@x12i/countex/insights";
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
No extra peer dependencies beyond what your core Countex setup already uses.
|
|
12
|
+
|
|
13
|
+
## Step 2: Construct insights with your Countex client
|
|
14
|
+
|
|
15
|
+
```ts
|
|
16
|
+
const insights = createCountexInsights({
|
|
17
|
+
countex,
|
|
18
|
+
durable, // optional; required only for estimateResourcesFromActuals
|
|
19
|
+
mapsByCounterId: new Map(maps.map((m) => [m.counterId, m])),
|
|
20
|
+
defaults: {
|
|
21
|
+
maxGroups: 10_000,
|
|
22
|
+
maxBuckets: 10_000,
|
|
23
|
+
queryTimeoutMs: 10_000,
|
|
24
|
+
minBaselineBuckets: 12,
|
|
25
|
+
resultLimit: 1_000,
|
|
26
|
+
},
|
|
27
|
+
});
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Passing **`mapsByCounterId`** enables map-aware advisors (for example map health).
|
|
31
|
+
|
|
32
|
+
## Step 3: Run an insight
|
|
33
|
+
|
|
34
|
+
Each method ultimately builds a `QuerySpec`, calls `client.query`, then post-processes rows.
|
|
35
|
+
|
|
36
|
+
```ts
|
|
37
|
+
const top = await insights.topN({
|
|
38
|
+
scope: "org_1",
|
|
39
|
+
counterId: "llm_usage",
|
|
40
|
+
timeframe: { type: "month", from: "2026-05", to: "2026-05" },
|
|
41
|
+
groupBy: ["subject"],
|
|
42
|
+
measure: "cost.sum",
|
|
43
|
+
n: 10,
|
|
44
|
+
});
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Step 4: Observe insight lifecycle events
|
|
48
|
+
|
|
49
|
+
`CountexInsights` emits:
|
|
50
|
+
|
|
51
|
+
- `insight_query` — timing per underlying query
|
|
52
|
+
- `anomaly`, `recommendation`, `insufficient_data` — depending on method
|
|
53
|
+
|
|
54
|
+
```ts
|
|
55
|
+
insights.on("insight_query", (e) => console.log(e));
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
## Step 5: Respect limits
|
|
59
|
+
|
|
60
|
+
Insights honor **`maxBuckets`** (aligned with core query expansion) and **`maxGroups`**. Prefer narrower filters when grouping heavily.
|
|
61
|
+
|
|
62
|
+
## Distributions, percentiles, and histograms
|
|
63
|
+
|
|
64
|
+
Core Countex cells store **additive** measure aggregates (`sum`, `count`, `min`, `max`) per sparse cell — not full distributions. Insights therefore reason about **rankings, deltas, thresholds, and derived averages**, not arbitrary percentile curves across arbitrary slices unless you add **explicit** histogram or sketch storage elsewhere.
|
|
65
|
+
|
|
66
|
+
Token and latency **advisors** use those summaries (and sometimes robust z-scores on bucketed series); treat percentile-style language in recommendations as **heuristic** unless you extend storage.
|
|
67
|
+
|
|
68
|
+
## Feature map
|
|
69
|
+
|
|
70
|
+
| Area | Methods |
|
|
71
|
+
|------|---------|
|
|
72
|
+
| Ranking | `topN`, `bottomN`, `topPercent`, `bottomPercent`, `pareto`, `contribution` |
|
|
73
|
+
| Comparison | `comparePeriods`, `trend` |
|
|
74
|
+
| Anomalies & rules | `detectAnomalies`, `evaluateRules` |
|
|
75
|
+
| Forecast & budget | `forecast`, `budgetProjection`, `costDrivers`, `recommendMaxTokens` |
|
|
76
|
+
| Advisors | `analyzeMapHealth`, `estimateResourcesFromActuals` |
|
|
77
|
+
|
|
78
|
+
## See also
|
|
79
|
+
|
|
80
|
+
- Domain-focused narratives: `docs/INSIGHTS.md`, `docs/INSIGHTS_ANOMALY_DETECTION.md`, etc.
|
|
81
|
+
- [Ranking](./insights-ranking.md)
|
|
82
|
+
- Internal: [Insights layer](../internals/internal-insights-layer.md)
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# Guide: Insights — ranking
|
|
2
|
+
|
|
3
|
+
Ranking helpers turn grouped query rows into **top/bottom lists**, **percent slices**, **Pareto** views, and **contribution** summaries.
|
|
4
|
+
|
|
5
|
+
## Prerequisites
|
|
6
|
+
|
|
7
|
+
- A `CountexInsights` instance — see [Insights overview](./insights-overview.md).
|
|
8
|
+
- A measure selector string such as `"count"` or `"cost.sum"`.
|
|
9
|
+
|
|
10
|
+
## Step 1: Top N
|
|
11
|
+
|
|
12
|
+
```ts
|
|
13
|
+
const out = await insights.topN({
|
|
14
|
+
scope: "org_1",
|
|
15
|
+
counterId: "usage",
|
|
16
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
17
|
+
groupBy: ["metric"],
|
|
18
|
+
measure: "count",
|
|
19
|
+
n: 10,
|
|
20
|
+
includeTotal: true,
|
|
21
|
+
});
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Step 2: Bottom N
|
|
25
|
+
|
|
26
|
+
```ts
|
|
27
|
+
const out = await insights.bottomN({
|
|
28
|
+
scope: "org_1",
|
|
29
|
+
counterId: "usage",
|
|
30
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
31
|
+
groupBy: ["dim:region"],
|
|
32
|
+
measure: "latencyMs.avg",
|
|
33
|
+
n: 5,
|
|
34
|
+
includeZeroes: false,
|
|
35
|
+
});
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Sparse counters omit zero rows unless you opt in.
|
|
39
|
+
|
|
40
|
+
## Step 3: Top / bottom percent slices
|
|
41
|
+
|
|
42
|
+
```ts
|
|
43
|
+
const slice = await insights.topPercent({
|
|
44
|
+
scope: "org_1",
|
|
45
|
+
counterId: "usage",
|
|
46
|
+
timeframe: { type: "month", from: "2026-05", to: "2026-05" },
|
|
47
|
+
groupBy: ["subject"],
|
|
48
|
+
measure: "cost.sum",
|
|
49
|
+
percent: 20,
|
|
50
|
+
});
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Step 4: Pareto
|
|
54
|
+
|
|
55
|
+
```ts
|
|
56
|
+
const pareto = await insights.pareto({
|
|
57
|
+
scope: "org_1",
|
|
58
|
+
counterId: "usage",
|
|
59
|
+
timeframe: { type: "month", from: "2026-05", to: "2026-05" },
|
|
60
|
+
groupBy: ["subject"],
|
|
61
|
+
measure: "cost.sum",
|
|
62
|
+
targetShare: 0.8,
|
|
63
|
+
});
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
## Step 5: Contribution
|
|
67
|
+
|
|
68
|
+
```ts
|
|
69
|
+
const contrib = await insights.contribution({
|
|
70
|
+
scope: "org_1",
|
|
71
|
+
counterId: "usage",
|
|
72
|
+
timeframe: { type: "month", from: "2026-05", to: "2026-05" },
|
|
73
|
+
groupBy: ["dim:model"],
|
|
74
|
+
measure: "tokens.sum",
|
|
75
|
+
});
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## See also
|
|
79
|
+
|
|
80
|
+
- [Querying](./querying.md)
|
|
81
|
+
- Internal: [Insights layer](../internals/internal-insights-layer.md)
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Guide: Measures
|
|
2
|
+
|
|
3
|
+
**Measures** are optional numeric fields per event (cost, latency, tokens, …) aggregated alongside counts.
|
|
4
|
+
|
|
5
|
+
## Step 1: Declare measures on the map
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const map = {
|
|
9
|
+
counterId: "llm_usage",
|
|
10
|
+
scope: { path: "$.scope.id" },
|
|
11
|
+
eventTime: { path: "$.request.createdAt" },
|
|
12
|
+
subjects: [{ type: "app", path: "$.app.id", required: true }],
|
|
13
|
+
metrics: [{ path: "$.request.status", required: true }],
|
|
14
|
+
dimensions: [{ name: "model", path: "$.model.name" }],
|
|
15
|
+
delta: { default: 1 },
|
|
16
|
+
timeframes: ["day"],
|
|
17
|
+
measures: [
|
|
18
|
+
{ name: "cost", path: "$.usage.costCents", aggregations: ["sum"], scale: 100 },
|
|
19
|
+
{ name: "latencyMs", path: "$.usage.latencyMs", aggregations: ["sum", "count", "min", "max"] },
|
|
20
|
+
],
|
|
21
|
+
} as const;
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
### Aggregations
|
|
25
|
+
|
|
26
|
+
- **`sum`**, **`count`**, **`min`**, **`max`** are stored.
|
|
27
|
+
- **`avg`** is derived at query time when both **`sum`** and **`count`** exist for that measure.
|
|
28
|
+
|
|
29
|
+
### Scale
|
|
30
|
+
|
|
31
|
+
`scale` multiplies then truncates for fixed-point storage (for example cents → integer micros).
|
|
32
|
+
|
|
33
|
+
### Missing values
|
|
34
|
+
|
|
35
|
+
Use `onMissing: "skip"` (default) or `"drop"` / `"zero"` per measure.
|
|
36
|
+
|
|
37
|
+
### Unique / distinct counts are not additive sums
|
|
38
|
+
|
|
39
|
+
Countex **counts** and **measure aggregations** (`sum`, `count`, `min`, `max`) behave like standard additive counters over cells. They do **not** answer “how many **distinct** users appeared in this bucket?” unless you model that explicitly (for example one subject id per event and interpret cell counts accordingly). Do **not** sum counts across overlapping populations and treat the result as a unique count. For large-cardinality uniques, plan on a dedicated approach (for example optional future HLL-style storage — see deployment notes); vanilla cells are the wrong tool for deduplicated set cardinality across arbitrary queries.
|
|
40
|
+
|
|
41
|
+
## Step 2: Ingest as usual
|
|
42
|
+
|
|
43
|
+
Each fan-out increment carries the **same** measure snapshot for that event.
|
|
44
|
+
|
|
45
|
+
## Step 3: Query measures
|
|
46
|
+
|
|
47
|
+
```ts
|
|
48
|
+
const result = await countex.query({
|
|
49
|
+
scope: "org_1",
|
|
50
|
+
counterId: "llm_usage",
|
|
51
|
+
timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
|
|
52
|
+
groupBy: ["subject", "metric"],
|
|
53
|
+
measures: ["cost", "latencyMs"],
|
|
54
|
+
format: "table",
|
|
55
|
+
});
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Use `measures: "*"` to include all declared measures.
|
|
59
|
+
|
|
60
|
+
### Ordering
|
|
61
|
+
|
|
62
|
+
`orderBy` can target measure aggregations (for example sort by `latencyMs` max).
|
|
63
|
+
|
|
64
|
+
### Cache vs durable
|
|
65
|
+
|
|
66
|
+
Redis keeps **sum** and **count** for measures. **min** / **max** live on durable; the query planner skips cache when min/max are required.
|
|
67
|
+
|
|
68
|
+
## See also
|
|
69
|
+
|
|
70
|
+
- [Querying](./querying.md)
|
|
71
|
+
- Internal: [Measures & scales](../internals/internal-measures-and-scales.md)
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
# Guide: Querying
|
|
2
|
+
|
|
3
|
+
`countex.query(spec)` reads aggregated cells for a scope and counter, with optional filters and output shaping.
|
|
4
|
+
|
|
5
|
+
## Step 1: Minimal query
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const result = await countex.query({
|
|
9
|
+
scope: "org_1",
|
|
10
|
+
counterId: "access_by_actor_label",
|
|
11
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
12
|
+
});
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
## Step 2: Filter
|
|
16
|
+
|
|
17
|
+
```ts
|
|
18
|
+
const result = await countex.query({
|
|
19
|
+
scope: "org_1",
|
|
20
|
+
counterId: "access_by_actor_label",
|
|
21
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
22
|
+
subject: { type: "actor", ids: [10001, 10002] },
|
|
23
|
+
metrics: [41, 42],
|
|
24
|
+
dimensions: { operation: ["read", "write"] },
|
|
25
|
+
});
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Subject and metric ids are **numeric dictionary ids** resolved during ingest.
|
|
29
|
+
|
|
30
|
+
## Step 3: Group and format
|
|
31
|
+
|
|
32
|
+
`groupBy` accepts:
|
|
33
|
+
|
|
34
|
+
- `subject`, `subjectType`, `metric`, `timeframeKey`
|
|
35
|
+
- `dim:NAME` for a dimension
|
|
36
|
+
|
|
37
|
+
`format`:
|
|
38
|
+
|
|
39
|
+
- `cells` — raw-ish cell list
|
|
40
|
+
- `table` / `matrix` — aggregated rows
|
|
41
|
+
- `timeseries` — points per bucket (and optional extra grouping)
|
|
42
|
+
|
|
43
|
+
```ts
|
|
44
|
+
const result = await countex.query({
|
|
45
|
+
scope: "org_1",
|
|
46
|
+
counterId: "access_by_actor_label",
|
|
47
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
48
|
+
groupBy: ["subject", "dim:operation"],
|
|
49
|
+
format: "table",
|
|
50
|
+
orderBy: "-count",
|
|
51
|
+
limit: 50,
|
|
52
|
+
});
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## Step 4: Measures and rates
|
|
56
|
+
|
|
57
|
+
```ts
|
|
58
|
+
await countex.query({
|
|
59
|
+
// ...
|
|
60
|
+
measures: ["cost"],
|
|
61
|
+
orderBy: { measure: "cost", agg: "sum", dir: "desc" },
|
|
62
|
+
rate: { measure: "cost", agg: "sum" },
|
|
63
|
+
});
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Do not use `rate` with timeframe **`all`**.
|
|
67
|
+
|
|
68
|
+
## Step 5: Cache control
|
|
69
|
+
|
|
70
|
+
- `bypassCache: true` forces durable reads.
|
|
71
|
+
- `signal` accepts `AbortSignal` for timeouts.
|
|
72
|
+
|
|
73
|
+
Result includes `source`: `"cache"`, `"durable"`, or `"merged"`.
|
|
74
|
+
|
|
75
|
+
### Query diagnostics
|
|
76
|
+
|
|
77
|
+
`result.diagnostics` (when present) adds planner detail:
|
|
78
|
+
|
|
79
|
+
| Field | Meaning |
|
|
80
|
+
|-------|---------|
|
|
81
|
+
| `cacheAttempted` | `true` if the engine consulted the cache adapter (cache configured, not `bypassCache`, and the planner allowed cache for this query shape). |
|
|
82
|
+
| `cacheCoverage` | When `cacheAttempted` is true: `"full"` if the cache answered completely; `"partial"` or `"miss"` if the planner fell through to durable for authoritative cells. |
|
|
83
|
+
| `cellsMaterialized` | Number of counter cells loaded from storage before grouping/formatting. |
|
|
84
|
+
|
|
85
|
+
Use this for tests, dashboards, and verifying Redis vs Mongo paths.
|
|
86
|
+
|
|
87
|
+
## Step 6: Totals, event volume, and “distinct entities”
|
|
88
|
+
|
|
89
|
+
### Grand total (sum all matching cells)
|
|
90
|
+
|
|
91
|
+
Omit **`groupBy`** (or use an empty list). For **`format: "table"`**, the engine merges every matching cell into **one row**; **`rows[0].count`** is the sum of **`count`** across those cells. **`result.total`** matches that sum for the returned rows (`sumRowCounts` in `src/query/formatter.ts`).
|
|
92
|
+
|
|
93
|
+
Use this for **total increment volume** in the query window. If one source event fans out into many cells (multiple subjects × metrics × timeframes), this total is **not** the same as “number of source events.”
|
|
94
|
+
|
|
95
|
+
### Distinct subjects / metrics as row cardinality
|
|
96
|
+
|
|
97
|
+
Countex does **not** store arbitrary **`COUNT(DISTINCT …)`** queries. **Exactly what to do** when you need “how many unique actors had activity?”:
|
|
98
|
+
|
|
99
|
+
1. **`subject: { type: "actor" }`** (optional **`ids`**) to restrict the subject family.
|
|
100
|
+
2. **`dimensions: { situation: ["external"] }`** (example) to restrict literals stored on cells.
|
|
101
|
+
3. **`groupBy: ["subjectType", "subject"]`** so **each row aggregates one dictionary subject id**.
|
|
102
|
+
4. **`rows.length`** after the query = number of distinct actors **with non-zero merged activity** in that slice (sparse — actors who never appeared produce no row).
|
|
103
|
+
|
|
104
|
+
**Critical:** if you set **`limit`**, **`rows.length ≤ limit`** and **`result.total`** only sum **returned** rows — you **cannot** infer global distinct counts from a capped query.
|
|
105
|
+
|
|
106
|
+
### Logical “one row per source event” totals
|
|
107
|
+
|
|
108
|
+
Maintain a **separate counter map** engineered for **one increment per deduped message** (single timeframe granularity, single synthetic metric field, minimal subjects). See [`docs/use-cases/audit-security-log-counts.md`](../use-cases/audit-security-log-counts.md) §10.6.
|
|
109
|
+
|
|
110
|
+
### Metric filters at rest
|
|
111
|
+
|
|
112
|
+
`MongoDurable` / `MemoryDurable` apply **`metrics`** filters using **numeric metric ids** parsed from `QuerySpec.metrics`. Pure string labels that are not finite numbers do **not** constrain the database filter — resolve ids from `countex_dict` or filter after read.
|
|
113
|
+
|
|
114
|
+
## See also
|
|
115
|
+
|
|
116
|
+
- [Measures](./measures.md)
|
|
117
|
+
- [Timeframes](./timeframes-and-bucketing.md)
|
|
118
|
+
- [Audit use case — analytics recipes](../use-cases/audit-security-log-counts.md) §10
|
|
119
|
+
- Internal: [Query path](../internals/internal-query-path.md)
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Guide: Simulation profiles
|
|
2
|
+
|
|
3
|
+
Profiles configure **what kind of events** the simulator emits and **how much fan-out** they produce.
|
|
4
|
+
|
|
5
|
+
## Built-in profile ids
|
|
6
|
+
|
|
7
|
+
| Profile id | Intent |
|
|
8
|
+
|------------|--------|
|
|
9
|
+
| `audit-log` | High-volume audit-style records: actors, resources, operations, optional label fan-out. |
|
|
10
|
+
| `product-activity` | Product analytics–style events: account scope, user subject, named event metric, feature/plan dimensions. |
|
|
11
|
+
| `processing-activity` | Jobs / pipelines / workflows: job id subject, task/step metrics, status dimensions, duration measures. |
|
|
12
|
+
| `llm-usage` | Token and cost measures with model/provider dimensions. |
|
|
13
|
+
| `custom` | Supply your own event factory; simulator handles scheduling and pools only. |
|
|
14
|
+
|
|
15
|
+
## Profile options (conceptual)
|
|
16
|
+
|
|
17
|
+
- **`durationMs`** — simulated wall-clock span of the run (logical time advancement between events).
|
|
18
|
+
- **`eventsPerTick`** — throughput knob for batch generators.
|
|
19
|
+
- **`poolSizes`** — caps on distinct subjects/metrics/dimension values (cardinality stress).
|
|
20
|
+
- **`weights`** — skew toward hot keys (Pareto-style) vs uniform draws.
|
|
21
|
+
|
|
22
|
+
Exact fields match `SimulationProfile` in `@x12i/countex/simulation` TypeScript exports.
|
|
23
|
+
|
|
24
|
+
## Choosing a profile
|
|
25
|
+
|
|
26
|
+
- Mirror the **same map** you plan for production; mismatched profiles hide JSONPath bugs.
|
|
27
|
+
- Start **uniform** pools, then add **skew** to reproduce hotspot behavior.
|
|
28
|
+
- Combine with **`validateMap(map, samples)`** from core so paths resolve on representative payloads.
|
|
29
|
+
|
|
30
|
+
## See also
|
|
31
|
+
|
|
32
|
+
- [Simulation overview](./simulation.md)
|
|
33
|
+
- [Simulation resource estimation](./simulation-resource-estimation.md)
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Guide: Simulation resource estimation
|
|
2
|
+
|
|
3
|
+
Use simulation outputs to reason about **Mongo**, **Redis**, and **process memory** before cutover.
|
|
4
|
+
|
|
5
|
+
## Signals from a run
|
|
6
|
+
|
|
7
|
+
After a simulator run (or CLI report):
|
|
8
|
+
|
|
9
|
+
- **Raw events** and **raw increments** (before cardinality drops) — multiply by your peak events/sec for ingress sizing.
|
|
10
|
+
- **Distinct dictionary keys** per kind (`subject`, `metric`, `dimensionValue`) — drives Mongo dictionary collection growth.
|
|
11
|
+
- **Distinct cells touched** — upper bound on sparse matrix cardinality for the window.
|
|
12
|
+
- **Flush batch sizes** — depends on buffer settings (`maxBufferedCells`, flush interval).
|
|
13
|
+
|
|
14
|
+
## Mapping to infrastructure
|
|
15
|
+
|
|
16
|
+
| Concern | What to measure |
|
|
17
|
+
|---------|-----------------|
|
|
18
|
+
| Mongo durable | Cell documents + dictionary docs; index sizes (see deployment guide). |
|
|
19
|
+
| Redis | Working set for hot timeframes; TTL/retention policy per timeframe type. |
|
|
20
|
+
| Application RAM | In-flight buffer + Node overhead; simulate worst-case burst with `onOverflow: "block"`. |
|
|
21
|
+
|
|
22
|
+
## Limits
|
|
23
|
+
|
|
24
|
+
Simulation **does not** replace load testing of your MongoDB cluster or Redis failover behavior; it estimates **logical** counter volume from maps and profiles.
|
|
25
|
+
|
|
26
|
+
## See also
|
|
27
|
+
|
|
28
|
+
- [Simulation overview](./simulation.md)
|
|
29
|
+
- [Ingestion & flush](./ingestion-and-flush.md)
|
|
30
|
+
- [`docs/DEPLOYMENT.md`](../DEPLOYMENT.md)
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Guide: Simulation overview
|
|
2
|
+
|
|
3
|
+
**Simulation** helps before production traffic exists: generate deterministic synthetic events, stress-test maps and cardinality, estimate storage and flush load, and validate that aggregates match expectations.
|
|
4
|
+
|
|
5
|
+
## Runtime entry point
|
|
6
|
+
|
|
7
|
+
When using the published package:
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
import { createCountexSimulator } from "@x12i/countex/simulation";
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The simulator is optional tooling on top of the core engine; it does not change ingest or query semantics.
|
|
14
|
+
|
|
15
|
+
## What simulation covers
|
|
16
|
+
|
|
17
|
+
- **Seeded randomness** — reproducible runs for CI and regression tests.
|
|
18
|
+
- **Pools & relations** — weighted entities (users, documents, models) with stable ids across events.
|
|
19
|
+
- **Profiles** — named scenarios (audit-style logs, product activity, LLM metering, processing pipelines); see [Simulation profiles](./simulation-profiles.md).
|
|
20
|
+
- **Expected aggregates** — optional tracker that compares simulated totals to counter reads for a window.
|
|
21
|
+
- **Reports** — structured summary (events emitted, fan-out estimates, errors) for sizing discussions.
|
|
22
|
+
|
|
23
|
+
## Relationship to core Countex
|
|
24
|
+
|
|
25
|
+
1. Build or load a `CounterMap` (same as production).
|
|
26
|
+
2. Instantiate `createCountexSimulator({ map, profile, seed, ... })`.
|
|
27
|
+
3. Drive **`generateEvents(n)`** or **`run()`** into `countex.ingest` / `ingestBatch` in tests or benchmarks.
|
|
28
|
+
4. Optionally attach **`ExpectedAggregateTracker`** and assert after flush + query.
|
|
29
|
+
|
|
30
|
+
See [Simulation resource estimation](./simulation-resource-estimation.md) for capacity hooks.
|
|
31
|
+
|
|
32
|
+
## See also
|
|
33
|
+
|
|
34
|
+
- [CLI](./cli.md) — `countex simulate:demo` for scripted dry-runs without a config module
|
|
35
|
+
- [Counter maps](./counter-maps.md)
|
|
36
|
+
- [Cardinality](./cardinality.md)
|