@x12i/countex 1.0.0 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +215 -4
- package/dist/backup/export.d.ts +2 -0
- package/dist/backup/export.d.ts.map +1 -1
- package/dist/backup/export.js +2 -0
- package/dist/backup/export.js.map +1 -1
- package/dist/backup/manifest.d.ts +49 -0
- package/dist/backup/manifest.d.ts.map +1 -0
- package/dist/backup/manifest.js +103 -0
- package/dist/backup/manifest.js.map +1 -0
- package/dist/backup/snapshot-encoding.d.ts +8 -0
- package/dist/backup/snapshot-encoding.d.ts.map +1 -0
- package/dist/backup/snapshot-encoding.js +27 -0
- package/dist/backup/snapshot-encoding.js.map +1 -0
- package/dist/cli.d.ts +12 -0
- package/dist/cli.d.ts.map +1 -0
- package/dist/cli.js +206 -0
- package/dist/cli.js.map +1 -0
- package/dist/core/buffer.d.ts +4 -3
- package/dist/core/buffer.d.ts.map +1 -1
- package/dist/core/buffer.js +65 -9
- package/dist/core/buffer.js.map +1 -1
- package/dist/core/countex.d.ts +14 -1
- package/dist/core/countex.d.ts.map +1 -1
- package/dist/core/countex.js +171 -48
- package/dist/core/countex.js.map +1 -1
- package/dist/core/extraction.d.ts +14 -6
- package/dist/core/extraction.d.ts.map +1 -1
- package/dist/core/extraction.js +117 -1
- package/dist/core/extraction.js.map +1 -1
- package/dist/core/mapping.d.ts +5 -0
- package/dist/core/mapping.d.ts.map +1 -1
- package/dist/core/mapping.js +71 -1
- package/dist/core/mapping.js.map +1 -1
- package/dist/core/timeframe.d.ts +7 -0
- package/dist/core/timeframe.d.ts.map +1 -1
- package/dist/core/timeframe.js +88 -0
- package/dist/core/timeframe.js.map +1 -1
- package/dist/index.d.ts +4 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +13 -1
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/fanout-health.d.ts +3 -0
- package/dist/insights/advisors/fanout-health.d.ts.map +1 -0
- package/dist/insights/advisors/fanout-health.js +3 -0
- package/dist/insights/advisors/fanout-health.js.map +1 -0
- package/dist/insights/advisors/map-health.d.ts +6 -0
- package/dist/insights/advisors/map-health.d.ts.map +1 -0
- package/dist/insights/advisors/map-health.js +154 -0
- package/dist/insights/advisors/map-health.js.map +1 -0
- package/dist/insights/advisors/resource-advisor.d.ts +4 -0
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -0
- package/dist/insights/advisors/resource-advisor.js +102 -0
- package/dist/insights/advisors/resource-advisor.js.map +1 -0
- package/dist/insights/advisors/retention-advisor.d.ts +2 -0
- package/dist/insights/advisors/retention-advisor.d.ts.map +1 -0
- package/dist/insights/advisors/retention-advisor.js +6 -0
- package/dist/insights/advisors/retention-advisor.js.map +1 -0
- package/dist/insights/advisors-subpath.d.ts +4 -0
- package/dist/insights/advisors-subpath.d.ts.map +1 -0
- package/dist/insights/advisors-subpath.js +10 -0
- package/dist/insights/advisors-subpath.js.map +1 -0
- package/dist/insights/anomaly/baseline.d.ts +4 -0
- package/dist/insights/anomaly/baseline.d.ts.map +1 -0
- package/dist/insights/anomaly/baseline.js +54 -0
- package/dist/insights/anomaly/baseline.js.map +1 -0
- package/dist/insights/anomaly/detect-anomalies.d.ts +6 -0
- package/dist/insights/anomaly/detect-anomalies.d.ts.map +1 -0
- package/dist/insights/anomaly/detect-anomalies.js +245 -0
- package/dist/insights/anomaly/detect-anomalies.js.map +1 -0
- package/dist/insights/anomaly/ewma.d.ts +2 -0
- package/dist/insights/anomaly/ewma.d.ts.map +1 -0
- package/dist/insights/anomaly/ewma.js +15 -0
- package/dist/insights/anomaly/ewma.js.map +1 -0
- package/dist/insights/anomaly/missing-activity.d.ts +3 -0
- package/dist/insights/anomaly/missing-activity.d.ts.map +1 -0
- package/dist/insights/anomaly/missing-activity.js +8 -0
- package/dist/insights/anomaly/missing-activity.js.map +1 -0
- package/dist/insights/anomaly/robust-zscore.d.ts +2 -0
- package/dist/insights/anomaly/robust-zscore.d.ts.map +1 -0
- package/dist/insights/anomaly/robust-zscore.js +9 -0
- package/dist/insights/anomaly/robust-zscore.js.map +1 -0
- package/dist/insights/anomaly-subpath.d.ts +7 -0
- package/dist/insights/anomaly-subpath.d.ts.map +1 -0
- package/dist/insights/anomaly-subpath.js +20 -0
- package/dist/insights/anomaly-subpath.js.map +1 -0
- package/dist/insights/comparison/compare-periods.d.ts +5 -0
- package/dist/insights/comparison/compare-periods.d.ts.map +1 -0
- package/dist/insights/comparison/compare-periods.js +64 -0
- package/dist/insights/comparison/compare-periods.js.map +1 -0
- package/dist/insights/comparison/growth-rate.d.ts +3 -0
- package/dist/insights/comparison/growth-rate.d.ts.map +1 -0
- package/dist/insights/comparison/growth-rate.js +3 -0
- package/dist/insights/comparison/growth-rate.js.map +1 -0
- package/dist/insights/comparison/moving-average.d.ts +2 -0
- package/dist/insights/comparison/moving-average.d.ts.map +1 -0
- package/dist/insights/comparison/moving-average.js +6 -0
- package/dist/insights/comparison/moving-average.js.map +1 -0
- package/dist/insights/comparison/trend.d.ts +5 -0
- package/dist/insights/comparison/trend.d.ts.map +1 -0
- package/dist/insights/comparison/trend.js +84 -0
- package/dist/insights/comparison/trend.js.map +1 -0
- package/dist/insights/errors.d.ts +25 -0
- package/dist/insights/errors.d.ts.map +1 -0
- package/dist/insights/errors.js +54 -0
- package/dist/insights/errors.js.map +1 -0
- package/dist/insights/forecast/budget.d.ts +2 -0
- package/dist/insights/forecast/budget.d.ts.map +1 -0
- package/dist/insights/forecast/budget.js +6 -0
- package/dist/insights/forecast/budget.js.map +1 -0
- package/dist/insights/forecast/capacity.d.ts +2 -0
- package/dist/insights/forecast/capacity.d.ts.map +1 -0
- package/dist/insights/forecast/capacity.js +3 -0
- package/dist/insights/forecast/capacity.js.map +1 -0
- package/dist/insights/forecast/forecast.d.ts +5 -0
- package/dist/insights/forecast/forecast.d.ts.map +1 -0
- package/dist/insights/forecast/forecast.js +103 -0
- package/dist/insights/forecast/forecast.js.map +1 -0
- package/dist/insights/forecast/retention.d.ts +2 -0
- package/dist/insights/forecast/retention.d.ts.map +1 -0
- package/dist/insights/forecast/retention.js +3 -0
- package/dist/insights/forecast/retention.js.map +1 -0
- package/dist/insights/index.d.ts +4 -0
- package/dist/insights/index.d.ts.map +1 -0
- package/dist/insights/index.js +23 -0
- package/dist/insights/index.js.map +1 -0
- package/dist/insights/insights-client.d.ts +33 -0
- package/dist/insights/insights-client.d.ts.map +1 -0
- package/dist/insights/insights-client.js +344 -0
- package/dist/insights/insights-client.js.map +1 -0
- package/dist/insights/llm-subpath.d.ts +4 -0
- package/dist/insights/llm-subpath.d.ts.map +1 -0
- package/dist/insights/llm-subpath.js +10 -0
- package/dist/insights/llm-subpath.js.map +1 -0
- package/dist/insights/optimization/cost-optimizer.d.ts +6 -0
- package/dist/insights/optimization/cost-optimizer.d.ts.map +1 -0
- package/dist/insights/optimization/cost-optimizer.js +98 -0
- package/dist/insights/optimization/cost-optimizer.js.map +1 -0
- package/dist/insights/optimization/latency-optimizer.d.ts +3 -0
- package/dist/insights/optimization/latency-optimizer.d.ts.map +1 -0
- package/dist/insights/optimization/latency-optimizer.js +3 -0
- package/dist/insights/optimization/latency-optimizer.js.map +1 -0
- package/dist/insights/optimization/threshold-advisor.d.ts +2 -0
- package/dist/insights/optimization/threshold-advisor.d.ts.map +1 -0
- package/dist/insights/optimization/threshold-advisor.js +3 -0
- package/dist/insights/optimization/threshold-advisor.js.map +1 -0
- package/dist/insights/optimization/token-optimizer.d.ts +4 -0
- package/dist/insights/optimization/token-optimizer.d.ts.map +1 -0
- package/dist/insights/optimization/token-optimizer.js +133 -0
- package/dist/insights/optimization/token-optimizer.js.map +1 -0
- package/dist/insights/ranking/contribution.d.ts +2 -0
- package/dist/insights/ranking/contribution.d.ts.map +1 -0
- package/dist/insights/ranking/contribution.js +6 -0
- package/dist/insights/ranking/contribution.js.map +1 -0
- package/dist/insights/ranking/helpers.d.ts +22 -0
- package/dist/insights/ranking/helpers.d.ts.map +1 -0
- package/dist/insights/ranking/helpers.js +88 -0
- package/dist/insights/ranking/helpers.js.map +1 -0
- package/dist/insights/ranking/pareto.d.ts +2 -0
- package/dist/insights/ranking/pareto.d.ts.map +1 -0
- package/dist/insights/ranking/pareto.js +6 -0
- package/dist/insights/ranking/pareto.js.map +1 -0
- package/dist/insights/ranking/slices.d.ts +2 -0
- package/dist/insights/ranking/slices.d.ts.map +1 -0
- package/dist/insights/ranking/slices.js +6 -0
- package/dist/insights/ranking/slices.js.map +1 -0
- package/dist/insights/ranking/top-bottom.d.ts +2 -0
- package/dist/insights/ranking/top-bottom.d.ts.map +1 -0
- package/dist/insights/ranking/top-bottom.js +7 -0
- package/dist/insights/ranking/top-bottom.js.map +1 -0
- package/dist/insights/recommendations/evidence.d.ts +2 -0
- package/dist/insights/recommendations/evidence.d.ts.map +1 -0
- package/dist/insights/recommendations/evidence.js +3 -0
- package/dist/insights/recommendations/evidence.js.map +1 -0
- package/dist/insights/recommendations/recommendation.d.ts +6 -0
- package/dist/insights/recommendations/recommendation.d.ts.map +1 -0
- package/dist/insights/recommendations/recommendation.js +25 -0
- package/dist/insights/recommendations/recommendation.js.map +1 -0
- package/dist/insights/recommendations/severity.d.ts +2 -0
- package/dist/insights/recommendations/severity.d.ts.map +1 -0
- package/dist/insights/recommendations/severity.js +3 -0
- package/dist/insights/recommendations/severity.js.map +1 -0
- package/dist/insights/rules/evaluate-rules.d.ts +5 -0
- package/dist/insights/rules/evaluate-rules.d.ts.map +1 -0
- package/dist/insights/rules/evaluate-rules.js +156 -0
- package/dist/insights/rules/evaluate-rules.js.map +1 -0
- package/dist/insights/types.d.ts +405 -0
- package/dist/insights/types.d.ts.map +1 -0
- package/dist/insights/types.js +3 -0
- package/dist/insights/types.js.map +1 -0
- package/dist/insights/utils/formatting.d.ts +4 -0
- package/dist/insights/utils/formatting.d.ts.map +1 -0
- package/dist/insights/utils/formatting.js +21 -0
- package/dist/insights/utils/formatting.js.map +1 -0
- package/dist/insights/utils/grouping.d.ts +2 -0
- package/dist/insights/utils/grouping.d.ts.map +1 -0
- package/dist/insights/utils/grouping.js +8 -0
- package/dist/insights/utils/grouping.js.map +1 -0
- package/dist/insights/utils/query-builder.d.ts +21 -0
- package/dist/insights/utils/query-builder.d.ts.map +1 -0
- package/dist/insights/utils/query-builder.js +107 -0
- package/dist/insights/utils/query-builder.js.map +1 -0
- package/dist/insights/utils/row-value.d.ts +5 -0
- package/dist/insights/utils/row-value.d.ts.map +1 -0
- package/dist/insights/utils/row-value.js +36 -0
- package/dist/insights/utils/row-value.js.map +1 -0
- package/dist/insights/utils/stats.d.ts +28 -0
- package/dist/insights/utils/stats.d.ts.map +1 -0
- package/dist/insights/utils/stats.js +125 -0
- package/dist/insights/utils/stats.js.map +1 -0
- package/dist/query/formatter.d.ts +2 -2
- package/dist/query/formatter.d.ts.map +1 -1
- package/dist/query/formatter.js +211 -30
- package/dist/query/formatter.js.map +1 -1
- package/dist/query/planner.d.ts +10 -2
- package/dist/query/planner.d.ts.map +1 -1
- package/dist/query/planner.js +55 -4
- package/dist/query/planner.js.map +1 -1
- package/dist/simulation/context.d.ts +5 -0
- package/dist/simulation/context.d.ts.map +1 -0
- package/dist/simulation/context.js +56 -0
- package/dist/simulation/context.js.map +1 -0
- package/dist/simulation/events.d.ts +3 -0
- package/dist/simulation/events.d.ts.map +1 -0
- package/dist/simulation/events.js +67 -0
- package/dist/simulation/events.js.map +1 -0
- package/dist/simulation/index.d.ts +7 -0
- package/dist/simulation/index.d.ts.map +1 -0
- package/dist/simulation/index.js +16 -0
- package/dist/simulation/index.js.map +1 -0
- package/dist/simulation/prng.d.ts +4 -0
- package/dist/simulation/prng.d.ts.map +1 -0
- package/dist/simulation/prng.js +18 -0
- package/dist/simulation/prng.js.map +1 -0
- package/dist/simulation/resource-estimate.d.ts +6 -0
- package/dist/simulation/resource-estimate.d.ts.map +1 -0
- package/dist/simulation/resource-estimate.js +11 -0
- package/dist/simulation/resource-estimate.js.map +1 -0
- package/dist/simulation/simulator.d.ts +15 -0
- package/dist/simulation/simulator.d.ts.map +1 -0
- package/dist/simulation/simulator.js +67 -0
- package/dist/simulation/simulator.js.map +1 -0
- package/dist/simulation/tracker.d.ts +8 -0
- package/dist/simulation/tracker.d.ts.map +1 -0
- package/dist/simulation/tracker.js +18 -0
- package/dist/simulation/tracker.js.map +1 -0
- package/dist/simulation/types.d.ts +49 -0
- package/dist/simulation/types.d.ts.map +1 -0
- package/dist/simulation/types.js +21 -0
- package/dist/simulation/types.js.map +1 -0
- package/dist/storage/memory.d.ts +3 -0
- package/dist/storage/memory.d.ts.map +1 -1
- package/dist/storage/memory.js +113 -4
- package/dist/storage/memory.js.map +1 -1
- package/dist/storage/mongo.d.ts +5 -0
- package/dist/storage/mongo.d.ts.map +1 -1
- package/dist/storage/mongo.js +129 -17
- package/dist/storage/mongo.js.map +1 -1
- package/dist/storage/redis.d.ts +12 -9
- package/dist/storage/redis.d.ts.map +1 -1
- package/dist/storage/redis.js +126 -47
- package/dist/storage/redis.js.map +1 -1
- package/dist/types/index.d.ts +139 -4
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +40 -2
- package/docs/INSIGHTS.md +59 -0
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -0
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -0
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +31 -0
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -0
- package/docs/guides/README.md +34 -0
- package/docs/guides/adapter-memory.md +48 -0
- package/docs/guides/adapter-mongo.md +62 -0
- package/docs/guides/adapter-redis.md +53 -0
- package/docs/guides/backup-export.md +83 -0
- package/docs/guides/cardinality.md +50 -0
- package/docs/guides/cli.md +53 -0
- package/docs/guides/client-events.md +49 -0
- package/docs/guides/counter-maps.md +122 -0
- package/docs/guides/idempotency.md +39 -0
- package/docs/guides/ingestion-and-flush.md +67 -0
- package/docs/guides/insights-advisors-and-optimization.md +39 -0
- package/docs/guides/insights-anomalies-and-rules.md +48 -0
- package/docs/guides/insights-comparison-and-trends.md +47 -0
- package/docs/guides/insights-forecasting-and-budget.md +61 -0
- package/docs/guides/insights-overview.md +82 -0
- package/docs/guides/insights-ranking.md +81 -0
- package/docs/guides/measures.md +71 -0
- package/docs/guides/querying.md +119 -0
- package/docs/guides/simulation-profiles.md +33 -0
- package/docs/guides/simulation-resource-estimation.md +30 -0
- package/docs/guides/simulation.md +36 -0
- package/docs/guides/timeframes-and-bucketing.md +55 -0
- package/docs/internals/README.md +21 -0
- package/docs/internals/internal-architecture-overview.md +65 -0
- package/docs/internals/internal-backup-stream.md +23 -0
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -0
- package/docs/internals/internal-cardinality.md +30 -0
- package/docs/internals/internal-dimension-hash.md +34 -0
- package/docs/internals/internal-idempotency.md +36 -0
- package/docs/internals/internal-ingest-pipeline.md +52 -0
- package/docs/internals/internal-insights-layer.md +36 -0
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -0
- package/docs/internals/internal-measures-and-scales.md +35 -0
- package/docs/internals/internal-query-path.md +46 -0
- package/docs/internals/internal-storage-memory.md +13 -0
- package/docs/internals/internal-storage-mongo.md +24 -0
- package/docs/internals/internal-storage-redis.md +31 -0
- package/docs/internals/internal-timeframe-bucketing.md +33 -0
- package/docs/use-cases/README.md +56 -0
- package/docs/use-cases/audit-security-log-counts.md +550 -0
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -0
- package/docs/use-cases/multi-tenant-isolation.md +203 -0
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -0
- package/docs/use-cases/processing-workflow-activity.md +165 -0
- package/docs/use-cases/product-feature-usage.md +240 -0
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -0
- package/package.json +61 -2
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Guide: Timeframes and bucketing
|
|
2
|
+
|
|
3
|
+
Timeframes control **how event time is bucketed** when Countex writes cells. Queries later ask for a **range of bucket keys** (`from` / `to`).
|
|
4
|
+
|
|
5
|
+
## Step 1: Pick timeframe types on the map
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const map = {
|
|
9
|
+
counterId: "events",
|
|
10
|
+
// ...
|
|
11
|
+
timeframes: ["minute", "hour", "day", "week", "month", "quarter", "year", "all"],
|
|
12
|
+
} as const;
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Only listed timeframes receive writes for each matching event.
|
|
16
|
+
|
|
17
|
+
## Step 2: Understand bucket keys (UTC)
|
|
18
|
+
|
|
19
|
+
| Type | Example key |
|
|
20
|
+
|------|----------------|
|
|
21
|
+
| minute | `2026-05-08T12:30` |
|
|
22
|
+
| hour | `2026-05-08T12` |
|
|
23
|
+
| day | `2026-05-08` |
|
|
24
|
+
| week | `2026-W19` (ISO week) |
|
|
25
|
+
| month | `2026-05` |
|
|
26
|
+
| quarter | `2026-Q2` |
|
|
27
|
+
| year | `2026` |
|
|
28
|
+
| all | `all` |
|
|
29
|
+
|
|
30
|
+
Buckets are derived from **`eventTime`**, not ingestion time, so late data lands in the correct historical bucket.
|
|
31
|
+
|
|
32
|
+
## Step 3: Query a range
|
|
33
|
+
|
|
34
|
+
```ts
|
|
35
|
+
const result = await countex.query({
|
|
36
|
+
scope: "org_1",
|
|
37
|
+
counterId: "events",
|
|
38
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
|
|
39
|
+
format: "table",
|
|
40
|
+
});
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
## Step 4: Respect the bucket cap
|
|
44
|
+
|
|
45
|
+
Range expansion is capped at **10,000** buckets. Narrow the window or coarsen the timeframe if you hit limits.
|
|
46
|
+
|
|
47
|
+
## Step 5: Rates
|
|
48
|
+
|
|
49
|
+
`rate` in queries divides counts or measure sums by elapsed seconds in the queried window. **`rate` is invalid when timeframe type is `all`** (no meaningful duration).
|
|
50
|
+
|
|
51
|
+
## See also
|
|
52
|
+
|
|
53
|
+
- [Querying](./querying.md)
|
|
54
|
+
- [Measures](./measures.md)
|
|
55
|
+
- Internal: [Timeframe bucketing](../internals/internal-timeframe-bucketing.md)
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# Countex internals
|
|
2
|
+
|
|
3
|
+
Deep dives into how the engine executes ingest, flush, storage IO, and insights post-processing. These documents are **not** imported as package subpaths; they ship in the npm tarball under `docs/internals/` for repository readers and advanced operators.
|
|
4
|
+
|
|
5
|
+
Start here:
|
|
6
|
+
|
|
7
|
+
| Topic | Document |
|
|
8
|
+
|-------|----------|
|
|
9
|
+
| End-to-end architecture | [Architecture overview](./internal-architecture-overview.md) |
|
|
10
|
+
| Map startup validation & JSON extraction | [Mapping validation & extraction](./internal-mapping-validation-and-extraction.md) |
|
|
11
|
+
| `ingestBatch` phases | [Ingest pipeline](./internal-ingest-pipeline.md) |
|
|
12
|
+
| In-memory merge | [Buffer merge & flush](./internal-buffer-merge-and-flush.md) |
|
|
13
|
+
| Reads | [Query path](./internal-query-path.md) |
|
|
14
|
+
| Bucket keys | [Timeframe bucketing](./internal-timeframe-bucketing.md) |
|
|
15
|
+
| Dimension hashing | [Dimension canonicalization & hash](./internal-dimension-hash.md) |
|
|
16
|
+
| Measures | [Measures & scales](./internal-measures-and-scales.md) |
|
|
17
|
+
| Cardinality | [Cardinality guard](./internal-cardinality.md) |
|
|
18
|
+
| Dedupe | [Idempotency](./internal-idempotency.md) |
|
|
19
|
+
| Adapters | [Mongo](./internal-storage-mongo.md), [Redis](./internal-storage-redis.md), [Memory](./internal-storage-memory.md) |
|
|
20
|
+
| Ops helpers | [Backup streaming](./internal-backup-stream.md) |
|
|
21
|
+
| Insights | [Insights layer](./internal-insights-layer.md) |
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Internal: Architecture overview
|
|
2
|
+
|
|
3
|
+
Countex separates **projection** (events → sparse cells), **durability** (authoritative totals), **cache** (hot reads + dedupe), and **query shaping**.
|
|
4
|
+
|
|
5
|
+
## Component diagram
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
flowchart LR
|
|
9
|
+
subgraph app [Host application]
|
|
10
|
+
CE[Countex client]
|
|
11
|
+
end
|
|
12
|
+
subgraph core [Core engine]
|
|
13
|
+
MAP[CounterMap registry]
|
|
14
|
+
EXT[extractIncrements]
|
|
15
|
+
CARD[CardinalityGuard]
|
|
16
|
+
BUF[DeltaBuffer]
|
|
17
|
+
PLAN[executeQuery]
|
|
18
|
+
FMT[formatResult]
|
|
19
|
+
end
|
|
20
|
+
subgraph io [Storage]
|
|
21
|
+
DUR[DurableStorage]
|
|
22
|
+
CACHE[CacheStorage]
|
|
23
|
+
end
|
|
24
|
+
CE --> MAP
|
|
25
|
+
CE --> EXT
|
|
26
|
+
EXT --> CARD
|
|
27
|
+
CARD --> DUR
|
|
28
|
+
CARD --> BUF
|
|
29
|
+
BUF --> DUR
|
|
30
|
+
BUF --> CACHE
|
|
31
|
+
CE --> PLAN
|
|
32
|
+
PLAN --> CACHE
|
|
33
|
+
PLAN --> DUR
|
|
34
|
+
PLAN --> FMT
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## Write trace (conceptual)
|
|
38
|
+
|
|
39
|
+
1. **Validate maps** at `createCountex` construction (`validateMaps`).
|
|
40
|
+
2. **Extract** raw increments per map/event (`extractIncrements`).
|
|
41
|
+
3. **Dedupe** optional (`cache.markEventSeen`).
|
|
42
|
+
4. **Cardinality** rewrite or drop (`CardinalityGuard.check`).
|
|
43
|
+
5. **Dictionary** allocate numeric ids (`durable.resolveIds`).
|
|
44
|
+
6. **Buffer** merge by hashed cell identity (`DeltaBuffer.add`).
|
|
45
|
+
7. **Flush** snapshot → `durable.applyDeltas` → best-effort `cache.applyDeltas`.
|
|
46
|
+
|
|
47
|
+
## Read trace (conceptual)
|
|
48
|
+
|
|
49
|
+
1. **Planner** tries cache when allowed (`cacheCanServe`).
|
|
50
|
+
2. On partial/miss, **durable.readCells**.
|
|
51
|
+
3. Optional **cache.warm** with durable rows.
|
|
52
|
+
4. **Formatter** applies `groupBy`, measures, rates, limits.
|
|
53
|
+
|
|
54
|
+
## Source anchors
|
|
55
|
+
|
|
56
|
+
- Client orchestration: `src/core/countex.ts`
|
|
57
|
+
- Extraction: `src/core/extraction.ts`
|
|
58
|
+
- Buffer: `src/core/buffer.ts`
|
|
59
|
+
- Planner: `src/query/planner.ts`
|
|
60
|
+
- Formatter: `src/query/formatter.ts`
|
|
61
|
+
|
|
62
|
+
## See also
|
|
63
|
+
|
|
64
|
+
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
65
|
+
- [Query path](./internal-query-path.md)
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# Internal: Backup streaming
|
|
2
|
+
|
|
3
|
+
`exportCells` and `rebuildCache` (`src/backup/export.ts`) are thin wrappers over `durable.streamCells`:
|
|
4
|
+
|
|
5
|
+
```mermaid
|
|
6
|
+
flowchart LR
|
|
7
|
+
DUR[durable.streamCells filter] --> EXPORT[AsyncIterable CounterCell]
|
|
8
|
+
DUR --> BATCH[batch warm loop]
|
|
9
|
+
BATCH --> CACHE[cache.warm batches]
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
## Contract expectations
|
|
13
|
+
|
|
14
|
+
Implementations must honor:
|
|
15
|
+
|
|
16
|
+
- Scope/counter/timeframe bounding
|
|
17
|
+
- Optional `AbortSignal` propagation
|
|
18
|
+
|
|
19
|
+
`rebuildCache` batches to limit Redis/Mongo round trips during warm operations.
|
|
20
|
+
|
|
21
|
+
## See also
|
|
22
|
+
|
|
23
|
+
- [Mongo durable](./internal-storage-mongo.md)
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Internal: Buffer merge & flush
|
|
2
|
+
|
|
3
|
+
## DeltaBuffer structure
|
|
4
|
+
|
|
5
|
+
`DeltaBuffer` (`src/core/buffer.ts`) stores:
|
|
6
|
+
|
|
7
|
+
- Map key: string from `cellKey(identity)`
|
|
8
|
+
- Value: `{ identity, delta, measures?, measureScales? }`
|
|
9
|
+
|
|
10
|
+
## Merge semantics
|
|
11
|
+
|
|
12
|
+
`add` merges:
|
|
13
|
+
|
|
14
|
+
- **Counts** — integer delta summed.
|
|
15
|
+
- **Measures** — per aggregation field:
|
|
16
|
+
- `sum`, `count` additive
|
|
17
|
+
- `min` / `max` rolling extrema
|
|
18
|
+
|
|
19
|
+
`measureScales` merge retains first-seen scale per measure name (consistent with map declaration).
|
|
20
|
+
|
|
21
|
+
## Flush lifecycle
|
|
22
|
+
|
|
23
|
+
```mermaid
|
|
24
|
+
stateDiagram-v2
|
|
25
|
+
[*] --> Idle
|
|
26
|
+
Idle --> Draining: flush() & buffer non-empty
|
|
27
|
+
Draining --> DurableWrite: snapshot taken
|
|
28
|
+
DurableWrite --> CacheWrite: durable OK
|
|
29
|
+
DurableWrite --> MergeBack: durable error
|
|
30
|
+
MergeBack --> Idle
|
|
31
|
+
CacheWrite --> Idle: cache success or logged warn
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
`flush()` returns the **same promise** while in-flight (`flushing` lock) so concurrent callers coalesce.
|
|
35
|
+
|
|
36
|
+
### Durable failure
|
|
37
|
+
|
|
38
|
+
Snapshot is **`mergeBack`** into the buffer so counts are not lost; `flush_error` emits.
|
|
39
|
+
|
|
40
|
+
### Cache failure
|
|
41
|
+
|
|
42
|
+
Logged as non-fatal — durable already committed.
|
|
43
|
+
|
|
44
|
+
## Event accounting
|
|
45
|
+
|
|
46
|
+
`noteEvent()` increments an event counter used solely for flush thresholds (reset implicitly when draining — payload uses duration only).
|
|
47
|
+
|
|
48
|
+
## See also
|
|
49
|
+
|
|
50
|
+
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
51
|
+
- [Measures & scales](./internal-measures-and-scales.md)
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# Internal: Cardinality guard
|
|
2
|
+
|
|
3
|
+
`CardinalityGuard` (`src/core/cardinality.ts`) tracks distinct source keys per `(scopeId, kind, qualifier)` using in-memory `Set`s.
|
|
4
|
+
|
|
5
|
+
## Check algorithm
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
flowchart TD
|
|
9
|
+
IN[sourceKey] --> KEY[keyFor scope/kind/qualifier]
|
|
10
|
+
KEY --> SET{Set contains key?}
|
|
11
|
+
SET -->|yes| OK[return original or existing overflow mapping]
|
|
12
|
+
SET -->|no| LIM{count < limit?}
|
|
13
|
+
LIM -->|yes| ADD[add to Set -> return key]
|
|
14
|
+
LIM -->|no| POL{onLimit}
|
|
15
|
+
POL -->|overflow| OVF[emit overflow event -> return sentinel]
|
|
16
|
+
POL -->|drop| DROP[return null]
|
|
17
|
+
POL -->|error| ERR[throw CardinalityLimitError]
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Dictionary kinds
|
|
21
|
+
|
|
22
|
+
Kinds align with subject/metric/dimension value namespaces; dimension checks include the dimension **name** as qualifier.
|
|
23
|
+
|
|
24
|
+
## Approximation warning
|
|
25
|
+
|
|
26
|
+
Commentary in source notes multi-worker deployments see approximate counts — acceptable because the guard targets runaway growth, not strict quotas.
|
|
27
|
+
|
|
28
|
+
## See also
|
|
29
|
+
|
|
30
|
+
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Internal: Dimension canonicalization and hashing
|
|
2
|
+
|
|
3
|
+
Each counter cell identity includes a **`dimensionsHash`** derived from the dimension key/value object for that increment.
|
|
4
|
+
|
|
5
|
+
## Normalization
|
|
6
|
+
|
|
7
|
+
Before hashing, dimension values pass through `normalizeDimensionValue` in `src/core/normalization.ts`:
|
|
8
|
+
|
|
9
|
+
- `null` / missing optional dimensions → `null`.
|
|
10
|
+
- Booleans and finite numbers preserved as typed values.
|
|
11
|
+
- Strings: optionally trimmed only (`normalizeValues` on the map); types are **not** coerced (`"1"` and `1` stay distinct).
|
|
12
|
+
|
|
13
|
+
Dictionary keys for subjects/metrics use **`normalizeKey`** (trim + lowercase strings) — that is separate from dimension hashing.
|
|
14
|
+
|
|
15
|
+
## Canonical string form
|
|
16
|
+
|
|
17
|
+
`canonicalizeDimensions` in `src/utils/hash.ts`:
|
|
18
|
+
|
|
19
|
+
1. Sort dimension **names** lexicographically.
|
|
20
|
+
2. Serialize as `key=value` pairs joined with `&`.
|
|
21
|
+
3. Escape `\`, `=`, and `&` in keys and string values with backslash.
|
|
22
|
+
4. Encode values with type prefixes: `n:` (null), `b:0|1`, `i:<number>`, `s:<escaped>`.
|
|
23
|
+
|
|
24
|
+
## Hash
|
|
25
|
+
|
|
26
|
+
`dimensionsHash` is **SHA-1** of the canonical string, truncated to **12 hex characters** (48 bits). Empty dimensions → literal `"0"`.
|
|
27
|
+
|
|
28
|
+
Collisions are statistically negligible at intended cardinalities; cells are further scoped by `(scopeId, counterId, timeframe, subject, metric)`.
|
|
29
|
+
|
|
30
|
+
## See also
|
|
31
|
+
|
|
32
|
+
- `src/utils/hash.ts`
|
|
33
|
+
- `src/core/normalization.ts`
|
|
34
|
+
- [Cardinality](../guides/cardinality.md)
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Internal: Idempotency
|
|
2
|
+
|
|
3
|
+
Idempotency spans three layers:
|
|
4
|
+
|
|
5
|
+
1. **Map config** — `idempotency.enabled` + `eventIdPath`.
|
|
6
|
+
2. **Extraction** — resolves optional `eventId` string (`src/core/extraction.ts`).
|
|
7
|
+
3. **Ingest gate** — parallel `cache.markEventSeen` before buffering (`src/core/countex.ts`).
|
|
8
|
+
|
|
9
|
+
## Sequence
|
|
10
|
+
|
|
11
|
+
```mermaid
|
|
12
|
+
sequenceDiagram
|
|
13
|
+
participant CX as Countex
|
|
14
|
+
participant EXT as extractIncrements
|
|
15
|
+
participant CACHE as CacheStorage
|
|
16
|
+
participant BUF as DeltaBuffer
|
|
17
|
+
CX->>EXT: increments + eventId
|
|
18
|
+
CX->>CACHE: markEventSeen(scope, counterId:eventId, ttl)
|
|
19
|
+
CACHE-->>CX: isNew boolean
|
|
20
|
+
alt duplicate
|
|
21
|
+
CX->>CX: emit duplicate
|
|
22
|
+
CX->>CX: zero increments for this map slice
|
|
23
|
+
else new
|
|
24
|
+
CX->>BUF: add deltas
|
|
25
|
+
end
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Namespace **`counterId:`** prefix prevents collisions across maps sharing physical cache.
|
|
29
|
+
|
|
30
|
+
## Failure modes
|
|
31
|
+
|
|
32
|
+
Missing event ids produce warnings during extraction but do not automatically dedupe unless your logic skips ingest.
|
|
33
|
+
|
|
34
|
+
## See also
|
|
35
|
+
|
|
36
|
+
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# Internal: Ingest pipeline (`ingestBatch`)
|
|
2
|
+
|
|
3
|
+
`Countex.ingest` delegates to `ingestBatch` (`src/core/countex.ts`). The implementation is intentionally phased so CPU work stays synchronous while IO stays bounded.
|
|
4
|
+
|
|
5
|
+
## Sequence diagram
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
sequenceDiagram
|
|
9
|
+
participant App
|
|
10
|
+
participant CX as Countex
|
|
11
|
+
participant EXT as extractIncrements
|
|
12
|
+
participant CACHE as CacheStorage
|
|
13
|
+
participant CARD as CardinalityGuard
|
|
14
|
+
participant DUR as DurableStorage
|
|
15
|
+
participant BUF as DeltaBuffer
|
|
16
|
+
App->>CX: ingestBatch(events)
|
|
17
|
+
loop each event × map
|
|
18
|
+
CX->>EXT: extractIncrements(map, ev)
|
|
19
|
+
EXT-->>CX: RawIncrement[]
|
|
20
|
+
end
|
|
21
|
+
CX->>CACHE: markEventSeen (optional, parallel)
|
|
22
|
+
CACHE-->>CX: boolean flags
|
|
23
|
+
CX->>CARD: check subject/metric/dimensions
|
|
24
|
+
CARD-->>CX: rewritten keys or drop
|
|
25
|
+
CX->>DUR: resolveIds(lookups)
|
|
26
|
+
DUR-->>CX: Map lookup → numeric id
|
|
27
|
+
loop each RawIncrement × timeframe
|
|
28
|
+
CX->>BUF: add(identity, delta, measures)
|
|
29
|
+
end
|
|
30
|
+
CX->>CX: maybe flush() on thresholds
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## Phase notes
|
|
34
|
+
|
|
35
|
+
1. **Extract** — warnings for measures surface as `measure_invalid` events when meta is present.
|
|
36
|
+
2. **Dedupe** — `markEventSeen` receives `${counterId}:${eventId}` to isolate maps.
|
|
37
|
+
3. **Cardinality** — dimension iteration may drop with `dropped` reason `cardinality:dimension`.
|
|
38
|
+
4. **Dictionary** — only **subjects** and **metrics** resolve via `resolveIds`; dimension values remain literal on cells.
|
|
39
|
+
5. **Buffer key** — identity includes `dimensionsHash` (`src/utils/hash.ts`).
|
|
40
|
+
6. **Timeframe expansion** — `bucketKey(tf, eventTime)` invoked per configured timeframe.
|
|
41
|
+
|
|
42
|
+
## Flush coupling
|
|
43
|
+
|
|
44
|
+
Threshold triggers call `flush()`:
|
|
45
|
+
|
|
46
|
+
- `onOverflow: "block"` awaits flush (back-pressure).
|
|
47
|
+
- `"drop"` schedules flush without awaiting.
|
|
48
|
+
|
|
49
|
+
## See also
|
|
50
|
+
|
|
51
|
+
- [Buffer merge & flush](./internal-buffer-merge-and-flush.md)
|
|
52
|
+
- [Cardinality](./internal-cardinality.md)
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
# Internal: Insights layer
|
|
2
|
+
|
|
3
|
+
`CountexInsights` (`src/insights/insights-client.ts`) wraps an existing `Countex` instance.
|
|
4
|
+
|
|
5
|
+
## Pattern
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
flowchart LR
|
|
9
|
+
IN[Insight input] --> QB[baseToQuerySpec +\nparseMeasureSelector]
|
|
10
|
+
QB --> Q[countex.query]
|
|
11
|
+
Q --> POST[ranking / anomaly /\nforecast helpers]
|
|
12
|
+
POST --> OUT[typed result]
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Shared utilities:
|
|
16
|
+
|
|
17
|
+
- `utils/query-builder.ts` translates insight inputs → `QuerySpec`.
|
|
18
|
+
- Ranking helpers sort rows, compute totals, enforce `maxGroups`.
|
|
19
|
+
- Anomaly pipeline fetches **eval** + **baseline** timeseries concurrently (`detectAnomalies`).
|
|
20
|
+
- Forecast/budget/token modules emit recommendations and bubble events.
|
|
21
|
+
|
|
22
|
+
## Event bridge
|
|
23
|
+
|
|
24
|
+
Methods emit:
|
|
25
|
+
|
|
26
|
+
- `insight_query` — performance telemetry
|
|
27
|
+
- `anomaly`, `recommendation`, `insufficient_data` — operator-facing signals
|
|
28
|
+
|
|
29
|
+
## Resource advisor
|
|
30
|
+
|
|
31
|
+
`estimateResourcesFromActuals` bypasses normal query formatting and streams cells via `durable.streamCells`, requiring the durable reference injected at construction.
|
|
32
|
+
|
|
33
|
+
## See also
|
|
34
|
+
|
|
35
|
+
- [Query path](./internal-query-path.md)
|
|
36
|
+
- `docs/INSIGHTS.md` for product-level narrative
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Internal: Mapping validation & extraction
|
|
2
|
+
|
|
3
|
+
## Startup validation
|
|
4
|
+
|
|
5
|
+
`validateMaps` runs once inside the `Countex` constructor (`src/core/mapping.ts`):
|
|
6
|
+
|
|
7
|
+
- Ensures unique `counterId` values.
|
|
8
|
+
- Parses every configured JSONPath with `parsePath`.
|
|
9
|
+
- Verifies required structural rules (non-empty subjects/metrics/timeframes, valid timeframe enums, measure aggregation enums, idempotency consistency).
|
|
10
|
+
|
|
11
|
+
Failures throw **`MappingError`** immediately.
|
|
12
|
+
|
|
13
|
+
## Extraction pipeline
|
|
14
|
+
|
|
15
|
+
`extractIncrements(map, event)` (`src/core/extraction.ts`) executes in order:
|
|
16
|
+
|
|
17
|
+
```mermaid
|
|
18
|
+
flowchart TD
|
|
19
|
+
A[extract scope] -->|missing| SKIP[skipped event]
|
|
20
|
+
A --> B[extract eventTime]
|
|
21
|
+
B -->|invalid| SKIP
|
|
22
|
+
B --> C[optional eventId for idempotency]
|
|
23
|
+
C --> D[resolve delta]
|
|
24
|
+
D -->|zero| SKIP
|
|
25
|
+
D --> E[fan-out subjects]
|
|
26
|
+
E -->|none| SKIP
|
|
27
|
+
E --> F[fan-out metrics]
|
|
28
|
+
F -->|none| SKIP
|
|
29
|
+
F --> G[dimensions]
|
|
30
|
+
G -->|required missing| SKIP
|
|
31
|
+
G --> H[measures]
|
|
32
|
+
H -->|drop policy| SKIP
|
|
33
|
+
H --> I[cross product increments]
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
### Fan-out
|
|
37
|
+
|
|
38
|
+
Nested loops build `RawIncrement` rows:
|
|
39
|
+
|
|
40
|
+
```
|
|
41
|
+
for subject in subjectFanout:
|
|
42
|
+
for metricKey in metricFanout:
|
|
43
|
+
push RawIncrement(... timeframes: map.timeframes ...)
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Each increment carries **copies** of dimensions and measures for that Cartesian tuple.
|
|
47
|
+
|
|
48
|
+
### Measures
|
|
49
|
+
|
|
50
|
+
`extractMeasuresForEvent` coerces numbers, applies `scale` truncation rules, and maps declared aggregations into pending contributions via `rawMeasuresToPendingDelta` during buffering.
|
|
51
|
+
|
|
52
|
+
### JSON paths
|
|
53
|
+
|
|
54
|
+
`extract` (`src/utils/jsonpath.ts`) evaluates the supported subset: properties, indices, and `[*]` expansion.
|
|
55
|
+
|
|
56
|
+
## See also
|
|
57
|
+
|
|
58
|
+
- [Ingest pipeline](./internal-ingest-pipeline.md)
|
|
59
|
+
- [Measures & scales](./internal-measures-and-scales.md)
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# Internal: Measures & scales
|
|
2
|
+
|
|
3
|
+
## Effective aggregations
|
|
4
|
+
|
|
5
|
+
`effectiveMeasureAggregations` (`src/core/extraction.ts`) ensures **`count`** exists whenever **`sum`**, **`min`**, or **`max`** are present so averages remain computable later.
|
|
6
|
+
|
|
7
|
+
## Pending deltas
|
|
8
|
+
|
|
9
|
+
`rawMeasuresToPendingDelta` converts each extracted measure into:
|
|
10
|
+
|
|
11
|
+
- `sum` contribution
|
|
12
|
+
- `count` increment (`1` when enabled)
|
|
13
|
+
- `min` / `max` seeds
|
|
14
|
+
|
|
15
|
+
These mirror what storage adapters must merge atomically.
|
|
16
|
+
|
|
17
|
+
## Buffer merging
|
|
18
|
+
|
|
19
|
+
`mergeMeasureDelta` (`src/core/buffer.ts`) applies associative merges identical to eventual durable semantics.
|
|
20
|
+
|
|
21
|
+
## Scale metadata
|
|
22
|
+
|
|
23
|
+
Maps may declare `scale` per measure. During ingest, numeric extraction truncates after multiply; `measureScales` propagate into buffer entries so adapters interpret stored integers consistently.
|
|
24
|
+
|
|
25
|
+
## Query-time averages
|
|
26
|
+
|
|
27
|
+
Formatter / row builders derive **`avg = sum / count`** when both components exist.
|
|
28
|
+
|
|
29
|
+
## Cache limitation
|
|
30
|
+
|
|
31
|
+
Planner denies cache for queries needing min/max fidelity — see `queryNeedsMinMax` in `src/query/planner.ts`.
|
|
32
|
+
|
|
33
|
+
## See also
|
|
34
|
+
|
|
35
|
+
- [Query path](./internal-query-path.md)
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# Internal: Query path
|
|
2
|
+
|
|
3
|
+
`Countex.query` delegates to `executeQuery` (`src/query/planner.ts`) with optional measure metadata from `buildMeasureAggIndex(map)`.
|
|
4
|
+
|
|
5
|
+
## Decision flow
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
flowchart TD
|
|
9
|
+
Q[QuerySpec] --> C{cache configured\nand not bypassed\nand cacheCanServe?}
|
|
10
|
+
C -->|yes| R1[cache.readCells]
|
|
11
|
+
R1 -->|coverage full| F[formatResult source=cache]
|
|
12
|
+
R1 -->|partial/miss| D[durable.readCells]
|
|
13
|
+
C -->|no| D
|
|
14
|
+
D --> W{cache && rows && not bypass?}
|
|
15
|
+
W -->|yes| WARM[cache.warm best-effort]
|
|
16
|
+
W -->|no| F2[formatResult source=merged or durable]
|
|
17
|
+
WARM --> F2
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
## Cache eligibility (`cacheCanServe`)
|
|
21
|
+
|
|
22
|
+
When the counter map is known, queries that require **min/max** measure accuracy skip cache:
|
|
23
|
+
|
|
24
|
+
- `orderBy` targeting min/max aggregation.
|
|
25
|
+
- Explicit measure list referencing min/max-maintained measures.
|
|
26
|
+
- `measures: "*"` while any declared measure tracks min/max.
|
|
27
|
+
|
|
28
|
+
This avoids stale extrema because Redis stores sum/count only for measures.
|
|
29
|
+
|
|
30
|
+
## Formatter responsibilities (`formatResult`)
|
|
31
|
+
|
|
32
|
+
`src/query/formatter.ts`:
|
|
33
|
+
|
|
34
|
+
- Groups cells by `groupBy` keys.
|
|
35
|
+
- Builds `table` / `matrix` / `timeseries` projections.
|
|
36
|
+
- Applies **`rate`** using `bucketDurationSeconds` except timeframe `all`.
|
|
37
|
+
- Sorts and truncates via `orderBy` / `limit`.
|
|
38
|
+
|
|
39
|
+
## Bucket expansion
|
|
40
|
+
|
|
41
|
+
Range queries call `bucketRange` (`src/core/timeframe.ts`) capped at **10,000** buckets.
|
|
42
|
+
|
|
43
|
+
## See also
|
|
44
|
+
|
|
45
|
+
- [Timeframe bucketing](./internal-timeframe-bucketing.md)
|
|
46
|
+
- [Measures & scales](./internal-measures-and-scales.md)
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
# Internal: Memory adapters
|
|
2
|
+
|
|
3
|
+
`MemoryDurable` and `MemoryCache` (`src/storage/memory.ts`) mirror production contracts using `Map` structures:
|
|
4
|
+
|
|
5
|
+
- Dictionary resolution mimics monotonic ids.
|
|
6
|
+
- Cells keyed similarly to durable identity tuples.
|
|
7
|
+
- Cache implements simplified coverage semantics for tests.
|
|
8
|
+
|
|
9
|
+
Used heavily by unit tests (`tests/unit/*.test.ts`) because they require no external services.
|
|
10
|
+
|
|
11
|
+
## See also
|
|
12
|
+
|
|
13
|
+
- [Architecture overview](./internal-architecture-overview.md)
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Internal: Mongo durable adapter
|
|
2
|
+
|
|
3
|
+
`MongoDurable` (`src/storage/mongo.ts`) implements:
|
|
4
|
+
|
|
5
|
+
- `applyDeltas` — upserts with `$inc` style operations for counts and compatible measure fields.
|
|
6
|
+
- `readCells` — satisfies `QuerySpec` filters using compound indexes.
|
|
7
|
+
- `resolveIds` — transactional or sequential allocation of dictionary ids (`countex_dict`, `countex_seq`).
|
|
8
|
+
- `streamCells` — bounded iteration for backup/export.
|
|
9
|
+
|
|
10
|
+
## Write path interaction
|
|
11
|
+
|
|
12
|
+
Flush snapshots arrive as arrays of `PendingDelta`; Mongo applies them idempotently per cell identity tuple.
|
|
13
|
+
|
|
14
|
+
## Indexes
|
|
15
|
+
|
|
16
|
+
Created lazily on first write paths (see README schema excerpt): uniqueness on full identity, secondary indexes for timeframe/subject/metric scans.
|
|
17
|
+
|
|
18
|
+
## Dictionary vs dimensions
|
|
19
|
+
|
|
20
|
+
Subjects/metrics map to integers; dimensions remain embedded documents/hashes for cells — matching core engine assumptions.
|
|
21
|
+
|
|
22
|
+
## See also
|
|
23
|
+
|
|
24
|
+
- [Backup streaming](./internal-backup-stream.md)
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Internal: Redis cache adapter
|
|
2
|
+
|
|
3
|
+
`RedisCache` (`src/storage/redis.ts`) maps each `(scope, counterId, timeframeType, timeframeKey)` bucket to a **HASH**:
|
|
4
|
+
|
|
5
|
+
- Field encodes `subjectType|subjectId|metricId|dimensionsHash`
|
|
6
|
+
- Value stores encoded counts / measure sums suitable for fast `HGETALL`
|
|
7
|
+
|
|
8
|
+
Metadata hashes may store dimension JSON for reconstruction.
|
|
9
|
+
|
|
10
|
+
## Read coverage
|
|
11
|
+
|
|
12
|
+
`readCells` returns:
|
|
13
|
+
|
|
14
|
+
- `coverage: "full"` when all requested buckets present
|
|
15
|
+
- `"partial"` / `"miss"` otherwise — planner falls back to durable
|
|
16
|
+
|
|
17
|
+
## Dedupe keys
|
|
18
|
+
|
|
19
|
+
`markEventSeen` writes ephemeral keys `countex:{scope}:dedupe:{eventId}` with TTL.
|
|
20
|
+
|
|
21
|
+
## Warm path
|
|
22
|
+
|
|
23
|
+
`warm` hydrates hashes from `CounterCell[]` after durable reads.
|
|
24
|
+
|
|
25
|
+
## Error stance
|
|
26
|
+
|
|
27
|
+
Apply failures during flush are swallowed at Countex level after durable success — subsequent reads rebuild cache.
|
|
28
|
+
|
|
29
|
+
## See also
|
|
30
|
+
|
|
31
|
+
- [Query path](./internal-query-path.md)
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Internal: Timeframe bucketing
|
|
2
|
+
|
|
3
|
+
## Bucket derivation
|
|
4
|
+
|
|
5
|
+
`bucketKey(type, date)` (`src/core/timeframe.ts`) formats UTC bucket strings:
|
|
6
|
+
|
|
7
|
+
- Minute/hour include `T` separators.
|
|
8
|
+
- Week uses **ISO week** numbering (`formatIsoWeek`).
|
|
9
|
+
- Quarter uses `YYYY-Qn`.
|
|
10
|
+
- `all` collapses to `"all"`.
|
|
11
|
+
|
|
12
|
+
Ingest **never** uses processing clock — only parsed **`eventTime`** from the map.
|
|
13
|
+
|
|
14
|
+
## Range iteration
|
|
15
|
+
|
|
16
|
+
`bucketRange(from, to)` walks forward with `advance`, stopping at:
|
|
17
|
+
|
|
18
|
+
- End boundary
|
|
19
|
+
- **MAX = 10_000** produced keys
|
|
20
|
+
|
|
21
|
+
Downstream query code relies on this bound for safety.
|
|
22
|
+
|
|
23
|
+
## Rates
|
|
24
|
+
|
|
25
|
+
`bucketDurationSeconds` supplies nominal seconds per bucket for rate calculations. **`all` throws** by design — mirrored in public API constraints.
|
|
26
|
+
|
|
27
|
+
## Parsing
|
|
28
|
+
|
|
29
|
+
`parseBucket` reconstructs `Date` objects for comparisons when reshaping query outputs.
|
|
30
|
+
|
|
31
|
+
## See also
|
|
32
|
+
|
|
33
|
+
- [Query path](./internal-query-path.md)
|