@x12i/countex 1.0.0 → 1.5.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +215 -4
- package/dist/backup/export.d.ts +2 -0
- package/dist/backup/export.d.ts.map +1 -1
- package/dist/backup/export.js +2 -0
- package/dist/backup/export.js.map +1 -1
- package/dist/backup/manifest.d.ts +49 -0
- package/dist/backup/manifest.d.ts.map +1 -0
- package/dist/backup/manifest.js +103 -0
- package/dist/backup/manifest.js.map +1 -0
- package/dist/backup/snapshot-encoding.d.ts +8 -0
- package/dist/backup/snapshot-encoding.d.ts.map +1 -0
- package/dist/backup/snapshot-encoding.js +27 -0
- package/dist/backup/snapshot-encoding.js.map +1 -0
- package/dist/cli.d.ts +12 -0
- package/dist/cli.d.ts.map +1 -0
- package/dist/cli.js +206 -0
- package/dist/cli.js.map +1 -0
- package/dist/core/buffer.d.ts +4 -3
- package/dist/core/buffer.d.ts.map +1 -1
- package/dist/core/buffer.js +65 -9
- package/dist/core/buffer.js.map +1 -1
- package/dist/core/countex.d.ts +14 -1
- package/dist/core/countex.d.ts.map +1 -1
- package/dist/core/countex.js +171 -48
- package/dist/core/countex.js.map +1 -1
- package/dist/core/extraction.d.ts +14 -6
- package/dist/core/extraction.d.ts.map +1 -1
- package/dist/core/extraction.js +117 -1
- package/dist/core/extraction.js.map +1 -1
- package/dist/core/mapping.d.ts +5 -0
- package/dist/core/mapping.d.ts.map +1 -1
- package/dist/core/mapping.js +71 -1
- package/dist/core/mapping.js.map +1 -1
- package/dist/core/timeframe.d.ts +7 -0
- package/dist/core/timeframe.d.ts.map +1 -1
- package/dist/core/timeframe.js +88 -0
- package/dist/core/timeframe.js.map +1 -1
- package/dist/index.d.ts +4 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +13 -1
- package/dist/index.js.map +1 -1
- package/dist/insights/advisors/fanout-health.d.ts +3 -0
- package/dist/insights/advisors/fanout-health.d.ts.map +1 -0
- package/dist/insights/advisors/fanout-health.js +3 -0
- package/dist/insights/advisors/fanout-health.js.map +1 -0
- package/dist/insights/advisors/map-health.d.ts +6 -0
- package/dist/insights/advisors/map-health.d.ts.map +1 -0
- package/dist/insights/advisors/map-health.js +154 -0
- package/dist/insights/advisors/map-health.js.map +1 -0
- package/dist/insights/advisors/resource-advisor.d.ts +4 -0
- package/dist/insights/advisors/resource-advisor.d.ts.map +1 -0
- package/dist/insights/advisors/resource-advisor.js +102 -0
- package/dist/insights/advisors/resource-advisor.js.map +1 -0
- package/dist/insights/advisors/retention-advisor.d.ts +2 -0
- package/dist/insights/advisors/retention-advisor.d.ts.map +1 -0
- package/dist/insights/advisors/retention-advisor.js +6 -0
- package/dist/insights/advisors/retention-advisor.js.map +1 -0
- package/dist/insights/advisors-subpath.d.ts +4 -0
- package/dist/insights/advisors-subpath.d.ts.map +1 -0
- package/dist/insights/advisors-subpath.js +10 -0
- package/dist/insights/advisors-subpath.js.map +1 -0
- package/dist/insights/anomaly/baseline.d.ts +4 -0
- package/dist/insights/anomaly/baseline.d.ts.map +1 -0
- package/dist/insights/anomaly/baseline.js +54 -0
- package/dist/insights/anomaly/baseline.js.map +1 -0
- package/dist/insights/anomaly/detect-anomalies.d.ts +6 -0
- package/dist/insights/anomaly/detect-anomalies.d.ts.map +1 -0
- package/dist/insights/anomaly/detect-anomalies.js +245 -0
- package/dist/insights/anomaly/detect-anomalies.js.map +1 -0
- package/dist/insights/anomaly/ewma.d.ts +2 -0
- package/dist/insights/anomaly/ewma.d.ts.map +1 -0
- package/dist/insights/anomaly/ewma.js +15 -0
- package/dist/insights/anomaly/ewma.js.map +1 -0
- package/dist/insights/anomaly/missing-activity.d.ts +3 -0
- package/dist/insights/anomaly/missing-activity.d.ts.map +1 -0
- package/dist/insights/anomaly/missing-activity.js +8 -0
- package/dist/insights/anomaly/missing-activity.js.map +1 -0
- package/dist/insights/anomaly/robust-zscore.d.ts +2 -0
- package/dist/insights/anomaly/robust-zscore.d.ts.map +1 -0
- package/dist/insights/anomaly/robust-zscore.js +9 -0
- package/dist/insights/anomaly/robust-zscore.js.map +1 -0
- package/dist/insights/anomaly-subpath.d.ts +7 -0
- package/dist/insights/anomaly-subpath.d.ts.map +1 -0
- package/dist/insights/anomaly-subpath.js +20 -0
- package/dist/insights/anomaly-subpath.js.map +1 -0
- package/dist/insights/comparison/compare-periods.d.ts +5 -0
- package/dist/insights/comparison/compare-periods.d.ts.map +1 -0
- package/dist/insights/comparison/compare-periods.js +64 -0
- package/dist/insights/comparison/compare-periods.js.map +1 -0
- package/dist/insights/comparison/growth-rate.d.ts +3 -0
- package/dist/insights/comparison/growth-rate.d.ts.map +1 -0
- package/dist/insights/comparison/growth-rate.js +3 -0
- package/dist/insights/comparison/growth-rate.js.map +1 -0
- package/dist/insights/comparison/moving-average.d.ts +2 -0
- package/dist/insights/comparison/moving-average.d.ts.map +1 -0
- package/dist/insights/comparison/moving-average.js +6 -0
- package/dist/insights/comparison/moving-average.js.map +1 -0
- package/dist/insights/comparison/trend.d.ts +5 -0
- package/dist/insights/comparison/trend.d.ts.map +1 -0
- package/dist/insights/comparison/trend.js +84 -0
- package/dist/insights/comparison/trend.js.map +1 -0
- package/dist/insights/errors.d.ts +25 -0
- package/dist/insights/errors.d.ts.map +1 -0
- package/dist/insights/errors.js +54 -0
- package/dist/insights/errors.js.map +1 -0
- package/dist/insights/forecast/budget.d.ts +2 -0
- package/dist/insights/forecast/budget.d.ts.map +1 -0
- package/dist/insights/forecast/budget.js +6 -0
- package/dist/insights/forecast/budget.js.map +1 -0
- package/dist/insights/forecast/capacity.d.ts +2 -0
- package/dist/insights/forecast/capacity.d.ts.map +1 -0
- package/dist/insights/forecast/capacity.js +3 -0
- package/dist/insights/forecast/capacity.js.map +1 -0
- package/dist/insights/forecast/forecast.d.ts +5 -0
- package/dist/insights/forecast/forecast.d.ts.map +1 -0
- package/dist/insights/forecast/forecast.js +103 -0
- package/dist/insights/forecast/forecast.js.map +1 -0
- package/dist/insights/forecast/retention.d.ts +2 -0
- package/dist/insights/forecast/retention.d.ts.map +1 -0
- package/dist/insights/forecast/retention.js +3 -0
- package/dist/insights/forecast/retention.js.map +1 -0
- package/dist/insights/index.d.ts +4 -0
- package/dist/insights/index.d.ts.map +1 -0
- package/dist/insights/index.js +23 -0
- package/dist/insights/index.js.map +1 -0
- package/dist/insights/insights-client.d.ts +33 -0
- package/dist/insights/insights-client.d.ts.map +1 -0
- package/dist/insights/insights-client.js +344 -0
- package/dist/insights/insights-client.js.map +1 -0
- package/dist/insights/llm-subpath.d.ts +4 -0
- package/dist/insights/llm-subpath.d.ts.map +1 -0
- package/dist/insights/llm-subpath.js +10 -0
- package/dist/insights/llm-subpath.js.map +1 -0
- package/dist/insights/optimization/cost-optimizer.d.ts +6 -0
- package/dist/insights/optimization/cost-optimizer.d.ts.map +1 -0
- package/dist/insights/optimization/cost-optimizer.js +98 -0
- package/dist/insights/optimization/cost-optimizer.js.map +1 -0
- package/dist/insights/optimization/latency-optimizer.d.ts +3 -0
- package/dist/insights/optimization/latency-optimizer.d.ts.map +1 -0
- package/dist/insights/optimization/latency-optimizer.js +3 -0
- package/dist/insights/optimization/latency-optimizer.js.map +1 -0
- package/dist/insights/optimization/threshold-advisor.d.ts +2 -0
- package/dist/insights/optimization/threshold-advisor.d.ts.map +1 -0
- package/dist/insights/optimization/threshold-advisor.js +3 -0
- package/dist/insights/optimization/threshold-advisor.js.map +1 -0
- package/dist/insights/optimization/token-optimizer.d.ts +4 -0
- package/dist/insights/optimization/token-optimizer.d.ts.map +1 -0
- package/dist/insights/optimization/token-optimizer.js +133 -0
- package/dist/insights/optimization/token-optimizer.js.map +1 -0
- package/dist/insights/ranking/contribution.d.ts +2 -0
- package/dist/insights/ranking/contribution.d.ts.map +1 -0
- package/dist/insights/ranking/contribution.js +6 -0
- package/dist/insights/ranking/contribution.js.map +1 -0
- package/dist/insights/ranking/helpers.d.ts +22 -0
- package/dist/insights/ranking/helpers.d.ts.map +1 -0
- package/dist/insights/ranking/helpers.js +88 -0
- package/dist/insights/ranking/helpers.js.map +1 -0
- package/dist/insights/ranking/pareto.d.ts +2 -0
- package/dist/insights/ranking/pareto.d.ts.map +1 -0
- package/dist/insights/ranking/pareto.js +6 -0
- package/dist/insights/ranking/pareto.js.map +1 -0
- package/dist/insights/ranking/slices.d.ts +2 -0
- package/dist/insights/ranking/slices.d.ts.map +1 -0
- package/dist/insights/ranking/slices.js +6 -0
- package/dist/insights/ranking/slices.js.map +1 -0
- package/dist/insights/ranking/top-bottom.d.ts +2 -0
- package/dist/insights/ranking/top-bottom.d.ts.map +1 -0
- package/dist/insights/ranking/top-bottom.js +7 -0
- package/dist/insights/ranking/top-bottom.js.map +1 -0
- package/dist/insights/recommendations/evidence.d.ts +2 -0
- package/dist/insights/recommendations/evidence.d.ts.map +1 -0
- package/dist/insights/recommendations/evidence.js +3 -0
- package/dist/insights/recommendations/evidence.js.map +1 -0
- package/dist/insights/recommendations/recommendation.d.ts +6 -0
- package/dist/insights/recommendations/recommendation.d.ts.map +1 -0
- package/dist/insights/recommendations/recommendation.js +25 -0
- package/dist/insights/recommendations/recommendation.js.map +1 -0
- package/dist/insights/recommendations/severity.d.ts +2 -0
- package/dist/insights/recommendations/severity.d.ts.map +1 -0
- package/dist/insights/recommendations/severity.js +3 -0
- package/dist/insights/recommendations/severity.js.map +1 -0
- package/dist/insights/rules/evaluate-rules.d.ts +5 -0
- package/dist/insights/rules/evaluate-rules.d.ts.map +1 -0
- package/dist/insights/rules/evaluate-rules.js +156 -0
- package/dist/insights/rules/evaluate-rules.js.map +1 -0
- package/dist/insights/types.d.ts +405 -0
- package/dist/insights/types.d.ts.map +1 -0
- package/dist/insights/types.js +3 -0
- package/dist/insights/types.js.map +1 -0
- package/dist/insights/utils/formatting.d.ts +4 -0
- package/dist/insights/utils/formatting.d.ts.map +1 -0
- package/dist/insights/utils/formatting.js +21 -0
- package/dist/insights/utils/formatting.js.map +1 -0
- package/dist/insights/utils/grouping.d.ts +2 -0
- package/dist/insights/utils/grouping.d.ts.map +1 -0
- package/dist/insights/utils/grouping.js +8 -0
- package/dist/insights/utils/grouping.js.map +1 -0
- package/dist/insights/utils/query-builder.d.ts +21 -0
- package/dist/insights/utils/query-builder.d.ts.map +1 -0
- package/dist/insights/utils/query-builder.js +107 -0
- package/dist/insights/utils/query-builder.js.map +1 -0
- package/dist/insights/utils/row-value.d.ts +5 -0
- package/dist/insights/utils/row-value.d.ts.map +1 -0
- package/dist/insights/utils/row-value.js +36 -0
- package/dist/insights/utils/row-value.js.map +1 -0
- package/dist/insights/utils/stats.d.ts +28 -0
- package/dist/insights/utils/stats.d.ts.map +1 -0
- package/dist/insights/utils/stats.js +125 -0
- package/dist/insights/utils/stats.js.map +1 -0
- package/dist/query/formatter.d.ts +2 -2
- package/dist/query/formatter.d.ts.map +1 -1
- package/dist/query/formatter.js +211 -30
- package/dist/query/formatter.js.map +1 -1
- package/dist/query/planner.d.ts +10 -2
- package/dist/query/planner.d.ts.map +1 -1
- package/dist/query/planner.js +55 -4
- package/dist/query/planner.js.map +1 -1
- package/dist/simulation/context.d.ts +5 -0
- package/dist/simulation/context.d.ts.map +1 -0
- package/dist/simulation/context.js +56 -0
- package/dist/simulation/context.js.map +1 -0
- package/dist/simulation/events.d.ts +3 -0
- package/dist/simulation/events.d.ts.map +1 -0
- package/dist/simulation/events.js +67 -0
- package/dist/simulation/events.js.map +1 -0
- package/dist/simulation/index.d.ts +7 -0
- package/dist/simulation/index.d.ts.map +1 -0
- package/dist/simulation/index.js +16 -0
- package/dist/simulation/index.js.map +1 -0
- package/dist/simulation/prng.d.ts +4 -0
- package/dist/simulation/prng.d.ts.map +1 -0
- package/dist/simulation/prng.js +18 -0
- package/dist/simulation/prng.js.map +1 -0
- package/dist/simulation/resource-estimate.d.ts +6 -0
- package/dist/simulation/resource-estimate.d.ts.map +1 -0
- package/dist/simulation/resource-estimate.js +11 -0
- package/dist/simulation/resource-estimate.js.map +1 -0
- package/dist/simulation/simulator.d.ts +15 -0
- package/dist/simulation/simulator.d.ts.map +1 -0
- package/dist/simulation/simulator.js +67 -0
- package/dist/simulation/simulator.js.map +1 -0
- package/dist/simulation/tracker.d.ts +8 -0
- package/dist/simulation/tracker.d.ts.map +1 -0
- package/dist/simulation/tracker.js +18 -0
- package/dist/simulation/tracker.js.map +1 -0
- package/dist/simulation/types.d.ts +49 -0
- package/dist/simulation/types.d.ts.map +1 -0
- package/dist/simulation/types.js +21 -0
- package/dist/simulation/types.js.map +1 -0
- package/dist/storage/memory.d.ts +3 -0
- package/dist/storage/memory.d.ts.map +1 -1
- package/dist/storage/memory.js +113 -4
- package/dist/storage/memory.js.map +1 -1
- package/dist/storage/mongo.d.ts +5 -0
- package/dist/storage/mongo.d.ts.map +1 -1
- package/dist/storage/mongo.js +129 -17
- package/dist/storage/mongo.js.map +1 -1
- package/dist/storage/redis.d.ts +12 -9
- package/dist/storage/redis.d.ts.map +1 -1
- package/dist/storage/redis.js +126 -47
- package/dist/storage/redis.js.map +1 -1
- package/dist/types/index.d.ts +139 -4
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +40 -2
- package/docs/INSIGHTS.md +59 -0
- package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -0
- package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -0
- package/docs/INSIGHTS_RESOURCE_ADVISOR.md +31 -0
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -0
- package/docs/guides/README.md +34 -0
- package/docs/guides/adapter-memory.md +48 -0
- package/docs/guides/adapter-mongo.md +62 -0
- package/docs/guides/adapter-redis.md +53 -0
- package/docs/guides/backup-export.md +83 -0
- package/docs/guides/cardinality.md +50 -0
- package/docs/guides/cli.md +53 -0
- package/docs/guides/client-events.md +49 -0
- package/docs/guides/counter-maps.md +122 -0
- package/docs/guides/idempotency.md +39 -0
- package/docs/guides/ingestion-and-flush.md +67 -0
- package/docs/guides/insights-advisors-and-optimization.md +39 -0
- package/docs/guides/insights-anomalies-and-rules.md +48 -0
- package/docs/guides/insights-comparison-and-trends.md +47 -0
- package/docs/guides/insights-forecasting-and-budget.md +61 -0
- package/docs/guides/insights-overview.md +82 -0
- package/docs/guides/insights-ranking.md +81 -0
- package/docs/guides/measures.md +71 -0
- package/docs/guides/querying.md +119 -0
- package/docs/guides/simulation-profiles.md +33 -0
- package/docs/guides/simulation-resource-estimation.md +30 -0
- package/docs/guides/simulation.md +36 -0
- package/docs/guides/timeframes-and-bucketing.md +55 -0
- package/docs/internals/README.md +21 -0
- package/docs/internals/internal-architecture-overview.md +65 -0
- package/docs/internals/internal-backup-stream.md +23 -0
- package/docs/internals/internal-buffer-merge-and-flush.md +51 -0
- package/docs/internals/internal-cardinality.md +30 -0
- package/docs/internals/internal-dimension-hash.md +34 -0
- package/docs/internals/internal-idempotency.md +36 -0
- package/docs/internals/internal-ingest-pipeline.md +52 -0
- package/docs/internals/internal-insights-layer.md +36 -0
- package/docs/internals/internal-mapping-validation-and-extraction.md +59 -0
- package/docs/internals/internal-measures-and-scales.md +35 -0
- package/docs/internals/internal-query-path.md +46 -0
- package/docs/internals/internal-storage-memory.md +13 -0
- package/docs/internals/internal-storage-mongo.md +24 -0
- package/docs/internals/internal-storage-redis.md +31 -0
- package/docs/internals/internal-timeframe-bucketing.md +33 -0
- package/docs/use-cases/README.md +56 -0
- package/docs/use-cases/audit-security-log-counts.md +550 -0
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -0
- package/docs/use-cases/multi-tenant-isolation.md +203 -0
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -0
- package/docs/use-cases/processing-workflow-activity.md +165 -0
- package/docs/use-cases/product-feature-usage.md +240 -0
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -0
- package/package.json +61 -2
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Countex feature guides
|
|
2
|
+
|
|
3
|
+
Step-by-step guides for using each major feature of `@x12i/countex`. Each document stands alone.
|
|
4
|
+
|
|
5
|
+
These files are published with the npm package and exposed as stable import subpaths (see root `README.md` and `package.json` `exports`).
|
|
6
|
+
|
|
7
|
+
| Guide | Package subpath |
|
|
8
|
+
|-------|-----------------|
|
|
9
|
+
| [Counter maps](./counter-maps.md) | `@x12i/countex/docs/guides/counter-maps` |
|
|
10
|
+
| [Ingestion and flush](./ingestion-and-flush.md) | `@x12i/countex/docs/guides/ingestion-and-flush` |
|
|
11
|
+
| [Timeframes](./timeframes-and-bucketing.md) | `@x12i/countex/docs/guides/timeframes-and-bucketing` |
|
|
12
|
+
| [Measures](./measures.md) | `@x12i/countex/docs/guides/measures` |
|
|
13
|
+
| [Querying](./querying.md) | `@x12i/countex/docs/guides/querying` |
|
|
14
|
+
| [Memory adapters](./adapter-memory.md) | `@x12i/countex/docs/guides/adapter-memory` |
|
|
15
|
+
| [Mongo durable](./adapter-mongo.md) | `@x12i/countex/docs/guides/adapter-mongo` |
|
|
16
|
+
| [Redis cache](./adapter-redis.md) | `@x12i/countex/docs/guides/adapter-redis` |
|
|
17
|
+
| [Cardinality](./cardinality.md) | `@x12i/countex/docs/guides/cardinality` |
|
|
18
|
+
| [Idempotency](./idempotency.md) | `@x12i/countex/docs/guides/idempotency` |
|
|
19
|
+
| [Backup export](./backup-export.md) | `@x12i/countex/docs/guides/backup-export` |
|
|
20
|
+
| [Client events](./client-events.md) | `@x12i/countex/docs/guides/client-events` |
|
|
21
|
+
| [Insights overview](./insights-overview.md) | `@x12i/countex/docs/guides/insights-overview` |
|
|
22
|
+
| [Insights ranking](./insights-ranking.md) | `@x12i/countex/docs/guides/insights-ranking` |
|
|
23
|
+
| [Insights comparison & trends](./insights-comparison-and-trends.md) | `@x12i/countex/docs/guides/insights-comparison-and-trends` |
|
|
24
|
+
| [Insights anomalies & rules](./insights-anomalies-and-rules.md) | `@x12i/countex/docs/guides/insights-anomalies-and-rules` |
|
|
25
|
+
| [Insights forecasting & budget](./insights-forecasting-and-budget.md) | `@x12i/countex/docs/guides/insights-forecasting-and-budget` |
|
|
26
|
+
| [Insights advisors](./insights-advisors-and-optimization.md) | `@x12i/countex/docs/guides/insights-advisors-and-optimization` |
|
|
27
|
+
| [CLI](./cli.md) | `@x12i/countex/docs/guides/cli` |
|
|
28
|
+
| [Simulation overview](./simulation.md) | `@x12i/countex/docs/guides/simulation` |
|
|
29
|
+
| [Simulation profiles](./simulation-profiles.md) | `@x12i/countex/docs/guides/simulation-profiles` |
|
|
30
|
+
| [Simulation resource estimation](./simulation-resource-estimation.md) | `@x12i/countex/docs/guides/simulation-resource-estimation` |
|
|
31
|
+
|
|
32
|
+
Scenario walkthroughs: [`docs/use-cases/`](../use-cases/README.md) (also exposed as `@x12i/countex/docs/use-cases/*`).
|
|
33
|
+
|
|
34
|
+
Additional narrative docs (also under `docs/`): `DEPLOYMENT.md`, `END_TO_END_EXAMPLE.md`, `INSIGHTS*.md`.
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# Guide: Memory adapters
|
|
2
|
+
|
|
3
|
+
In-process **`MemoryDurable`** and **`MemoryCache`** implement the storage contracts for tests and local development.
|
|
4
|
+
|
|
5
|
+
## Step 1: Import
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory";
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
## Step 2: Construct Countex
|
|
12
|
+
|
|
13
|
+
```ts
|
|
14
|
+
import { createCountex } from "@x12i/countex";
|
|
15
|
+
|
|
16
|
+
const countex = createCountex({
|
|
17
|
+
maps: [myMap],
|
|
18
|
+
durable: new MemoryDurable(),
|
|
19
|
+
cache: new MemoryCache(),
|
|
20
|
+
});
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
## Step 3: Use like production
|
|
24
|
+
|
|
25
|
+
```ts
|
|
26
|
+
await countex.ingest(event);
|
|
27
|
+
await countex.flush();
|
|
28
|
+
const q = await countex.query({ /* ... */ });
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
No network, no peer dependencies.
|
|
32
|
+
|
|
33
|
+
## When to use
|
|
34
|
+
|
|
35
|
+
- Unit tests
|
|
36
|
+
- Demos and notebooks
|
|
37
|
+
- CI without Docker services
|
|
38
|
+
|
|
39
|
+
## Limitations
|
|
40
|
+
|
|
41
|
+
- State dies with the process.
|
|
42
|
+
- Not suitable for multi-process concurrency testing at scale.
|
|
43
|
+
|
|
44
|
+
## See also
|
|
45
|
+
|
|
46
|
+
- [Mongo adapter](./adapter-mongo.md)
|
|
47
|
+
- [Redis adapter](./adapter-redis.md)
|
|
48
|
+
- Internal: [Memory storage](../internals/internal-storage-memory.md)
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Guide: MongoDB durable adapter
|
|
2
|
+
|
|
3
|
+
`MongoDurable` persists cells and the subject/metric dictionary to MongoDB.
|
|
4
|
+
|
|
5
|
+
## Step 1: Install peer dependency
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
npm install mongodb
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
## Step 2: Create a client (you own lifecycle)
|
|
12
|
+
|
|
13
|
+
```ts
|
|
14
|
+
import { MongoClient } from "mongodb";
|
|
15
|
+
import { MongoDurable } from "@x12i/countex/adapters/mongo";
|
|
16
|
+
import { createCountex } from "@x12i/countex";
|
|
17
|
+
|
|
18
|
+
const mongo = new MongoClient(process.env.MONGO_URI!);
|
|
19
|
+
await mongo.connect();
|
|
20
|
+
|
|
21
|
+
const durable = new MongoDurable({
|
|
22
|
+
client: mongo,
|
|
23
|
+
dbResolver: ({ scope }) => `countex_${scope}`,
|
|
24
|
+
});
|
|
25
|
+
|
|
26
|
+
const countex = createCountex({
|
|
27
|
+
maps: [myMap],
|
|
28
|
+
durable,
|
|
29
|
+
cache: /* optional Redis */,
|
|
30
|
+
});
|
|
31
|
+
|
|
32
|
+
await countex.init();
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
## Step 3: Collections and indexes
|
|
36
|
+
|
|
37
|
+
On first write the adapter ensures collections such as **`countex_cells`** and **`countex_dict`**, plus indexes for identity and range scans. Per-scope databases isolate tenant data when using `dbResolver`.
|
|
38
|
+
|
|
39
|
+
## Step 4: Flush semantics
|
|
40
|
+
|
|
41
|
+
Flushes call `applyDeltas` with atomic increment operations so parallel writers can safely aggregate.
|
|
42
|
+
|
|
43
|
+
## Dictionary allocation (`resolveIds`) and concurrency
|
|
44
|
+
|
|
45
|
+
Dictionary rows map **`(scopeId, kind, qualifier, sourceKey)` → `internalId`** (integer). The adapter:
|
|
46
|
+
|
|
47
|
+
1. **Reads** existing rows for the batch.
|
|
48
|
+
2. **Allocates** a contiguous block of ids from a per-scope sequence document (`countex_seq`).
|
|
49
|
+
3. **Inserts** new rows with `insertMany({ ordered: false })`.
|
|
50
|
+
|
|
51
|
+
A **unique index** (`uniq_dict_key` on `scopeId`, `kind`, `qualifier`, `sourceKey`) prevents duplicate keys. If two workers race on the same new key, one insert wins and the other hits a duplicate-key error; the adapter **re-reads** and continues. Always create indexes via `init()` / first write (unless `skipIndexInit`).
|
|
52
|
+
|
|
53
|
+
Sequence allocation uses MongoDB **`findOneAndUpdate` `$inc`**, which is atomic per sequence document. Very large parallel bursts should still be safe; id blocks may interleave across workers but remain unique.
|
|
54
|
+
|
|
55
|
+
## Step 5: Shutdown
|
|
56
|
+
|
|
57
|
+
Close Countex, then close the Mongo client when your process exits.
|
|
58
|
+
|
|
59
|
+
## See also
|
|
60
|
+
|
|
61
|
+
- [Deployment](../DEPLOYMENT.md)
|
|
62
|
+
- Internal: [Mongo durable](../internals/internal-storage-mongo.md)
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# Guide: Redis cache adapter
|
|
2
|
+
|
|
3
|
+
`RedisCache` stores hot counter cells for fast reads. It pairs with a durable store (typically Mongo).
|
|
4
|
+
|
|
5
|
+
## Step 1: Install peer dependency
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
npm install ioredis
|
|
9
|
+
```
|
|
10
|
+
|
|
11
|
+
## Step 2: Wire client and adapter
|
|
12
|
+
|
|
13
|
+
```ts
|
|
14
|
+
import IORedis from "ioredis";
|
|
15
|
+
import { RedisCache } from "@x12i/countex/adapters/redis";
|
|
16
|
+
|
|
17
|
+
const redis = new IORedis(process.env.REDIS_URL!);
|
|
18
|
+
|
|
19
|
+
const cache = new RedisCache({
|
|
20
|
+
client: redis,
|
|
21
|
+
keyPrefix: "countex",
|
|
22
|
+
ttlSeconds: 92 * 24 * 3600,
|
|
23
|
+
// Optional: shorter hot buckets, longer rollups
|
|
24
|
+
ttlSecondsByTimeframe: {
|
|
25
|
+
minute: 48 * 3600,
|
|
26
|
+
hour: 14 * 24 * 3600,
|
|
27
|
+
day: 92 * 24 * 3600,
|
|
28
|
+
},
|
|
29
|
+
});
|
|
30
|
+
|
|
31
|
+
const countex = createCountex({
|
|
32
|
+
maps: [myMap],
|
|
33
|
+
durable,
|
|
34
|
+
cache,
|
|
35
|
+
});
|
|
36
|
+
|
|
37
|
+
await countex.init();
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Step 3: Idempotency hook
|
|
41
|
+
|
|
42
|
+
When maps declare **idempotency**, the cache adapter can implement **`markEventSeen`** so duplicates are skipped before buffering.
|
|
43
|
+
|
|
44
|
+
## Step 4: Operational expectations
|
|
45
|
+
|
|
46
|
+
- Cache misses fall back to durable reads; results may warm the cache afterward.
|
|
47
|
+
- Cache flush failures during write path are **non-fatal** (durable is source of truth).
|
|
48
|
+
|
|
49
|
+
## See also
|
|
50
|
+
|
|
51
|
+
- [Idempotency](./idempotency.md)
|
|
52
|
+
- [Querying](./querying.md) (cache vs durable for min/max measures)
|
|
53
|
+
- Internal: [Redis cache](../internals/internal-storage-redis.md)
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# Guide: Backup export and cache rebuild
|
|
2
|
+
|
|
3
|
+
Helpers **`exportCells`** and **`rebuildCache`** stream data from durable storage and optionally warm a cache.
|
|
4
|
+
|
|
5
|
+
## Step 1: Export cells
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
import { exportCells } from "@x12i/countex";
|
|
9
|
+
|
|
10
|
+
for await (const cell of exportCells(durable, {
|
|
11
|
+
scope: "org_1",
|
|
12
|
+
counterId: "events",
|
|
13
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
14
|
+
})) {
|
|
15
|
+
// serialize, audit, backup...
|
|
16
|
+
}
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
Streaming is bounded by the `(scope, counterId, timeframe)` filter you provide.
|
|
20
|
+
|
|
21
|
+
## Step 2: Rebuild cache after outage
|
|
22
|
+
|
|
23
|
+
```ts
|
|
24
|
+
import { rebuildCache } from "@x12i/countex";
|
|
25
|
+
|
|
26
|
+
const { cellsWarmed } = await rebuildCache(durable, cache, {
|
|
27
|
+
scope: "org_1",
|
|
28
|
+
counterId: "events",
|
|
29
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
30
|
+
}, { batchSize: 1000 });
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## Step 3: Manifest and verification
|
|
34
|
+
|
|
35
|
+
After streaming an export, compute a stable fingerprint and save a small JSON manifest (e.g. next to your NDJSON dump):
|
|
36
|
+
|
|
37
|
+
```ts
|
|
38
|
+
import {
|
|
39
|
+
computeCellsFingerprint,
|
|
40
|
+
createBackupManifest,
|
|
41
|
+
verifyBackupAgainstManifest,
|
|
42
|
+
exportCells,
|
|
43
|
+
COUNTEX_CELL_SCHEMA_VERSION,
|
|
44
|
+
} from "@x12i/countex";
|
|
45
|
+
|
|
46
|
+
const filter = {
|
|
47
|
+
scope: "org_1",
|
|
48
|
+
counterId: "events",
|
|
49
|
+
timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
|
|
50
|
+
};
|
|
51
|
+
|
|
52
|
+
const fp = await computeCellsFingerprint(exportCells(durable, filter));
|
|
53
|
+
const manifest = createBackupManifest(filter, fp, COUNTEX_CELL_SCHEMA_VERSION);
|
|
54
|
+
// persist manifest JSON + cell lines...
|
|
55
|
+
|
|
56
|
+
const check = await verifyBackupAgainstManifest(manifest, exportCells(durable, filter));
|
|
57
|
+
if (!check.ok) throw new Error(check.detail);
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Checksums sort canonical JSON lines in memory — use **timeframe-bounded** exports for very large datasets.
|
|
61
|
+
|
|
62
|
+
## Step 4: Restore snapshot
|
|
63
|
+
|
|
64
|
+
**Target:** durable storage only (`importSnapshotCells` on `MongoDurable` / `MemoryDurable`). After restore, **rebuild Redis** with `rebuildCache` for affected windows.
|
|
65
|
+
|
|
66
|
+
```ts
|
|
67
|
+
import { restoreSnapshotCells, exportCells } from "@x12i/countex";
|
|
68
|
+
|
|
69
|
+
const { restored } = await restoreSnapshotCells(targetDurable, exportCells(sourceDurable, filter), {
|
|
70
|
+
batchSize: 500,
|
|
71
|
+
});
|
|
72
|
+
// dryRun: true counts cells without writing
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
Restore replaces absolute cell state for each identity in the stream — it is not an incremental delta merge.
|
|
76
|
+
|
|
77
|
+
## Step 5: Cancellation
|
|
78
|
+
|
|
79
|
+
Pass `AbortSignal` where supported (`streamCells` / `warm` / restore options propagate through adapters).
|
|
80
|
+
|
|
81
|
+
## See also
|
|
82
|
+
|
|
83
|
+
- Internal: [Backup streaming](../internals/internal-backup-stream.md)
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# Guide: Cardinality protection
|
|
2
|
+
|
|
3
|
+
Cardinality limits stop runaway distinct **subjects**, **metrics**, or **dimension values** from exploding storage when a map accidentally points at high-cardinality fields (URLs, UUIDs everywhere, etc.).
|
|
4
|
+
|
|
5
|
+
## Step 1: Configure engine defaults
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const countex = createCountex({
|
|
9
|
+
maps: [myMap],
|
|
10
|
+
durable,
|
|
11
|
+
cache,
|
|
12
|
+
cardinality: {
|
|
13
|
+
maxDimensionValuesPerName: 10_000,
|
|
14
|
+
maxSubjectsPerScope: 10_000_000,
|
|
15
|
+
onLimit: "overflow",
|
|
16
|
+
overflowValue: "__OTHER__",
|
|
17
|
+
highCardinalityWarnings: true,
|
|
18
|
+
},
|
|
19
|
+
});
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
## Step 2: Override per map (optional)
|
|
23
|
+
|
|
24
|
+
Add `cardinality` on a `CounterMap` for stricter rules on sensitive projections.
|
|
25
|
+
|
|
26
|
+
## Step 3: Understand behaviors (`onLimit`)
|
|
27
|
+
|
|
28
|
+
| Mode | Behavior |
|
|
29
|
+
|------|-----------|
|
|
30
|
+
| `overflow` | Rewrite additional distinct keys to `overflowValue` |
|
|
31
|
+
| `drop` | Skip increments that would exceed limits |
|
|
32
|
+
| `error` | Throw (surrounding ingest handles via logging; prefer rare use) |
|
|
33
|
+
|
|
34
|
+
## Step 4: Listen for signals
|
|
35
|
+
|
|
36
|
+
```ts
|
|
37
|
+
countex.on("cardinality_warning", (w) => { /* nearing limit */ });
|
|
38
|
+
countex.on("overflow", (o) => { /* key rewritten */ });
|
|
39
|
+
countex.on("dropped", (d) => { /* increment skipped */ });
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
## Practical tips
|
|
43
|
+
|
|
44
|
+
- Prefer bounded dimensions (enums, categories).
|
|
45
|
+
- Use `overflow` in production pipelines so one bad event shape does not halt ingestion.
|
|
46
|
+
|
|
47
|
+
## See also
|
|
48
|
+
|
|
49
|
+
- [Counter maps](./counter-maps.md)
|
|
50
|
+
- Internal: [Cardinality guard](../internals/internal-cardinality.md)
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# Guide: CLI (`countex`)
|
|
2
|
+
|
|
3
|
+
The package exposes a minimal **`countex`** binary for running selected insights against a **config module** you provide.
|
|
4
|
+
|
|
5
|
+
## Step 1: Build or install
|
|
6
|
+
|
|
7
|
+
After install, the binary resolves to `dist/cli.js`. For local development:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
npm run build
|
|
11
|
+
npx countex insights:top ./my-context.mjs --scope org_1 --counter my_counter
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
## Step 2: Author a config module
|
|
15
|
+
|
|
16
|
+
Export one of:
|
|
17
|
+
|
|
18
|
+
- `export async function createContext()` returning `{ insights: CountexInsights }`
|
|
19
|
+
- `export default { insights }`
|
|
20
|
+
- `export const insights`
|
|
21
|
+
|
|
22
|
+
## Step 3: Run a command
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
countex insights:top ./ctx.mjs --scope org_1 --counter usage
|
|
26
|
+
countex insights:pareto ./ctx.mjs --format markdown
|
|
27
|
+
countex insights:anomalies ./ctx.mjs
|
|
28
|
+
countex insights:cost ./ctx.mjs
|
|
29
|
+
countex insights:tokens ./ctx.mjs
|
|
30
|
+
countex insights:map-health ./ctx.mjs
|
|
31
|
+
countex insights:resources ./ctx.mjs
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Use `--format json|markdown` for output shaping.
|
|
35
|
+
|
|
36
|
+
## Simulation demo (no config file)
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
countex simulate:demo --events 200 --seed 1
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Emits JSON **`SimulationReport`** from a dry-run against the bundled demo map (`audit-log` profile). No Mongo/Redis required.
|
|
43
|
+
|
|
44
|
+
## Step 4: TypeScript configs
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
node --import=tsx node_modules/@x12i/countex/dist/cli.js insights:top ./ctx.ts --scope org_1
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## See also
|
|
51
|
+
|
|
52
|
+
- [Insights overview](./insights-overview.md)
|
|
53
|
+
- `src/cli.ts` for the authoritative command list
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Guide: Client events (`Countex`)
|
|
2
|
+
|
|
3
|
+
`Countex` extends `EventEmitter`. Subscribe for operational visibility.
|
|
4
|
+
|
|
5
|
+
## Step 1: Subscribe
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
countex.on("flush", ({ cells, durationMs }) => {
|
|
9
|
+
console.log("flushed", cells, durationMs);
|
|
10
|
+
});
|
|
11
|
+
|
|
12
|
+
countex.on("flush_error", ({ error }) => {
|
|
13
|
+
console.error(error);
|
|
14
|
+
});
|
|
15
|
+
|
|
16
|
+
countex.on("cardinality_warning", (w) => {
|
|
17
|
+
console.warn("cardinality_warning", w);
|
|
18
|
+
});
|
|
19
|
+
|
|
20
|
+
countex.on("overflow", (o) => {
|
|
21
|
+
console.warn("overflow", o);
|
|
22
|
+
});
|
|
23
|
+
|
|
24
|
+
countex.on("dropped", (d) => {
|
|
25
|
+
console.warn("dropped", d);
|
|
26
|
+
});
|
|
27
|
+
|
|
28
|
+
countex.on("duplicate", ({ scopeId, eventId }) => {
|
|
29
|
+
console.info("duplicate", scopeId, eventId);
|
|
30
|
+
});
|
|
31
|
+
|
|
32
|
+
countex.on("measure_invalid", (m) => {
|
|
33
|
+
console.warn("measure_invalid", m);
|
|
34
|
+
});
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## Step 2: Type payloads
|
|
38
|
+
|
|
39
|
+
Import `CountexEvents` from types if you want compile-time checks when wiring handlers.
|
|
40
|
+
|
|
41
|
+
## Step 3: Unsubscribe
|
|
42
|
+
|
|
43
|
+
Use `off` with the same listener reference when tearing down tests.
|
|
44
|
+
|
|
45
|
+
## See also
|
|
46
|
+
|
|
47
|
+
- [Ingestion and flush](./ingestion-and-flush.md)
|
|
48
|
+
- [Cardinality](./cardinality.md)
|
|
49
|
+
- [Measures](./measures.md)
|
|
@@ -0,0 +1,122 @@
|
|
|
1
|
+
# Guide: Counter maps
|
|
2
|
+
|
|
3
|
+
A **counter map** (`CounterMap`) tells Countex how to read one JSON event and turn it into sparse counter updates. This guide walks through defining a map from scratch.
|
|
4
|
+
|
|
5
|
+
## Metrics vs domain “labels”
|
|
6
|
+
|
|
7
|
+
Countex uses **`metrics`** in the map config: paths that resolve to the category or label you want one counter split per value (for example `$.labels[*]` or `$.event.operation`). That field name is generic — your payload might call them labels, statuses, tags, or event types. They all map to Countex **metrics**.
|
|
8
|
+
|
|
9
|
+
Documentation refers to **`sourceKey`** for the string form of a subject or metric before dictionary compaction; that is not necessarily a UUID unless you choose UUID strings as keys.
|
|
10
|
+
|
|
11
|
+
## What you will configure
|
|
12
|
+
|
|
13
|
+
- **scope** — tenant or isolation boundary (string id).
|
|
14
|
+
- **eventTime** — when the event happened (drives timeframe buckets).
|
|
15
|
+
- **subjects** — one or more “things” the count attaches to (fan-out supported).
|
|
16
|
+
- **metrics** — labels or categories being counted (fan-out supported).
|
|
17
|
+
- **dimensions** — optional breakdown fields stored on each cell.
|
|
18
|
+
- **delta** — how much to add (default `1`).
|
|
19
|
+
- **timeframes** — which bucket granularities to write (`minute` … `all`).
|
|
20
|
+
- Optional: **measures**, **idempotency**, **cardinality**, **options**.
|
|
21
|
+
|
|
22
|
+
## Step 1: Choose a stable `counterId`
|
|
23
|
+
|
|
24
|
+
Use a string that identifies this projection family across deployments, for example `access_by_actor_label`.
|
|
25
|
+
|
|
26
|
+
## Step 2: Point at scope and time
|
|
27
|
+
|
|
28
|
+
```ts
|
|
29
|
+
const map = {
|
|
30
|
+
counterId: "access_by_actor_label",
|
|
31
|
+
scope: { path: "$.tenant.id" },
|
|
32
|
+
eventTime: { path: "$.event.createdAt" },
|
|
33
|
+
// ...
|
|
34
|
+
} as const;
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Rules:
|
|
38
|
+
|
|
39
|
+
- Both paths must resolve to usable values or the event is skipped for this map.
|
|
40
|
+
- `eventTime` accepts `Date`, ISO strings, or epoch numbers.
|
|
41
|
+
|
|
42
|
+
## Step 3: Declare subjects
|
|
43
|
+
|
|
44
|
+
Each subject has a `type` (your label) and a `path`. Multi-value paths fan out.
|
|
45
|
+
|
|
46
|
+
```ts
|
|
47
|
+
subjects: [
|
|
48
|
+
{ type: "actor", path: "$.actor.id", required: true },
|
|
49
|
+
{ type: "resource", path: "$.resource.id", required: false },
|
|
50
|
+
],
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
- **`required: true`** — if nothing matches, the whole event is skipped.
|
|
54
|
+
- **`required: false`** — missing values omit that subject type for this event.
|
|
55
|
+
|
|
56
|
+
## Step 4: Declare metrics
|
|
57
|
+
|
|
58
|
+
```ts
|
|
59
|
+
metrics: [
|
|
60
|
+
{ path: "$.labels[*]", required: true },
|
|
61
|
+
],
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Wildcards (`[*]`) expand to one increment per array element (or object values).
|
|
65
|
+
|
|
66
|
+
## Step 5: Add dimensions (optional)
|
|
67
|
+
|
|
68
|
+
```ts
|
|
69
|
+
dimensions: [
|
|
70
|
+
{ name: "operation", path: "$.event.operation" },
|
|
71
|
+
{ name: "situation", path: "$.event.situation", required: false },
|
|
72
|
+
],
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## Step 6: Delta and timeframes
|
|
76
|
+
|
|
77
|
+
```ts
|
|
78
|
+
delta: { default: 1 },
|
|
79
|
+
timeframes: ["day", "week", "month"],
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
Use `delta: { path: "$.weight" }` when counts are not always `1`.
|
|
83
|
+
|
|
84
|
+
## Step 7: Pass maps into `createCountex`
|
|
85
|
+
|
|
86
|
+
```ts
|
|
87
|
+
import { createCountex } from "@x12i/countex";
|
|
88
|
+
import { MemoryDurable, MemoryCache } from "@x12i/countex/adapters/memory";
|
|
89
|
+
|
|
90
|
+
const countex = createCountex({
|
|
91
|
+
maps: [map],
|
|
92
|
+
durable: new MemoryDurable(),
|
|
93
|
+
cache: new MemoryCache(),
|
|
94
|
+
});
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Maps are validated at construction time; invalid paths or duplicate `counterId` values throw immediately.
|
|
98
|
+
|
|
99
|
+
## Path syntax (subset)
|
|
100
|
+
|
|
101
|
+
| Syntax | Meaning |
|
|
102
|
+
|--------|---------|
|
|
103
|
+
| `$` | Root |
|
|
104
|
+
| `.name` | Property |
|
|
105
|
+
| `['name']` | Bracket property |
|
|
106
|
+
| `[N]` | Array index |
|
|
107
|
+
| `[*]` | Fan-out wildcard |
|
|
108
|
+
|
|
109
|
+
## Fan-out math
|
|
110
|
+
|
|
111
|
+
Total raw increments per event ≈
|
|
112
|
+
|
|
113
|
+
`(subjects) × (metrics) × (number of timeframes)`.
|
|
114
|
+
|
|
115
|
+
Plan capacity accordingly.
|
|
116
|
+
|
|
117
|
+
## See also
|
|
118
|
+
|
|
119
|
+
- [Timeframes](./timeframes-and-bucketing.md)
|
|
120
|
+
- [Measures](./measures.md)
|
|
121
|
+
- [Cardinality](./cardinality.md)
|
|
122
|
+
- Internal: [Mapping validation & extraction](../internals/internal-mapping-validation-and-extraction.md)
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
# Guide: Idempotency
|
|
2
|
+
|
|
3
|
+
Idempotency prevents **duplicate events** from incrementing counters twice when retries or at-least-once delivery replay the same logical event.
|
|
4
|
+
|
|
5
|
+
## Step 1: Enable on the map
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
const map = {
|
|
9
|
+
counterId: "orders",
|
|
10
|
+
// ...
|
|
11
|
+
idempotency: {
|
|
12
|
+
enabled: true,
|
|
13
|
+
eventIdPath: "$.event.id",
|
|
14
|
+
ttlSeconds: 86_400,
|
|
15
|
+
},
|
|
16
|
+
} as const;
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
If `enabled` is true, `eventIdPath` is required (validated at startup).
|
|
20
|
+
|
|
21
|
+
## Step 2: Use a cache adapter that supports dedupe
|
|
22
|
+
|
|
23
|
+
Implement **`markEventSeen(scopeId, eventId, ttlSeconds): Promise<boolean>`** on your cache adapter. The built-in **`RedisCache`** provides this using Redis keys with TTL.
|
|
24
|
+
|
|
25
|
+
Countex namespaces ids as **`${counterId}:${eventId}`** per map so different counters do not collide.
|
|
26
|
+
|
|
27
|
+
## Step 3: Ingest normally
|
|
28
|
+
|
|
29
|
+
When `markEventSeen` returns **false** (already seen), increments for that map/event are dropped and a **`duplicate`** event is emitted.
|
|
30
|
+
|
|
31
|
+
## Step 4: Handle missing event ids
|
|
32
|
+
|
|
33
|
+
If the path resolves empty while idempotency is enabled, Countex records a warning and still processes unless your extraction logic skips the event — verify your producer always sets ids.
|
|
34
|
+
|
|
35
|
+
## See also
|
|
36
|
+
|
|
37
|
+
- [Redis adapter](./adapter-redis.md)
|
|
38
|
+
- [Client events](./client-events.md)
|
|
39
|
+
- Internal: [Idempotency](../internals/internal-idempotency.md)
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
# Guide: Ingestion and flush
|
|
2
|
+
|
|
3
|
+
Countex keeps **deltas in memory**, then **flushes** them to durable storage and optionally a cache. This guide covers the write path from your application’s perspective.
|
|
4
|
+
|
|
5
|
+
## Step 1: Ingest events
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
await countex.ingest({
|
|
9
|
+
tenant: { id: "org_1" },
|
|
10
|
+
event: { createdAt: "2026-05-08T12:00:00.000Z", operation: "read" },
|
|
11
|
+
actor: { id: "user_a" },
|
|
12
|
+
labels: ["docs"],
|
|
13
|
+
});
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
`ingest` accepts `unknown`. Each configured map runs extraction independently.
|
|
17
|
+
|
|
18
|
+
Use **`ingestBatch(events)`** when you already have an array; dictionary lookups are batched across the batch.
|
|
19
|
+
|
|
20
|
+
## Step 2: Understand buffering
|
|
21
|
+
|
|
22
|
+
After ingestion:
|
|
23
|
+
|
|
24
|
+
- Raw increments are merged into an in-memory **`DeltaBuffer`** keyed by cell identity.
|
|
25
|
+
- Nothing hits Mongo/Redis until flush (except dictionary resolution during ingest, which uses durable `resolveIds`).
|
|
26
|
+
|
|
27
|
+
## Step 3: Flush explicitly or on schedule
|
|
28
|
+
|
|
29
|
+
```ts
|
|
30
|
+
await countex.flush();
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Flush behavior is controlled by `flush` in config:
|
|
34
|
+
|
|
35
|
+
| Option | Role |
|
|
36
|
+
|--------|------|
|
|
37
|
+
| `everyMs` | Periodic background flush (0 disables timer) |
|
|
38
|
+
| `maxBufferedCells` | Threshold on distinct buffered cells |
|
|
39
|
+
| `maxBufferedEvents` | Threshold on ingested events since last drain |
|
|
40
|
+
| `flushOnShutdown` | Drain on `shutdown()` |
|
|
41
|
+
| `onOverflow` | `"block"` awaits flush at threshold; `"drop"` fires flush without blocking |
|
|
42
|
+
|
|
43
|
+
## Step 4: Shutdown cleanly
|
|
44
|
+
|
|
45
|
+
```ts
|
|
46
|
+
await countex.shutdown();
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Typically clears the timer, optionally flushes, then closes adapters.
|
|
50
|
+
|
|
51
|
+
## Step 5: Initialize adapters in production
|
|
52
|
+
|
|
53
|
+
```ts
|
|
54
|
+
await countex.init(); // calls durable.init?.() and cache.init?.()
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Call once after constructing clients.
|
|
58
|
+
|
|
59
|
+
## Observability
|
|
60
|
+
|
|
61
|
+
Subscribe to `flush`, `flush_error`, `dropped`, and related events — see [Client events](./client-events.md).
|
|
62
|
+
|
|
63
|
+
## See also
|
|
64
|
+
|
|
65
|
+
- [Counter maps](./counter-maps.md)
|
|
66
|
+
- [Adapter guides](./adapter-mongo.md)
|
|
67
|
+
- Internal: [Ingest pipeline](../internals/internal-ingest-pipeline.md), [Buffer & flush](../internals/internal-buffer-merge-and-flush.md)
|