@x12i/countex 1.0.0 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (256) hide show
  1. package/README.md +192 -3
  2. package/dist/backup/export.d.ts +2 -0
  3. package/dist/backup/export.d.ts.map +1 -1
  4. package/dist/backup/export.js +2 -0
  5. package/dist/backup/export.js.map +1 -1
  6. package/dist/cli.d.ts +12 -0
  7. package/dist/cli.d.ts.map +1 -0
  8. package/dist/cli.js +190 -0
  9. package/dist/cli.js.map +1 -0
  10. package/dist/core/buffer.d.ts +4 -3
  11. package/dist/core/buffer.d.ts.map +1 -1
  12. package/dist/core/buffer.js +65 -9
  13. package/dist/core/buffer.js.map +1 -1
  14. package/dist/core/countex.d.ts.map +1 -1
  15. package/dist/core/countex.js +30 -3
  16. package/dist/core/countex.js.map +1 -1
  17. package/dist/core/extraction.d.ts +14 -2
  18. package/dist/core/extraction.d.ts.map +1 -1
  19. package/dist/core/extraction.js +117 -1
  20. package/dist/core/extraction.js.map +1 -1
  21. package/dist/core/mapping.d.ts.map +1 -1
  22. package/dist/core/mapping.js +44 -1
  23. package/dist/core/mapping.js.map +1 -1
  24. package/dist/core/timeframe.d.ts +7 -0
  25. package/dist/core/timeframe.d.ts.map +1 -1
  26. package/dist/core/timeframe.js +88 -0
  27. package/dist/core/timeframe.js.map +1 -1
  28. package/dist/index.d.ts +1 -1
  29. package/dist/index.d.ts.map +1 -1
  30. package/dist/insights/advisors/fanout-health.d.ts +3 -0
  31. package/dist/insights/advisors/fanout-health.d.ts.map +1 -0
  32. package/dist/insights/advisors/fanout-health.js +3 -0
  33. package/dist/insights/advisors/fanout-health.js.map +1 -0
  34. package/dist/insights/advisors/map-health.d.ts +6 -0
  35. package/dist/insights/advisors/map-health.d.ts.map +1 -0
  36. package/dist/insights/advisors/map-health.js +154 -0
  37. package/dist/insights/advisors/map-health.js.map +1 -0
  38. package/dist/insights/advisors/resource-advisor.d.ts +4 -0
  39. package/dist/insights/advisors/resource-advisor.d.ts.map +1 -0
  40. package/dist/insights/advisors/resource-advisor.js +102 -0
  41. package/dist/insights/advisors/resource-advisor.js.map +1 -0
  42. package/dist/insights/advisors/retention-advisor.d.ts +2 -0
  43. package/dist/insights/advisors/retention-advisor.d.ts.map +1 -0
  44. package/dist/insights/advisors/retention-advisor.js +6 -0
  45. package/dist/insights/advisors/retention-advisor.js.map +1 -0
  46. package/dist/insights/advisors-subpath.d.ts +4 -0
  47. package/dist/insights/advisors-subpath.d.ts.map +1 -0
  48. package/dist/insights/advisors-subpath.js +10 -0
  49. package/dist/insights/advisors-subpath.js.map +1 -0
  50. package/dist/insights/anomaly/baseline.d.ts +4 -0
  51. package/dist/insights/anomaly/baseline.d.ts.map +1 -0
  52. package/dist/insights/anomaly/baseline.js +54 -0
  53. package/dist/insights/anomaly/baseline.js.map +1 -0
  54. package/dist/insights/anomaly/detect-anomalies.d.ts +6 -0
  55. package/dist/insights/anomaly/detect-anomalies.d.ts.map +1 -0
  56. package/dist/insights/anomaly/detect-anomalies.js +245 -0
  57. package/dist/insights/anomaly/detect-anomalies.js.map +1 -0
  58. package/dist/insights/anomaly/ewma.d.ts +2 -0
  59. package/dist/insights/anomaly/ewma.d.ts.map +1 -0
  60. package/dist/insights/anomaly/ewma.js +15 -0
  61. package/dist/insights/anomaly/ewma.js.map +1 -0
  62. package/dist/insights/anomaly/missing-activity.d.ts +3 -0
  63. package/dist/insights/anomaly/missing-activity.d.ts.map +1 -0
  64. package/dist/insights/anomaly/missing-activity.js +8 -0
  65. package/dist/insights/anomaly/missing-activity.js.map +1 -0
  66. package/dist/insights/anomaly/robust-zscore.d.ts +2 -0
  67. package/dist/insights/anomaly/robust-zscore.d.ts.map +1 -0
  68. package/dist/insights/anomaly/robust-zscore.js +9 -0
  69. package/dist/insights/anomaly/robust-zscore.js.map +1 -0
  70. package/dist/insights/anomaly-subpath.d.ts +7 -0
  71. package/dist/insights/anomaly-subpath.d.ts.map +1 -0
  72. package/dist/insights/anomaly-subpath.js +20 -0
  73. package/dist/insights/anomaly-subpath.js.map +1 -0
  74. package/dist/insights/comparison/compare-periods.d.ts +5 -0
  75. package/dist/insights/comparison/compare-periods.d.ts.map +1 -0
  76. package/dist/insights/comparison/compare-periods.js +64 -0
  77. package/dist/insights/comparison/compare-periods.js.map +1 -0
  78. package/dist/insights/comparison/growth-rate.d.ts +3 -0
  79. package/dist/insights/comparison/growth-rate.d.ts.map +1 -0
  80. package/dist/insights/comparison/growth-rate.js +3 -0
  81. package/dist/insights/comparison/growth-rate.js.map +1 -0
  82. package/dist/insights/comparison/moving-average.d.ts +2 -0
  83. package/dist/insights/comparison/moving-average.d.ts.map +1 -0
  84. package/dist/insights/comparison/moving-average.js +6 -0
  85. package/dist/insights/comparison/moving-average.js.map +1 -0
  86. package/dist/insights/comparison/trend.d.ts +5 -0
  87. package/dist/insights/comparison/trend.d.ts.map +1 -0
  88. package/dist/insights/comparison/trend.js +84 -0
  89. package/dist/insights/comparison/trend.js.map +1 -0
  90. package/dist/insights/errors.d.ts +25 -0
  91. package/dist/insights/errors.d.ts.map +1 -0
  92. package/dist/insights/errors.js +54 -0
  93. package/dist/insights/errors.js.map +1 -0
  94. package/dist/insights/forecast/budget.d.ts +2 -0
  95. package/dist/insights/forecast/budget.d.ts.map +1 -0
  96. package/dist/insights/forecast/budget.js +6 -0
  97. package/dist/insights/forecast/budget.js.map +1 -0
  98. package/dist/insights/forecast/capacity.d.ts +2 -0
  99. package/dist/insights/forecast/capacity.d.ts.map +1 -0
  100. package/dist/insights/forecast/capacity.js +3 -0
  101. package/dist/insights/forecast/capacity.js.map +1 -0
  102. package/dist/insights/forecast/forecast.d.ts +5 -0
  103. package/dist/insights/forecast/forecast.d.ts.map +1 -0
  104. package/dist/insights/forecast/forecast.js +103 -0
  105. package/dist/insights/forecast/forecast.js.map +1 -0
  106. package/dist/insights/forecast/retention.d.ts +2 -0
  107. package/dist/insights/forecast/retention.d.ts.map +1 -0
  108. package/dist/insights/forecast/retention.js +3 -0
  109. package/dist/insights/forecast/retention.js.map +1 -0
  110. package/dist/insights/index.d.ts +4 -0
  111. package/dist/insights/index.d.ts.map +1 -0
  112. package/dist/insights/index.js +23 -0
  113. package/dist/insights/index.js.map +1 -0
  114. package/dist/insights/insights-client.d.ts +33 -0
  115. package/dist/insights/insights-client.d.ts.map +1 -0
  116. package/dist/insights/insights-client.js +344 -0
  117. package/dist/insights/insights-client.js.map +1 -0
  118. package/dist/insights/llm-subpath.d.ts +4 -0
  119. package/dist/insights/llm-subpath.d.ts.map +1 -0
  120. package/dist/insights/llm-subpath.js +10 -0
  121. package/dist/insights/llm-subpath.js.map +1 -0
  122. package/dist/insights/optimization/cost-optimizer.d.ts +6 -0
  123. package/dist/insights/optimization/cost-optimizer.d.ts.map +1 -0
  124. package/dist/insights/optimization/cost-optimizer.js +98 -0
  125. package/dist/insights/optimization/cost-optimizer.js.map +1 -0
  126. package/dist/insights/optimization/latency-optimizer.d.ts +3 -0
  127. package/dist/insights/optimization/latency-optimizer.d.ts.map +1 -0
  128. package/dist/insights/optimization/latency-optimizer.js +3 -0
  129. package/dist/insights/optimization/latency-optimizer.js.map +1 -0
  130. package/dist/insights/optimization/threshold-advisor.d.ts +2 -0
  131. package/dist/insights/optimization/threshold-advisor.d.ts.map +1 -0
  132. package/dist/insights/optimization/threshold-advisor.js +3 -0
  133. package/dist/insights/optimization/threshold-advisor.js.map +1 -0
  134. package/dist/insights/optimization/token-optimizer.d.ts +4 -0
  135. package/dist/insights/optimization/token-optimizer.d.ts.map +1 -0
  136. package/dist/insights/optimization/token-optimizer.js +133 -0
  137. package/dist/insights/optimization/token-optimizer.js.map +1 -0
  138. package/dist/insights/ranking/contribution.d.ts +2 -0
  139. package/dist/insights/ranking/contribution.d.ts.map +1 -0
  140. package/dist/insights/ranking/contribution.js +6 -0
  141. package/dist/insights/ranking/contribution.js.map +1 -0
  142. package/dist/insights/ranking/helpers.d.ts +22 -0
  143. package/dist/insights/ranking/helpers.d.ts.map +1 -0
  144. package/dist/insights/ranking/helpers.js +88 -0
  145. package/dist/insights/ranking/helpers.js.map +1 -0
  146. package/dist/insights/ranking/pareto.d.ts +2 -0
  147. package/dist/insights/ranking/pareto.d.ts.map +1 -0
  148. package/dist/insights/ranking/pareto.js +6 -0
  149. package/dist/insights/ranking/pareto.js.map +1 -0
  150. package/dist/insights/ranking/slices.d.ts +2 -0
  151. package/dist/insights/ranking/slices.d.ts.map +1 -0
  152. package/dist/insights/ranking/slices.js +6 -0
  153. package/dist/insights/ranking/slices.js.map +1 -0
  154. package/dist/insights/ranking/top-bottom.d.ts +2 -0
  155. package/dist/insights/ranking/top-bottom.d.ts.map +1 -0
  156. package/dist/insights/ranking/top-bottom.js +7 -0
  157. package/dist/insights/ranking/top-bottom.js.map +1 -0
  158. package/dist/insights/recommendations/evidence.d.ts +2 -0
  159. package/dist/insights/recommendations/evidence.d.ts.map +1 -0
  160. package/dist/insights/recommendations/evidence.js +3 -0
  161. package/dist/insights/recommendations/evidence.js.map +1 -0
  162. package/dist/insights/recommendations/recommendation.d.ts +6 -0
  163. package/dist/insights/recommendations/recommendation.d.ts.map +1 -0
  164. package/dist/insights/recommendations/recommendation.js +25 -0
  165. package/dist/insights/recommendations/recommendation.js.map +1 -0
  166. package/dist/insights/recommendations/severity.d.ts +2 -0
  167. package/dist/insights/recommendations/severity.d.ts.map +1 -0
  168. package/dist/insights/recommendations/severity.js +3 -0
  169. package/dist/insights/recommendations/severity.js.map +1 -0
  170. package/dist/insights/rules/evaluate-rules.d.ts +5 -0
  171. package/dist/insights/rules/evaluate-rules.d.ts.map +1 -0
  172. package/dist/insights/rules/evaluate-rules.js +156 -0
  173. package/dist/insights/rules/evaluate-rules.js.map +1 -0
  174. package/dist/insights/types.d.ts +405 -0
  175. package/dist/insights/types.d.ts.map +1 -0
  176. package/dist/insights/types.js +3 -0
  177. package/dist/insights/types.js.map +1 -0
  178. package/dist/insights/utils/formatting.d.ts +4 -0
  179. package/dist/insights/utils/formatting.d.ts.map +1 -0
  180. package/dist/insights/utils/formatting.js +21 -0
  181. package/dist/insights/utils/formatting.js.map +1 -0
  182. package/dist/insights/utils/grouping.d.ts +2 -0
  183. package/dist/insights/utils/grouping.d.ts.map +1 -0
  184. package/dist/insights/utils/grouping.js +8 -0
  185. package/dist/insights/utils/grouping.js.map +1 -0
  186. package/dist/insights/utils/query-builder.d.ts +21 -0
  187. package/dist/insights/utils/query-builder.d.ts.map +1 -0
  188. package/dist/insights/utils/query-builder.js +107 -0
  189. package/dist/insights/utils/query-builder.js.map +1 -0
  190. package/dist/insights/utils/row-value.d.ts +5 -0
  191. package/dist/insights/utils/row-value.d.ts.map +1 -0
  192. package/dist/insights/utils/row-value.js +36 -0
  193. package/dist/insights/utils/row-value.js.map +1 -0
  194. package/dist/insights/utils/stats.d.ts +28 -0
  195. package/dist/insights/utils/stats.d.ts.map +1 -0
  196. package/dist/insights/utils/stats.js +125 -0
  197. package/dist/insights/utils/stats.js.map +1 -0
  198. package/dist/query/formatter.d.ts.map +1 -1
  199. package/dist/query/formatter.js +202 -24
  200. package/dist/query/formatter.js.map +1 -1
  201. package/dist/query/planner.d.ts +10 -2
  202. package/dist/query/planner.d.ts.map +1 -1
  203. package/dist/query/planner.js +39 -2
  204. package/dist/query/planner.js.map +1 -1
  205. package/dist/storage/memory.d.ts.map +1 -1
  206. package/dist/storage/memory.js +78 -4
  207. package/dist/storage/memory.js.map +1 -1
  208. package/dist/storage/mongo.d.ts.map +1 -1
  209. package/dist/storage/mongo.js +61 -16
  210. package/dist/storage/mongo.js.map +1 -1
  211. package/dist/storage/redis.d.ts +6 -7
  212. package/dist/storage/redis.d.ts.map +1 -1
  213. package/dist/storage/redis.js +108 -39
  214. package/dist/storage/redis.js.map +1 -1
  215. package/dist/types/index.d.ts +88 -4
  216. package/dist/types/index.d.ts.map +1 -1
  217. package/docs/INSIGHTS.md +59 -0
  218. package/docs/INSIGHTS_ANOMALY_DETECTION.md +33 -0
  219. package/docs/INSIGHTS_LLM_COST_AND_TOKENS.md +26 -0
  220. package/docs/INSIGHTS_RESOURCE_ADVISOR.md +31 -0
  221. package/docs/guides/README.md +29 -0
  222. package/docs/guides/adapter-memory.md +48 -0
  223. package/docs/guides/adapter-mongo.md +50 -0
  224. package/docs/guides/adapter-redis.md +47 -0
  225. package/docs/guides/backup-export.md +39 -0
  226. package/docs/guides/cardinality.md +50 -0
  227. package/docs/guides/cli.md +45 -0
  228. package/docs/guides/client-events.md +49 -0
  229. package/docs/guides/counter-maps.md +116 -0
  230. package/docs/guides/idempotency.md +39 -0
  231. package/docs/guides/ingestion-and-flush.md +67 -0
  232. package/docs/guides/insights-advisors-and-optimization.md +39 -0
  233. package/docs/guides/insights-anomalies-and-rules.md +48 -0
  234. package/docs/guides/insights-comparison-and-trends.md +47 -0
  235. package/docs/guides/insights-forecasting-and-budget.md +61 -0
  236. package/docs/guides/insights-overview.md +76 -0
  237. package/docs/guides/insights-ranking.md +81 -0
  238. package/docs/guides/measures.md +67 -0
  239. package/docs/guides/querying.md +79 -0
  240. package/docs/guides/timeframes-and-bucketing.md +55 -0
  241. package/docs/internals/README.md +20 -0
  242. package/docs/internals/internal-architecture-overview.md +65 -0
  243. package/docs/internals/internal-backup-stream.md +23 -0
  244. package/docs/internals/internal-buffer-merge-and-flush.md +51 -0
  245. package/docs/internals/internal-cardinality.md +30 -0
  246. package/docs/internals/internal-idempotency.md +36 -0
  247. package/docs/internals/internal-ingest-pipeline.md +52 -0
  248. package/docs/internals/internal-insights-layer.md +36 -0
  249. package/docs/internals/internal-mapping-validation-and-extraction.md +59 -0
  250. package/docs/internals/internal-measures-and-scales.md +35 -0
  251. package/docs/internals/internal-query-path.md +46 -0
  252. package/docs/internals/internal-storage-memory.md +13 -0
  253. package/docs/internals/internal-storage-mongo.md +24 -0
  254. package/docs/internals/internal-storage-redis.md +31 -0
  255. package/docs/internals/internal-timeframe-bucketing.md +33 -0
  256. package/package.json +45 -2
@@ -0,0 +1,67 @@
1
+ # Guide: Measures
2
+
3
+ **Measures** are optional numeric fields per event (cost, latency, tokens, …) aggregated alongside counts.
4
+
5
+ ## Step 1: Declare measures on the map
6
+
7
+ ```ts
8
+ const map = {
9
+ counterId: "llm_usage",
10
+ scope: { path: "$.scope.id" },
11
+ eventTime: { path: "$.request.createdAt" },
12
+ subjects: [{ type: "app", path: "$.app.id", required: true }],
13
+ metrics: [{ path: "$.request.status", required: true }],
14
+ dimensions: [{ name: "model", path: "$.model.name" }],
15
+ delta: { default: 1 },
16
+ timeframes: ["day"],
17
+ measures: [
18
+ { name: "cost", path: "$.usage.costCents", aggregations: ["sum"], scale: 100 },
19
+ { name: "latencyMs", path: "$.usage.latencyMs", aggregations: ["sum", "count", "min", "max"] },
20
+ ],
21
+ } as const;
22
+ ```
23
+
24
+ ### Aggregations
25
+
26
+ - **`sum`**, **`count`**, **`min`**, **`max`** are stored.
27
+ - **`avg`** is derived at query time when both **`sum`** and **`count`** exist for that measure.
28
+
29
+ ### Scale
30
+
31
+ `scale` multiplies then truncates for fixed-point storage (for example cents → integer micros).
32
+
33
+ ### Missing values
34
+
35
+ Use `onMissing: "skip"` (default) or `"drop"` / `"zero"` per measure.
36
+
37
+ ## Step 2: Ingest as usual
38
+
39
+ Each fan-out increment carries the **same** measure snapshot for that event.
40
+
41
+ ## Step 3: Query measures
42
+
43
+ ```ts
44
+ const result = await countex.query({
45
+ scope: "org_1",
46
+ counterId: "llm_usage",
47
+ timeframe: { type: "day", from: "2026-05-08", to: "2026-05-08" },
48
+ groupBy: ["subject", "metric"],
49
+ measures: ["cost", "latencyMs"],
50
+ format: "table",
51
+ });
52
+ ```
53
+
54
+ Use `measures: "*"` to include all declared measures.
55
+
56
+ ### Ordering
57
+
58
+ `orderBy` can target measure aggregations (for example sort by `latencyMs` max).
59
+
60
+ ### Cache vs durable
61
+
62
+ Redis keeps **sum** and **count** for measures. **min** / **max** live on durable; the query planner skips cache when min/max are required.
63
+
64
+ ## See also
65
+
66
+ - [Querying](./querying.md)
67
+ - Internal: [Measures & scales](../internals/internal-measures-and-scales.md)
@@ -0,0 +1,79 @@
1
+ # Guide: Querying
2
+
3
+ `countex.query(spec)` reads aggregated cells for a scope and counter, with optional filters and output shaping.
4
+
5
+ ## Step 1: Minimal query
6
+
7
+ ```ts
8
+ const result = await countex.query({
9
+ scope: "org_1",
10
+ counterId: "access_by_actor_label",
11
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
12
+ });
13
+ ```
14
+
15
+ ## Step 2: Filter
16
+
17
+ ```ts
18
+ const result = await countex.query({
19
+ scope: "org_1",
20
+ counterId: "access_by_actor_label",
21
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
22
+ subject: { type: "actor", ids: [10001, 10002] },
23
+ metrics: [41, 42],
24
+ dimensions: { operation: ["read", "write"] },
25
+ });
26
+ ```
27
+
28
+ Subject and metric ids are **numeric dictionary ids** resolved during ingest.
29
+
30
+ ## Step 3: Group and format
31
+
32
+ `groupBy` accepts:
33
+
34
+ - `subject`, `subjectType`, `metric`, `timeframeKey`
35
+ - `dim:NAME` for a dimension
36
+
37
+ `format`:
38
+
39
+ - `cells` — raw-ish cell list
40
+ - `table` / `matrix` — aggregated rows
41
+ - `timeseries` — points per bucket (and optional extra grouping)
42
+
43
+ ```ts
44
+ const result = await countex.query({
45
+ scope: "org_1",
46
+ counterId: "access_by_actor_label",
47
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-07" },
48
+ groupBy: ["subject", "dim:operation"],
49
+ format: "table",
50
+ orderBy: "-count",
51
+ limit: 50,
52
+ });
53
+ ```
54
+
55
+ ## Step 4: Measures and rates
56
+
57
+ ```ts
58
+ await countex.query({
59
+ // ...
60
+ measures: ["cost"],
61
+ orderBy: { measure: "cost", agg: "sum", dir: "desc" },
62
+ rate: { measure: "cost", agg: "sum" },
63
+ });
64
+ ```
65
+
66
+ Do not use `rate` with timeframe **`all`**.
67
+
68
+ ## Step 5: Cache control
69
+
70
+ - `bypassCache: true` forces durable reads.
71
+ - `signal` accepts `AbortSignal` for timeouts.
72
+
73
+ Result includes `source`: `"cache"`, `"durable"`, or `"merged"`.
74
+
75
+ ## See also
76
+
77
+ - [Measures](./measures.md)
78
+ - [Timeframes](./timeframes-and-bucketing.md)
79
+ - Internal: [Query path](../internals/internal-query-path.md)
@@ -0,0 +1,55 @@
1
+ # Guide: Timeframes and bucketing
2
+
3
+ Timeframes control **how event time is bucketed** when Countex writes cells. Queries later ask for a **range of bucket keys** (`from` / `to`).
4
+
5
+ ## Step 1: Pick timeframe types on the map
6
+
7
+ ```ts
8
+ const map = {
9
+ counterId: "events",
10
+ // ...
11
+ timeframes: ["minute", "hour", "day", "week", "month", "quarter", "year", "all"],
12
+ } as const;
13
+ ```
14
+
15
+ Only listed timeframes receive writes for each matching event.
16
+
17
+ ## Step 2: Understand bucket keys (UTC)
18
+
19
+ | Type | Example key |
20
+ |------|----------------|
21
+ | minute | `2026-05-08T12:30` |
22
+ | hour | `2026-05-08T12` |
23
+ | day | `2026-05-08` |
24
+ | week | `2026-W19` (ISO week) |
25
+ | month | `2026-05` |
26
+ | quarter | `2026-Q2` |
27
+ | year | `2026` |
28
+ | all | `all` |
29
+
30
+ Buckets are derived from **`eventTime`**, not ingestion time, so late data lands in the correct historical bucket.
31
+
32
+ ## Step 3: Query a range
33
+
34
+ ```ts
35
+ const result = await countex.query({
36
+ scope: "org_1",
37
+ counterId: "events",
38
+ timeframe: { type: "day", from: "2026-05-01", to: "2026-05-08" },
39
+ format: "table",
40
+ });
41
+ ```
42
+
43
+ ## Step 4: Respect the bucket cap
44
+
45
+ Range expansion is capped at **10,000** buckets. Narrow the window or coarsen the timeframe if you hit limits.
46
+
47
+ ## Step 5: Rates
48
+
49
+ `rate` in queries divides counts or measure sums by elapsed seconds in the queried window. **`rate` is invalid when timeframe type is `all`** (no meaningful duration).
50
+
51
+ ## See also
52
+
53
+ - [Querying](./querying.md)
54
+ - [Measures](./measures.md)
55
+ - Internal: [Timeframe bucketing](../internals/internal-timeframe-bucketing.md)
@@ -0,0 +1,20 @@
1
+ # Countex internals
2
+
3
+ Deep dives into how the engine executes ingest, flush, storage IO, and insights post-processing. These documents are **not** imported as package subpaths; they ship in the npm tarball under `docs/internals/` for repository readers and advanced operators.
4
+
5
+ Start here:
6
+
7
+ | Topic | Document |
8
+ |-------|----------|
9
+ | End-to-end architecture | [Architecture overview](./internal-architecture-overview.md) |
10
+ | Map startup validation & JSON extraction | [Mapping validation & extraction](./internal-mapping-validation-and-extraction.md) |
11
+ | `ingestBatch` phases | [Ingest pipeline](./internal-ingest-pipeline.md) |
12
+ | In-memory merge | [Buffer merge & flush](./internal-buffer-merge-and-flush.md) |
13
+ | Reads | [Query path](./internal-query-path.md) |
14
+ | Bucket keys | [Timeframe bucketing](./internal-timeframe-bucketing.md) |
15
+ | Measures | [Measures & scales](./internal-measures-and-scales.md) |
16
+ | Cardinality | [Cardinality guard](./internal-cardinality.md) |
17
+ | Dedupe | [Idempotency](./internal-idempotency.md) |
18
+ | Adapters | [Mongo](./internal-storage-mongo.md), [Redis](./internal-storage-redis.md), [Memory](./internal-storage-memory.md) |
19
+ | Ops helpers | [Backup streaming](./internal-backup-stream.md) |
20
+ | Insights | [Insights layer](./internal-insights-layer.md) |
@@ -0,0 +1,65 @@
1
+ # Internal: Architecture overview
2
+
3
+ Countex separates **projection** (events → sparse cells), **durability** (authoritative totals), **cache** (hot reads + dedupe), and **query shaping**.
4
+
5
+ ## Component diagram
6
+
7
+ ```mermaid
8
+ flowchart LR
9
+ subgraph app [Host application]
10
+ CE[Countex client]
11
+ end
12
+ subgraph core [Core engine]
13
+ MAP[CounterMap registry]
14
+ EXT[extractIncrements]
15
+ CARD[CardinalityGuard]
16
+ BUF[DeltaBuffer]
17
+ PLAN[executeQuery]
18
+ FMT[formatResult]
19
+ end
20
+ subgraph io [Storage]
21
+ DUR[DurableStorage]
22
+ CACHE[CacheStorage]
23
+ end
24
+ CE --> MAP
25
+ CE --> EXT
26
+ EXT --> CARD
27
+ CARD --> DUR
28
+ CARD --> BUF
29
+ BUF --> DUR
30
+ BUF --> CACHE
31
+ CE --> PLAN
32
+ PLAN --> CACHE
33
+ PLAN --> DUR
34
+ PLAN --> FMT
35
+ ```
36
+
37
+ ## Write trace (conceptual)
38
+
39
+ 1. **Validate maps** at `createCountex` construction (`validateMaps`).
40
+ 2. **Extract** raw increments per map/event (`extractIncrements`).
41
+ 3. **Dedupe** optional (`cache.markEventSeen`).
42
+ 4. **Cardinality** rewrite or drop (`CardinalityGuard.check`).
43
+ 5. **Dictionary** allocate numeric ids (`durable.resolveIds`).
44
+ 6. **Buffer** merge by hashed cell identity (`DeltaBuffer.add`).
45
+ 7. **Flush** snapshot → `durable.applyDeltas` → best-effort `cache.applyDeltas`.
46
+
47
+ ## Read trace (conceptual)
48
+
49
+ 1. **Planner** tries cache when allowed (`cacheCanServe`).
50
+ 2. On partial/miss, **durable.readCells**.
51
+ 3. Optional **cache.warm** with durable rows.
52
+ 4. **Formatter** applies `groupBy`, measures, rates, limits.
53
+
54
+ ## Source anchors
55
+
56
+ - Client orchestration: `src/core/countex.ts`
57
+ - Extraction: `src/core/extraction.ts`
58
+ - Buffer: `src/core/buffer.ts`
59
+ - Planner: `src/query/planner.ts`
60
+ - Formatter: `src/query/formatter.ts`
61
+
62
+ ## See also
63
+
64
+ - [Ingest pipeline](./internal-ingest-pipeline.md)
65
+ - [Query path](./internal-query-path.md)
@@ -0,0 +1,23 @@
1
+ # Internal: Backup streaming
2
+
3
+ `exportCells` and `rebuildCache` (`src/backup/export.ts`) are thin wrappers over `durable.streamCells`:
4
+
5
+ ```mermaid
6
+ flowchart LR
7
+ DUR[durable.streamCells filter] --> EXPORT[AsyncIterable CounterCell]
8
+ DUR --> BATCH[batch warm loop]
9
+ BATCH --> CACHE[cache.warm batches]
10
+ ```
11
+
12
+ ## Contract expectations
13
+
14
+ Implementations must honor:
15
+
16
+ - Scope/counter/timeframe bounding
17
+ - Optional `AbortSignal` propagation
18
+
19
+ `rebuildCache` batches to limit Redis/Mongo round trips during warm operations.
20
+
21
+ ## See also
22
+
23
+ - [Mongo durable](./internal-storage-mongo.md)
@@ -0,0 +1,51 @@
1
+ # Internal: Buffer merge & flush
2
+
3
+ ## DeltaBuffer structure
4
+
5
+ `DeltaBuffer` (`src/core/buffer.ts`) stores:
6
+
7
+ - Map key: string from `cellKey(identity)`
8
+ - Value: `{ identity, delta, measures?, measureScales? }`
9
+
10
+ ## Merge semantics
11
+
12
+ `add` merges:
13
+
14
+ - **Counts** — integer delta summed.
15
+ - **Measures** — per aggregation field:
16
+ - `sum`, `count` additive
17
+ - `min` / `max` rolling extrema
18
+
19
+ `measureScales` merge retains first-seen scale per measure name (consistent with map declaration).
20
+
21
+ ## Flush lifecycle
22
+
23
+ ```mermaid
24
+ stateDiagram-v2
25
+ [*] --> Idle
26
+ Idle --> Draining: flush() & buffer non-empty
27
+ Draining --> DurableWrite: snapshot taken
28
+ DurableWrite --> CacheWrite: durable OK
29
+ DurableWrite --> MergeBack: durable error
30
+ MergeBack --> Idle
31
+ CacheWrite --> Idle: cache success or logged warn
32
+ ```
33
+
34
+ `flush()` returns the **same promise** while in-flight (`flushing` lock) so concurrent callers coalesce.
35
+
36
+ ### Durable failure
37
+
38
+ Snapshot is **`mergeBack`** into the buffer so counts are not lost; `flush_error` emits.
39
+
40
+ ### Cache failure
41
+
42
+ Logged as non-fatal — durable already committed.
43
+
44
+ ## Event accounting
45
+
46
+ `noteEvent()` increments an event counter used solely for flush thresholds (reset implicitly when draining — payload uses duration only).
47
+
48
+ ## See also
49
+
50
+ - [Ingest pipeline](./internal-ingest-pipeline.md)
51
+ - [Measures & scales](./internal-measures-and-scales.md)
@@ -0,0 +1,30 @@
1
+ # Internal: Cardinality guard
2
+
3
+ `CardinalityGuard` (`src/core/cardinality.ts`) tracks distinct source keys per `(scopeId, kind, qualifier)` using in-memory `Set`s.
4
+
5
+ ## Check algorithm
6
+
7
+ ```mermaid
8
+ flowchart TD
9
+ IN[sourceKey] --> KEY[keyFor scope/kind/qualifier]
10
+ KEY --> SET{Set contains key?}
11
+ SET -->|yes| OK[return original or existing overflow mapping]
12
+ SET -->|no| LIM{count < limit?}
13
+ LIM -->|yes| ADD[add to Set -> return key]
14
+ LIM -->|no| POL{onLimit}
15
+ POL -->|overflow| OVF[emit overflow event -> return sentinel]
16
+ POL -->|drop| DROP[return null]
17
+ POL -->|error| ERR[throw CardinalityLimitError]
18
+ ```
19
+
20
+ ## Dictionary kinds
21
+
22
+ Kinds align with subject/metric/dimension value namespaces; dimension checks include the dimension **name** as qualifier.
23
+
24
+ ## Approximation warning
25
+
26
+ Commentary in source notes multi-worker deployments see approximate counts — acceptable because the guard targets runaway growth, not strict quotas.
27
+
28
+ ## See also
29
+
30
+ - [Ingest pipeline](./internal-ingest-pipeline.md)
@@ -0,0 +1,36 @@
1
+ # Internal: Idempotency
2
+
3
+ Idempotency spans three layers:
4
+
5
+ 1. **Map config** — `idempotency.enabled` + `eventIdPath`.
6
+ 2. **Extraction** — resolves optional `eventId` string (`src/core/extraction.ts`).
7
+ 3. **Ingest gate** — parallel `cache.markEventSeen` before buffering (`src/core/countex.ts`).
8
+
9
+ ## Sequence
10
+
11
+ ```mermaid
12
+ sequenceDiagram
13
+ participant CX as Countex
14
+ participant EXT as extractIncrements
15
+ participant CACHE as CacheStorage
16
+ participant BUF as DeltaBuffer
17
+ CX->>EXT: increments + eventId
18
+ CX->>CACHE: markEventSeen(scope, counterId:eventId, ttl)
19
+ CACHE-->>CX: isNew boolean
20
+ alt duplicate
21
+ CX->>CX: emit duplicate
22
+ CX->>CX: zero increments for this map slice
23
+ else new
24
+ CX->>BUF: add deltas
25
+ end
26
+ ```
27
+
28
+ Namespace **`counterId:`** prefix prevents collisions across maps sharing physical cache.
29
+
30
+ ## Failure modes
31
+
32
+ Missing event ids produce warnings during extraction but do not automatically dedupe unless your logic skips ingest.
33
+
34
+ ## See also
35
+
36
+ - [Ingest pipeline](./internal-ingest-pipeline.md)
@@ -0,0 +1,52 @@
1
+ # Internal: Ingest pipeline (`ingestBatch`)
2
+
3
+ `Countex.ingest` delegates to `ingestBatch` (`src/core/countex.ts`). The implementation is intentionally phased so CPU work stays synchronous while IO stays bounded.
4
+
5
+ ## Sequence diagram
6
+
7
+ ```mermaid
8
+ sequenceDiagram
9
+ participant App
10
+ participant CX as Countex
11
+ participant EXT as extractIncrements
12
+ participant CACHE as CacheStorage
13
+ participant CARD as CardinalityGuard
14
+ participant DUR as DurableStorage
15
+ participant BUF as DeltaBuffer
16
+ App->>CX: ingestBatch(events)
17
+ loop each event × map
18
+ CX->>EXT: extractIncrements(map, ev)
19
+ EXT-->>CX: RawIncrement[]
20
+ end
21
+ CX->>CACHE: markEventSeen (optional, parallel)
22
+ CACHE-->>CX: boolean flags
23
+ CX->>CARD: check subject/metric/dimensions
24
+ CARD-->>CX: rewritten keys or drop
25
+ CX->>DUR: resolveIds(lookups)
26
+ DUR-->>CX: Map lookup → numeric id
27
+ loop each RawIncrement × timeframe
28
+ CX->>BUF: add(identity, delta, measures)
29
+ end
30
+ CX->>CX: maybe flush() on thresholds
31
+ ```
32
+
33
+ ## Phase notes
34
+
35
+ 1. **Extract** — warnings for measures surface as `measure_invalid` events when meta is present.
36
+ 2. **Dedupe** — `markEventSeen` receives `${counterId}:${eventId}` to isolate maps.
37
+ 3. **Cardinality** — dimension iteration may drop with `dropped` reason `cardinality:dimension`.
38
+ 4. **Dictionary** — only **subjects** and **metrics** resolve via `resolveIds`; dimension values remain literal on cells.
39
+ 5. **Buffer key** — identity includes `dimensionsHash` (`src/utils/hash.ts`).
40
+ 6. **Timeframe expansion** — `bucketKey(tf, eventTime)` invoked per configured timeframe.
41
+
42
+ ## Flush coupling
43
+
44
+ Threshold triggers call `flush()`:
45
+
46
+ - `onOverflow: "block"` awaits flush (back-pressure).
47
+ - `"drop"` schedules flush without awaiting.
48
+
49
+ ## See also
50
+
51
+ - [Buffer merge & flush](./internal-buffer-merge-and-flush.md)
52
+ - [Cardinality](./internal-cardinality.md)
@@ -0,0 +1,36 @@
1
+ # Internal: Insights layer
2
+
3
+ `CountexInsights` (`src/insights/insights-client.ts`) wraps an existing `Countex` instance.
4
+
5
+ ## Pattern
6
+
7
+ ```mermaid
8
+ flowchart LR
9
+ IN[Insight input] --> QB[baseToQuerySpec +\nparseMeasureSelector]
10
+ QB --> Q[countex.query]
11
+ Q --> POST[ranking / anomaly /\nforecast helpers]
12
+ POST --> OUT[typed result]
13
+ ```
14
+
15
+ Shared utilities:
16
+
17
+ - `utils/query-builder.ts` translates insight inputs → `QuerySpec`.
18
+ - Ranking helpers sort rows, compute totals, enforce `maxGroups`.
19
+ - Anomaly pipeline fetches **eval** + **baseline** timeseries concurrently (`detectAnomalies`).
20
+ - Forecast/budget/token modules emit recommendations and bubble events.
21
+
22
+ ## Event bridge
23
+
24
+ Methods emit:
25
+
26
+ - `insight_query` — performance telemetry
27
+ - `anomaly`, `recommendation`, `insufficient_data` — operator-facing signals
28
+
29
+ ## Resource advisor
30
+
31
+ `estimateResourcesFromActuals` bypasses normal query formatting and streams cells via `durable.streamCells`, requiring the durable reference injected at construction.
32
+
33
+ ## See also
34
+
35
+ - [Query path](./internal-query-path.md)
36
+ - `docs/INSIGHTS.md` for product-level narrative
@@ -0,0 +1,59 @@
1
+ # Internal: Mapping validation & extraction
2
+
3
+ ## Startup validation
4
+
5
+ `validateMaps` runs once inside the `Countex` constructor (`src/core/mapping.ts`):
6
+
7
+ - Ensures unique `counterId` values.
8
+ - Parses every configured JSONPath with `parsePath`.
9
+ - Verifies required structural rules (non-empty subjects/metrics/timeframes, valid timeframe enums, measure aggregation enums, idempotency consistency).
10
+
11
+ Failures throw **`MappingError`** immediately.
12
+
13
+ ## Extraction pipeline
14
+
15
+ `extractIncrements(map, event)` (`src/core/extraction.ts`) executes in order:
16
+
17
+ ```mermaid
18
+ flowchart TD
19
+ A[extract scope] -->|missing| SKIP[skipped event]
20
+ A --> B[extract eventTime]
21
+ B -->|invalid| SKIP
22
+ B --> C[optional eventId for idempotency]
23
+ C --> D[resolve delta]
24
+ D -->|zero| SKIP
25
+ D --> E[fan-out subjects]
26
+ E -->|none| SKIP
27
+ E --> F[fan-out metrics]
28
+ F -->|none| SKIP
29
+ F --> G[dimensions]
30
+ G -->|required missing| SKIP
31
+ G --> H[measures]
32
+ H -->|drop policy| SKIP
33
+ H --> I[cross product increments]
34
+ ```
35
+
36
+ ### Fan-out
37
+
38
+ Nested loops build `RawIncrement` rows:
39
+
40
+ ```
41
+ for subject in subjectFanout:
42
+ for metricKey in metricFanout:
43
+ push RawIncrement(... timeframes: map.timeframes ...)
44
+ ```
45
+
46
+ Each increment carries **copies** of dimensions and measures for that Cartesian tuple.
47
+
48
+ ### Measures
49
+
50
+ `extractMeasuresForEvent` coerces numbers, applies `scale` truncation rules, and maps declared aggregations into pending contributions via `rawMeasuresToPendingDelta` during buffering.
51
+
52
+ ### JSON paths
53
+
54
+ `extract` (`src/utils/jsonpath.ts`) evaluates the supported subset: properties, indices, and `[*]` expansion.
55
+
56
+ ## See also
57
+
58
+ - [Ingest pipeline](./internal-ingest-pipeline.md)
59
+ - [Measures & scales](./internal-measures-and-scales.md)
@@ -0,0 +1,35 @@
1
+ # Internal: Measures & scales
2
+
3
+ ## Effective aggregations
4
+
5
+ `effectiveMeasureAggregations` (`src/core/extraction.ts`) ensures **`count`** exists whenever **`sum`**, **`min`**, or **`max`** are present so averages remain computable later.
6
+
7
+ ## Pending deltas
8
+
9
+ `rawMeasuresToPendingDelta` converts each extracted measure into:
10
+
11
+ - `sum` contribution
12
+ - `count` increment (`1` when enabled)
13
+ - `min` / `max` seeds
14
+
15
+ These mirror what storage adapters must merge atomically.
16
+
17
+ ## Buffer merging
18
+
19
+ `mergeMeasureDelta` (`src/core/buffer.ts`) applies associative merges identical to eventual durable semantics.
20
+
21
+ ## Scale metadata
22
+
23
+ Maps may declare `scale` per measure. During ingest, numeric extraction truncates after multiply; `measureScales` propagate into buffer entries so adapters interpret stored integers consistently.
24
+
25
+ ## Query-time averages
26
+
27
+ Formatter / row builders derive **`avg = sum / count`** when both components exist.
28
+
29
+ ## Cache limitation
30
+
31
+ Planner denies cache for queries needing min/max fidelity — see `queryNeedsMinMax` in `src/query/planner.ts`.
32
+
33
+ ## See also
34
+
35
+ - [Query path](./internal-query-path.md)
@@ -0,0 +1,46 @@
1
+ # Internal: Query path
2
+
3
+ `Countex.query` delegates to `executeQuery` (`src/query/planner.ts`) with optional measure metadata from `buildMeasureAggIndex(map)`.
4
+
5
+ ## Decision flow
6
+
7
+ ```mermaid
8
+ flowchart TD
9
+ Q[QuerySpec] --> C{cache configured\nand not bypassed\nand cacheCanServe?}
10
+ C -->|yes| R1[cache.readCells]
11
+ R1 -->|coverage full| F[formatResult source=cache]
12
+ R1 -->|partial/miss| D[durable.readCells]
13
+ C -->|no| D
14
+ D --> W{cache && rows && not bypass?}
15
+ W -->|yes| WARM[cache.warm best-effort]
16
+ W -->|no| F2[formatResult source=merged or durable]
17
+ WARM --> F2
18
+ ```
19
+
20
+ ## Cache eligibility (`cacheCanServe`)
21
+
22
+ When the counter map is known, queries that require **min/max** measure accuracy skip cache:
23
+
24
+ - `orderBy` targeting min/max aggregation.
25
+ - Explicit measure list referencing min/max-maintained measures.
26
+ - `measures: "*"` while any declared measure tracks min/max.
27
+
28
+ This avoids stale extrema because Redis stores sum/count only for measures.
29
+
30
+ ## Formatter responsibilities (`formatResult`)
31
+
32
+ `src/query/formatter.ts`:
33
+
34
+ - Groups cells by `groupBy` keys.
35
+ - Builds `table` / `matrix` / `timeseries` projections.
36
+ - Applies **`rate`** using `bucketDurationSeconds` except timeframe `all`.
37
+ - Sorts and truncates via `orderBy` / `limit`.
38
+
39
+ ## Bucket expansion
40
+
41
+ Range queries call `bucketRange` (`src/core/timeframe.ts`) capped at **10,000** buckets.
42
+
43
+ ## See also
44
+
45
+ - [Timeframe bucketing](./internal-timeframe-bucketing.md)
46
+ - [Measures & scales](./internal-measures-and-scales.md)
@@ -0,0 +1,13 @@
1
+ # Internal: Memory adapters
2
+
3
+ `MemoryDurable` and `MemoryCache` (`src/storage/memory.ts`) mirror production contracts using `Map` structures:
4
+
5
+ - Dictionary resolution mimics monotonic ids.
6
+ - Cells keyed similarly to durable identity tuples.
7
+ - Cache implements simplified coverage semantics for tests.
8
+
9
+ Used heavily by unit tests (`tests/unit/*.test.ts`) because they require no external services.
10
+
11
+ ## See also
12
+
13
+ - [Architecture overview](./internal-architecture-overview.md)