@x12i/countex 1.2.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +80 -18
- package/dist/backup/manifest.d.ts +70 -0
- package/dist/backup/manifest.d.ts.map +1 -0
- package/dist/backup/manifest.js +131 -0
- package/dist/backup/manifest.js.map +1 -0
- package/dist/backup/snapshot-encoding.d.ts +8 -0
- package/dist/backup/snapshot-encoding.d.ts.map +1 -0
- package/dist/backup/snapshot-encoding.js +27 -0
- package/dist/backup/snapshot-encoding.js.map +1 -0
- package/dist/cli.js +16 -0
- package/dist/cli.js.map +1 -1
- package/dist/core/countex.d.ts +79 -1
- package/dist/core/countex.d.ts.map +1 -1
- package/dist/core/countex.js +254 -49
- package/dist/core/countex.js.map +1 -1
- package/dist/core/extraction.d.ts +1 -5
- package/dist/core/extraction.d.ts.map +1 -1
- package/dist/core/extraction.js.map +1 -1
- package/dist/core/mapping.d.ts +5 -0
- package/dist/core/mapping.d.ts.map +1 -1
- package/dist/core/mapping.js +27 -0
- package/dist/core/mapping.js.map +1 -1
- package/dist/index.d.ts +5 -2
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +15 -1
- package/dist/index.js.map +1 -1
- package/dist/query/formatter.d.ts +2 -2
- package/dist/query/formatter.d.ts.map +1 -1
- package/dist/query/formatter.js +9 -6
- package/dist/query/formatter.js.map +1 -1
- package/dist/query/planner.d.ts +15 -1
- package/dist/query/planner.d.ts.map +1 -1
- package/dist/query/planner.js +152 -3
- package/dist/query/planner.js.map +1 -1
- package/dist/simulation/context.d.ts +5 -0
- package/dist/simulation/context.d.ts.map +1 -0
- package/dist/simulation/context.js +56 -0
- package/dist/simulation/context.js.map +1 -0
- package/dist/simulation/events.d.ts +3 -0
- package/dist/simulation/events.d.ts.map +1 -0
- package/dist/simulation/events.js +67 -0
- package/dist/simulation/events.js.map +1 -0
- package/dist/simulation/index.d.ts +7 -0
- package/dist/simulation/index.d.ts.map +1 -0
- package/dist/simulation/index.js +16 -0
- package/dist/simulation/index.js.map +1 -0
- package/dist/simulation/prng.d.ts +4 -0
- package/dist/simulation/prng.d.ts.map +1 -0
- package/dist/simulation/prng.js +18 -0
- package/dist/simulation/prng.js.map +1 -0
- package/dist/simulation/resource-estimate.d.ts +6 -0
- package/dist/simulation/resource-estimate.d.ts.map +1 -0
- package/dist/simulation/resource-estimate.js +11 -0
- package/dist/simulation/resource-estimate.js.map +1 -0
- package/dist/simulation/simulator.d.ts +15 -0
- package/dist/simulation/simulator.d.ts.map +1 -0
- package/dist/simulation/simulator.js +67 -0
- package/dist/simulation/simulator.js.map +1 -0
- package/dist/simulation/tracker.d.ts +8 -0
- package/dist/simulation/tracker.d.ts.map +1 -0
- package/dist/simulation/tracker.js +18 -0
- package/dist/simulation/tracker.js.map +1 -0
- package/dist/simulation/types.d.ts +49 -0
- package/dist/simulation/types.d.ts.map +1 -0
- package/dist/simulation/types.js +21 -0
- package/dist/simulation/types.js.map +1 -0
- package/dist/storage/coverage.d.ts +13 -0
- package/dist/storage/coverage.d.ts.map +1 -0
- package/dist/storage/coverage.js +37 -0
- package/dist/storage/coverage.js.map +1 -0
- package/dist/storage/memory.d.ts +23 -5
- package/dist/storage/memory.d.ts.map +1 -1
- package/dist/storage/memory.js +162 -3
- package/dist/storage/memory.js.map +1 -1
- package/dist/storage/mongo.d.ts +5 -0
- package/dist/storage/mongo.d.ts.map +1 -1
- package/dist/storage/mongo.js +68 -1
- package/dist/storage/mongo.js.map +1 -1
- package/dist/storage/redis.d.ts +40 -11
- package/dist/storage/redis.d.ts.map +1 -1
- package/dist/storage/redis.js +202 -44
- package/dist/storage/redis.js.map +1 -1
- package/dist/types/index.d.ts +165 -7
- package/dist/types/index.d.ts.map +1 -1
- package/docs/DEPLOYMENT.md +125 -3
- package/docs/countex_gap_analysis_complete_checklist.md +1340 -0
- package/docs/guides/README.md +5 -0
- package/docs/guides/adapter-mongo.md +12 -0
- package/docs/guides/adapter-redis.md +46 -4
- package/docs/guides/backup-export.md +106 -10
- package/docs/guides/cli.md +8 -0
- package/docs/guides/counter-maps.md +14 -0
- package/docs/guides/insights-overview.md +6 -0
- package/docs/guides/measures.md +4 -0
- package/docs/guides/preview.md +92 -0
- package/docs/guides/querying.md +70 -0
- package/docs/guides/simulation-profiles.md +33 -0
- package/docs/guides/simulation-resource-estimation.md +30 -0
- package/docs/guides/simulation.md +36 -0
- package/docs/internals/README.md +1 -0
- package/docs/internals/internal-dimension-hash.md +34 -0
- package/docs/internals/internal-storage-mongo.md +24 -4
- package/docs/internals/internal-storage-redis.md +41 -9
- package/docs/use-cases/README.md +56 -0
- package/docs/use-cases/audit-security-log-counts.md +550 -0
- package/docs/use-cases/llm-cost-and-token-metering.md +255 -0
- package/docs/use-cases/multi-tenant-isolation.md +203 -0
- package/docs/use-cases/ops-dashboard-daily-rollups.md +255 -0
- package/docs/use-cases/processing-workflow-activity.md +165 -0
- package/docs/use-cases/product-feature-usage.md +240 -0
- package/docs/use-cases/webhook-idempotent-ingestion.md +188 -0
- package/package.json +18 -2
|
@@ -4,21 +4,41 @@
|
|
|
4
4
|
|
|
5
5
|
- `applyDeltas` — upserts with `$inc` style operations for counts and compatible measure fields.
|
|
6
6
|
- `readCells` — satisfies `QuerySpec` filters using compound indexes.
|
|
7
|
-
- `resolveIds` —
|
|
7
|
+
- `resolveIds` — concurrency-safe allocation of dictionary ids (`countex_dict`, `countex_seq`).
|
|
8
8
|
- `streamCells` — bounded iteration for backup/export.
|
|
9
|
+
- `importSnapshotCells` — replace-window restore target (called by `restoreSnapshotCells`).
|
|
9
10
|
|
|
10
11
|
## Write path interaction
|
|
11
12
|
|
|
12
|
-
Flush snapshots arrive as arrays of `PendingDelta`; Mongo applies them idempotently per cell identity tuple
|
|
13
|
+
Flush snapshots arrive as arrays of `PendingDelta`; Mongo applies them idempotently per cell identity tuple via `bulkWrite` of `updateOne` operations with `$inc` and `upsert: true`.
|
|
13
14
|
|
|
14
15
|
## Indexes
|
|
15
16
|
|
|
16
|
-
Created lazily on first write paths
|
|
17
|
+
Created lazily on first write paths:
|
|
18
|
+
|
|
19
|
+
- `countex_cells` — `uniq_cell_identity` (unique on the full identity tuple), `by_timeframe`, `by_subject`, `by_metric`.
|
|
20
|
+
- `countex_dict` — `uniq_dict_key` (unique on `scopeId, kind, qualifier, sourceKey`).
|
|
21
|
+
|
|
22
|
+
## Dictionary allocation under concurrency
|
|
23
|
+
|
|
24
|
+
`resolveIds` follows a strict five-step recipe so 20+ workers seeing the same new `sourceKey` at once still converge on a single internal id:
|
|
25
|
+
|
|
26
|
+
1. **Read** existing dict rows for the requested `(scopeId, kind, qualifier, sourceKey)` batch in one round-trip.
|
|
27
|
+
2. **Allocate** a contiguous block of ids from the per-scope sequence document via `findOneAndUpdate({ _id }, { $inc: { seq: count } })` — atomic per sequence document.
|
|
28
|
+
3. **Insert** the new dict rows with `insertMany({ ordered: false })`. The `uniq_dict_key` index forbids duplicates.
|
|
29
|
+
4. On **duplicate-key error** (another worker won the race), **re-read** the dict for the same identities — the winning row is now visible.
|
|
30
|
+
5. **Cache** the resolved id in the local LRU and (optionally) Redis.
|
|
31
|
+
|
|
32
|
+
Bad outcome avoided: two workers each minting `user_42 → 10001` and `user_42 → 10002`. With this protocol one insert wins, the loser re-reads the winner's id, and counts never fragment across two ids.
|
|
33
|
+
|
|
34
|
+
The user-facing recipe lives in [`docs/guides/adapter-mongo.md`](../guides/adapter-mongo.md) §"Dictionary allocation (`resolveIds`) and concurrency".
|
|
17
35
|
|
|
18
36
|
## Dictionary vs dimensions
|
|
19
37
|
|
|
20
|
-
Subjects/metrics map to integers
|
|
38
|
+
Subjects/metrics map to integers via `countex_dict`. Dimension VALUES are stored as raw values on cells (not via the dictionary), keeping cells self-describing and avoiding dictionary churn for low-cardinality dimensions.
|
|
21
39
|
|
|
22
40
|
## See also
|
|
23
41
|
|
|
24
42
|
- [Backup streaming](./internal-backup-stream.md)
|
|
43
|
+
- [Dimension hash](./internal-dimension-hash.md) — how the `dimensionsHash` part of the cell identity is computed.
|
|
44
|
+
- [Mongo adapter guide](../guides/adapter-mongo.md)
|
|
@@ -1,18 +1,49 @@
|
|
|
1
1
|
# Internal: Redis cache adapter
|
|
2
2
|
|
|
3
|
-
`RedisCache` (`src/storage/redis.ts`) maps each `(scope, counterId, timeframeType, timeframeKey)` bucket to a **HASH
|
|
3
|
+
`RedisCache` (`src/storage/redis.ts`) maps each `(scope, counterId, timeframeType, timeframeKey)` bucket to a **HASH** of cell entries plus a sibling **`:cov` HASH** of coverage metadata:
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
5
|
+
```
|
|
6
|
+
countex:{scope}:{counterId}:{tf}:{bucket} → HASH (cells)
|
|
7
|
+
field = "{subjectType}|{subjectId}|{metricId}|{dimensionsHash}"
|
|
8
|
+
value = count (integer)
|
|
9
|
+
field = "{base}:m:{measureName}:sum" → measure sum
|
|
10
|
+
field = "{base}:m:{measureName}:count" → measure event count
|
|
7
11
|
|
|
8
|
-
|
|
12
|
+
countex:{scope}:{counterId}:{tf}:{bucket}:cov → HASH (coverage meta)
|
|
13
|
+
status = "complete" | "partial" | "rebuilding"
|
|
14
|
+
lastWarmAt = epoch ms
|
|
15
|
+
cellCount = number of cells last warmed (set on warm)
|
|
16
|
+
schemaVersion = REDIS_CACHE_SCHEMA_VERSION
|
|
17
|
+
mapVersion = optional opaque integer
|
|
18
|
+
|
|
19
|
+
countex:{scope}:{counterId}:{tf}:{bucket}:meta:{baseField} → JSON of dimensions
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Per-bucket coverage gives the planner authoritative information about which buckets the cache can answer with confidence, instead of guessing from "did `HGETALL` return anything?".
|
|
23
|
+
|
|
24
|
+
## Coverage state machine
|
|
25
|
+
|
|
26
|
+
| Triggered by | Effect on `:cov.status` |
|
|
27
|
+
|---|---|
|
|
28
|
+
| First `applyDeltas` to a previously-empty bucket | `partial` (via `HSETNX`) — never downgrades a `complete` or `rebuilding` bucket |
|
|
29
|
+
| Subsequent `applyDeltas` | unchanged status; `lastWarmAt` updated |
|
|
30
|
+
| `warm()` | `complete`, with `cellCount`, `schemaVersion`, `mapVersion`, `lastWarmAt` written atomically |
|
|
31
|
+
| `markBucketRebuilding()` | `rebuilding` (called by `rebuildCache` before refilling) |
|
|
32
|
+
| TTL expiry | key disappears → `missing` on next read |
|
|
9
33
|
|
|
10
34
|
## Read coverage
|
|
11
35
|
|
|
12
|
-
`readCells`
|
|
36
|
+
`readCells` pipelines both `HGETALL bucket` and `HGETALL :cov`, then derives per-bucket coverage from the meta:
|
|
37
|
+
|
|
38
|
+
- All buckets `complete` → aggregate `full`.
|
|
39
|
+
- Any `complete | partial | rebuilding` mixed with the rest → aggregate `partial`.
|
|
40
|
+
- All `missing` → aggregate `miss`.
|
|
41
|
+
|
|
42
|
+
When the planner supplies `expectedMapVersion` or `expectedSchemaVersion` and the bucket's stored version differs, `complete` is demoted to `partial` (cache is treated as stale for that bucket). The planner then falls back to durable.
|
|
43
|
+
|
|
44
|
+
## Probe-only path
|
|
13
45
|
|
|
14
|
-
|
|
15
|
-
- `"partial"` / `"miss"` otherwise — planner falls back to durable
|
|
46
|
+
`probeBucketCoverage(scope, counterId, tf, [keys])` returns just the `:cov` hashes without touching cell data. The `Countex.explainQuery(spec)` API uses this to predict the cache hit rate without materializing rows.
|
|
16
47
|
|
|
17
48
|
## Dedupe keys
|
|
18
49
|
|
|
@@ -20,12 +51,13 @@ Metadata hashes may store dimension JSON for reconstruction.
|
|
|
20
51
|
|
|
21
52
|
## Warm path
|
|
22
53
|
|
|
23
|
-
`warm` hydrates hashes from `CounterCell[]` after durable reads.
|
|
54
|
+
`warm` hydrates hashes from `CounterCell[]` after durable reads, then sets `:cov.status=complete`. Used by `rebuildCache` and by the query planner's best-effort warm-after-fallback.
|
|
24
55
|
|
|
25
56
|
## Error stance
|
|
26
57
|
|
|
27
|
-
Apply failures during flush are swallowed at Countex level after durable success — subsequent reads rebuild cache.
|
|
58
|
+
Apply failures during flush are swallowed at Countex level after durable success — subsequent reads rebuild cache. Coverage meta is best-effort: a stale `:cov.status=complete` only causes a temporary divergence (caller sees stale-but-consistent reads until TTL expires).
|
|
28
59
|
|
|
29
60
|
## See also
|
|
30
61
|
|
|
31
62
|
- [Query path](./internal-query-path.md)
|
|
63
|
+
- [Storage memory](./internal-storage-memory.md) — mirrors the same coverage contract for tests.
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# Countex use cases — technical playbooks
|
|
2
|
+
|
|
3
|
+
These documents are **end-to-end operational specifications**: they start from a concrete delivery problem, define **exact** input contracts and Countex primitives, then walk through **numbered steps** you can implement without improvising.
|
|
4
|
+
|
|
5
|
+
They are published with `@x12i/countex` under `docs/use-cases/` and exposed as stable `package.json` exports (see root `README.md`).
|
|
6
|
+
|
|
7
|
+
## How each playbook is structured
|
|
8
|
+
|
|
9
|
+
Every use-case file follows the same skeleton:
|
|
10
|
+
|
|
11
|
+
1. **Problem & constraints** — what you are counting, latency/durability expectations, and what Countex does *not* solve.
|
|
12
|
+
2. **Primitives mapping** — explicit table: JSON field → `scope` / `subject` / `metric` / `dimension` / `eventTime` / `delta`.
|
|
13
|
+
3. **Canonical payload** — JSON example plus a **required-field matrix** (what must be present or ingest skips).
|
|
14
|
+
4. **`CounterMap`** — copy-paste literal with a **field-by-field rationale**.
|
|
15
|
+
5. **Fan-out arithmetic** — formula and a worked example for capacity planning.
|
|
16
|
+
6. **Implementation steps** — repository layout, dependencies, `createCountex` options, `init` / `ingest` / `flush` / `shutdown`, event subscriptions.
|
|
17
|
+
7. **Query specifications** — literal `QuerySpec` objects; how to read **`QueryRow.keys`** (dictionary ids vs literal dimension strings).
|
|
18
|
+
8. **Production hardening** — Mongo / Redis wiring patterns, cardinality, idempotency TTL math, observability.
|
|
19
|
+
9. **Verification** — checks you can run after deploy.
|
|
20
|
+
10. **Failure modes** — symptom → cause → exact mitigation.
|
|
21
|
+
|
|
22
|
+
## Stable import paths (npm)
|
|
23
|
+
|
|
24
|
+
| Document | Subpath |
|
|
25
|
+
|----------|---------|
|
|
26
|
+
| Index (this list) | `@x12i/countex/docs/use-cases` |
|
|
27
|
+
| [Audit & security logs](./audit-security-log-counts.md) (includes **volume, situations, distinct entities, logical event counts**) | `@x12i/countex/docs/use-cases/audit-security-log-counts` |
|
|
28
|
+
| [Product & feature activity](./product-feature-usage.md) | `@x12i/countex/docs/use-cases/product-feature-usage` |
|
|
29
|
+
| [LLM metering](./llm-cost-and-token-metering.md) | `@x12i/countex/docs/use-cases/llm-cost-and-token-metering` |
|
|
30
|
+
| [Multi-tenant isolation](./multi-tenant-isolation.md) | `@x12i/countex/docs/use-cases/multi-tenant-isolation` |
|
|
31
|
+
| [Webhooks & queues](./webhook-idempotent-ingestion.md) | `@x12i/countex/docs/use-cases/webhook-idempotent-ingestion` |
|
|
32
|
+
| [Ops dashboards](./ops-dashboard-daily-rollups.md) | `@x12i/countex/docs/use-cases/ops-dashboard-daily-rollups` |
|
|
33
|
+
| [Processing & workflow activity](./processing-workflow-activity.md) | `@x12i/countex/docs/use-cases/processing-workflow-activity` |
|
|
34
|
+
|
|
35
|
+
## Prerequisites (global)
|
|
36
|
+
|
|
37
|
+
- **Node.js** ≥ 18 (see `package.json` `engines`).
|
|
38
|
+
- Core package: `@x12i/countex`.
|
|
39
|
+
- Optional peers for production paths: `mongodb`, `ioredis` (your process owns client lifecycle).
|
|
40
|
+
|
|
41
|
+
## Terminology (read this once)
|
|
42
|
+
|
|
43
|
+
| Term | Meaning |
|
|
44
|
+
|------|---------|
|
|
45
|
+
| **Scope** | Tenant boundary string (`scopeId` in storage). Every query **must** pin `scope`. |
|
|
46
|
+
| **Subject** | `(subjectType, subjectId)` after dictionary resolution; `subjectId` is an **integer**. |
|
|
47
|
+
| **Metric** | `(metricId)` after dictionary resolution; labels/strings become stable numeric ids per scope. |
|
|
48
|
+
| **Dimension** | Named breakdown; values stored **literally** on cells (not dictionary-encoded). |
|
|
49
|
+
| **Timeframe key** | UTC bucket string, e.g. `2026-05-08` for `day` (see `bucketKey` in `src/core/timeframe.ts`). |
|
|
50
|
+
| **Fan-out** | Cartesian product of subject extractions × metric extractions × number of `timeframes` entries. |
|
|
51
|
+
|
|
52
|
+
## Related documentation
|
|
53
|
+
|
|
54
|
+
- Feature reference: [`docs/guides/`](../guides/README.md)
|
|
55
|
+
- Narrative tutorial: [`END_TO_END_EXAMPLE.md`](../END_TO_END_EXAMPLE.md)
|
|
56
|
+
- Deployment: [`DEPLOYMENT.md`](../DEPLOYMENT.md)
|