@iann29/rastro 0.1.0-alpha.0 → 0.1.0-alpha.2

package/README.md

@@ -4,7 +4,7 @@
 
 Privacy-first, real-time web analytics built as an isolated Convex component.
 
-Amage Rastro combines a 991-byte gzip browser tracker with reactive traffic
+Amage Rastro combines a 1,022-byte gzip browser tracker with reactive traffic
 metrics, a live visitor atlas, journey timelines, trusted payment attribution,
 goals, funnels, affiliates, and multi-site reports. Data and functions stay
 inside your Convex deployment.
@@ -24,7 +24,7 @@ inside your Convex deployment.
 | Cross-site analytics        | Reports over up to ten authorized site IDs with a preloaded pseudonymous visitor ID                 |
 | Auto-captured clicks        | Anchors, buttons, outbound links, and explicit `data-rastro-event` controls                         |
 | Cookieless mode             | Session-scoped pseudonymous IDs in `sessionStorage`; no cookies                                     |
-| Tiny tracker                | 1,890 bytes raw, 991 bytes gzip; the test suite enforces `< 1,024` bytes                            |
+| Tiny tracker                | 1,909 bytes raw, 1,022 bytes gzip; the test suite enforces `< 1,024` bytes                          |
 
 ## Install
 
@@ -172,6 +172,12 @@ Properties accept bounded string, number, boolean, or null values. Never send
 emails, names, form values, secrets, or other personal data as analytics
 properties.
 
+Browser telemetry exact retries are idempotent within
+`(siteId, sessionId, event-minute, eventId)`. Reusing an event ID in another
+session or with a timestamp in another minute is allowed. The tracker preserves
+timestamps when retrying. Trusted payment event IDs remain separately
+payload-bound by the site's financial ledger.
+
 ## Attribute trusted payments
 
 Browser-created conversion events are untrusted telemetry and never affect
@@ -271,25 +277,99 @@ The `Rastro` class wraps the component boundary for host functions:
 - Retention: `cleanup`
 
 Every public function has argument and return validators. Growing reads are
-indexed and bounded. Component pagination uses validated opaque manual cursors
-because native `.paginate()` is not supported inside Convex components.
+indexed and bounded. Component pagination uses scope-bound opaque keyset cursors
+because native `.paginate()` is not supported inside Convex components; page
+cost stays constant instead of increasing with an offset.
 
 ## Operational limits
 
 - HTTP body: 64 KiB
-- Events per batch: 20
-- Aggregate shards: 16, derived from stable session/visitor identity
-- Site ingress budget: eight deterministic per-minute shards with bounded event
-  and byte caps
-- Live session timeout: 60 seconds with one self-rescheduling expiry chain per
-  session
+- Events per batch: 50
+- Session-minute groups per batch: 4
+- Active goals per exact matcher: 1
+- Primary aggregate shards: 256 hourly and 128 daily, derived from stable
+  session identity
+- Funnel and affiliate aggregate shards: 16
+- Site ingress budget: 4,096 deterministic per-minute shards, each capped at
+  300 events and 512 KiB (1,228,800 admitted events/minute at uniform load)
+- Rate-window storage: one reusable row per active session and site shard
+- Live session timeout: 60 seconds with one bounded five-second expiry sweep per
+  site; legacy per-session jobs hand off to that sweep during the cutover
 - Dashboard overview: up to ten sites and bounded report ranges
+- Hourly overview: up to 24 hours; wider ranges use daily buckets
 - Retention cleanup: bounded batches; schedule host maintenance for the data
-  classes your policy retains
+  classes your policy retains. The `events` cleanup kind covers both legacy
+  event rows and new event batches and processes at most 32 heavy batches per
+  transaction.
+
+Capacity claims use declared workload profiles. Complete release evidence
+reports ten-minute `lean`, `realistic`, `heavy`, and browser `feature` profiles
+separately. The browser `feature` profile exercises configured goals, funnels,
+and affiliate attribution; trusted conversions require a separate server-side
+driver and certification result.
+
+The historical development evidence and its certification caveats are recorded
+in [`docs/benchmarks/2026-08-20-realistic.md`](docs/benchmarks/2026-08-20-realistic.md).
+
+From a repository checkout, run a duration-based profile against a confirmed
+non-production deployment and a dedicated idle benchmark site:
+
+```bash
+npm run --silent benchmark:ingest -- \
+  --url https://example.site/rastro/events \
+  --convex-url https://example.convex-backend \
+  --site-id your_site_id \
+  --origin https://your-allowed-origin.example \
+  --profile realistic \
+  --duration 600 \
+  --concurrency 100 \
+  --certify
+```
+
+Profiles are `lean`, `realistic`, `heavy`, and `feature`. The realistic profile
+reuses sessions and sends geographic and user-agent headers so the same load is
+visible in Live Atlas. Only `feature` intentionally matches the demo goal,
+funnel, and affiliate definitions; it does not call the trusted conversion API.
+Use `--dry-run` to inspect generated request size without sending traffic. When
+`--convex-url` is supplied, it must identify the same deployment as `--url` and
+expose this repository's `example:overview` query. The final reconciliation is
+a site-wide before/after delta, so unrelated traffic on that site invalidates
+the result. Reports distinguish accepted ingress, heartbeats, and stored events.
+Shorter completed runs report extrapolated daily estimates only. `--certify`
+requires at least ten minutes and successful reconciliation before emitting
+certified daily capacity fields or exiting successfully as a certification.
+
+The ingestion transaction folds metric/dimension deltas, session rate counters,
+site-shard counters, session/live state, visitor attribution, and exact funnel
+progress before writing. Accepted non-heartbeat telemetry from one request is
+grouped into one immutable raw document per session and event-minute, without
+duplicating session geo or user-agent context. A 50-event same-session,
+same-minute baseline stays within 10 document writes. Each touched aggregate
+bucket is written at most once per batch, and dimension overflow folds into an
+explicit `(other)` bucket instead of relabeling old counts.
+Click/outbound/browser-conversion telemetry skips goal and funnel work that
+cannot match those event types.
+
+New telemetry is written only to `eventBatches`. Existing `events` rows remain
+readable until an operator runs the resumable migration mounted inside the
+Rastro component:
+
+```bash
+npx convex run --component rastroAnalytics migrations:migrateLegacyEvents \
+  '{"dryRun":true,"reset":true}'
+npx convex run --component rastroAnalytics migrations:migrateLegacyEvents '{}'
+npx convex run --component rastroAnalytics migrations:legacyEventsStatus '{}'
+```
 
-The ingestion transaction folds metric/dimension deltas and writes each touched
-aggregate bucket at most once per batch. Click/outbound/browser-conversion
-telemetry skips goal and funnel work that cannot match those event types.
+Replace `rastroAnalytics` if the host mounted the component under another name.
+Proceed only when status returns `complete: true`, which requires both durable
+migration success and an empty legacy source. A source refilled after completion
+requires an explicit `{"reset":true}` rerun. Migrated rows are compacted into
+bounded canonical batches and retain their original per-event browser and geo
+context. Journey `_id` values remain stable strings for both formats during the
+cutover, and session-journey cursors use timestamp, creation time, and ID. The
+compatibility logic remains isolated in `src/component/eventStore.ts` until a
+later release removes the empty legacy table and its read paths.
 
 ## Privacy contract
 
@@ -336,10 +416,10 @@ cd example && npx vite build
 ```
 
 The test suite covers origin enforcement, event/privacy sanitization,
-idempotency, session/site rate limits, live expiry scheduling, folded aggregate
-writes, goals, funnels, trusted payments, currency safety, affiliate commission,
-cross-site journeys, manual pagination, retention, HTTP routes, React
-integration, and tracker size/behavior.
+idempotency, reusable session/site rate limits, bounded live expiry chains,
+folded aggregate writes, goals, funnels, trusted payments, currency safety,
+affiliate commission, cross-site journeys, keyset pagination beyond 1,000 rows,
+retention, HTTP routes, React integration, and tracker size/behavior.
 
 ## Synapse demo
 

package/dist/component/_generated/api.d.ts

@@ -9,12 +9,14 @@
 import type * as affiliates from "../affiliates.js";
 import type * as constants from "../constants.js";
 import type * as errors from "../errors.js";
+import type * as eventStore from "../eventStore.js";
 import type * as funnels from "../funnels.js";
 import type * as goals from "../goals.js";
 import type * as guards from "../guards.js";
 import type * as http from "../http.js";
 import type * as ingest from "../ingest.js";
 import type * as live from "../live.js";
+import type * as migrations from "../migrations.js";
 import type * as reports from "../reports.js";
 import type * as retention from "../retention.js";
 import type * as sanitize from "../sanitize.js";
@@ -25,12 +27,14 @@ declare const fullApi: ApiFromModules<{
     affiliates: typeof affiliates;
     constants: typeof constants;
     errors: typeof errors;
+    eventStore: typeof eventStore;
     funnels: typeof funnels;
     goals: typeof goals;
     guards: typeof guards;
     http: typeof http;
     ingest: typeof ingest;
     live: typeof live;
+    migrations: typeof migrations;
     reports: typeof reports;
     retention: typeof retention;
     sanitize: typeof sanitize;
@@ -55,6 +59,8 @@ export declare const api: FilterApi<typeof fullApi, FunctionReference<any, "publ
  * ```
  */
 export declare const internal: FilterApi<typeof fullApi, FunctionReference<any, "internal">>;
-export declare const components: {};
+export declare const components: {
+    migrations: import("@convex-dev/migrations/_generated/component.js").ComponentApi<"migrations">;
+};
 export {};
 //# sourceMappingURL=api.d.ts.map

package/dist/component/_generated/api.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"api.d.ts","sourceRoot":"","sources":["../../../src/component/_generated/api.ts"],"names":[],"mappings":"AACA;;;;;;;GAOG;AAEH,OAAO,KAAK,KAAK,UAAU,MAAM,kBAAkB,CAAC;AACpD,OAAO,KAAK,KAAK,SAAS,MAAM,iBAAiB,CAAC;AAClD,OAAO,KAAK,KAAK,MAAM,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,KAAK,OAAO,MAAM,eAAe,CAAC;AAC9C,OAAO,KAAK,KAAK,KAAK,MAAM,aAAa,CAAC;AAC1C,OAAO,KAAK,KAAK,MAAM,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,KAAK,IAAI,MAAM,YAAY,CAAC;AACxC,OAAO,KAAK,KAAK,MAAM,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,KAAK,IAAI,MAAM,YAAY,CAAC;AACxC,OAAO,KAAK,KAAK,OAAO,MAAM,eAAe,CAAC;AAC9C,OAAO,KAAK,KAAK,SAAS,MAAM,iBAAiB,CAAC;AAClD,OAAO,KAAK,KAAK,QAAQ,MAAM,gBAAgB,CAAC;AAChD,OAAO,KAAK,KAAK,KAAK,MAAM,aAAa,CAAC;AAC1C,OAAO,KAAK,KAAK,UAAU,MAAM,kBAAkB,CAAC;AAEpD,OAAO,KAAK,EACV,cAAc,EACd,SAAS,EACT,iBAAiB,EAClB,MAAM,eAAe,CAAC;AAGvB,QAAA,MAAM,OAAO,EAAE,cAAc,CAAC;IAC5B,UAAU,EAAE,OAAO,UAAU,CAAC;IAC9B,SAAS,EAAE,OAAO,SAAS,CAAC;IAC5B,MAAM,EAAE,OAAO,MAAM,CAAC;IACtB,OAAO,EAAE,OAAO,OAAO,CAAC;IACxB,KAAK,EAAE,OAAO,KAAK,CAAC;IACpB,MAAM,EAAE,OAAO,MAAM,CAAC;IACtB,IAAI,EAAE,OAAO,IAAI,CAAC;IAClB,MAAM,EAAE,OAAO,MAAM,CAAC;IACtB,IAAI,EAAE,OAAO,IAAI,CAAC;IAClB,OAAO,EAAE,OAAO,OAAO,CAAC;IACxB,SAAS,EAAE,OAAO,SAAS,CAAC;IAC5B,QAAQ,EAAE,OAAO,QAAQ,CAAC;IAC1B,KAAK,EAAE,OAAO,KAAK,CAAC;IACpB,UAAU,EAAE,OAAO,UAAU,CAAC;CAC/B,CAAiB,CAAC;AAEnB;;;;;;;GAOG;AACH,eAAO,MAAM,GAAG,EAAE,SAAS,CACzB,OAAO,OAAO,EACd,iBAAiB,CAAC,GAAG,EAAE,QAAQ,CAAC,CACjB,CAAC;AAElB;;;;;;;GAOG;AACH,eAAO,MAAM,QAAQ,EAAE,SAAS,CAC9B,OAAO,OAAO,EACd,iBAAiB,CAAC,GAAG,EAAE,UAAU,CAAC,CACnB,CAAC;AAElB,eAAO,MAAM,UAAU,EAAqC,EAAE,CAAC"}
+{"version":3,"file":"api.d.ts","sourceRoot":"","sources":["../../../src/component/_generated/api.ts"],"names":[],"mappings":"AACA;;;;;;;GAOG;AAEH,OAAO,KAAK,KAAK,UAAU,MAAM,kBAAkB,CAAC;AACpD,OAAO,KAAK,KAAK,SAAS,MAAM,iBAAiB,CAAC;AAClD,OAAO,KAAK,KAAK,MAAM,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,KAAK,UAAU,MAAM,kBAAkB,CAAC;AACpD,OAAO,KAAK,KAAK,OAAO,MAAM,eAAe,CAAC;AAC9C,OAAO,KAAK,KAAK,KAAK,MAAM,aAAa,CAAC;AAC1C,OAAO,KAAK,KAAK,MAAM,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,KAAK,IAAI,MAAM,YAAY,CAAC;AACxC,OAAO,KAAK,KAAK,MAAM,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,KAAK,IAAI,MAAM,YAAY,CAAC;AACxC,OAAO,KAAK,KAAK,UAAU,MAAM,kBAAkB,CAAC;AACpD,OAAO,KAAK,KAAK,OAAO,MAAM,eAAe,CAAC;AAC9C,OAAO,KAAK,KAAK,SAAS,MAAM,iBAAiB,CAAC;AAClD,OAAO,KAAK,KAAK,QAAQ,MAAM,gBAAgB,CAAC;AAChD,OAAO,KAAK,KAAK,KAAK,MAAM,aAAa,CAAC;AAC1C,OAAO,KAAK,KAAK,UAAU,MAAM,kBAAkB,CAAC;AAEpD,OAAO,KAAK,EACV,cAAc,EACd,SAAS,EACT,iBAAiB,EAClB,MAAM,eAAe,CAAC;AAGvB,QAAA,MAAM,OAAO,EAAE,cAAc,CAAC;IAC5B,UAAU,EAAE,OAAO,UAAU,CAAC;IAC9B,SAAS,EAAE,OAAO,SAAS,CAAC;IAC5B,MAAM,EAAE,OAAO,MAAM,CAAC;IACtB,UAAU,EAAE,OAAO,UAAU,CAAC;IAC9B,OAAO,EAAE,OAAO,OAAO,CAAC;IACxB,KAAK,EAAE,OAAO,KAAK,CAAC;IACpB,MAAM,EAAE,OAAO,MAAM,CAAC;IACtB,IAAI,EAAE,OAAO,IAAI,CAAC;IAClB,MAAM,EAAE,OAAO,MAAM,CAAC;IACtB,IAAI,EAAE,OAAO,IAAI,CAAC;IAClB,UAAU,EAAE,OAAO,UAAU,CAAC;IAC9B,OAAO,EAAE,OAAO,OAAO,CAAC;IACxB,SAAS,EAAE,OAAO,SAAS,CAAC;IAC5B,QAAQ,EAAE,OAAO,QAAQ,CAAC;IAC1B,KAAK,EAAE,OAAO,KAAK,CAAC;IACpB,UAAU,EAAE,OAAO,UAAU,CAAC;CAC/B,CAAiB,CAAC;AAEnB;;;;;;;GAOG;AACH,eAAO,MAAM,GAAG,EAAE,SAAS,CACzB,OAAO,OAAO,EACd,iBAAiB,CAAC,GAAG,EAAE,QAAQ,CAAC,CACjB,CAAC;AAElB;;;;;;;GAOG;AACH,eAAO,MAAM,QAAQ,EAAE,SAAS,CAC9B,OAAO,OAAO,EACd,iBAAiB,CAAC,GAAG,EAAE,UAAU,CAAC,CACnB,CAAC;AAElB,eAAO,MAAM,UAAU,EAAqC;IAC1D,UAAU,EAAE,OAAO,gDAAgD,EAAE,YAAY,CAAC,YAAY,CAAC,CAAC;CACjG,CAAC"}

package/dist/component/_generated/api.js.map

@@ -1 +1 @@
-{"version":3,"file":"api.js","sourceRoot":"","sources":["../../../src/component/_generated/api.ts"],"names":[],"mappings":"AAAA,oBAAoB;AACpB;;;;;;;GAOG;AAsBH,OAAO,EAAE,MAAM,EAAE,iBAAiB,EAAE,MAAM,eAAe,CAAC;AAE1D,MAAM,OAAO,GAeR,MAAa,CAAC;AAEnB;;;;;;;GAOG;AACH,MAAM,CAAC,MAAM,GAAG,GAGZ,MAAa,CAAC;AAElB;;;;;;;GAOG;AACH,MAAM,CAAC,MAAM,QAAQ,GAGjB,MAAa,CAAC;AAElB,MAAM,CAAC,MAAM,UAAU,GAAG,iBAAiB,EAAmB,CAAC"}
+{"version":3,"file":"api.js","sourceRoot":"","sources":["../../../src/component/_generated/api.ts"],"names":[],"mappings":"AAAA,oBAAoB;AACpB;;;;;;;GAOG;AAwBH,OAAO,EAAE,MAAM,EAAE,iBAAiB,EAAE,MAAM,eAAe,CAAC;AAE1D,MAAM,OAAO,GAiBR,MAAa,CAAC;AAEnB;;;;;;;GAOG;AACH,MAAM,CAAC,MAAM,GAAG,GAGZ,MAAa,CAAC;AAElB;;;;;;;GAOG;AACH,MAAM,CAAC,MAAM,QAAQ,GAGjB,MAAa,CAAC;AAElB,MAAM,CAAC,MAAM,UAAU,GAAG,iBAAiB,EAE1C,CAAC"}

package/dist/component/constants.d.ts

@@ -1,13 +1,17 @@
-export declare const MAX_BATCH_EVENTS = 20;
+export declare const MAX_BATCH_EVENTS = 50;
+export declare const MAX_BATCH_EVENT_GROUPS = 4;
 export declare const MAX_BATCH_BYTES: number;
 export declare const MAX_EVENTS_PER_SESSION_WINDOW = 120;
 export declare const RATE_LIMIT_WINDOW_MS = 60000;
 export declare const MAX_BYTES_PER_SESSION_WINDOW: number;
-export declare const SITE_INGEST_SHARDS = 8;
-export declare const MAX_SITE_EVENTS_PER_SHARD_WINDOW = 120;
+export declare const SITE_INGEST_SHARDS = 4096;
+export declare const MAX_SITE_EVENTS_PER_SHARD_WINDOW = 300;
 export declare const MAX_SITE_BYTES_PER_SHARD_WINDOW: number;
 export declare const LIVE_SESSION_TTL_MS = 60000;
-export declare const AGGREGATE_SHARDS = 16;
+export declare const LIVE_SWEEP_INTERVAL_MS = 5000;
+export declare const AGGREGATE_SHARDS = 128;
+export declare const HOURLY_AGGREGATE_SHARDS = 256;
+export declare const SECONDARY_AGGREGATE_SHARDS = 16;
 export declare const DIMENSION_SLOTS = 16;
 export declare const MAX_SITE_DOMAINS = 32;
 export declare const MAX_SITES_PER_OWNER = 100;
@@ -20,9 +24,10 @@ export declare const MAX_REPORT_SITES = 10;
 export declare const MAX_REPORT_RANGE_DAYS = 366;
 export declare const MAX_LIVE_VISITORS = 500;
 export declare const MAX_JOURNEY_EVENTS = 500;
+export declare const JOURNEY_READ_BYTE_RESERVE: number;
+export declare const JOURNEY_READ_DOCUMENT_RESERVE = 1000;
 export declare const MAX_RETENTION_BATCH = 500;
-export declare const MAX_OVERVIEW_AGGREGATE_ROWS = 12000;
+export declare const MAX_OVERVIEW_AGGREGATE_ROWS = 13000;
 export declare const DAY_MS = 86400000;
 export declare const HOUR_MS = 3600000;
-export declare const MAX_MANUAL_PAGINATION_OFFSET = 1000;
 //# sourceMappingURL=constants.d.ts.map

package/dist/component/constants.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"constants.d.ts","sourceRoot":"","sources":["../../src/component/constants.ts"],"names":[],"mappings":"AAAA,eAAO,MAAM,gBAAgB,KAAK,CAAC;AACnC,eAAO,MAAM,eAAe,QAAY,CAAC;AACzC,eAAO,MAAM,6BAA6B,MAAM,CAAC;AACjD,eAAO,MAAM,oBAAoB,QAAS,CAAC;AAC3C,eAAO,MAAM,4BAA4B,QAAa,CAAC;AACvD,eAAO,MAAM,kBAAkB,IAAI,CAAC;AACpC,eAAO,MAAM,gCAAgC,MAAM,CAAC;AACpD,eAAO,MAAM,+BAA+B,QAAa,CAAC;AAC1D,eAAO,MAAM,mBAAmB,QAAS,CAAC;AAC1C,eAAO,MAAM,gBAAgB,KAAK,CAAC;AACnC,eAAO,MAAM,eAAe,KAAK,CAAC;AAClC,eAAO,MAAM,gBAAgB,KAAK,CAAC;AACnC,eAAO,MAAM,mBAAmB,MAAM,CAAC;AACvC,eAAO,MAAM,kBAAkB,KAAK,CAAC;AACrC,eAAO,MAAM,oBAAoB,KAAK,CAAC;AACvC,eAAO,MAAM,uBAAuB,MAAM,CAAC;AAC3C,eAAO,MAAM,gBAAgB,KAAK,CAAC;AACnC,eAAO,MAAM,cAAc,KAAK,CAAC;AACjC,eAAO,MAAM,gBAAgB,KAAK,CAAC;AACnC,eAAO,MAAM,qBAAqB,MAAM,CAAC;AACzC,eAAO,MAAM,iBAAiB,MAAM,CAAC;AACrC,eAAO,MAAM,kBAAkB,MAAM,CAAC;AACtC,eAAO,MAAM,mBAAmB,MAAM,CAAC;AACvC,eAAO,MAAM,2BAA2B,QAAS,CAAC;AAElD,eAAO,MAAM,MAAM,WAAa,CAAC;AACjC,eAAO,MAAM,OAAO,UAAY,CAAC;AACjC,eAAO,MAAM,4BAA4B,OAAQ,CAAC"}
+{"version":3,"file":"constants.d.ts","sourceRoot":"","sources":["../../src/component/constants.ts"],"names":[],"mappings":"AAAA,eAAO,MAAM,gBAAgB,KAAK,CAAC;AACnC,eAAO,MAAM,sBAAsB,IAAI,CAAC;AACxC,eAAO,MAAM,eAAe,QAAY,CAAC;AACzC,eAAO,MAAM,6BAA6B,MAAM,CAAC;AACjD,eAAO,MAAM,oBAAoB,QAAS,CAAC;AAC3C,eAAO,MAAM,4BAA4B,QAAa,CAAC;AACvD,eAAO,MAAM,kBAAkB,OAAQ,CAAC;AACxC,eAAO,MAAM,gCAAgC,MAAM,CAAC;AACpD,eAAO,MAAM,+BAA+B,QAAa,CAAC;AAC1D,eAAO,MAAM,mBAAmB,QAAS,CAAC;AAC1C,eAAO,MAAM,sBAAsB,OAAQ,CAAC;AAC5C,eAAO,MAAM,gBAAgB,MAAM,CAAC;AACpC,eAAO,MAAM,uBAAuB,MAAM,CAAC;AAC3C,eAAO,MAAM,0BAA0B,KAAK,CAAC;AAC7C,eAAO,MAAM,eAAe,KAAK,CAAC;AAClC,eAAO,MAAM,gBAAgB,KAAK,CAAC;AACnC,eAAO,MAAM,mBAAmB,MAAM,CAAC;AACvC,eAAO,MAAM,kBAAkB,KAAK,CAAC;AACrC,eAAO,MAAM,oBAAoB,KAAK,CAAC;AACvC,eAAO,MAAM,uBAAuB,MAAM,CAAC;AAC3C,eAAO,MAAM,gBAAgB,KAAK,CAAC;AACnC,eAAO,MAAM,cAAc,KAAK,CAAC;AACjC,eAAO,MAAM,gBAAgB,KAAK,CAAC;AACnC,eAAO,MAAM,qBAAqB,MAAM,CAAC;AACzC,eAAO,MAAM,iBAAiB,MAAM,CAAC;AACrC,eAAO,MAAM,kBAAkB,MAAM,CAAC;AACtC,eAAO,MAAM,yBAAyB,QAAkB,CAAC;AACzD,eAAO,MAAM,6BAA6B,OAAQ,CAAC;AACnD,eAAO,MAAM,mBAAmB,MAAM,CAAC;AACvC,eAAO,MAAM,2BAA2B,QAAS,CAAC;AAElD,eAAO,MAAM,MAAM,WAAa,CAAC;AACjC,eAAO,MAAM,OAAO,UAAY,CAAC"}

package/dist/component/constants.js

@@ -1,13 +1,17 @@
-export const MAX_BATCH_EVENTS = 20;
+export const MAX_BATCH_EVENTS = 50;
+export const MAX_BATCH_EVENT_GROUPS = 4;
 export const MAX_BATCH_BYTES = 64 * 1024;
 export const MAX_EVENTS_PER_SESSION_WINDOW = 120;
 export const RATE_LIMIT_WINDOW_MS = 60_000;
 export const MAX_BYTES_PER_SESSION_WINDOW = 256 * 1024;
-export const SITE_INGEST_SHARDS = 8;
-export const MAX_SITE_EVENTS_PER_SHARD_WINDOW = 120;
+export const SITE_INGEST_SHARDS = 4_096;
+export const MAX_SITE_EVENTS_PER_SHARD_WINDOW = 300;
 export const MAX_SITE_BYTES_PER_SHARD_WINDOW = 512 * 1024;
 export const LIVE_SESSION_TTL_MS = 60_000;
-export const AGGREGATE_SHARDS = 16;
+export const LIVE_SWEEP_INTERVAL_MS = 5_000;
+export const AGGREGATE_SHARDS = 128;
+export const HOURLY_AGGREGATE_SHARDS = 256;
+export const SECONDARY_AGGREGATE_SHARDS = 16;
 export const DIMENSION_SLOTS = 16;
 export const MAX_SITE_DOMAINS = 32;
 export const MAX_SITES_PER_OWNER = 100;
@@ -20,9 +24,10 @@ export const MAX_REPORT_SITES = 10;
 export const MAX_REPORT_RANGE_DAYS = 366;
 export const MAX_LIVE_VISITORS = 500;
 export const MAX_JOURNEY_EVENTS = 500;
+export const JOURNEY_READ_BYTE_RESERVE = 4 * 1024 * 1024;
+export const JOURNEY_READ_DOCUMENT_RESERVE = 1_000;
 export const MAX_RETENTION_BATCH = 500;
-export const MAX_OVERVIEW_AGGREGATE_ROWS = 12_000;
+export const MAX_OVERVIEW_AGGREGATE_ROWS = 13_000;
 export const DAY_MS = 86_400_000;
 export const HOUR_MS = 3_600_000;
-export const MAX_MANUAL_PAGINATION_OFFSET = 1_000;
 //# sourceMappingURL=constants.js.map

package/dist/component/constants.js.map

@@ -1 +1 @@
-{"version":3,"file":"constants.js","sourceRoot":"","sources":["../../src/component/constants.ts"],"names":[],"mappings":"AAAA,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AACnC,MAAM,CAAC,MAAM,eAAe,GAAG,EAAE,GAAG,IAAI,CAAC;AACzC,MAAM,CAAC,MAAM,6BAA6B,GAAG,GAAG,CAAC;AACjD,MAAM,CAAC,MAAM,oBAAoB,GAAG,MAAM,CAAC;AAC3C,MAAM,CAAC,MAAM,4BAA4B,GAAG,GAAG,GAAG,IAAI,CAAC;AACvD,MAAM,CAAC,MAAM,kBAAkB,GAAG,CAAC,CAAC;AACpC,MAAM,CAAC,MAAM,gCAAgC,GAAG,GAAG,CAAC;AACpD,MAAM,CAAC,MAAM,+BAA+B,GAAG,GAAG,GAAG,IAAI,CAAC;AAC1D,MAAM,CAAC,MAAM,mBAAmB,GAAG,MAAM,CAAC;AAC1C,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AACnC,MAAM,CAAC,MAAM,eAAe,GAAG,EAAE,CAAC;AAClC,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AACnC,MAAM,CAAC,MAAM,mBAAmB,GAAG,GAAG,CAAC;AACvC,MAAM,CAAC,MAAM,kBAAkB,GAAG,EAAE,CAAC;AACrC,MAAM,CAAC,MAAM,oBAAoB,GAAG,EAAE,CAAC;AACvC,MAAM,CAAC,MAAM,uBAAuB,GAAG,GAAG,CAAC;AAC3C,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AACnC,MAAM,CAAC,MAAM,cAAc,GAAG,EAAE,CAAC;AACjC,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AACnC,MAAM,CAAC,MAAM,qBAAqB,GAAG,GAAG,CAAC;AACzC,MAAM,CAAC,MAAM,iBAAiB,GAAG,GAAG,CAAC;AACrC,MAAM,CAAC,MAAM,kBAAkB,GAAG,GAAG,CAAC;AACtC,MAAM,CAAC,MAAM,mBAAmB,GAAG,GAAG,CAAC;AACvC,MAAM,CAAC,MAAM,2BAA2B,GAAG,MAAM,CAAC;AAElD,MAAM,CAAC,MAAM,MAAM,GAAG,UAAU,CAAC;AACjC,MAAM,CAAC,MAAM,OAAO,GAAG,SAAS,CAAC;AACjC,MAAM,CAAC,MAAM,4BAA4B,GAAG,KAAK,CAAC"}
+{"version":3,"file":"constants.js","sourceRoot":"","sources":["../../src/component/constants.ts"],"names":[],"mappings":"AAAA,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AACnC,MAAM,CAAC,MAAM,sBAAsB,GAAG,CAAC,CAAC;AACxC,MAAM,CAAC,MAAM,eAAe,GAAG,EAAE,GAAG,IAAI,CAAC;AACzC,MAAM,CAAC,MAAM,6BAA6B,GAAG,GAAG,CAAC;AACjD,MAAM,CAAC,MAAM,oBAAoB,GAAG,MAAM,CAAC;AAC3C,MAAM,CAAC,MAAM,4BAA4B,GAAG,GAAG,GAAG,IAAI,CAAC;AACvD,MAAM,CAAC,MAAM,kBAAkB,GAAG,KAAK,CAAC;AACxC,MAAM,CAAC,MAAM,gCAAgC,GAAG,GAAG,CAAC;AACpD,MAAM,CAAC,MAAM,+BAA+B,GAAG,GAAG,GAAG,IAAI,CAAC;AAC1D,MAAM,CAAC,MAAM,mBAAmB,GAAG,MAAM,CAAC;AAC1C,MAAM,CAAC,MAAM,sBAAsB,GAAG,KAAK,CAAC;AAC5C,MAAM,CAAC,MAAM,gBAAgB,GAAG,GAAG,CAAC;AACpC,MAAM,CAAC,MAAM,uBAAuB,GAAG,GAAG,CAAC;AAC3C,MAAM,CAAC,MAAM,0BAA0B,GAAG,EAAE,CAAC;AAC7C,MAAM,CAAC,MAAM,eAAe,GAAG,EAAE,CAAC;AAClC,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AACnC,MAAM,CAAC,MAAM,mBAAmB,GAAG,GAAG,CAAC;AACvC,MAAM,CAAC,MAAM,kBAAkB,GAAG,EAAE,CAAC;AACrC,MAAM,CAAC,MAAM,oBAAoB,GAAG,EAAE,CAAC;AACvC,MAAM,CAAC,MAAM,uBAAuB,GAAG,GAAG,CAAC;AAC3C,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AACnC,MAAM,CAAC,MAAM,cAAc,GAAG,EAAE,CAAC;AACjC,MAAM,CAAC,MAAM,gBAAgB,GAAG,EAAE,CAAC;AACnC,MAAM,CAAC,MAAM,qBAAqB,GAAG,GAAG,CAAC;AACzC,MAAM,CAAC,MAAM,iBAAiB,GAAG,GAAG,CAAC;AACrC,MAAM,CAAC,MAAM,kBAAkB,GAAG,GAAG,CAAC;AACtC,MAAM,CAAC,MAAM,yBAAyB,GAAG,CAAC,GAAG,IAAI,GAAG,IAAI,CAAC;AACzD,MAAM,CAAC,MAAM,6BAA6B,GAAG,KAAK,CAAC;AACnD,MAAM,CAAC,MAAM,mBAAmB,GAAG,GAAG,CAAC;AACvC,MAAM,CAAC,MAAM,2BAA2B,GAAG,MAAM,CAAC;AAElD,MAAM,CAAC,MAAM,MAAM,GAAG,UAAU,CAAC;AACjC,MAAM,CAAC,MAAM,OAAO,GAAG,SAAS,CAAC"}

package/dist/component/convex.config.d.ts

@@ -1,3 +1,3 @@
-declare const _default: import("convex/server").ComponentDefinition<any, {}>;
-export default _default;
+declare const component: import("convex/server").ComponentDefinition<any, {}>;
+export default component;
 //# sourceMappingURL=convex.config.d.ts.map

package/dist/component/convex.config.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"convex.config.d.ts","sourceRoot":"","sources":["../../src/component/convex.config.ts"],"names":[],"mappings":""}
+{"version":3,"file":"convex.config.d.ts","sourceRoot":"","sources":["../../src/component/convex.config.ts"],"names":[],"mappings":"AAGA,QAAA,MAAM,SAAS,sDAAqC,CAAC;eAGtC,SAAS"}

package/dist/component/convex.config.js

@@ -1,3 +1,6 @@
+import migrations from "@convex-dev/migrations/convex.config.js";
 import { defineComponent } from "convex/server";
-export default defineComponent("rastroAnalytics");
+const component = defineComponent("rastroAnalytics");
+component.use(migrations);
+export default component;
 //# sourceMappingURL=convex.config.js.map

package/dist/component/convex.config.js.map

@@ -1 +1 @@
-{"version":3,"file":"convex.config.js","sourceRoot":"","sources":["../../src/component/convex.config.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAEhD,eAAe,eAAe,CAAC,iBAAiB,CAAC,CAAC"}
+{"version":3,"file":"convex.config.js","sourceRoot":"","sources":["../../src/component/convex.config.ts"],"names":[],"mappings":"AAAA,OAAO,UAAU,MAAM,yCAAyC,CAAC;AACjE,OAAO,EAAE,eAAe,EAAE,MAAM,eAAe,CAAC;AAEhD,MAAM,SAAS,GAAG,eAAe,CAAC,iBAAiB,CAAC,CAAC;AACrD,SAAS,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;AAE1B,eAAe,SAAS,CAAC"}

package/dist/component/eventStore.d.ts

@@ -0,0 +1,102 @@
+import type { Doc, Id } from "./_generated/dataModel.js";
+import type { MutationCtx, QueryCtx } from "./_generated/server.js";
+import type { TrackerEvent } from "./validators.js";
+export type TelemetryMatch = {
+    sessionId: string;
+    visitorId: string;
+    country?: string;
+    city?: string;
+    latitude?: number;
+    longitude?: number;
+    browser?: string;
+    os?: string;
+    device?: string;
+};
+type BufferedEventBatch = {
+    siteId: Id<"sites">;
+    sessionId: string;
+    visitorId: string;
+    bucketStart: number;
+    events: TrackerEvent[];
+};
+export type EventBatchBuffer = Map<string, BufferedEventBatch>;
+export type JourneyCursor = {
+    timestamp: number;
+    creationTime: number;
+    id: string;
+};
+export type RawJourneyEvent = Omit<Doc<"events">, "_id"> & {
+    _id: string;
+};
+export declare function eventBucketStart(timestamp: number): number;
+export declare function eventBatchKey(sessionId: string, bucketStart: number): string;
+export declare function inspectTelemetryMinute(ctx: MutationCtx, input: {
+    siteId: Id<"sites">;
+    sessionId: string;
+    bucketStart: number;
+    candidateEventIds: Set<string>;
+}): Promise<{
+    eventIds: Set<string>;
+    persistedCount: number;
+}>;
+export declare function findTelemetry(ctx: MutationCtx, siteId: Id<"sites">, event: TrackerEvent): Promise<{
+    match: TelemetryMatch | null;
+    saturated: boolean;
+}>;
+export declare function bufferEventBatch(siteId: Id<"sites">, event: TrackerEvent, buffer: EventBatchBuffer): void;
+export declare function insertEventBatch(ctx: MutationCtx, siteId: Id<"sites">, event: TrackerEvent): Promise<void>;
+export declare function flushEventBatches(ctx: MutationCtx, buffer: EventBatchBuffer): Promise<void>;
+export declare function migrateLegacyEventToBatch(ctx: MutationCtx, event: Doc<"events">): Promise<void>;
+export declare function loadSessionJourneyEvents(ctx: QueryCtx, input: {
+    siteId: Id<"sites">;
+    sessionId: string;
+    from: number;
+    to: number;
+    cursor: JourneyCursor | null;
+    resultLimit: number;
+}): Promise<RawJourneyEvent[]>;
+export declare function loadVisitorJourneyEvents(ctx: QueryCtx, input: {
+    siteId: Id<"sites">;
+    visitorId: string;
+    from: number;
+    to: number;
+    resultLimit: number;
+}): Promise<RawJourneyEvent[]>;
+export declare function compareJourneyEvents(left: Pick<RawJourneyEvent, "timestamp" | "_creationTime" | "_id">, right: Pick<RawJourneyEvent, "timestamp" | "_creationTime" | "_id">): number;
+export declare function hydrateEventContexts(ctx: QueryCtx, rows: RawJourneyEvent[]): Promise<{
+    _creationTime: number;
+    _id: string;
+    country?: string | undefined;
+    city?: string | undefined;
+    latitude?: number | undefined;
+    longitude?: number | undefined;
+    browser: string;
+    os: string;
+    device: string;
+    affiliateSlug?: string | undefined;
+    currency?: string | undefined;
+    eventId: string;
+    href?: string | undefined;
+    name?: string | undefined;
+    path: string;
+    properties?: Record<string, string | number | boolean | null> | undefined;
+    referrer?: string | undefined;
+    revenueCents?: number | undefined;
+    sequence: number;
+    sessionId: string;
+    siteId: import("convex/values").GenericId<"sites">;
+    target?: string | undefined;
+    timestamp: number;
+    type: "click" | "conversion" | "custom" | "heartbeat" | "outbound" | "pageview";
+    visitorId: string;
+}[]>;
+export declare function deleteTelemetryBefore(ctx: MutationCtx, input: {
+    siteId: Id<"sites">;
+    before: number;
+    batchSize: number;
+}): Promise<{
+    deleted: number;
+    hasMore: boolean;
+}>;
+export {};
+//# sourceMappingURL=eventStore.d.ts.map

package/dist/component/eventStore.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"eventStore.d.ts","sourceRoot":"","sources":["../../src/component/eventStore.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,GAAG,EAAE,EAAE,EAAE,MAAM,2BAA2B,CAAC;AACzD,OAAO,KAAK,EAAE,WAAW,EAAE,QAAQ,EAAE,MAAM,wBAAwB,CAAC;AASpE,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,iBAAiB,CAAC;AAEpD,MAAM,MAAM,cAAc,GAAG;IAC3B,SAAS,EAAE,MAAM,CAAC;IAClB,SAAS,EAAE,MAAM,CAAC;IAClB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,EAAE,CAAC,EAAE,MAAM,CAAC;IACZ,MAAM,CAAC,EAAE,MAAM,CAAC;CACjB,CAAC;AAEF,KAAK,kBAAkB,GAAG;IACxB,MAAM,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC;IACpB,SAAS,EAAE,MAAM,CAAC;IAClB,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,MAAM,EAAE,YAAY,EAAE,CAAC;CACxB,CAAC;AAEF,MAAM,MAAM,gBAAgB,GAAG,GAAG,CAAC,MAAM,EAAE,kBAAkB,CAAC,CAAC;AAY/D,MAAM,MAAM,aAAa,GAAG;IAC1B,SAAS,EAAE,MAAM,CAAC;IAClB,YAAY,EAAE,MAAM,CAAC;IACrB,EAAE,EAAE,MAAM,CAAC;CACZ,CAAC;AAEF,MAAM,MAAM,eAAe,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,KAAK,CAAC,GAAG;IAAE,GAAG,EAAE,MAAM,CAAA;CAAE,CAAC;AAE3E,wBAAgB,gBAAgB,CAAC,SAAS,EAAE,MAAM,GAAG,MAAM,CAE1D;AAED,wBAAgB,aAAa,CAAC,SAAS,EAAE,MAAM,EAAE,WAAW,EAAE,MAAM,GAAG,MAAM,CAE5E;AAED,wBAAsB,sBAAsB,CAC1C,GAAG,EAAE,WAAW,EAChB,KAAK,EAAE;IACL,MAAM,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC;IACpB,SAAS,EAAE,MAAM,CAAC;IAClB,WAAW,EAAE,MAAM,CAAC;IACpB,iBAAiB,EAAE,GAAG,CAAC,MAAM,CAAC,CAAC;CAChC;;;GA0BF;AAED,wBAAsB,aAAa,CACjC,GAAG,EAAE,WAAW,EAChB,MAAM,EAAE,EAAE,CAAC,OAAO,CAAC,EACnB,KAAK,EAAE,YAAY,GAClB,OAAO,CAAC;IAAE,KAAK,EAAE,cAAc,GAAG,IAAI,CAAC;IAAC,SAAS,EAAE,OAAO,CAAA;CAAE,CAAC,CA6C/D;AAED,wBAAgB,gBAAgB,CAC9B,MAAM,EAAE,EAAE,CAAC,OAAO,CAAC,EACnB,KAAK,EAAE,YAAY,EACnB,MAAM,EAAE,gBAAgB,QAmBzB;AAED,wBAAsB,gBAAgB,CACpC,GAAG,EAAE,WAAW,EAChB,MAAM,EAAE,EAAE,CAAC,OAAO,CAAC,EACnB,KAAK,EAAE,YAAY,iBASpB;AAED,wBAAsB,iBAAiB,CACrC,GAAG,EAAE,WAAW,EAChB,MAAM,EAAE,gBAAgB,iBAKzB;AAED,wBAAsB,yBAAyB,CAC7C,GAAG,EAAE,WAAW,EAChB,KAAK,EAAE,GAAG,CAAC,QAAQ,CAAC,iBA0ErB;AAED,wBAAsB,wBAAwB,CAC5C,GAAG,EAAE,QAAQ,EACb,KAAK,EAAE;IACL,MAAM,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC;IACpB,SAAS,EAAE,MAAM,CAAC;IAClB,IAAI,EAAE,MAAM,CAAC;IACb,EAAE,EAAE,MAAM,CAAC;IACX,MAAM,EAAE,aAAa,GAAG,IAAI,CAAC;IAC7B,WAAW,EAAE,MAAM,CAAC;CACrB,GACA,OAAO,CAAC,eAAe,EAAE,CAAC,CAyF5B;AAED,wBAAsB,wBAAwB,CAC5C,GAAG,EAAE,QAAQ,EACb,KAAK,EAAE;IACL,MAAM,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC;IACpB,SAAS,EAAE,MAAM,CAAC;IAClB,IAAI,EAAE,MAAM,CAAC;IACb,EAAE,EAAE,MAAM,CAAC;IACX,WAAW,EAAE,MAAM,CAAC;CACrB,GACA,OAAO,CAAC,eAAe,EAAE,CAAC,CA4D5B;AAED,wBAAgB,oBAAoB,CAClC,IAAI,EAAE,IAAI,CAAC,eAAe,EAAE,WAAW,GAAG,eAAe,GAAG,KAAK,CAAC,EAClE,KAAK,EAAE,IAAI,CAAC,eAAe,EAAE,WAAW,GAAG,eAAe,GAAG,KAAK,CAAC,GAClE,MAAM,CAIR;AAED,wBAAsB,oBAAoB,CACxC,GAAG,EAAE,QAAQ,EACb,IAAI,EAAE,eAAe,EAAE;;SApZyC,MAAM;;;;;;;;;;;;;;;;;;;;;;;;KA6bvE;AAED,wBAAsB,qBAAqB,CACzC,GAAG,EAAE,WAAW,EAChB,KAAK,EAAE;IACL,MAAM,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC;IACpB,MAAM,EAAE,MAAM,CAAC;IACf,SAAS,EAAE,MAAM,CAAC;CACnB;;;GA+BF"}