@iann29/rastro 0.1.0-alpha.2 → 0.1.0-alpha.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (124) hide show
  1. package/README.md +325 -63
  2. package/agent/integration.md +651 -0
  3. package/agent/manifest.json +183 -0
  4. package/agent/manifest.schema.json +405 -0
  5. package/dist/client/federation.d.ts +205 -0
  6. package/dist/client/federation.d.ts.map +1 -0
  7. package/dist/client/federation.js +179 -0
  8. package/dist/client/federation.js.map +1 -0
  9. package/dist/client/index.d.ts +1600 -8
  10. package/dist/client/index.d.ts.map +1 -1
  11. package/dist/client/index.js +210 -2
  12. package/dist/client/index.js.map +1 -1
  13. package/dist/component/_generated/api.d.ts +9 -5
  14. package/dist/component/_generated/api.d.ts.map +1 -1
  15. package/dist/component/_generated/api.js.map +1 -1
  16. package/dist/component/_generated/component.d.ts +161 -1
  17. package/dist/component/_generated/component.d.ts.map +1 -1
  18. package/dist/component/_generated/server.d.ts +4 -0
  19. package/dist/component/_generated/server.d.ts.map +1 -1
  20. package/dist/component/_generated/server.js.map +1 -1
  21. package/dist/component/cardinality.d.ts +12 -0
  22. package/dist/component/cardinality.d.ts.map +1 -0
  23. package/dist/component/cardinality.js +94 -0
  24. package/dist/component/cardinality.js.map +1 -0
  25. package/dist/component/constants.d.ts +10 -0
  26. package/dist/component/constants.d.ts.map +1 -1
  27. package/dist/component/constants.js +10 -0
  28. package/dist/component/constants.js.map +1 -1
  29. package/dist/component/convex.config.d.ts +7 -2
  30. package/dist/component/convex.config.d.ts.map +1 -1
  31. package/dist/component/convex.config.js +9 -4
  32. package/dist/component/convex.config.js.map +1 -1
  33. package/dist/component/coverage.d.ts +18 -0
  34. package/dist/component/coverage.d.ts.map +1 -0
  35. package/dist/component/coverage.js +19 -0
  36. package/dist/component/coverage.js.map +1 -0
  37. package/dist/component/diagnostics.d.ts +9 -0
  38. package/dist/component/diagnostics.d.ts.map +1 -0
  39. package/dist/component/diagnostics.js +47 -0
  40. package/dist/component/diagnostics.js.map +1 -0
  41. package/dist/component/errors.d.ts +1 -1
  42. package/dist/component/errors.d.ts.map +1 -1
  43. package/dist/component/errors.js.map +1 -1
  44. package/dist/component/eventStore.d.ts +11 -9
  45. package/dist/component/eventStore.d.ts.map +1 -1
  46. package/dist/component/eventStore.js +55 -227
  47. package/dist/component/eventStore.js.map +1 -1
  48. package/dist/component/geo.d.ts +71 -0
  49. package/dist/component/geo.d.ts.map +1 -0
  50. package/dist/component/geo.js +610 -0
  51. package/dist/component/geo.js.map +1 -0
  52. package/dist/component/http.d.ts.map +1 -1
  53. package/dist/component/http.js +168 -17
  54. package/dist/component/http.js.map +1 -1
  55. package/dist/component/ingest.d.ts.map +1 -1
  56. package/dist/component/ingest.js +272 -32
  57. package/dist/component/ingest.js.map +1 -1
  58. package/dist/component/live.d.ts.map +1 -1
  59. package/dist/component/live.js +6 -3
  60. package/dist/component/live.js.map +1 -1
  61. package/dist/component/reports.d.ts +117 -8
  62. package/dist/component/reports.d.ts.map +1 -1
  63. package/dist/component/reports.js +641 -79
  64. package/dist/component/reports.js.map +1 -1
  65. package/dist/component/retention.d.ts +75 -1
  66. package/dist/component/retention.d.ts.map +1 -1
  67. package/dist/component/retention.js +517 -54
  68. package/dist/component/retention.js.map +1 -1
  69. package/dist/component/sanitize.d.ts +4 -1
  70. package/dist/component/sanitize.d.ts.map +1 -1
  71. package/dist/component/sanitize.js +9 -4
  72. package/dist/component/sanitize.js.map +1 -1
  73. package/dist/component/schema.d.ts +202 -53
  74. package/dist/component/schema.js +113 -16
  75. package/dist/component/schema.js.map +1 -1
  76. package/dist/component/sites.d.ts.map +1 -1
  77. package/dist/component/sites.js +5 -1
  78. package/dist/component/sites.js.map +1 -1
  79. package/dist/component/validators.d.ts +29 -51
  80. package/dist/component/validators.d.ts.map +1 -1
  81. package/dist/component/validators.js +11 -12
  82. package/dist/component/validators.js.map +1 -1
  83. package/dist/tracker/generated.d.ts +4 -4
  84. package/dist/tracker/generated.d.ts.map +1 -1
  85. package/dist/tracker/generated.js +4 -4
  86. package/dist/tracker/generated.js.map +1 -1
  87. package/dist/tracker/tracker.js +7 -6
  88. package/dist/tracker/tracker.js.map +1 -1
  89. package/dist/tracker.min.js +1 -1
  90. package/docs/benchmarks/2026-08-20-realistic.md +5 -5
  91. package/docs/benchmarks/2026-08-21-formal-certification.md +333 -0
  92. package/docs/federation-setup.md +395 -0
  93. package/docs/federation.md +258 -0
  94. package/docs/upgrading.md +130 -0
  95. package/llms.txt +65 -0
  96. package/package.json +29 -7
  97. package/scripts/benchmark-ingest.mjs +81 -32
  98. package/scripts/generate-federation-keys.mjs +20 -0
  99. package/src/component/_generated/api.ts +9 -5
  100. package/src/component/_generated/component.ts +224 -1
  101. package/src/component/_generated/server.ts +4 -0
  102. package/src/component/cardinality.ts +116 -0
  103. package/src/component/constants.ts +10 -0
  104. package/src/component/convex.config.ts +11 -5
  105. package/src/component/coverage.ts +25 -0
  106. package/src/component/diagnostics.ts +64 -0
  107. package/src/component/errors.ts +2 -1
  108. package/src/component/eventStore.ts +78 -280
  109. package/src/component/geo.ts +779 -0
  110. package/src/component/http.ts +259 -22
  111. package/src/component/ingest.ts +436 -27
  112. package/src/component/live.ts +8 -3
  113. package/src/component/reports.ts +895 -87
  114. package/src/component/retention.ts +624 -88
  115. package/src/component/sanitize.ts +15 -4
  116. package/src/component/schema.ts +125 -16
  117. package/src/component/sites.ts +5 -1
  118. package/src/component/validators.ts +12 -13
  119. package/src/test.ts +0 -2
  120. package/dist/component/migrations.d.ts +0 -17
  121. package/dist/component/migrations.d.ts.map +0 -1
  122. package/dist/component/migrations.js +0 -44
  123. package/dist/component/migrations.js.map +0 -1
  124. package/src/component/migrations.ts +0 -52
package/README.md CHANGED
@@ -4,35 +4,79 @@
4
4
 
5
5
  Privacy-first, real-time web analytics built as an isolated Convex component.
6
6
 
7
- Amage Rastro combines a 1,022-byte gzip browser tracker with reactive traffic
7
+ Amage Rastro combines a 1,000-byte gzip browser tracker with reactive traffic
8
8
  metrics, a live visitor atlas, journey timelines, trusted payment attribution,
9
9
  goals, funnels, affiliates, and multi-site reports. Data and functions stay
10
10
  inside your Convex deployment.
11
11
 
12
12
  ![Amage Rastro Live Atlas](assets/rastro-live-atlas.png)
13
13
 
14
+ ## Project status
15
+
16
+ Amage Rastro is an actively developed alpha with a canonical batched event
17
+ store, protected CI, exact run-scoped benchmark reconciliation, and published
18
+ npm prereleases. The untagged `latest` channel intentionally remains on the
19
+ first public alpha. Federation consumers must statically inspect an
20
+ operator-approved exact registry artifact for every required runtime and type
21
+ export before changing a host; follow the gate in the
22
+ [federation setup runbook](docs/federation-setup.md). Never substitute a mutable
23
+ branch or website-supplied source tarball.
24
+
25
+ Public integration documentation is available at
26
+ [`www.amagerastro.com/docs/`](https://www.amagerastro.com/docs/). Coding agents
27
+ should start with [`llms.txt`](https://www.amagerastro.com/llms.txt) and verify
28
+ the [machine manifest](https://www.amagerastro.com/agent/manifest.json).
29
+
30
+ Current evidence supports more than 120 million stored events/day for realistic
31
+ traffic. Near-limit heavy payloads are certified at 76.62 million/day, and the
32
+ feature-saturated profile at 33.22 million/day. Workload-specific results and
33
+ caveats are reported separately rather than collapsed into one capacity claim.
34
+
14
35
  ## What ships
15
36
 
16
- | Capability | Implementation |
17
- | --------------------------- | --------------------------------------------------------------------------------------------------- |
18
- | Live visitor map | Reactive live sessions with coarse city-level coordinates and TTL expiry |
19
- | Traffic analytics | Sharded hour/day rollups, time series, top pages, sources, countries, devices, browsers, and events |
20
- | Journey replay | Bounded pageview/click/custom-event timelines; no DOM, keystroke, or form-value recording |
21
- | Payment conversion tracking | Trusted server-side ledger with idempotent event IDs and canonical site currency |
22
- | Goals and funnels | Exact pageview/custom-event goals and ordered conversion funnels |
23
- | Affiliates | Referral attribution, trusted revenue, and commission accounting |
24
- | Cross-site analytics | Reports over up to ten authorized site IDs with a preloaded pseudonymous visitor ID |
25
- | Auto-captured clicks | Anchors, buttons, outbound links, and explicit `data-rastro-event` controls |
26
- | Cookieless mode | Session-scoped pseudonymous IDs in `sessionStorage`; no cookies |
27
- | Tiny tracker | 1,909 bytes raw, 1,022 bytes gzip; the test suite enforces `< 1,024` bytes |
28
-
29
- ## Install
37
+ | Capability | Implementation |
38
+ | --------------------------- | ---------------------------------------------------------------------------------------------------- |
39
+ | Live visitor map | Reactive live sessions with coarse city-level coordinates and TTL expiry |
40
+ | Traffic analytics | Sharded hour/day rollups, time series, top pages, sources, countries, devices, browsers, and events |
41
+ | Journey replay | Bounded pageview/click/custom-event timelines; no DOM, keystroke, or form-value recording |
42
+ | Payment conversion tracking | Trusted server-side ledger with idempotent event IDs and canonical site currency |
43
+ | Goals and funnels | Exact pageview/custom-event goals and ordered conversion funnels |
44
+ | Affiliates | Referral attribution, trusted revenue, and commission accounting |
45
+ | Cross-site analytics | Reports over up to ten authorized site IDs with a preloaded pseudonymous visitor ID |
46
+ | Automatic GeoIP | Opt-in native request metadata plus IPinfo/IPWhois fallback; bounded quota and no raw-IP persistence |
47
+ | Auto-captured clicks | Anchors, buttons, outbound links, and explicit `data-rastro-event` controls |
48
+ | Cookieless mode | Session-scoped pseudonymous IDs in `sessionStorage`; no cookies |
49
+ | Tiny tracker | 1,846 bytes raw, 1,000 bytes gzip; the test suite enforces `< 1,024` bytes |
50
+
51
+ ## Quickstart
52
+
53
+ ### Prerequisites
54
+
55
+ - Convex `^1.43.0` or a compatible Synapse deployment.
56
+ - An existing host authentication provider for administrative and local report
57
+ functions. The browser ingestion route itself is public and origin-checked.
58
+ - Two deployment origins recorded separately:
59
+
60
+ | Purpose | Convex Cloud example | Synapse example |
61
+ | ------------------------------ | ------------------------------ | -------------------------------------- |
62
+ | Functions and reactive queries | `https://project.convex.cloud` | `https://project.synapse.example` |
63
+ | HTTP actions and tracker | `https://project.convex.site` | `https://project.site.synapse.example` |
64
+
65
+ Never append `/rastro/tracker.js` to the functions URL.
66
+
67
+ ### 1. Install
30
68
 
31
69
  ```bash
32
- npm install @iann29/rastro
70
+ npm install @iann29/rastro@alpha
33
71
  ```
34
72
 
35
- Mount the component under an HTTP prefix:
73
+ The untagged npm `latest` channel intentionally remains on the first public
74
+ alpha. Use `@alpha` until a stable release is promoted.
75
+
76
+ ### 2. Mount the component
77
+
78
+ Merge the component into the project's existing configuration; preserve every
79
+ component and environment binding already registered there.
36
80
 
37
81
  ```ts
38
82
  // convex/convex.config.ts
@@ -45,14 +89,93 @@ app.use(rastro, { httpPrefix: "/rastro/" });
45
89
  export default app;
46
90
  ```
47
91
 
48
- This creates isolated component tables and three HTTP routes:
92
+ This creates isolated component tables and four HTTP routes:
49
93
 
50
94
  - `POST /rastro/events` — browser event ingestion
51
95
  - `OPTIONS /rastro/events` — CORS preflight
52
96
  - `GET /rastro/tracker.js` — the pre-gzipped tracker
53
97
  - `GET /rastro/health` — integration health
54
98
 
55
- ## Create a site
99
+ Push the component and regenerate host bindings before importing
100
+ `components.rastroAnalytics`:
101
+
102
+ ```bash
103
+ # Convex Cloud development deployment
104
+ npx convex dev
105
+
106
+ # Synapse development deployment
107
+ synapse dev --once
108
+ ```
109
+
110
+ For production, use the product's existing release workflow. On an uncoordinated
111
+ Convex project that is `npx convex deploy`; on Synapse use
112
+ `synapse release --execute` when the project has a release manifest, otherwise
113
+ `synapse deploy`. Never deploy before confirming the target.
114
+
115
+ Verify the HTTP origin independently from the functions origin:
116
+
117
+ ```bash
118
+ curl -fsS https://YOUR_HTTP_ORIGIN/rastro/health
119
+ ```
120
+
121
+ A successful health response proves that the component HTTP routes are mounted;
122
+ it does not prove that a browser event was accepted.
123
+
124
+ ### Automatic GeoIP
125
+
126
+ Rastro uses Convex request metadata as the canonical client IP and can resolve
127
+ coarse city-level coordinates on the first request of each session. GeoIP is
128
+ opt-in because the configured provider receives the request IP. For IPinfo Core:
129
+
130
+ ```ts
131
+ // convex/convex.config.ts
132
+ import { defineApp } from "convex/server";
133
+ import { v } from "convex/values";
134
+ import rastro from "@iann29/rastro/convex.config.js";
135
+
136
+ const app = defineApp({ env: { IPINFO_TOKEN: v.string() } });
137
+ app.use(rastro, {
138
+ httpPrefix: "/rastro/",
139
+ env: {
140
+ RASTRO_GEOIP_PROVIDER: "ipinfo",
141
+ RASTRO_GEOIP_TOKEN: app.env.IPINFO_TOKEN,
142
+ RASTRO_GEOIP_DAILY_LIMIT: "1000",
143
+ },
144
+ });
145
+
146
+ export default app;
147
+ ```
148
+
149
+ `ipwhois` is also supported. Its token is optional for the documented
150
+ 1,000-request/day free endpoint and recommended for production. Bind it as
151
+ `RASTRO_GEOIP_TOKEN` in the same way as the IPinfo example.
152
+ `RASTRO_GEOIP_DAILY_LIMIT` defaults to 1,000 lookups per site per UTC day and
153
+ may be set from `0` through `1000000` to match the provider plan. Admission is
154
+ transactional, so concurrent first-session requests cannot exceed the cap.
155
+
156
+ When Rastro runs behind a reverse proxy that supplies `X-Forwarded-For` or
157
+ Vercel/Cloudflare geographic headers, explicitly bind
158
+ `RASTRO_TRUST_PROXY: "true"`. Leave it unset for direct Convex deployments.
159
+ Rastro uses forwarded data only when this opt-in is enabled and the runtime IP
160
+ identifies a private immediate peer. Caller-supplied proxy or geographic headers
161
+ are ignored when native metadata is absent or public.
162
+
163
+ The lookup has a 1.5-second timeout and fails open: telemetry ingestion
164
+ continues without location when the provider is unavailable. A session reserves
165
+ its single lookup attempt atomically only after its initial event is persisted;
166
+ heartbeats and concurrent duplicate requests do not consume provider quota. The
167
+ raw IP is never passed to a mutation or stored in component tables; only
168
+ sanitized country, city, and coordinates rounded to one decimal place are
169
+ persisted.
170
+
171
+ `GET /rastro/health` reports whether the runtime exposes a usable client IP and
172
+ whether GeoIP is configured and the daily cap, without returning the address or
173
+ token. Deployment administrators can exercise the configured provider through
174
+ the internal `geo:probe` action. If health reports no usable client IP, use
175
+ trusted Vercel/Cloudflare geographic headers with `RASTRO_TRUST_PROXY` or fix
176
+ the deployment's proxy forwarding.
177
+
178
+ ### 3. Create a site
56
179
 
57
180
  Components cannot read the host app's authentication state. The host must
58
181
  authenticate the caller and pass an opaque owner ID.
@@ -78,7 +201,7 @@ export const createSite = mutation({
78
201
  if (!identity) throw new Error("Not authenticated");
79
202
 
80
203
  return await analytics.createSite(ctx, {
81
- ownerId: identity.subject,
204
+ ownerId: identity.tokenIdentifier,
82
205
  name: args.name,
83
206
  domains: args.domains,
84
207
  currency: args.currency,
@@ -88,12 +211,30 @@ export const createSite = mutation({
88
211
  });
89
212
  ```
90
213
 
214
+ Call this mutation from the host application's authenticated administration UI
215
+ and persist or display the returned `siteId`:
216
+
217
+ ```tsx
218
+ const createSite = useMutation(api.analyticsAdmin.createSite);
219
+ const siteId = await createSite({
220
+ name: "Amage Product",
221
+ domains: ["product.example.com", "localhost"],
222
+ currency: "BRL",
223
+ });
224
+ ```
225
+
226
+ Do not call the authenticated example with a bare CLI command: without an
227
+ explicit test identity, `ctx.auth.getUserIdentity()` correctly returns `null`.
228
+
91
229
  Allowed domains are checked at ingestion. Exact hosts and wildcard subdomains
92
230
  such as `*.example.com` are supported. `Origin` is an abuse signal, not
93
231
  authentication; Amage Rastro also enforces per-session and fixed-shard per-site
94
232
  event/byte budgets.
95
233
 
96
- ## Add the tracker
234
+ Domain entries are hostnames, not host-and-port pairs. Use `localhost` or a full
235
+ origin such as `http://localhost:3000`; both normalize to the `localhost` host.
236
+
237
+ ### 4. Add the tracker
97
238
 
98
239
  Use the HTTP-actions URL for your deployment, not the Convex function URL.
99
240
 
@@ -125,6 +266,9 @@ serves the script and ingestion from different origins:
125
266
 
126
267
  ### React
127
268
 
269
+ React is an optional peer dependency. It is required only when importing the
270
+ `@iann29/rastro/react` entry point; backend-only consumers do not need it.
271
+
128
272
  ```tsx
129
273
  import { useState } from "react";
130
274
  import { RastroScript, useRastro } from "@iann29/rastro/react";
@@ -154,6 +298,26 @@ the same value on authorized sites to connect their journeys. Amage Rastro
154
298
  intentionally has no mutable post-load `identify` command: changing identity
155
299
  after a session starts would make the session ambiguous.
156
300
 
301
+ ### 5. Verify ingestion
302
+
303
+ 1. Open an allowed site origin in a fresh browser tab.
304
+ 2. In browser developer tools, filter Network requests by `rastro/events`.
305
+ 3. Wait for the normal tracker flush, which is debounced by approximately 1.2
306
+ seconds.
307
+ 4. Confirm `POST /rastro/events` returns HTTP 200 and a response containing
308
+ `accepted`, `duplicates`, and `rejected` counters.
309
+ 5. Run one authenticated local report, such as `analytics:overview`, for the
310
+ returned `siteId` and a complete UTC hour/day range.
311
+
312
+ The tracker intentionally does not interrupt the product UI when transport
313
+ fails. A loaded `tracker.js` or successful `/health` response alone is not proof
314
+ of ingestion. The usual causes of a rejected or missing request are:
315
+
316
+ - using the functions URL instead of the HTTP-actions URL;
317
+ - omitting the exact browser host from the site's `domains`;
318
+ - including query strings or paths in a configured base URL;
319
+ - querying a partial UTC bucket or a site the authenticated user does not own.
320
+
157
321
  ## Track custom events
158
322
 
159
323
  ```js
@@ -246,12 +410,12 @@ export const {
246
410
 
247
411
  for (const siteId of request.siteIds) {
248
412
  const site = await analytics.getSite(ctx, siteId);
249
- if (!site || site.ownerId !== identity.subject) {
413
+ if (!site || site.ownerId !== identity.tokenIdentifier) {
250
414
  throw new Error("Not authorized for analytics site");
251
415
  }
252
416
  }
253
417
 
254
- if (request.ownerId && request.ownerId !== identity.subject) {
418
+ if (request.ownerId && request.ownerId !== identity.tokenIdentifier) {
255
419
  throw new Error("Not authorized for analytics owner");
256
420
  }
257
421
  },
@@ -262,6 +426,30 @@ Do not copy the unauthenticated demo boundary into production. The included demo
262
426
  exposes only fixed-scope read reports plus bounded, idempotent seed/live-refresh
263
427
  functions.
264
428
 
429
+ ### Federated dashboard backend
430
+
431
+ The reference control plane in `example/` can connect a central dashboard while
432
+ telemetry remains in the customer's deployment. It uses
433
+ `exposeFederatedAnalyticsApi` with issuer and connection-resolver options. The
434
+ helper constructs its authorization boundary internally; it cannot accept a
435
+ permissive host authorizer. Its surface contains a public protocol v1 manifest,
436
+ authenticated redacted connection status, redacted site summaries, and only the
437
+ reactive dashboard report queries.
438
+
439
+ The JWT only identifies the requested connection. Site access remains
440
+ authoritative in a local host record resolved on every query, so revocation is
441
+ immediate and a token cannot grant itself another site. `listSites` accepts no
442
+ site arguments and omits owner, domain, and network configuration. Stable
443
+ manifest names, capabilities, limits, and authorization error codes are exported
444
+ for connector clients. The host must already have Convex authentication and a
445
+ local authoritative grant. Follow
446
+ [`docs/federation-setup.md`](docs/federation-setup.md) for the executable setup
447
+ and verification sequence; use [`docs/federation.md`](docs/federation.md) as the
448
+ protocol reference. The production control plane is available at
449
+ `https://www.amagerastro.com`; before connecting a customer deployment, verify
450
+ the selected package exports the federation helper as shown in the setup
451
+ runbook.
452
+
265
453
  ## Backend API
266
454
 
267
455
  The `Rastro` class wraps the component boundary for host functions:
@@ -270,17 +458,46 @@ The `Rastro` class wraps the component boundary for host functions:
270
458
  - Ingestion: `ingest`, `trackConversion`
271
459
  - Reports: `overview`, `liveVisitors`, `listSessions`, `sessionJourney`,
272
460
  `listConversions`, `visitorJourney`, `goalsReport`, `funnelsReport`,
273
- `affiliatesReport`
461
+ `affiliatesReport`, `dataCoverage`
274
462
  - Configuration: `upsertGoal`, `listGoals`, `removeGoal`, `upsertFunnel`,
275
463
  `listFunnels`, `removeFunnel`, `upsertAffiliate`, `listAffiliates`,
276
464
  `removeAffiliate`
277
- - Retention: `cleanup`
465
+ - Retention: `cleanup`, `setRetentionPolicy`, `disableRetentionPolicy`,
466
+ `runRetentionPolicy`, `retentionStatus`
278
467
 
279
468
  Every public function has argument and return validators. Growing reads are
280
469
  indexed and bounded. Component pagination uses scope-bound opaque keyset cursors
281
470
  because native `.paginate()` is not supported inside Convex components; page
282
471
  cost stays constant instead of increasing with an offset.
283
472
 
473
+ Overview responses label visitor values as either HLL-estimated unique visitors
474
+ or legacy `bucketOccurrences`; legacy aggregate rows are never presented as
475
+ exact uniques. Breakdown metadata likewise identifies page, source, country,
476
+ device, browser, and event rankings as event-volume counts. Feature status is
477
+ explicit: outbound-link event volume is supported, while session replay, error
478
+ insights, and bot detection are currently unsupported.
479
+
480
+ All report ranges use integer Unix milliseconds with inclusive `from` and `to`
481
+ boundaries. Range metadata reports `boundary: "inclusive"` and
482
+ `timezone: "UTC"`; the site's timezone is presentation metadata and never
483
+ changes report bucket boundaries. Overview queries must contain complete UTC
484
+ hours or days, and funnel and affiliate reports must contain complete UTC days.
485
+ Those aggregate-backed reports fail with `REPORT_INCOMPLETE` instead of silently
486
+ including the rest of a partial bucket. Goal reports retain exact millisecond
487
+ filtering while bounded goal-completion rows are available.
488
+
489
+ Goal, funnel, and affiliate reports use definition-ID keyed daily rollups when
490
+ the low-churn rollout control proves that generation is available. Each loader
491
+ enforces day, row, document, and byte headroom before returning a complete
492
+ result; it never returns a silently truncated report. Ordinary ingestion never
493
+ patches or schedules work against a shared coverage document. `dataCoverage`
494
+ derives availability through bounded indexed reads of source and rollup records,
495
+ while the control document stores only rollout generation and durable retention
496
+ watermarks. Heartbeats can advance session availability but do not claim event
497
+ or aggregate availability. Hourly and daily overview availability are reported
498
+ separately as `overviewHour` and `overviewDay` because their retention
499
+ boundaries use different bucket alignment.
500
+
284
501
  ## Operational limits
285
502
 
286
503
  - HTTP body: 64 KiB
@@ -290,17 +507,25 @@ cost stays constant instead of increasing with an offset.
290
507
  - Primary aggregate shards: 256 hourly and 128 daily, derived from stable
291
508
  session identity
292
509
  - Funnel and affiliate aggregate shards: 16
293
- - Site ingress budget: 4,096 deterministic per-minute shards, each capped at
294
- 300 events and 512 KiB (1,228,800 admitted events/minute at uniform load)
510
+ - Site ingress budget: 4,096 deterministic per-minute shards, each capped at 300
511
+ events and 512 KiB (1,228,800 admitted events/minute at uniform load)
295
512
  - Rate-window storage: one reusable row per active session and site shard
296
513
  - Live session timeout: 60 seconds with one bounded five-second expiry sweep per
297
514
  site; legacy per-session jobs hand off to that sweep during the cutover
298
515
  - Dashboard overview: up to ten sites and bounded report ranges
299
516
  - Hourly overview: up to 24 hours; wider ranges use daily buckets
300
- - Retention cleanup: bounded batches; schedule host maintenance for the data
301
- classes your policy retains. The `events` cleanup kind covers both legacy
302
- event rows and new event batches and processes at most 32 heavy batches per
303
- transaction.
517
+ - Retention cleanup: bounded batches. `setRetentionPolicy` automatically starts
518
+ leased, persisted cleanup chains for every policy-managed dataset; interrupted
519
+ attempts resume after lease expiry and stale generations cannot mutate the
520
+ active chain. A completed chain schedules its next cycle after one day and
521
+ recomputes its cutoff from the then-current policy. Policy updates replace the
522
+ generation, and `disableRetentionPolicy` makes every queued attempt stale.
523
+ `runRetentionPolicy` remains available for an explicit bounded run. The
524
+ `events` cleanup kind removes canonical event-batch documents and processes at
525
+ most 32 heavy batches per transaction.
526
+ - Retention cutoffs are bucket-safe: event minutes, hourly aggregates, daily
527
+ aggregates and feature rollups preserve the bucket containing a cutoff and
528
+ return the exact effective `before` boundary they applied.
304
529
 
305
530
  Capacity claims use declared workload profiles. Complete release evidence
306
531
  reports ten-minute `lean`, `realistic`, `heavy`, and browser `feature` profiles
@@ -308,12 +533,52 @@ separately. The browser `feature` profile exercises configured goals, funnels,
308
533
  and affiliate attribution; trusted conversions require a separate server-side
309
534
  driver and certification result.
310
535
 
311
- The historical development evidence and its certification caveats are recorded
312
- in [`docs/benchmarks/2026-08-20-realistic.md`](docs/benchmarks/2026-08-20-realistic.md).
536
+ ### Historical ingest evidence requiring recertification
537
+
538
+ | Profile | Workload | Certified stored/day | p95 latency | 100M/day target |
539
+ | --------- | ------------------------------------------------- | -------------------: | ----------: | --------------- |
540
+ | Lean | Minimal events, maximum batches | 136.65M | 4,543 ms | Pass |
541
+ | Realistic | Reused sessions, browser mix, context, heartbeats | 120.81M | 4,247 ms | Pass |
542
+ | Heavy | 62 KiB requests with bounded custom properties | 76.62M | 3,670 ms | Below target |
543
+ | Feature | Goals, funnels, and affiliate attribution | 33.22M | 6,226 ms | Below target |
544
+
545
+ Every listed run lasted at least ten minutes, completed with zero request
546
+ failures, duplicates, or rejected events, and exactly reconciled the expected
547
+ stored events. Results came from one client process against a non-production
548
+ Synapse DEV deployment; they are engineering evidence, not a production SLA. The
549
+ heavy row is the formal recertification after removing redundant payload
550
+ serialization. GeoIP with a provider enabled and trusted server-side conversions
551
+ remain separate pending certifications.
552
+
553
+ These runs predate the current source-derived coverage and recurring-retention
554
+ changes. They remain historical evidence for the named revisions and profiles,
555
+ not certification of this uncommitted implementation. Recertify the final code
556
+ before presenting any row as current capacity.
557
+
558
+ The latest formal results are recorded in
559
+ [`docs/benchmarks/2026-08-21-formal-certification.md`](docs/benchmarks/2026-08-21-formal-certification.md).
560
+ Earlier development evidence remains available in
561
+ [`docs/benchmarks/2026-08-20-realistic.md`](docs/benchmarks/2026-08-20-realistic.md).
313
562
 
314
563
  From a repository checkout, run a duration-based profile against a confirmed
315
564
  non-production deployment and a dedicated idle benchmark site:
316
565
 
566
+ For a deployment-free CI smoke benchmark of write-heavy ingestion plus reactive
567
+ overview and feature report queries, run:
568
+
569
+ ```bash
570
+ npm run benchmark:local
571
+ ```
572
+
573
+ The local harness uses `convex-test`, enforces a 12-document ingest transaction
574
+ budget, checks reactive totals before and after a write, and queries proven
575
+ daily rollups with the maximum configured 50 goals, 20 funnels, and 100
576
+ affiliates under deterministic 1,000-document and 4 MiB read ceilings. It also
577
+ drives concurrent ingest calls and asserts that their shared rollout/retention
578
+ control document is unchanged. It fails when its generous CI latency budgets are
579
+ exceeded. This is a deterministic local regression guard, not evidence of live
580
+ deployment throughput or capacity.
581
+
317
582
  ```bash
318
583
  npm run --silent benchmark:ingest -- \
319
584
  --url https://example.site/rastro/events \
@@ -322,7 +587,8 @@ npm run --silent benchmark:ingest -- \
322
587
  --origin https://your-allowed-origin.example \
323
588
  --profile realistic \
324
589
  --duration 600 \
325
- --concurrency 100 \
590
+ --concurrency 75 \
591
+ --batch-size 50 \
326
592
  --certify
327
593
  ```
328
594
 
@@ -332,12 +598,15 @@ visible in Live Atlas. Only `feature` intentionally matches the demo goal,
332
598
  funnel, and affiliate definitions; it does not call the trusted conversion API.
333
599
  Use `--dry-run` to inspect generated request size without sending traffic. When
334
600
  `--convex-url` is supplied, it must identify the same deployment as `--url` and
335
- expose this repository's `example:overview` query. The final reconciliation is
336
- a site-wide before/after delta, so unrelated traffic on that site invalidates
337
- the result. Reports distinguish accepted ingress, heartbeats, and stored events.
338
- Shorter completed runs report extrapolated daily estimates only. `--certify`
339
- requires at least ten minutes and successful reconciliation before emitting
340
- certified daily capacity fields or exiting successfully as a certification.
601
+ expose this repository's `benchmark:reconcileSessionsPage` query. Reconciliation
602
+ generates exact session IDs from a cryptographic run UUID and reads them through
603
+ bounded index lookups, so unrelated traffic cannot alter the result. The query
604
+ uses the session counters persisted transactionally with event batches; it
605
+ certifies ingest accounting, not raw-event availability after later retention.
606
+ Reports distinguish accepted ingress, heartbeats, and stored events. Shorter
607
+ completed runs report extrapolated daily estimates only. `--certify` requires at
608
+ least ten minutes and successful reconciliation before emitting certified daily
609
+ capacity fields or exiting successfully as a certification.
341
610
 
342
611
  The ingestion transaction folds metric/dimension deltas, session rate counters,
343
612
  site-shard counters, session/live state, visitor attribution, and exact funnel
@@ -350,26 +619,16 @@ explicit `(other)` bucket instead of relabeling old counts.
350
619
  Click/outbound/browser-conversion telemetry skips goal and funnel work that
351
620
  cannot match those event types.
352
621
 
353
- New telemetry is written only to `eventBatches`. Existing `events` rows remain
354
- readable until an operator runs the resumable migration mounted inside the
355
- Rastro component:
356
-
357
- ```bash
358
- npx convex run --component rastroAnalytics migrations:migrateLegacyEvents \
359
- '{"dryRun":true,"reset":true}'
360
- npx convex run --component rastroAnalytics migrations:migrateLegacyEvents '{}'
361
- npx convex run --component rastroAnalytics migrations:legacyEventsStatus '{}'
362
- ```
622
+ Raw telemetry is stored only in immutable `eventBatches`. Migrated events retain
623
+ their original per-event browser and geo context inside those batches, while new
624
+ events hydrate context from their session. Journey `_id` values are stable
625
+ strings derived from batch and nested-event identity, and session-journey
626
+ cursors use timestamp, creation time, and ID.
363
627
 
364
- Replace `rastroAnalytics` if the host mounted the component under another name.
365
- Proceed only when status returns `complete: true`, which requires both durable
366
- migration success and an empty legacy source. A source refilled after completion
367
- requires an explicit `{"reset":true}` rerun. Migrated rows are compacted into
368
- bounded canonical batches and retain their original per-event browser and geo
369
- context. Journey `_id` values remain stable strings for both formats during the
370
- cutover, and session-journey cursors use timestamp, creation time, and ID. The
371
- compatibility logic remains isolated in `src/component/eventStore.ts` until a
372
- later release removes the empty legacy table and its read paths.
628
+ Existing deployments that still contain the retired `events` table must not
629
+ upgrade directly to this cutover. Follow the exact intermediate release and
630
+ verification sequence in [`docs/upgrading.md`](docs/upgrading.md). Fresh
631
+ deployments need no legacy-event migration.
373
632
 
374
633
  ## Privacy contract
375
634
 
@@ -377,8 +636,10 @@ Amage Rastro's default tracker:
377
636
 
378
637
  - sets no cookies;
379
638
  - stores a pseudonymous session ID and referral slug in `sessionStorage`;
380
- - does not collect or persist raw visitor IP addresses;
381
- - stores coarse location only when the host edge provides it;
639
+ - never persists raw visitor IP addresses in component tables;
640
+ - processes an IP only when an edge supplies geography or an operator enables a
641
+ GeoIP provider, which receives that IP under its own privacy terms;
642
+ - stores only country, city, and coordinates rounded to one decimal place;
382
643
  - does not record the DOM, screenshots, keystrokes, form values, or arbitrary
383
644
  page text;
384
645
  - captures only bounded paths, explicit properties, button/anchor targets, and
@@ -412,14 +673,15 @@ npm test
412
673
  npm run typecheck
413
674
  npm run lint
414
675
  npm run build
415
- cd example && npx vite build
676
+ npm exec vite -- build example
416
677
  ```
417
678
 
418
679
  The test suite covers origin enforcement, event/privacy sanitization,
419
680
  idempotency, reusable session/site rate limits, bounded live expiry chains,
420
681
  folded aggregate writes, goals, funnels, trusted payments, currency safety,
421
682
  affiliate commission, cross-site journeys, keyset pagination beyond 1,000 rows,
422
- retention, HTTP routes, React integration, and tracker size/behavior.
683
+ retention, request metadata and GeoIP, exact byte boundaries, HTTP routes, React
684
+ integration, and tracker size/behavior.
423
685
 
424
686
  ## Synapse demo
425
687