@iann29/rastro 0.1.0-alpha.3 → 0.1.0-alpha.4

package/README.md

@@ -11,28 +11,72 @@ inside your Convex deployment.
 
 ![Amage Rastro Live Atlas](assets/rastro-live-atlas.png)
 
+## Project status
+
+Amage Rastro is an actively developed alpha with a canonical batched event
+store, protected CI, exact run-scoped benchmark reconciliation, and published
+npm prereleases. The untagged `latest` channel intentionally remains on the
+first public alpha. Federation consumers must statically inspect an
+operator-approved exact registry artifact for every required runtime and type
+export before changing a host; follow the gate in the
+[federation setup runbook](docs/federation-setup.md). Never substitute a mutable
+branch or website-supplied source tarball.
+
+Public integration documentation is available at
+[`www.amagerastro.com/docs/`](https://www.amagerastro.com/docs/). Coding agents
+should start with [`llms.txt`](https://www.amagerastro.com/llms.txt) and verify
+the [machine manifest](https://www.amagerastro.com/agent/manifest.json).
+
+Current evidence supports more than 120 million stored events/day for realistic
+traffic. Near-limit heavy payloads are certified at 76.62 million/day, and the
+feature-saturated profile at 33.22 million/day. Workload-specific results and
+caveats are reported separately rather than collapsed into one capacity claim.
+
 ## What ships
 
-| Capability                  | Implementation                                                                                      |
-| --------------------------- | --------------------------------------------------------------------------------------------------- |
-| Live visitor map            | Reactive live sessions with coarse city-level coordinates and TTL expiry                            |
-| Traffic analytics           | Sharded hour/day rollups, time series, top pages, sources, countries, devices, browsers, and events |
-| Journey replay              | Bounded pageview/click/custom-event timelines; no DOM, keystroke, or form-value recording           |
-| Payment conversion tracking | Trusted server-side ledger with idempotent event IDs and canonical site currency                    |
-| Goals and funnels           | Exact pageview/custom-event goals and ordered conversion funnels                                    |
-| Affiliates                  | Referral attribution, trusted revenue, and commission accounting                                    |
-| Cross-site analytics        | Reports over up to ten authorized site IDs with a preloaded pseudonymous visitor ID                 |
-| Auto-captured clicks        | Anchors, buttons, outbound links, and explicit `data-rastro-event` controls                         |
-| Cookieless mode             | Session-scoped pseudonymous IDs in `sessionStorage`; no cookies                                     |
-| Tiny tracker                | 1,846 bytes raw, 1,000 bytes gzip; the test suite enforces `< 1,024` bytes                          |
-
-## Install
+| Capability                  | Implementation                                                                                       |
+| --------------------------- | ---------------------------------------------------------------------------------------------------- |
+| Live visitor map            | Reactive live sessions with coarse city-level coordinates and TTL expiry                             |
+| Traffic analytics           | Sharded hour/day rollups, time series, top pages, sources, countries, devices, browsers, and events  |
+| Journey replay              | Bounded pageview/click/custom-event timelines; no DOM, keystroke, or form-value recording            |
+| Payment conversion tracking | Trusted server-side ledger with idempotent event IDs and canonical site currency                     |
+| Goals and funnels           | Exact pageview/custom-event goals and ordered conversion funnels                                     |
+| Affiliates                  | Referral attribution, trusted revenue, and commission accounting                                     |
+| Cross-site analytics        | Reports over up to ten authorized site IDs with a preloaded pseudonymous visitor ID                  |
+| Automatic GeoIP             | Opt-in native request metadata plus IPinfo/IPWhois fallback; bounded quota and no raw-IP persistence |
+| Auto-captured clicks        | Anchors, buttons, outbound links, and explicit `data-rastro-event` controls                          |
+| Cookieless mode             | Session-scoped pseudonymous IDs in `sessionStorage`; no cookies                                      |
+| Tiny tracker                | 1,846 bytes raw, 1,000 bytes gzip; the test suite enforces `< 1,024` bytes                           |
+
+## Quickstart
+
+### Prerequisites
+
+- Convex `^1.43.0` or a compatible Synapse deployment.
+- An existing host authentication provider for administrative and local report
+  functions. The browser ingestion route itself is public and origin-checked.
+- Two deployment origins recorded separately:
+
+| Purpose                        | Convex Cloud example           | Synapse example                        |
+| ------------------------------ | ------------------------------ | -------------------------------------- |
+| Functions and reactive queries | `https://project.convex.cloud` | `https://project.synapse.example`      |
+| HTTP actions and tracker       | `https://project.convex.site`  | `https://project.site.synapse.example` |
+
+Never append `/rastro/tracker.js` to the functions URL.
+
+### 1. Install
 
 ```bash
-npm install @iann29/rastro
+npm install @iann29/rastro@alpha
 ```
 
-Mount the component under an HTTP prefix:
+The untagged npm `latest` channel intentionally remains on the first public
+alpha. Use `@alpha` until a stable release is promoted.
+
+### 2. Mount the component
+
+Merge the component into the project's existing configuration; preserve every
+component and environment binding already registered there.
 
 ```ts
 // convex/convex.config.ts
@@ -52,6 +96,31 @@ This creates isolated component tables and four HTTP routes:
 - `GET /rastro/tracker.js` — the pre-gzipped tracker
 - `GET /rastro/health` — integration health
 
+Push the component and regenerate host bindings before importing
+`components.rastroAnalytics`:
+
+```bash
+# Convex Cloud development deployment
+npx convex dev
+
+# Synapse development deployment
+synapse dev --once
+```
+
+For production, use the product's existing release workflow. On an uncoordinated
+Convex project that is `npx convex deploy`; on Synapse use
+`synapse release --execute` when the project has a release manifest, otherwise
+`synapse deploy`. Never deploy before confirming the target.
+
+Verify the HTTP origin independently from the functions origin:
+
+```bash
+curl -fsS https://YOUR_HTTP_ORIGIN/rastro/health
+```
+
+A successful health response proves that the component HTTP routes are mounted;
+it does not prove that a browser event was accepted.
+
 ### Automatic GeoIP
 
 Rastro uses Convex request metadata as the canonical client IP and can resolve
@@ -80,9 +149,9 @@ export default app;
 `ipwhois` is also supported. Its token is optional for the documented
 1,000-request/day free endpoint and recommended for production. Bind it as
 `RASTRO_GEOIP_TOKEN` in the same way as the IPinfo example.
-`RASTRO_GEOIP_DAILY_LIMIT` defaults to 1,000 lookups per site per UTC day
-and may be set from `0` through `1000000` to match the provider plan. Admission
-is transactional, so concurrent first-session requests cannot exceed the cap.
+`RASTRO_GEOIP_DAILY_LIMIT` defaults to 1,000 lookups per site per UTC day and
+may be set from `0` through `1000000` to match the provider plan. Admission is
+transactional, so concurrent first-session requests cannot exceed the cap.
 
 When Rastro runs behind a reverse proxy that supplies `X-Forwarded-For` or
 Vercel/Cloudflare geographic headers, explicitly bind
@@ -100,13 +169,13 @@ sanitized country, city, and coordinates rounded to one decimal place are
 persisted.
 
 `GET /rastro/health` reports whether the runtime exposes a usable client IP and
-whether GeoIP is configured and the daily cap, without returning the address or token. Deployment
-administrators can exercise the configured provider through the internal
-`geo:probe` action. If health reports no usable client IP, use trusted
-Vercel/Cloudflare geographic headers with `RASTRO_TRUST_PROXY` or fix the
-deployment's proxy forwarding.
+whether GeoIP is configured and the daily cap, without returning the address or
+token. Deployment administrators can exercise the configured provider through
+the internal `geo:probe` action. If health reports no usable client IP, use
+trusted Vercel/Cloudflare geographic headers with `RASTRO_TRUST_PROXY` or fix
+the deployment's proxy forwarding.
 
-## Create a site
+### 3. Create a site
 
 Components cannot read the host app's authentication state. The host must
 authenticate the caller and pass an opaque owner ID.
@@ -132,7 +201,7 @@ export const createSite = mutation({
     if (!identity) throw new Error("Not authenticated");
 
     return await analytics.createSite(ctx, {
-      ownerId: identity.subject,
+      ownerId: identity.tokenIdentifier,
       name: args.name,
       domains: args.domains,
       currency: args.currency,
@@ -142,12 +211,30 @@ export const createSite = mutation({
 });
 ```
 
+Call this mutation from the host application's authenticated administration UI
+and persist or display the returned `siteId`:
+
+```tsx
+const createSite = useMutation(api.analyticsAdmin.createSite);
+const siteId = await createSite({
+  name: "Amage Product",
+  domains: ["product.example.com", "localhost"],
+  currency: "BRL",
+});
+```
+
+Do not call the authenticated example with a bare CLI command: without an
+explicit test identity, `ctx.auth.getUserIdentity()` correctly returns `null`.
+
 Allowed domains are checked at ingestion. Exact hosts and wildcard subdomains
 such as `*.example.com` are supported. `Origin` is an abuse signal, not
 authentication; Amage Rastro also enforces per-session and fixed-shard per-site
 event/byte budgets.
 
-## Add the tracker
+Domain entries are hostnames, not host-and-port pairs. Use `localhost` or a full
+origin such as `http://localhost:3000`; both normalize to the `localhost` host.
+
+### 4. Add the tracker
 
 Use the HTTP-actions URL for your deployment, not the Convex function URL.
 
@@ -179,6 +266,9 @@ serves the script and ingestion from different origins:
 
 ### React
 
+React is an optional peer dependency. It is required only when importing the
+`@iann29/rastro/react` entry point; backend-only consumers do not need it.
+
 ```tsx
 import { useState } from "react";
 import { RastroScript, useRastro } from "@iann29/rastro/react";
@@ -208,6 +298,26 @@ the same value on authorized sites to connect their journeys. Amage Rastro
 intentionally has no mutable post-load `identify` command: changing identity
 after a session starts would make the session ambiguous.
 
+### 5. Verify ingestion
+
+1. Open an allowed site origin in a fresh browser tab.
+2. In browser developer tools, filter Network requests by `rastro/events`.
+3. Wait for the normal tracker flush, which is debounced by approximately 1.2
+   seconds.
+4. Confirm `POST /rastro/events` returns HTTP 200 and a response containing
+   `accepted`, `duplicates`, and `rejected` counters.
+5. Run one authenticated local report, such as `analytics:overview`, for the
+   returned `siteId` and a complete UTC hour/day range.
+
+The tracker intentionally does not interrupt the product UI when transport
+fails. A loaded `tracker.js` or successful `/health` response alone is not proof
+of ingestion. The usual causes of a rejected or missing request are:
+
+- using the functions URL instead of the HTTP-actions URL;
+- omitting the exact browser host from the site's `domains`;
+- including query strings or paths in a configured base URL;
+- querying a partial UTC bucket or a site the authenticated user does not own.
+
 ## Track custom events
 
 ```js
@@ -300,12 +410,12 @@ export const {
 
     for (const siteId of request.siteIds) {
       const site = await analytics.getSite(ctx, siteId);
-      if (!site || site.ownerId !== identity.subject) {
+      if (!site || site.ownerId !== identity.tokenIdentifier) {
         throw new Error("Not authorized for analytics site");
       }
     }
 
-    if (request.ownerId && request.ownerId !== identity.subject) {
+    if (request.ownerId && request.ownerId !== identity.tokenIdentifier) {
       throw new Error("Not authorized for analytics owner");
     }
   },
@@ -316,6 +426,30 @@ Do not copy the unauthenticated demo boundary into production. The included demo
 exposes only fixed-scope read reports plus bounded, idempotent seed/live-refresh
 functions.
 
+### Federated dashboard backend
+
+The reference control plane in `example/` can connect a central dashboard while
+telemetry remains in the customer's deployment. It uses
+`exposeFederatedAnalyticsApi` with issuer and connection-resolver options. The
+helper constructs its authorization boundary internally; it cannot accept a
+permissive host authorizer. Its surface contains a public protocol v1 manifest,
+authenticated redacted connection status, redacted site summaries, and only the
+reactive dashboard report queries.
+
+The JWT only identifies the requested connection. Site access remains
+authoritative in a local host record resolved on every query, so revocation is
+immediate and a token cannot grant itself another site. `listSites` accepts no
+site arguments and omits owner, domain, and network configuration. Stable
+manifest names, capabilities, limits, and authorization error codes are exported
+for connector clients. The host must already have Convex authentication and a
+local authoritative grant. Follow
+[`docs/federation-setup.md`](docs/federation-setup.md) for the executable setup
+and verification sequence; use [`docs/federation.md`](docs/federation.md) as the
+protocol reference. The production control plane is available at
+`https://www.amagerastro.com`; before connecting a customer deployment, verify
+the selected package exports the federation helper as shown in the setup
+runbook.
+
 ## Backend API
 
 The `Rastro` class wraps the component boundary for host functions:
@@ -324,17 +458,46 @@ The `Rastro` class wraps the component boundary for host functions:
 - Ingestion: `ingest`, `trackConversion`
 - Reports: `overview`, `liveVisitors`, `listSessions`, `sessionJourney`,
   `listConversions`, `visitorJourney`, `goalsReport`, `funnelsReport`,
-  `affiliatesReport`
+  `affiliatesReport`, `dataCoverage`
 - Configuration: `upsertGoal`, `listGoals`, `removeGoal`, `upsertFunnel`,
   `listFunnels`, `removeFunnel`, `upsertAffiliate`, `listAffiliates`,
   `removeAffiliate`
-- Retention: `cleanup`
+- Retention: `cleanup`, `setRetentionPolicy`, `disableRetentionPolicy`,
+  `runRetentionPolicy`, `retentionStatus`
 
 Every public function has argument and return validators. Growing reads are
 indexed and bounded. Component pagination uses scope-bound opaque keyset cursors
 because native `.paginate()` is not supported inside Convex components; page
 cost stays constant instead of increasing with an offset.
 
+Overview responses label visitor values as either HLL-estimated unique visitors
+or legacy `bucketOccurrences`; legacy aggregate rows are never presented as
+exact uniques. Breakdown metadata likewise identifies page, source, country,
+device, browser, and event rankings as event-volume counts. Feature status is
+explicit: outbound-link event volume is supported, while session replay, error
+insights, and bot detection are currently unsupported.
+
+All report ranges use integer Unix milliseconds with inclusive `from` and `to`
+boundaries. Range metadata reports `boundary: "inclusive"` and
+`timezone: "UTC"`; the site's timezone is presentation metadata and never
+changes report bucket boundaries. Overview queries must contain complete UTC
+hours or days, and funnel and affiliate reports must contain complete UTC days.
+Those aggregate-backed reports fail with `REPORT_INCOMPLETE` instead of silently
+including the rest of a partial bucket. Goal reports retain exact millisecond
+filtering while bounded goal-completion rows are available.
+
+Goal, funnel, and affiliate reports use definition-ID keyed daily rollups when
+the low-churn rollout control proves that generation is available. Each loader
+enforces day, row, document, and byte headroom before returning a complete
+result; it never returns a silently truncated report. Ordinary ingestion never
+patches or schedules work against a shared coverage document. `dataCoverage`
+derives availability through bounded indexed reads of source and rollup records,
+while the control document stores only rollout generation and durable retention
+watermarks. Heartbeats can advance session availability but do not claim event
+or aggregate availability. Hourly and daily overview availability are reported
+separately as `overviewHour` and `overviewDay` because their retention
+boundaries use different bucket alignment.
+
 ## Operational limits
 
 - HTTP body: 64 KiB
@@ -344,16 +507,25 @@ cost stays constant instead of increasing with an offset.
 - Primary aggregate shards: 256 hourly and 128 daily, derived from stable
   session identity
 - Funnel and affiliate aggregate shards: 16
-- Site ingress budget: 4,096 deterministic per-minute shards, each capped at
-  300 events and 512 KiB (1,228,800 admitted events/minute at uniform load)
+- Site ingress budget: 4,096 deterministic per-minute shards, each capped at 300
+  events and 512 KiB (1,228,800 admitted events/minute at uniform load)
 - Rate-window storage: one reusable row per active session and site shard
 - Live session timeout: 60 seconds with one bounded five-second expiry sweep per
   site; legacy per-session jobs hand off to that sweep during the cutover
 - Dashboard overview: up to ten sites and bounded report ranges
 - Hourly overview: up to 24 hours; wider ranges use daily buckets
-- Retention cleanup: bounded batches; schedule host maintenance for the data
-  classes your policy retains. The `events` cleanup kind removes canonical
-  event-batch documents and processes at most 32 heavy batches per transaction.
+- Retention cleanup: bounded batches. `setRetentionPolicy` automatically starts
+  leased, persisted cleanup chains for every policy-managed dataset; interrupted
+  attempts resume after lease expiry and stale generations cannot mutate the
+  active chain. A completed chain schedules its next cycle after one day and
+  recomputes its cutoff from the then-current policy. Policy updates replace the
+  generation, and `disableRetentionPolicy` makes every queued attempt stale.
+  `runRetentionPolicy` remains available for an explicit bounded run. The
+  `events` cleanup kind removes canonical event-batch documents and processes at
+  most 32 heavy batches per transaction.
+- Retention cutoffs are bucket-safe: event minutes, hourly aggregates, daily
+  aggregates and feature rollups preserve the bucket containing a cutoff and
+  return the exact effective `before` boundary they applied.
 
 Capacity claims use declared workload profiles. Complete release evidence
 reports ten-minute `lean`, `realistic`, `heavy`, and browser `feature` profiles
@@ -361,12 +533,52 @@ separately. The browser `feature` profile exercises configured goals, funnels,
 and affiliate attribution; trusted conversions require a separate server-side
 driver and certification result.
 
-The historical development evidence and its certification caveats are recorded
-in [`docs/benchmarks/2026-08-20-realistic.md`](docs/benchmarks/2026-08-20-realistic.md).
+### Historical ingest evidence requiring recertification
+
+| Profile   | Workload                                          | Certified stored/day | p95 latency | 100M/day target |
+| --------- | ------------------------------------------------- | -------------------: | ----------: | --------------- |
+| Lean      | Minimal events, maximum batches                   |              136.65M |    4,543 ms | Pass            |
+| Realistic | Reused sessions, browser mix, context, heartbeats |              120.81M |    4,247 ms | Pass            |
+| Heavy     | 62 KiB requests with bounded custom properties    |               76.62M |    3,670 ms | Below target    |
+| Feature   | Goals, funnels, and affiliate attribution         |               33.22M |    6,226 ms | Below target    |
+
+Every listed run lasted at least ten minutes, completed with zero request
+failures, duplicates, or rejected events, and exactly reconciled the expected
+stored events. Results came from one client process against a non-production
+Synapse DEV deployment; they are engineering evidence, not a production SLA. The
+heavy row is the formal recertification after removing redundant payload
+serialization. GeoIP with a provider enabled and trusted server-side conversions
+remain separate pending certifications.
+
+These runs predate the current source-derived coverage and recurring-retention
+changes. They remain historical evidence for the named revisions and profiles,
+not certification of this uncommitted implementation. Recertify the final code
+before presenting any row as current capacity.
+
+The latest formal results are recorded in
+[`docs/benchmarks/2026-08-21-formal-certification.md`](docs/benchmarks/2026-08-21-formal-certification.md).
+Earlier development evidence remains available in
+[`docs/benchmarks/2026-08-20-realistic.md`](docs/benchmarks/2026-08-20-realistic.md).
 
 From a repository checkout, run a duration-based profile against a confirmed
 non-production deployment and a dedicated idle benchmark site:
 
+For a deployment-free CI smoke benchmark of write-heavy ingestion plus reactive
+overview and feature report queries, run:
+
+```bash
+npm run benchmark:local
+```
+
+The local harness uses `convex-test`, enforces a 12-document ingest transaction
+budget, checks reactive totals before and after a write, and queries proven
+daily rollups with the maximum configured 50 goals, 20 funnels, and 100
+affiliates under deterministic 1,000-document and 4 MiB read ceilings. It also
+drives concurrent ingest calls and asserts that their shared rollout/retention
+control document is unchanged. It fails when its generous CI latency budgets are
+exceeded. This is a deterministic local regression guard, not evidence of live
+deployment throughput or capacity.
+
 ```bash
 npm run --silent benchmark:ingest -- \
   --url https://example.site/rastro/events \
@@ -375,7 +587,8 @@ npm run --silent benchmark:ingest -- \
   --origin https://your-allowed-origin.example \
   --profile realistic \
   --duration 600 \
-  --concurrency 100 \
+  --concurrency 75 \
+  --batch-size 50 \
   --certify
 ```
 
@@ -390,10 +603,10 @@ generates exact session IDs from a cryptographic run UUID and reads them through
 bounded index lookups, so unrelated traffic cannot alter the result. The query
 uses the session counters persisted transactionally with event batches; it
 certifies ingest accounting, not raw-event availability after later retention.
-Reports distinguish accepted ingress, heartbeats, and stored events.
-Shorter completed runs report extrapolated daily estimates only. `--certify`
-requires at least ten minutes and successful reconciliation before emitting
-certified daily capacity fields or exiting successfully as a certification.
+Reports distinguish accepted ingress, heartbeats, and stored events. Shorter
+completed runs report extrapolated daily estimates only. `--certify` requires at
+least ten minutes and successful reconciliation before emitting certified daily
+capacity fields or exiting successfully as a certification.
 
 The ingestion transaction folds metric/dimension deltas, session rate counters,
 site-shard counters, session/live state, visitor attribution, and exact funnel
@@ -407,16 +620,15 @@ Click/outbound/browser-conversion telemetry skips goal and funnel work that
 cannot match those event types.
 
 Raw telemetry is stored only in immutable `eventBatches`. Migrated events retain
-their original per-event browser and geo context inside those batches, while
-new events hydrate context from their session. Journey `_id` values are stable
+their original per-event browser and geo context inside those batches, while new
+events hydrate context from their session. Journey `_id` values are stable
 strings derived from batch and nested-event identity, and session-journey
 cursors use timestamp, creation time, and ID.
 
 Existing deployments that still contain the retired `events` table must not
-upgrade directly to this cutover. First deploy the migration-capable predecessor,
-run its documented migration until both durable success and an empty source are
-reported, and only then install the canonical-only release. Fresh deployments
-need no migration.
+upgrade directly to this cutover. Follow the exact intermediate release and
+verification sequence in [`docs/upgrading.md`](docs/upgrading.md). Fresh
+deployments need no legacy-event migration.
 
 ## Privacy contract
 
@@ -461,14 +673,15 @@ npm test
 npm run typecheck
 npm run lint
 npm run build
-cd example && npx vite build
+npm exec vite -- build example
 ```
 
 The test suite covers origin enforcement, event/privacy sanitization,
 idempotency, reusable session/site rate limits, bounded live expiry chains,
 folded aggregate writes, goals, funnels, trusted payments, currency safety,
 affiliate commission, cross-site journeys, keyset pagination beyond 1,000 rows,
-retention, HTTP routes, React integration, and tracker size/behavior.
+retention, request metadata and GeoIP, exact byte boundaries, HTTP routes, React
+integration, and tracker size/behavior.
 
 ## Synapse demo