@iann29/rastro 0.1.0-alpha.9 → 0.1.0

package/README.md

@@ -13,12 +13,12 @@ inside your Convex deployment.
 
 ## Project status
 
-Amage Rastro is an actively developed alpha with a canonical batched event
-store, protected CI, exact run-scoped benchmark reconciliation, and published
-npm prereleases. The untagged `latest` channel intentionally remains on the
-first public alpha. Federation consumers must statically inspect an
-operator-approved exact registry artifact for every required runtime and type
-export before changing a host; follow the gate in the
+Amage Rastro is actively developed, with a canonical batched event store,
+protected CI, exact run-scoped benchmark reconciliation, and `0.1.0` as the
+first release on the untagged npm `latest` channel; the alphas before it stay
+available under their exact versions. Federation consumers must statically
+inspect an operator-approved exact registry artifact for every required runtime
+and type export before changing a host; follow the gate in the
 [federation setup runbook](docs/federation-setup.md). Never substitute a mutable
 branch or website-supplied source tarball.
 
@@ -30,9 +30,11 @@ the [machine manifest](https://www.amagerastro.com/agent/manifest.json).
 Current evidence supports more than 140 million stored events/day for realistic
 traffic and more than 178 million for lean traffic. Near-limit heavy payloads
 are certified at 55.26 million/day on a data-heavy store (76.62 million/day on a
-fresh store, 2026-08-22). The feature-saturated profile currently fails
-certification on a known rollup-contention defect
-([#52](https://github.com/amageweb/amage-rastro/issues/52)). Workload-specific
+fresh store, 2026-08-22). The feature-saturated profile failed its last
+certification on a rollup-contention defect
+([#52](https://github.com/amageweb/amage-rastro/issues/52)); the contended write
+path no longer exists (report rollups are folded by one job per site, off the
+ingestion transaction) and the profile awaits recertification. Workload-specific
 results and caveats are reported separately rather than collapsed into one
 capacity claim.
 
@@ -60,8 +62,11 @@ capacity claim.
 ### Prerequisites
 
 - Convex `^1.43.0` or a compatible Synapse deployment.
-- An existing host authentication provider for administrative and local report
-  functions. The browser ingestion route itself is public and origin-checked.
+- A way to authorize site administration and local report functions: the host's
+  existing authentication provider, or, for a site with no user accounts, a
+  fixed opaque owner ID and internal functions run with deployment credentials
+  (see [Hosts without user authentication](#hosts-without-user-authentication)).
+  The browser ingestion route itself is public and origin-checked.
 - Two deployment origins recorded separately:
 
 | Purpose                        | Convex Cloud example           | Synapse example                        |
@@ -74,11 +79,11 @@ Never append `/rastro/tracker.js` to the functions URL.
 ### 1. Install
 
 ```bash
-npm install @iann29/rastro@alpha
+npm install @iann29/rastro
 ```
 
-The untagged npm `latest` channel intentionally remains on the first public
-alpha. Use `@alpha` until a stable release is promoted.
+The untagged npm `latest` channel is `0.1.0`. Pin the exact version in the host;
+the changelog and the upgrading guide say what each release changes.
 
 ### 2. Mount the component
 
@@ -133,7 +138,9 @@ it does not prove that a browser event was accepted.
 
 Rastro uses Convex request metadata as the canonical client IP and can resolve
 coarse city-level coordinates on the first request of each session. GeoIP is
-opt-in because the configured provider receives the request IP. For IPinfo Core:
+opt-in because the configured provider receives the request IP. Declare the
+settings as optional host environment variables and pass them through, so each
+deployment decides on its own provider, token, and cap without a code change:
 
 ```ts
 // convex/convex.config.ts
@@ -141,19 +148,47 @@ import { defineApp } from "convex/server";
 import { v } from "convex/values";
 import rastro from "@iann29/rastro/convex.config.js";
 
-const app = defineApp({ env: { IPINFO_TOKEN: v.string() } });
+const app = defineApp({
+  env: {
+    RASTRO_GEOIP_PROVIDER: v.optional(
+      v.union(v.literal("ipinfo"), v.literal("ipwhois")),
+    ),
+    RASTRO_GEOIP_TOKEN: v.optional(v.string()),
+    RASTRO_GEOIP_DAILY_LIMIT: v.optional(v.string()),
+    RASTRO_TRUST_PROXY: v.optional(v.literal("true")),
+  },
+});
 app.use(rastro, {
   httpPrefix: "/rastro/",
   env: {
-    RASTRO_GEOIP_PROVIDER: "ipinfo",
-    RASTRO_GEOIP_TOKEN: app.env.IPINFO_TOKEN,
-    RASTRO_GEOIP_DAILY_LIMIT: "1000",
+    RASTRO_GEOIP_PROVIDER: app.env.RASTRO_GEOIP_PROVIDER,
+    RASTRO_GEOIP_TOKEN: app.env.RASTRO_GEOIP_TOKEN,
+    RASTRO_GEOIP_DAILY_LIMIT: app.env.RASTRO_GEOIP_DAILY_LIMIT,
+    RASTRO_TRUST_PROXY: app.env.RASTRO_TRUST_PROXY,
   },
 });
 
 export default app;
 ```
 
+This is the shape of the [example host](example/convex/convex.config.ts). With
+nothing set, provider lookups stay off. To enable IPinfo Core on one deployment:
+
+```bash
+# Convex Cloud development deployment
+npx convex env set --deployment dev RASTRO_GEOIP_PROVIDER ipinfo
+npx convex env set --deployment dev RASTRO_GEOIP_TOKEN 'YOUR_IPINFO_TOKEN'
+npx convex env set --deployment dev RASTRO_GEOIP_DAILY_LIMIT 1000
+
+# Synapse development deployment
+synapse convex --dev env set RASTRO_GEOIP_PROVIDER ipinfo
+synapse convex --dev env set RASTRO_GEOIP_TOKEN 'YOUR_IPINFO_TOKEN'
+synapse convex --dev env set RASTRO_GEOIP_DAILY_LIMIT 1000
+```
+
+Binding a literal instead of a passthrough (`RASTRO_GEOIP_PROVIDER: "ipinfo"`)
+is also valid when every deployment of the project shares the provider.
+
 `ipwhois` is also supported. Its token is optional for the documented
 1,000-request/day free endpoint and recommended for production. Bind it as
 `RASTRO_GEOIP_TOKEN` in the same way as the IPinfo example.
@@ -162,11 +197,11 @@ may be set from `0` through `1000000` to match the provider plan. Admission is
 transactional, so concurrent first-session requests cannot exceed the cap.
 
 When Rastro runs behind a reverse proxy that supplies `X-Forwarded-For` or
-Vercel/Cloudflare geographic headers, explicitly bind
-`RASTRO_TRUST_PROXY: "true"`. Leave it unset for direct Convex deployments.
-Rastro uses forwarded data only when this opt-in is enabled and the runtime IP
-identifies a private immediate peer. Caller-supplied proxy or geographic headers
-are ignored when native metadata is absent or public.
+Vercel/Cloudflare geographic headers, set `RASTRO_TRUST_PROXY` to `true` on that
+deployment. Leave it unset for direct Convex deployments. Rastro uses forwarded
+data only when this opt-in is enabled and the runtime IP identifies a private
+immediate peer. Caller-supplied proxy or geographic headers are ignored when
+native metadata is absent or public.
 
 The lookup has a 1.5-second timeout and fails open: telemetry ingestion
 continues without location when the provider is unavailable. A session reserves
@@ -181,7 +216,13 @@ whether GeoIP is configured and the daily cap, without returning the address or
 token. Deployment administrators can exercise the configured provider through
 the internal `geo:probe` action. If health reports no usable client IP, use
 trusted Vercel/Cloudflare geographic headers with `RASTRO_TRUST_PROXY` or fix
-the deployment's proxy forwarding.
+the deployment's proxy forwarding. Health also names the build the deployment
+serves under `tracker`: the package `version` plus the content `hash` of
+`tracker.js` and the `vitalsHash` of `vitals.js`. Both scripts carry that hash
+as a strong `ETag`, answer `If-None-Match` with `304`, and are cached for five
+minutes with up to one hour of `stale-while-revalidate`, so a returning visitor
+runs a new build after at most one stale page load; an open tab keeps its script
+until it reloads.
 
 ### Browser and device classification
 
@@ -275,6 +316,66 @@ event/byte budgets.
 
 Domain entries are hostnames, not host-and-port pairs. Use `localhost` or a full
 origin such as `http://localhost:3000`; both normalize to the `localhost` host.
+A site deployed on Vercel needs its preview hosts too: list `*.vercel.app` next
+to the production host, or preview deployments are rejected as
+`ORIGIN_NOT_ALLOWED`.
+
+#### Hosts without user authentication
+
+An institutional site with no user accounts has nobody to authenticate. Use one
+fixed opaque owner ID and internal functions, which only deployment credentials
+can run:
+
+```ts
+// convex/analyticsAdmin.ts
+import { v } from "convex/values";
+import { Rastro } from "@iann29/rastro";
+import { components } from "./_generated/api";
+import { internalMutation } from "./_generated/server";
+
+const analytics = new Rastro(components.rastroAnalytics);
+
+// Opaque and stable. Reuse the same value as `ownerId` when provisioning a
+// federation grant.
+const SITE_OWNER_ID = "acme-site-owner";
+
+export const createSite = internalMutation({
+  args: {
+    name: v.string(),
+    domains: v.array(v.string()),
+    currency: v.string(),
+  },
+  returns: v.string(),
+  handler: async (ctx, args) =>
+    await analytics.createSite(ctx, {
+      ownerId: SITE_OWNER_ID,
+      name: args.name,
+      domains: args.domains,
+      currency: args.currency,
+      cookieless: true,
+    }),
+});
+```
+
+Run it once per site with the deployment's own credentials, never from a
+browser:
+
+```bash
+# Convex Cloud development deployment
+npx convex run --deployment dev analyticsAdmin:createSite \
+  '{"name":"Acme","domains":["acme.example","*.vercel.app"],"currency":"BRL"}'
+
+# Synapse development deployment
+synapse convex --dev run analyticsAdmin:createSite \
+  '{"name":"Acme","domains":["acme.example","*.vercel.app"],"currency":"BRL"}'
+```
+
+Such a host needs no local report functions: the central dashboard reads the
+[federated surface](#federated-dashboard-backend), and its `auth.config.ts`
+lists only the Rastro federation provider, exactly as the
+[example host](example/convex/auth.config.ts) does. Goals, funnels, affiliates,
+and retention policies are configured the same way, through `internalMutation`s
+that call the `Rastro` class with `SITE_OWNER_ID`.
 
 ### 4. Add the tracker
 
@@ -306,6 +407,33 @@ serves the script and ingestion from different origins:
 ></script>
 ```
 
+In a Next.js App Router layout, `next/script` injects the same tag;
+`document.currentScript` is set when it runs, so `data-site` and the inferred
+endpoint work exactly as with a plain tag:
+
+```tsx
+// app/layout.tsx
+import Script from "next/script";
+
+export default function RootLayout({ children }) {
+  return (
+    <html lang="pt-BR">
+      <body>
+        {children}
+        <Script
+          src="https://YOUR_DEPLOYMENT.convex.site/rastro/tracker.js"
+          data-site="YOUR_SITE_ID"
+          strategy="afterInteractive"
+        />
+      </body>
+    </html>
+  );
+}
+```
+
+Remember the site's `domains`: a Vercel preview runs on `*.vercel.app`, not on
+the production host.
+
 ### Field Web Vitals (optional)
 
 A second, separate module measures real-visitor LCP, CLS, INP, FCP, and TTFB
@@ -371,17 +499,35 @@ after a session starts would make the session ambiguous.
    seconds.
 4. Confirm `POST /rastro/events` returns HTTP 200 and a response containing
    `accepted`, `duplicates`, and `rejected` counters.
-5. Run one authenticated local report, such as `analytics:overview`, for the
-   returned `siteId` and a complete UTC hour/day range.
+5. Run `analytics:liveVisitors` (with `siteIds` and `now`) or
+   `analytics:sessionJourney` for the returned `siteId`: both reflect the
+   request as soon as it is accepted. `analytics:listSessions` and the
+   aggregate-backed `analytics:overview` can trail the live view by a few
+   seconds, and the overview only reports complete UTC hour/day buckets, so an
+   empty result from those two right after the first request is not a failed
+   ingestion.
 
 The tracker intentionally does not interrupt the product UI when transport
 fails. A loaded `tracker.js` or successful `/health` response alone is not proof
 of ingestion. The usual causes of a rejected or missing request are:
 
 - using the functions URL instead of the HTTP-actions URL;
-- omitting the exact browser host from the site's `domains`;
+- omitting the exact browser host from the site's `domains`, including
+  `*.vercel.app` for preview deployments;
 - including query strings or paths in a configured base URL;
-- querying a partial UTC bucket or a site the authenticated user does not own.
+- querying a partial UTC bucket or a site the authenticated user does not own;
+- reading `listSessions` or `overview` seconds after the first request instead
+  of `liveVisitors`.
+
+Since 0.1.0 a live row also carries the intent facts of its session:
+`entryPath`, `returning` (the visitor had a history before this session; a host
+id linked to anonymous history at sign-up is a return), `utmCampaign`,
+`affiliateSlug`, `conversionCount`, and the furthest `funnel` step the visitor
+reached (`{ funnelId, step, steps }`). `visitorHeat` from the package turns
+those, plus the pages that complete a goal or a funnel step, into the
+hot/warm/cold score the dashboard's live view shows; a host may apply the same
+rules to its own signals. Rows written before the upgrade lack the facts and
+score from what they have.
 
 ## Track custom events
 
@@ -421,8 +567,13 @@ of the session, and sends them as properties of every pageview; no other query
 parameter ever leaves the page. A session's `source` is the lowercase
 `utm_source` when present and the referrer host otherwise, so campaign traffic
 no longer reads as `direct`. `utmSource`, `utmMedium`, and `utmCampaign` stay on
-the session for journeys, and `overview.topCampaigns` ranks `utm_campaign` by
-event volume next to `topSources`.
+the session for journeys; `overview.topCampaigns` ranks `utm_campaign` and
+`overview.topMediums` the lowercase `utm_medium` next to `topSources`, by
+sessions once the range is covered by the counters. `overview.topOutbound` ranks
+the hosts outbound clicks reached — the host alone, never the path or query — by
+clicks. Both lists are recorded from `metadata.dimensionsSince` (each site's
+first ingest on 0.1.0 plus the rollup delay); a range that starts earlier ranks
+them from partial data, and the dashboard says so.
 
 ## Identify visitors after signup
 
@@ -546,16 +697,26 @@ while telemetry remains in the customer's deployment. It uses
 `exposeFederatedAnalyticsApi` with issuer and connection-resolver options. The
 helper constructs its authorization boundary internally; it cannot accept a
 permissive host authorizer. Its surface contains a public protocol v1 manifest,
-authenticated redacted connection status, redacted site summaries, and only the
-reactive dashboard report queries.
+authenticated redacted connection status, redacted site summaries, the reactive
+dashboard report queries, and — behind the optional `configure` capability — the
+functions that manage goals, funnels, affiliates, a site's name, domains and
+timezone, and the retention policy from the dashboard.
 
 The JWT only identifies the requested connection. Site access remains
 authoritative in a local host record resolved on every query, so revocation is
-immediate and a token cannot grant itself another site. `listSites` accepts no
-site arguments and omits owner, domain, and network configuration. Stable
-manifest names, capabilities, limits, and authorization error codes are exported
-for connector clients. The host must already have Convex authentication and a
-local authoritative grant. Follow
+immediate and a token cannot grant itself another site. The same record decides
+whether the dashboard may configure: a configure function runs only when the
+token carries `analytics:configure` (the control plane claims it for
+organization owners and admins alone) and the host's grant lists it too;
+otherwise the dashboard shows the same forms but hands over the code the host
+runs itself. A third permission, `analytics:public`, lets the organization
+publish a read-only public link of the granted sites — the overview without
+revenue, conversions, goals, campaigns, or custom events, and the live visitors
+without identities — and, like the configure scope, it works only when the
+host's grant lists it. `listSites` accepts no site arguments and omits owner,
+domain, and network configuration. Stable manifest names, capabilities, limits,
+and authorization error codes are exported for connector clients. The host must
+already have Convex authentication and a local authoritative grant. Follow
 [`docs/federation-setup.md`](docs/federation-setup.md) for the executable setup
 and verification sequence; use [`docs/federation.md`](docs/federation.md) as the
 protocol reference. The production control plane is available at
@@ -602,18 +763,36 @@ returns. When storage is blocked or an old cached tracker reports no id,
 sessions from the same device and network within one UTC day still fold into one
 visitor through a salted daily key; runtimes without a client IP fall back to
 one visitor per session, which `/rastro/health` reports as
-`uniqueVisitors: "session"` instead of `"dailyHash"`. Breakdown metadata
-likewise identifies page, source, country, device, browser, and event rankings
-as event-volume counts. Feature status is explicit: outbound-link event volume
-is supported, bot detection rejects self-declared agents by `User-Agent`, and
-session replay and error insights are currently unsupported.
+`uniqueVisitors: "session"` instead of `"dailyHash"`. Breakdown metadata says
+what each ranking counts: pages by pageviews and sources, campaigns, countries,
+devices, and browsers by sessions once every bucket in the range carries those
+counters (recorded since alpha.11), and raw event volume for ranges that reach
+back before them; events always rank by volume. Totals carry the engagement
+counters recorded since the same stamp — summed session duration, sessions that
+reached a second pageview, and first-ever sessions of a visitor — and
+`metadata.engagement` reports `counted` when the whole range has them, so bounce
+rate, average duration, pages per session, and the share of new visitors derive
+from one response; `unavailable` means the range reaches back before the
+counters and those four stay blank. Feature status is explicit: outbound-link
+event volume is supported (with `topOutbound` naming the destinations), bot
+detection rejects self-declared agents by `User-Agent`, and session replay and
+error insights are currently unsupported.
 
 All report ranges use integer Unix milliseconds with inclusive `from` and `to`
-boundaries. Range metadata reports `boundary: "inclusive"` and
-`timezone: "UTC"`; the site's timezone is presentation metadata and never
-changes report bucket boundaries. Overview queries must contain complete UTC
-hours or days, and funnel and affiliate reports must contain complete UTC days.
-Those aggregate-backed reports fail with `REPORT_INCOMPLETE` instead of silently
+boundaries, and range metadata reports `boundary: "inclusive"`. Hourly overview
+queries must contain complete UTC hours. A daily overview query must contain
+either complete UTC days or, when every requested site has the same timezone,
+complete calendar days in that zone: a site with a timezone gets one aggregate
+bucket per local calendar day at ingestion, next to the UTC hour and day
+buckets, so "yesterday" in São Paulo ends at midnight in São Paulo. Range
+metadata then reports `timezone` as the zone the buckets follow, `"UTC"`
+otherwise. Local-day coverage starts at the site's first ingest after the
+upgrade (`analyticsCoverage.localDaySince`) and restarts when its timezone
+changes; a range reaching further back is reported as partial coverage, and
+`sites.localDays` (federated `listSites.localDays`) tells a dashboard the zone
+and the stamp up front. A zone that never leaves UTC keeps no extra buckets.
+Funnel and affiliate reports must contain complete UTC days. Those
+aggregate-backed reports fail with `REPORT_INCOMPLETE` instead of silently
 including the rest of a partial bucket. Goal reports retain exact millisecond
 filtering while bounded goal-completion rows are available.
 
@@ -647,6 +826,13 @@ boundaries use different bucket alignment.
 - Primary aggregate shards: 256 hourly and 128 daily, derived from stable
   session identity
 - Funnel and affiliate aggregate shards: 16
+- Report rollups (goal, funnel, affiliate, route, and Web Vitals daily rows) are
+  never written by the ingestion transaction: a batch appends one document to
+  the site's rollup queue and one leased fold job per site merges up to 200
+  queued batches per run into the rows, so every rollup row has a single writer
+  and concurrent ingestion cannot conflict on it. Reports see a batch's rollups
+  a few hundred milliseconds after ingestion accepts it; a fold chain that dies
+  is restarted by the next batch once its 60-second lease lapses
 - Site ingress budget: 4,096 deterministic per-minute shards, each capped at 300
   events and 512 KiB (1,228,800 admitted events/minute at uniform load)
 - Rate-window storage: one reusable row per active session and site shard
@@ -705,12 +891,13 @@ evidence, not a production SLA.
 Two caveats stand deliberately. The heavy row measured a store already holding
 the battery's own multi-million-event history; the 2026-08-22 fresh-store result
 (76.62M/day) remains valid for that state, and neither number overwrites the
-other. The feature profile cannot currently be certified at all: affiliate
-daily-rollup contention loses ~4–5% of batches under concurrent load
-([#52](https://github.com/amageweb/amage-rastro/issues/52)), so no
-feature-profile capacity is claimed until the fix lands. GeoIP with a provider
-enabled, trusted server-side conversions, and vitals-bearing traffic remain
-separate pending certifications.
+other. The feature profile is not certified: on alpha.6, affiliate daily-rollup
+contention lost ~4–5% of batches under concurrent load
+([#52](https://github.com/amageweb/amage-rastro/issues/52)). Since alpha.11
+ingestion no longer writes rollup rows at all (see the operational limits), but
+no feature-profile capacity is claimed until a new battery certifies it. GeoIP
+with a provider enabled, trusted server-side conversions, and vitals-bearing
+traffic remain separate pending certifications.
 
 The full evidence is recorded in
 [`docs/benchmarks/2026-08-30-alpha6-recertification.md`](docs/benchmarks/2026-08-30-alpha6-recertification.md).
@@ -772,9 +959,11 @@ site-shard counters, session/live state, visitor attribution, and exact funnel
 progress before writing. Accepted non-heartbeat telemetry from one request is
 grouped into one immutable raw document per session and event-minute, without
 duplicating session geo or user-agent context. A 50-event same-session,
-same-minute baseline stays within 10 document writes. Each touched aggregate
-bucket is written at most once per batch, and dimension overflow folds into an
-explicit `(other)` bucket instead of relabeling old counts.
+same-minute baseline stays within 10 document writes plus the rollup queue entry
+and its fold lease. Each touched aggregate bucket is written at most once per
+batch, and dimension overflow folds into an explicit `(other)` bucket instead of
+relabeling old counts. Goal, funnel, affiliate, route, and Web Vitals rollup
+rows are written only by the site's fold job, never by ingestion.
 Click/outbound/browser-conversion telemetry skips goal and funnel work that
 cannot match those event types.