@iann29/rastro 0.1.0-alpha.0 → 0.1.0-alpha.10

package/docs/benchmarks/2026-08-30-alpha6-recertification.md

@@ -0,0 +1,206 @@
+# Ingest recertification, 2026-08-30
+
+This report recertifies the four workload profiles ahead of `0.1.0-alpha.6`,
+against the same deployment and benchmark site as the
+[2026-08-21 formal certification](2026-08-21-formal-certification.md). It is
+engineering evidence for the tested revisions and deployment state, not a
+production SLA.
+
+## Environment
+
+| Item                | Value                                                            |
+| ------------------- | ---------------------------------------------------------------- |
+| Deployment          | DEV canary `fast-finch-7454` (unchanged since 2026-08-21)        |
+| Benchmark site      | `kx75pnr9wap522n0zmh4nnb4jh8cv110` (unchanged)                   |
+| Lean run revision   | `ba9eff1` (main before Web Vitals)                               |
+| Later run revisions | Web Vitals build; component sources identical to merge `6250526` |
+| Driver              | One local Node.js process                                        |
+| Reconciliation      | Exact run-scoped session IDs, indexed pages of 100               |
+| GeoIP provider      | Disabled                                                         |
+| Health before runs  | `ok: true`                                                       |
+
+Since the 2026-08-21 certification the wire path gained salted daily visitor
+keys, User-Agent bot rejection, client-hint browser detection, the federation
+hardening, and the `vital` ingest branch (present but idle: no profile sends
+vitals). Unlike 2026-08-21, the canary was not fresh: it accumulated the
+battery's own multi-million-event history while the profiles ran, which is the
+honest way most deployments look after a real day.
+
+All certified runs used `--duration 600 --certify`, ran sequentially with at
+least five minutes of cooldown, and had no competing local load. Two runs were
+discarded and repeated because the local DNS resolver failed their first
+connection wave (a client-side condition — the deployment was idle and healthy
+each time); one more was discarded after a mid-run deploy from this same
+machine. Discarded runs are not reported as capacity.
+
+## Results
+
+Meeting the 100 million stored-events/day target requires at least 1,157.41
+stored events/second.
+
+| Profile   | Concurrency | Batch | Sessions |  Accepted |    Stored | Stored/s | Certified stored/day |      p95 | Exact reconciliation  | 100M/day target  |
+| --------- | ----------: | ----: | -------: | --------: | --------: | -------: | -------------------: | -------: | --------------------- | ---------------- |
+| Lean      |         100 |    50 |   unique | 1,247,000 | 1,247,000 |    2,071 |          178,948,552 | 4,167 ms | 1,247,000 / 1,247,000 | Pass             |
+| Realistic |          75 |    50 |    4,096 | 1,039,050 |   988,995 |    1,644 |          142,036,529 | 4,953 ms | 988,995 / 988,995     | Pass             |
+| Heavy     |         100 |    20 |    2,048 |   388,040 |   388,040 |      640 |           55,259,642 | 7,702 ms | 388,040 / 388,040     | **Below target** |
+| Feature   |         100 |    15 |    2,048 |         — |         — |        — |           **failed** |        — | —                     | **Failed**       |
+
+### Lean
+
+Certified by the harness: 24,940 requests, zero failures, duplicates, or
+rejections; exact reconciliation. 178.95M stored events/day is 31% above the
+2026-08-21 result (136.65M) on the same deployment — the byte-measurement reuse
+first recertified for heavy on 2026-08-22 now shows up across the board, and the
+new per-request work (bot check, client hints, salted visitor keys) did not cost
+the hot path its headroom.
+
+### Realistic
+
+The driver's session pool was doubled to 4,096 (default 2,048) and that change
+is part of this row's workload definition. At the current deployment speed the
+2,048-session pool drives individual sessions past the documented 120
+events/session/minute ingest budget, so a prior attempt lost 2 of 20,477
+requests to `429 RATE_LIMITED` — the limiter working as documented, but not a
+clean certification. With 4,096 sessions the run completed 20,781 requests with
+**zero failures** of any kind.
+
+The harness itself did not stamp this run: after the clean 600 s window, its
+first reconciliation query lost a fetch to the local resolver
+(`reconciliation:fetch failed`) and gave up. Reconciliation was re-run
+immediately afterwards with the same `benchmark:reconcileSessionsPage` pages the
+harness uses, with per-page retries:
+
+```json
+{
+  "runId": "3c81abe6-60cd-4b40-828f-cedf03fedd39",
+  "requested": 4096,
+  "found": 4096,
+  "stored": 988995,
+  "expectedStored": 988995
+}
+```
+
+988,995 expected stored events (1,039,050 accepted − 50,055 heartbeats), 988,995
+found — exact. 988,995 stored over the 601.6 s window is 1,644 stored/s, 142.04M
+stored events/day, 17.6% above the 2026-08-21 result (120.81M).
+
+### Heavy
+
+Certified by the harness: 19,402 requests, zero failures, exact reconciliation,
+1,990,190 request bytes/second under the 64 KiB body limit. 55.26M stored
+events/day is 28% below the 2026-08-22 fresh-store recertification (76.62M). The
+payload and validation are unchanged; the store was not. This run followed hours
+of sustained battery load on a canary holding millions of freshly written rows,
+where 2026-08-22 hit a much lighter store. Both numbers stand side by side:
+76.62M remains the fresh-store ceiling, 55.26M is the same workload on a loaded
+store. Neither is overwritten by the other without a matched-state rerun.
+
+### Feature — certification FAILED (reproducible defect)
+
+The feature profile (goals, funnels, and affiliate attribution all matching)
+cannot currently be certified: a steady ~4–5% of requests fail with
+`500 INTERNAL_ERROR` at every concurrency tried (100, 15, and 5), and each
+failed request loses its whole batch — reconciliation of a 60 s smoke found only
+946 of 997 run sessions, so this is data loss under load. Backend logs show OCC
+retry exhaustion on the affiliate daily rollups, always on the same two
+documents — the two `REPORT_ROLLUP_SHARDS` rows for the demo affiliate's day:
+
+```text
+[CONVEX M(ingest:ingestBatch)] Documents read from or written to the
+"affiliateDailyRollups" table changed while this mutation was being run and on
+every subsequent retry. Another call to this mutation changed the document
+with ID "mh735zse9410dt27ftc8t3tv4s8deazf".
+```
+
+The defect postdates alpha.3 (the 2026-08-21 feature run predates the
+generation-1 daily rollups and passed clean at 384 stored/s) and predates the
+Web Vitals change, whose fold path is separate. It is tracked as
+[issue #52](https://github.com/amageweb/amage-rastro/issues/52); the fix needs
+wider rollup sharding, rebalanced feature-report read budgets, and a
+rollout-generation bump. Until it lands, no feature-profile capacity is claimed
+at all — the 2026-08-21 33.22M row is historical evidence for alpha.3 only.
+
+## Certified run JSON
+
+### Lean (`cert-lean-alpha5`, run `862673e3-ff44-492e-a078-5d8c33f70101`)
+
+```json
+{
+  "requests": { "started": 24940, "completed": 24940, "failures": 0 },
+  "events": {
+    "sent": 1247000,
+    "heartbeats": 0,
+    "accepted": 1247000,
+    "stored": 1247000,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "throughput": {
+    "storedEventsPerSecond": 2071,
+    "certifiedStoredEventsPerDay": 178948552,
+    "requestBytesPerSecond": 575501
+  },
+  "latencyMs": { "p50": 2156, "p95": 4167, "p99": 5925, "max": 7189 },
+  "reconciliation": {
+    "expectedStoredEvents": 1247000,
+    "storedEvents": 1247000,
+    "matches": true
+  }
+}
+```
+
+### Heavy (`cert-heavy-alpha5-r2`)
+
+```json
+{
+  "requests": { "started": 19402, "completed": 19402, "failures": 0 },
+  "events": {
+    "sent": 388040,
+    "accepted": 388040,
+    "stored": 388040,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "throughput": {
+    "storedEventsPerSecond": 640,
+    "certifiedStoredEventsPerDay": 55259642,
+    "requestBytesPerSecond": 1990190
+  },
+  "latencyMs": { "p50": 2545, "p95": 7702, "p99": 10325, "max": 11220 },
+  "reconciliation": {
+    "expectedStoredEvents": 388040,
+    "storedEvents": 388040,
+    "matches": true
+  }
+}
+```
+
+### Realistic (`cert-realistic-alpha5-r3-a1`, run `3c81abe6-60cd-4b40-828f-cedf03fedd39`)
+
+```json
+{
+  "requests": { "started": 20781, "completed": 20781, "failures": 0 },
+  "events": {
+    "sent": 1039050,
+    "heartbeats": 50055,
+    "accepted": 1039050,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "throughput": {
+    "eventsPerSecond": 1727,
+    "requestBytesPerSecond": 523501
+  },
+  "latencyMs": { "p50": 1790, "p95": 4953, "p99": 7096, "max": 9274 },
+  "reconciliation": "lost to a client-side DNS failure; re-run exactly, see above"
+}
+```
+
+## What still lacks certification
+
+GeoIP with a provider enabled, trusted server-side conversions, vitals-bearing
+traffic (the `vital` ingest branch ran idle here), and the feature profile until
+issue #52 is fixed.

package/docs/federation-setup.md

@@ -0,0 +1,397 @@
+# Federated dashboard setup
+
+This runbook connects an existing Convex or Synapse project to the Amage Rastro
+control plane without moving telemetry out of that project. Read
+[`federation.md`](federation.md) for the protocol and threat model.
+
+## Package eligibility requirement
+
+Do not infer federation availability from this website, a dist-tag, or a source
+checkout. Discover the current tag without changing the host:
+
+```bash
+npm view @iann29/rastro@alpha version
+```
+
+After the operator approves that exact version, execute the complete static
+tarball, export-map, AST, and TypeScript consumer probe in the **Package
+eligibility gate** of the canonical
+[`agent/integration.md`](https://www.amagerastro.com/agent/integration.md). Do
+not replace it with an `import()` test, which executes package code, or a text
+search, which can accept comments and broken re-exports.
+
+If verification fails, stop and request an eligible release. If it passes,
+record that exact version but do not install it yet. Detect the currently
+installed version with `npm ls @iann29/rastro --depth=0`, then follow
+[`upgrading.md`](upgrading.md): snapshot/rehearse maintained deployments and
+complete every required migration bridge. Only then install with
+`npm install --ignore-scripts --save-exact @iann29/rastro@EXACT_VERSION` and
+commit the lockfile. Never fetch a mutable branch or install a source tarball
+supplied by this website.
+
+## 1. Verify local analytics first
+
+Do not configure federation until the customer deployment already has:
+
+- the Rastro component mounted under `/rastro/`;
+- at least one site with the correct allowed browser domains;
+- a tracker using the deployment's HTTP-actions URL;
+- an accepted `POST /rastro/events` request;
+- a protected local report returning that site's telemetry.
+- an audit through authorized local reports of retained routes, visitor/session
+  IDs, live location, campaigns, affiliates, conversions, and custom properties,
+  proving that no unexpected personal data, credentials, secrets, or form values
+  will become visible to the intended dashboard organization members and that
+  retention is acceptable.
+
+The README quickstart covers those steps. Federation only exposes existing
+reports; it does not install the tracker or create sites.
+
+## 2. Record both deployment URLs
+
+The connection form requires two different base URLs with no path, query, hash,
+username, or password:
+
+| Field          | Used for                          | Convex Cloud example           | Synapse example                        |
+| -------------- | --------------------------------- | ------------------------------ | -------------------------------------- |
+| URL de funções | Reactive queries and JWT audience | `https://product.convex.cloud` | `https://product.synapse.example`      |
+| URL HTTP       | Tracker and event ingestion       | `https://product.convex.site`  | `https://product.site.synapse.example` |
+
+Custom domains are valid when they route to the corresponding Convex surface.
+The control-plane operator must add each custom functions origin to
+`RASTRO_FEDERATION_ALLOWED_DEPLOYMENT_ORIGINS` as an exact, comma-separated
+HTTPS origin before it can be registered. Do not derive one URL by appending
+paths to the other or derive the allowlist from browser input. If the dashboard
+returns `UNTRUSTED_DEPLOYMENT_HOST`, stop and ask the Amage Rastro operator who
+issued the organization access to add the exact functions origin. Provide only
+that public HTTPS origin, never credentials; retry after the operator confirms
+the allowlist update.
+
+## 3. Create the pending connection
+
+In the central dashboard:
+
+1. Sign in and select the intended organization.
+2. Open **Conexões**.
+3. Enter a recognizable name, the functions URL, and the HTTP URL.
+4. Create the connection, but do not verify it yet.
+5. Copy the displayed issuer, audience, connection ID, and organization ID.
+
+The audience must exactly equal the normalized functions URL. The connection
+remains pending until its local grant is installed and verification succeeds.
+
+## 4. Add the authoritative local grant
+
+Before editing, inventory every existing schema table/index, component mount/env
+declaration, and authentication provider. After editing, prove each remains in
+the diff and typechecks. The snippets in this runbook are insertion fragments,
+not replacement files.
+
+Insert this table property into the pilot project's existing
+`defineSchema({ ... })` object:
+
+```ts
+rastroFederationGrants: defineTable({
+  connectionId: v.string(),
+  organizationId: v.string(),
+  siteIds: v.array(v.string()),
+  revokedAt: v.optional(v.number()),
+  createdAt: v.number(),
+  updatedAt: v.number(),
+}).index("by_connectionId", ["connectionId"]),
+```
+
+The grant, not the JWT, owns the allowed `siteIds`. Add operator-only
+provisioning functions:
+
+```ts
+// convex/rastroFederationAdmin.ts
+import { ConvexError, v } from "convex/values";
+import { Rastro } from "@iann29/rastro";
+import { components } from "./_generated/api";
+import { internalMutation } from "./_generated/server";
+
+const rastro = new Rastro(components.rastroAnalytics);
+
+export const provisionConnection = internalMutation({
+  args: {
+    connectionId: v.string(),
+    organizationId: v.string(),
+    ownerId: v.string(),
+    siteIds: v.array(v.string()),
+  },
+  returns: v.null(),
+  handler: async (ctx, args) => {
+    const siteIds = [...new Set(args.siteIds)];
+    if (
+      args.connectionId.length === 0 ||
+      args.connectionId.length > 128 ||
+      args.organizationId.length === 0 ||
+      args.organizationId.length > 128 ||
+      siteIds.length === 0 ||
+      siteIds.length > 10 ||
+      siteIds.length !== args.siteIds.length
+    ) {
+      throw new ConvexError({ code: "INVALID_FEDERATION_GRANT" });
+    }
+
+    for (const siteId of siteIds) {
+      const site = await rastro.getSite(ctx, siteId);
+      if (!site || site.ownerId !== args.ownerId) {
+        throw new ConvexError({ code: "FEDERATION_SITE_NOT_OWNED" });
+      }
+    }
+
+    const existing = await ctx.db
+      .query("rastroFederationGrants")
+      .withIndex("by_connectionId", (query) =>
+        query.eq("connectionId", args.connectionId),
+      )
+      .unique();
+    const now = Date.now();
+    const grant = {
+      connectionId: args.connectionId,
+      organizationId: args.organizationId,
+      siteIds,
+      revokedAt: undefined,
+      updatedAt: now,
+    };
+    if (existing) {
+      await ctx.db.patch("rastroFederationGrants", existing._id, grant);
+    } else {
+      await ctx.db.insert("rastroFederationGrants", {
+        ...grant,
+        createdAt: now,
+      });
+    }
+    return null;
+  },
+});
+
+export const revokeConnection = internalMutation({
+  args: { connectionId: v.string() },
+  returns: v.null(),
+  handler: async (ctx, args) => {
+    const grant = await ctx.db
+      .query("rastroFederationGrants")
+      .withIndex("by_connectionId", (query) =>
+        query.eq("connectionId", args.connectionId),
+      )
+      .unique();
+    if (!grant) {
+      throw new ConvexError({ code: "FEDERATION_GRANT_NOT_FOUND" });
+    }
+    if (grant.revokedAt === undefined) {
+      const now = Date.now();
+      await ctx.db.patch("rastroFederationGrants", grant._id, {
+        revokedAt: now,
+        updatedAt: now,
+      });
+    }
+    return null;
+  },
+});
+```
+
+These mutations are `internalMutation`s so a browser cannot call them. Run them
+only with deployment administrator credentials. A host that needs self-service
+pairing may wrap equivalent logic in its own authenticated owner/admin mutation;
+never accept an owner or user ID from an untrusted caller.
+
+## 5. Trust the federation issuer
+
+The displayed issuer must exactly equal the independently verified production
+manifest issuer, `https://site.api.amagerastro.com/federation`; abort on any
+mismatch. Derive the audience locally from the normalized functions URL and
+compare it with the displayed audience. For Convex Cloud DEV:
+
+```bash
+npx convex env set --deployment dev RASTRO_FEDERATION_ISSUER \
+  'https://site.api.amagerastro.com/federation'
+npx convex env set --deployment dev RASTRO_FEDERATION_AUDIENCE \
+  'https://product.convex.cloud'
+```
+
+Use `--deployment prod` only with fresh production consent. For an existing
+Synapse-managed deployment, target its live Convex environment explicitly:
+
+```bash
+synapse convex --dev env set \
+  RASTRO_FEDERATION_ISSUER 'https://site.api.amagerastro.com/federation'
+synapse convex --dev env set \
+  RASTRO_FEDERATION_AUDIENCE 'https://product.synapse.example'
+```
+
+Use `synapse env set NAME=value --for=dev` only when defining project defaults
+before a new deployment is created; defaults do not update an existing live
+deployment.
+
+Insert only these two properties into the pilot project's existing
+`defineApp({ env: { ... } })` object. If no `env` object exists, add it without
+changing any other `defineApp` option. Do not add a second Rastro mount:
+
+```ts
+RASTRO_FEDERATION_ISSUER: v.string(),
+RASTRO_FEDERATION_AUDIENCE: v.string(),
+```
+
+Insert this conditional spread into the existing `providers` array; never
+replace the array or its current entries:
+
+```ts
+// convex/auth.config.ts
+import type { AuthConfig } from "convex/server";
+
+const issuer = process.env.RASTRO_FEDERATION_ISSUER;
+const audience = process.env.RASTRO_FEDERATION_AUDIENCE;
+
+...(issuer && audience
+  ? [
+      {
+        domain: issuer.replace(/\/$/, ""),
+        applicationID: audience.replace(/\/$/, ""),
+      },
+    ]
+  : []),
+```
+
+An exact issuer mismatch is rejected even when the JWT signature is otherwise
+valid. Convex discovers keys from `<issuer>/.well-known/openid-configuration`.
+
+## 6. Expose the read-only host module
+
+Create the canonical module with this exact filename:
+
+```ts
+// convex/rastroFederation.ts
+import {
+  exposeFederatedAnalyticsApi,
+  type FederatedAnalyticsConnection,
+} from "@iann29/rastro";
+import { components } from "./_generated/api";
+import { env, type QueryCtx } from "./_generated/server";
+
+const federated = exposeFederatedAnalyticsApi(components.rastroAnalytics, {
+  issuer: env.RASTRO_FEDERATION_ISSUER.replace(/\/$/, ""),
+  resolveConnection: async (ctx, identity) => {
+    const db = ctx.db as unknown as QueryCtx["db"];
+    const grant = await db
+      .query("rastroFederationGrants")
+      .withIndex("by_connectionId", (query) =>
+        query.eq("connectionId", identity.connectionId),
+      )
+      .unique();
+    if (!grant || grant.organizationId !== identity.organizationId) return null;
+    return {
+      connectionId: grant.connectionId,
+      organizationId: grant.organizationId,
+      siteIds: grant.siteIds,
+      revokedAt: grant.revokedAt,
+    } satisfies FederatedAnalyticsConnection;
+  },
+});
+
+export const {
+  manifest,
+  connectionStatus,
+  listSites,
+  overview,
+  liveVisitors,
+  listSessions,
+  sessionJourney,
+  listConversions,
+  visitorJourney,
+  goalsReport,
+  funnelsReport,
+  affiliatesReport,
+  vitalsReport,
+  siteMap,
+  dataCoverage,
+} = federated;
+```
+
+The manifest is public static metadata. Every other function is read-only and
+requires both a valid JWT and a matching non-revoked local grant.
+
+## 7. Deploy and provision
+
+Push the customer host changes first:
+
+```bash
+# Convex Cloud DEV, one push
+npx convex dev --once
+
+# Synapse DEV
+synapse dev --once
+```
+
+Then provision the exact site scope. Replace every placeholder with values
+copied from the pending connection and the host's existing site records:
+
+```bash
+# Convex Cloud DEV
+npx convex run --deployment dev rastroFederationAdmin:provisionConnection \
+  '{"connectionId":"CONNECTION_ID","organizationId":"ORGANIZATION_ID","ownerId":"OPAQUE_HOST_OWNER_ID","siteIds":["SITE_ID"]}'
+
+# Synapse DEV
+synapse convex --dev run rastroFederationAdmin:provisionConnection \
+  '{"connectionId":"CONNECTION_ID","organizationId":"ORGANIZATION_ID","ownerId":"OPAQUE_HOST_OWNER_ID","siteIds":["SITE_ID"]}'
+```
+
+Use the corresponding explicit production target only after rehearsing DEV. The
+command is idempotent for one `connectionId` and replaces its site scope.
+
+## 8. Verify in order
+
+1. Run `rastroFederation:manifest` without auth and confirm protocol version 1.
+2. Return to **Conexões** in the central dashboard.
+3. Click **Verificar conexão**.
+4. Confirm the connection becomes active and reports the expected site count.
+5. Open Visão geral and confirm the deployment selector names the new host.
+6. Trigger one real browser pageview and confirm Ao vivo updates reactively.
+7. Check Metas/Afiliados with a complete UTC-day range when those features are
+   configured.
+
+The central verification signs a short-lived RS256 token, calls authenticated
+`connectionStatus` and `listSites`, and refuses incompatible manifests or
+inconsistent site counts.
+
+## 9. Revoke safely
+
+For immediate denial, revoke the local grant first:
+
+```bash
+npx convex run --deployment dev rastroFederationAdmin:revokeConnection \
+  '{"connectionId":"CONNECTION_ID"}'
+
+# Synapse DEV
+synapse convex --dev run rastroFederationAdmin:revokeConnection \
+  '{"connectionId":"CONNECTION_ID"}'
+```
+
+For production, repeat the rehearsed command with the platform's explicit
+production target: `npx convex run --deployment prod ...` or
+`synapse convex --prod run ...`. Never infer the target from the current shell
+state.
+
+Then revoke the connection in the central dashboard. Central revocation stops
+new tokens; local revocation rejects already-issued tokens immediately. Without
+the local step, an existing token remains usable until its ten-minute expiry.
+
+## Troubleshooting
+
+| Symptom or code                | Check                                                                         |
+| ------------------------------ | ----------------------------------------------------------------------------- |
+| Manifest not found             | File must be `convex/rastroFederation.ts`; deploy/codegen must have completed |
+| `FEDERATION_UNAUTHENTICATED`   | Federation provider missing, token absent, or deployment not redeployed       |
+| `FEDERATION_ISSUER_MISMATCH`   | `auth.config.ts` domain and helper issuer differ, often by a trailing slash   |
+| JWT audience failure           | `RASTRO_FEDERATION_AUDIENCE` must equal the functions URL, not the HTTP URL   |
+| `FEDERATION_CONNECTION_DENIED` | Grant missing, organization mismatched, empty, malformed, or revoked          |
+| `FEDERATION_SITE_DENIED`       | Dashboard requested a site outside the local grant                            |
+| Tracker 404                    | Tracker uses the functions URL instead of the HTTP-actions URL                |
+| `ORIGIN_NOT_ALLOWED`           | Browser host is absent from the site's exact/wildcard domains                 |
+| `REPORT_INCOMPLETE`            | Use complete UTC buckets or reduce a range that exceeds a bounded read budget |
+| `UNTRUSTED_DEPLOYMENT_HOST`    | Ask the control-plane operator to allow the exact custom functions origin     |
+
+Do not fix authorization errors by weakening the resolver, copying site IDs into
+JWT claims, or exposing a permissive report API.