@iann29/rastro 0.1.0-alpha.5 → 0.1.0-alpha.7

package/docs/benchmarks/2026-08-30-alpha6-recertification.md

@@ -0,0 +1,206 @@
+# Ingest recertification, 2026-08-30
+
+This report recertifies the four workload profiles ahead of `0.1.0-alpha.6`,
+against the same deployment and benchmark site as the
+[2026-08-21 formal certification](2026-08-21-formal-certification.md). It is
+engineering evidence for the tested revisions and deployment state, not a
+production SLA.
+
+## Environment
+
+| Item                | Value                                                            |
+| ------------------- | ---------------------------------------------------------------- |
+| Deployment          | DEV canary `fast-finch-7454` (unchanged since 2026-08-21)        |
+| Benchmark site      | `kx75pnr9wap522n0zmh4nnb4jh8cv110` (unchanged)                   |
+| Lean run revision   | `ba9eff1` (main before Web Vitals)                               |
+| Later run revisions | Web Vitals build; component sources identical to merge `6250526` |
+| Driver              | One local Node.js process                                        |
+| Reconciliation      | Exact run-scoped session IDs, indexed pages of 100               |
+| GeoIP provider      | Disabled                                                         |
+| Health before runs  | `ok: true`                                                       |
+
+Since the 2026-08-21 certification the wire path gained salted daily visitor
+keys, User-Agent bot rejection, client-hint browser detection, the federation
+hardening, and the `vital` ingest branch (present but idle: no profile sends
+vitals). Unlike 2026-08-21, the canary was not fresh: it accumulated the
+battery's own multi-million-event history while the profiles ran, which is the
+honest way most deployments look after a real day.
+
+All certified runs used `--duration 600 --certify`, ran sequentially with at
+least five minutes of cooldown, and had no competing local load. Two runs were
+discarded and repeated because the local DNS resolver failed their first
+connection wave (a client-side condition — the deployment was idle and healthy
+each time); one more was discarded after a mid-run deploy from this same
+machine. Discarded runs are not reported as capacity.
+
+## Results
+
+Meeting the 100 million stored-events/day target requires at least 1,157.41
+stored events/second.
+
+| Profile   | Concurrency | Batch | Sessions |  Accepted |    Stored | Stored/s | Certified stored/day |      p95 | Exact reconciliation  | 100M/day target  |
+| --------- | ----------: | ----: | -------: | --------: | --------: | -------: | -------------------: | -------: | --------------------- | ---------------- |
+| Lean      |         100 |    50 |   unique | 1,247,000 | 1,247,000 |    2,071 |          178,948,552 | 4,167 ms | 1,247,000 / 1,247,000 | Pass             |
+| Realistic |          75 |    50 |    4,096 | 1,039,050 |   988,995 |    1,644 |          142,036,529 | 4,953 ms | 988,995 / 988,995     | Pass             |
+| Heavy     |         100 |    20 |    2,048 |   388,040 |   388,040 |      640 |           55,259,642 | 7,702 ms | 388,040 / 388,040     | **Below target** |
+| Feature   |         100 |    15 |    2,048 |         — |         — |        — |           **failed** |        — | —                     | **Failed**       |
+
+### Lean
+
+Certified by the harness: 24,940 requests, zero failures, duplicates, or
+rejections; exact reconciliation. 178.95M stored events/day is 31% above the
+2026-08-21 result (136.65M) on the same deployment — the byte-measurement reuse
+first recertified for heavy on 2026-08-22 now shows up across the board, and the
+new per-request work (bot check, client hints, salted visitor keys) did not cost
+the hot path its headroom.
+
+### Realistic
+
+The driver's session pool was doubled to 4,096 (default 2,048) and that change
+is part of this row's workload definition. At the current deployment speed the
+2,048-session pool drives individual sessions past the documented 120
+events/session/minute ingest budget, so a prior attempt lost 2 of 20,477
+requests to `429 RATE_LIMITED` — the limiter working as documented, but not a
+clean certification. With 4,096 sessions the run completed 20,781 requests with
+**zero failures** of any kind.
+
+The harness itself did not stamp this run: after the clean 600 s window, its
+first reconciliation query lost a fetch to the local resolver
+(`reconciliation:fetch failed`) and gave up. Reconciliation was re-run
+immediately afterwards with the same `benchmark:reconcileSessionsPage` pages the
+harness uses, with per-page retries:
+
+```json
+{
+  "runId": "3c81abe6-60cd-4b40-828f-cedf03fedd39",
+  "requested": 4096,
+  "found": 4096,
+  "stored": 988995,
+  "expectedStored": 988995
+}
+```
+
+988,995 expected stored events (1,039,050 accepted − 50,055 heartbeats), 988,995
+found — exact. 988,995 stored over the 601.6 s window is 1,644 stored/s, 142.04M
+stored events/day, 17.6% above the 2026-08-21 result (120.81M).
+
+### Heavy
+
+Certified by the harness: 19,402 requests, zero failures, exact reconciliation,
+1,990,190 request bytes/second under the 64 KiB body limit. 55.26M stored
+events/day is 28% below the 2026-08-22 fresh-store recertification (76.62M). The
+payload and validation are unchanged; the store was not. This run followed hours
+of sustained battery load on a canary holding millions of freshly written rows,
+where 2026-08-22 hit a much lighter store. Both numbers stand side by side:
+76.62M remains the fresh-store ceiling, 55.26M is the same workload on a loaded
+store. Neither is overwritten by the other without a matched-state rerun.
+
+### Feature — certification FAILED (reproducible defect)
+
+The feature profile (goals, funnels, and affiliate attribution all matching)
+cannot currently be certified: a steady ~4–5% of requests fail with
+`500 INTERNAL_ERROR` at every concurrency tried (100, 15, and 5), and each
+failed request loses its whole batch — reconciliation of a 60 s smoke found only
+946 of 997 run sessions, so this is data loss under load. Backend logs show OCC
+retry exhaustion on the affiliate daily rollups, always on the same two
+documents — the two `REPORT_ROLLUP_SHARDS` rows for the demo affiliate's day:
+
+```text
+[CONVEX M(ingest:ingestBatch)] Documents read from or written to the
+"affiliateDailyRollups" table changed while this mutation was being run and on
+every subsequent retry. Another call to this mutation changed the document
+with ID "mh735zse9410dt27ftc8t3tv4s8deazf".
+```
+
+The defect postdates alpha.3 (the 2026-08-21 feature run predates the
+generation-1 daily rollups and passed clean at 384 stored/s) and predates the
+Web Vitals change, whose fold path is separate. It is tracked as
+[issue #52](https://github.com/amageweb/amage-rastro/issues/52); the fix needs
+wider rollup sharding, rebalanced feature-report read budgets, and a
+rollout-generation bump. Until it lands, no feature-profile capacity is claimed
+at all — the 2026-08-21 33.22M row is historical evidence for alpha.3 only.
+
+## Certified run JSON
+
+### Lean (`cert-lean-alpha5`, run `862673e3-ff44-492e-a078-5d8c33f70101`)
+
+```json
+{
+  "requests": { "started": 24940, "completed": 24940, "failures": 0 },
+  "events": {
+    "sent": 1247000,
+    "heartbeats": 0,
+    "accepted": 1247000,
+    "stored": 1247000,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "throughput": {
+    "storedEventsPerSecond": 2071,
+    "certifiedStoredEventsPerDay": 178948552,
+    "requestBytesPerSecond": 575501
+  },
+  "latencyMs": { "p50": 2156, "p95": 4167, "p99": 5925, "max": 7189 },
+  "reconciliation": {
+    "expectedStoredEvents": 1247000,
+    "storedEvents": 1247000,
+    "matches": true
+  }
+}
+```
+
+### Heavy (`cert-heavy-alpha5-r2`)
+
+```json
+{
+  "requests": { "started": 19402, "completed": 19402, "failures": 0 },
+  "events": {
+    "sent": 388040,
+    "accepted": 388040,
+    "stored": 388040,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "throughput": {
+    "storedEventsPerSecond": 640,
+    "certifiedStoredEventsPerDay": 55259642,
+    "requestBytesPerSecond": 1990190
+  },
+  "latencyMs": { "p50": 2545, "p95": 7702, "p99": 10325, "max": 11220 },
+  "reconciliation": {
+    "expectedStoredEvents": 388040,
+    "storedEvents": 388040,
+    "matches": true
+  }
+}
+```
+
+### Realistic (`cert-realistic-alpha5-r3-a1`, run `3c81abe6-60cd-4b40-828f-cedf03fedd39`)
+
+```json
+{
+  "requests": { "started": 20781, "completed": 20781, "failures": 0 },
+  "events": {
+    "sent": 1039050,
+    "heartbeats": 50055,
+    "accepted": 1039050,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "throughput": {
+    "eventsPerSecond": 1727,
+    "requestBytesPerSecond": 523501
+  },
+  "latencyMs": { "p50": 1790, "p95": 4953, "p99": 7096, "max": 9274 },
+  "reconciliation": "lost to a client-side DNS failure; re-run exactly, see above"
+}
+```
+
+## What still lacks certification
+
+GeoIP with a provider enabled, trusted server-side conversions, vitals-bearing
+traffic (the `vital` ingest branch ran idle here), and the feature profile until
+issue #52 is fixed.

package/docs/upgrading.md

@@ -6,14 +6,16 @@ deployment.
 
 ## Release matrix
 
-| Current state                                         | Required path                                                             | Success condition                                               |
-| ----------------------------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------- |
-| Fresh deployment                                      | Install the intended release directly                                     | Component deploys and `/rastro/health` succeeds                 |
-| `0.1.0-alpha.0` with maintained data                  | Install `0.1.0-alpha.1` first and wait for staged visitor-index backfills | Every staged index reports complete before installing `alpha.2` |
-| `0.1.0-alpha.1`                                       | Wait for staged visitor-index backfills, then install `alpha.2`           | Backfills complete before active readers deploy                 |
-| `0.1.0-alpha.2` with legacy `events` rows             | Stay on `alpha.2` and run/resume migration                                | Durable success plus empty source                               |
-| `0.1.0-alpha.3` fresh/canonical deployment            | Rehearse the next alpha normally                                          | Schema, reports, tracker, and retention checks pass             |
-| Any release whose packed declarations lack federation | Stop and request an eligible exact registry release                       | Packed declarations contain the federation exports              |
+| Current state                                         | Required path                                                             | Success condition                                                |
+| ----------------------------------------------------- | ------------------------------------------------------------------------- | ---------------------------------------------------------------- |
+| Fresh deployment                                      | Install the intended release directly                                     | Component deploys and `/rastro/health` succeeds                  |
+| `0.1.0-alpha.0` with maintained data                  | Install `0.1.0-alpha.1` first and wait for staged visitor-index backfills | Every staged index reports complete before installing `alpha.2`  |
+| `0.1.0-alpha.1`                                       | Wait for staged visitor-index backfills, then install `alpha.2`           | Backfills complete before active readers deploy                  |
+| `0.1.0-alpha.2` with legacy `events` rows             | Stay on `alpha.2` and run/resume migration                                | Durable success plus empty source                                |
+| `0.1.0-alpha.3` fresh/canonical deployment            | Rehearse the next alpha normally                                          | Schema, reports, tracker, and retention checks pass              |
+| `0.1.0-alpha.3` through `alpha.5` with canonical data | Install `0.1.0-alpha.6` directly; Web Vitals tables are additive          | Schema deploys and `/rastro/health` reports `webVitals: "optIn"` |
+| `0.1.0-alpha.6`                                       | Install `0.1.0-alpha.7` directly; no schema change                        | `GET /rastro/tracker.js` contains `sentAt`                       |
+| Any release whose packed declarations lack federation | Stop and request an eligible exact registry release                       | Packed declarations contain the federation exports               |
 
 Do not upgrade a populated legacy deployment directly to a release that removes
 the `events` table. Schema acceptance alone does not prove that telemetry was
@@ -93,6 +95,40 @@ The visitor-time indexes on legacy and canonical telemetry follow this rule.
 Check the release notes for the exact intermediate version before upgrading a
 large deployment.
 
+## Web Vitals (alpha.6)
+
+`0.1.0-alpha.6` adds field Web Vitals as a purely additive upgrade: a new
+`vitalRollups` component table, the `vital` event type, the `vitalsReport`
+query, the retention kind `vitalRollups` (governed by `reportsDays`), and the
+pre-gzipped `GET /rastro/vitals.js` module. No data migration and no staged
+indexes are involved — install and deploy through the host's normal workflow.
+
+Measurement is opt-in per page. Nothing reports until the host adds the second
+script tag documented in the README ("Field Web Vitals (optional)"); the base
+tracker is byte-for-byte unchanged. After deploying, the federation manifest
+advertises the optional `vitals` capability; in the central dashboard, open
+**Conexões** and click **Verificar novamente** on the active connection so the
+stored capability list picks it up and the **Vitais** view replaces its upgrade
+notice. A dashboard talking to an older host shows that notice by design — the
+protocol-v1 baseline does not require the capability.
+
+Acceptance for this upgrade: `/rastro/health` reports
+`features.webVitals: "optIn"`, `GET /rastro/vitals.js` serves the module, and
+after one real page load is hidden, `vitalsReport` shows the measurement.
+
+## Clock alignment (alpha.7)
+
+`0.1.0-alpha.7` changes no schema. The served `tracker.js` and `vitals.js` stamp
+each batch with `sentAt`, and `POST /rastro/events` re-anchors a batch whose
+sender clock is more than 30 seconds from the server's before ingestion. Live
+presence lasts five minutes after the last event or heartbeat instead of one.
+Browsers keep the previous tracker for at most the five-minute cache plus one
+background revalidation; a batch without `sentAt` is ingested exactly as before.
+
+Acceptance for this upgrade: `/rastro/health` succeeds, `GET /rastro/tracker.js`
+contains `sentAt`, and a visitor whose device clock is minutes off appears in
+**Ao vivo** within seconds of loading a page.
+
 ## Federation releases
 
 Federation is additive to component telemetry but requires host tables,
@@ -120,6 +156,8 @@ control-plane issuer.
 - Staged indexes completed before their readers were activated.
 - `/rastro/health` succeeds on the HTTP-actions URL.
 - A real tracker request is accepted from an allowed origin.
+- When the host opts into Web Vitals: `/rastro/vitals.js` serves and
+  `vitalsReport` reflects a real page load.
 - Protected overview, journey, goal, funnel, and affiliate reports used by the
   host still run.
 - Trusted conversion idempotency and currency checks still pass.

package/llms.txt

@@ -16,6 +16,7 @@ Do not infer federation availability from this website, a dist-tag, or a source
 ## Architecture
 
 - Tracker and telemetry: browser -> customer HTTP-actions URL -> customer Rastro component.
+- Optional field Web Vitals (alpha.6+): a second browser module at `/rastro/vitals.js` reports LCP, CLS, INP, FCP, and TTFB to the same events endpoint, into bounded daily histograms.
 - Reactive dashboard reads: amagerastro.com browser -> customer functions URL -> `rastroFederation:*`.
 - Control plane: account, organization, connection metadata, short-lived token issuance, and revocation.
 - The control plane does not proxy or persist customer analytics events.
@@ -29,7 +30,7 @@ Do not infer federation availability from this website, a dist-tag, or a source
 - Resolve an indexed, authoritative local grant on every federated query.
 - Grant one through ten explicit site IDs. Never grant by owner-wide enumeration.
 - Use the canonical module name `convex/rastroFederation.ts`.
-- Expose only the manifest, connection status, site summaries, reports, journeys, conversions, affiliates, funnels, goals, and coverage listed in the manifest.
+- Expose only the manifest, connection status, site summaries, reports, journeys, conversions, affiliates, funnels, goals, web vitals, and coverage listed in the manifest.
 - Use the functions URL as JWT audience and the HTTP-actions URL for tracker ingestion.
 - Rehearse on an explicit development target. Production changes require fresh operator consent.
 - Validate manifest, authenticated connection status, redacted site list, real ingestion, reactive live data, and local-first revocation.
@@ -51,7 +52,9 @@ Do not infer federation availability from this website, a dist-tag, or a source
 - Hourly overview maximum: 24 hours.
 - Funnel report maximum: 90 complete UTC days.
 - Affiliate reports require complete UTC-day boundaries.
+- Vitals report maximum: 90 complete UTC days; `vitals` is an optional capability that hosts before `alpha.6` do not advertise.
 - Maximum live visitors: 500.
+- Live window: five minutes since the visitor's last event or heartbeat. `POST /rastro/events` re-anchors a batch whose `sentAt` is more than 30 seconds from server time, so a wrong device clock decides neither liveness nor hour buckets.
 - Maximum journey events: 500.
 - Token lifetime: 10 minutes, RS256, permission `analytics:read`.
 
@@ -63,3 +66,4 @@ Do not infer federation availability from this website, a dist-tag, or a source
 - Audit retained routes, identity strategy, sessions, live location, campaigns, affiliates, conversions, custom properties, retention, and intended organization membership before granting federation; stop if their sensitivity or audience cannot be established.
 - Overview ranges require complete UTC hour or day buckets. Manual pagination honors `numItems` and `cursor` only.
 - `dataCoverage` echoes the authorized site ID but returns no event, visitor, or session IDs.
+- `vitalsReport` returns p75, averages, and exact good/needs-improvement/poor counts per metric, page, and device — never individual measurements.

package/package.json

@@ -6,7 +6,7 @@
   "bugs": {
     "url": "https://github.com/amageweb/amage-rastro/issues"
   },
-  "version": "0.1.0-alpha.5",
+  "version": "0.1.0-alpha.7",
   "license": "Apache-2.0",
   "publishConfig": {
     "access": "public"
@@ -37,7 +37,9 @@
     "federation:keys": "node ./scripts/generate-federation-keys.mjs",
     "benchmark:ingest": "node ./scripts/benchmark-ingest.mjs",
     "benchmark:local": "vitest run src/component/benchmark.test.ts",
+    "format:check": "prettier --check .",
     "verify:agent-docs": "node ./scripts/verify-agent-docs.mjs",
+    "verify:skill-paths": "node ./scripts/verify-skill-paths.mjs",
     "build:codegen": "npm run build && synapse convex codegen --component-dir ./src/component && npm run build && npm run codegen:example && synapse dev --once",
     "codegen:example": "npm run --workspace example codegen",
     "build:clean": "npm run build:codegen",
@@ -47,8 +49,9 @@
     "test:watch": "vitest --typecheck --clearScreen false",
     "test:debug": "vitest --inspect-brk --no-file-parallelism",
     "test:coverage": "vitest run --coverage --coverage.reporter=text",
+    "e2e": "playwright test -c control-plane/playwright.config.ts",
     "prepare": "npm run build",
-    "preversion": "npm run build && npm exec vite -- build control-plane && npm run test && npm run lint && npm run typecheck && npm run benchmark:local && npm run verify:agent-docs"
+    "preversion": "npm run build && VITE_CONVEX_URL=https://preflight.example.convex.cloud VITE_CONVEX_SITE_URL=https://preflight.example.convex.site npm exec vite -- build control-plane && npm run test && npm run format:check && npm run lint && npm run typecheck && npm run benchmark:local && npm run verify:agent-docs && npm run verify:skill-paths"
   },
   "files": [
     "dist",
@@ -78,6 +81,7 @@
       "default": "./dist/react/index.js"
     },
     "./tracker.js": "./dist/tracker.min.js",
+    "./vitals.js": "./dist/vitals.min.js",
     "./test": "./src/test.ts",
     "./_generated/component.js": {
       "types": "./dist/component/_generated/component.d.ts"
@@ -106,12 +110,19 @@
   "devDependencies": {
     "@convex-dev/eslint-plugin": "^4.0.0",
     "@convex-dev/static-hosting": "^0.2.1",
+    "@dicebear/core": "^9.4.3",
+    "@dicebear/lorelei": "^9.4.2",
     "@edge-runtime/vm": "^5.0.0",
     "@eslint/eslintrc": "^3.3.6",
     "@eslint/js": "9.39.5",
     "@fontsource-variable/instrument-sans": "^5.3.0",
     "@fontsource/ibm-plex-mono": "^5.3.0",
+    "@playwright/test": "^1.62.1",
     "@tailwindcss/vite": "^4.3.3",
+    "@testing-library/dom": "^10.4.1",
+    "@testing-library/jest-dom": "^7.0.1",
+    "@testing-library/react": "^16.3.3",
+    "@testing-library/user-event": "^14.6.6",
     "@types/node": "^24.13.3",
     "@types/react": "^19.2.18",
     "@types/react-dom": "^19.2.4",
@@ -129,6 +140,7 @@
     "eslint-plugin-react-hooks": "^7.1.1",
     "eslint-plugin-react-refresh": "^0.5.3",
     "globals": "^17.9.0",
+    "jsdom": "^30.0.1",
     "lucide-react": "^1.32.0",
     "maplibre-gl": "^6.4.1",
     "pkg-pr-new": "^0.0.86",
@@ -149,7 +161,9 @@
   "dependencies": {
     "@convex-dev/better-auth": "^0.12.5",
     "better-auth": "~1.6.15",
-    "jose": "^6.2.10"
+    "country-flag-icons": "^1.6.20",
+    "jose": "^6.2.10",
+    "simple-icons": "^16.29.0"
   },
   "workspaces": [
     "example"