@iann29/rastro 0.1.0-alpha.1 → 0.1.0-alpha.11

package/docs/benchmarks/2026-08-21-formal-certification.md

@@ -0,0 +1,353 @@
+# Formal ingest certification, 2026-08-21
+
+This report preserves four isolated ten-minute benchmark runs against the
+non-production Synapse canary. It is engineering evidence for the tested commit
+and deployment, not a production SLA.
+
+## Environment
+
+| Item                | Value                                              |
+| ------------------- | -------------------------------------------------- |
+| Git commit          | `e6ae92658c86c1931e697fa2a38c80fd902f7afa`         |
+| Package version     | `0.1.0-alpha.3`                                    |
+| Deployment          | DEV `fast-finch-7454`                              |
+| Benchmark site      | `kx75pnr9wap522n0zmh4nnb4jh8cv110`                 |
+| Driver              | One local Node.js process                          |
+| Reconciliation      | Exact run-scoped session IDs, indexed pages of 100 |
+| Health before/after | `ok: true`                                         |
+| GeoIP provider      | Disabled                                           |
+
+All runs used `--duration 600 --certify`. Runs were sequential so the profiles
+did not compete with each other. A harness certification means the run lasted
+the required duration, had clean request accounting, and reconciled every
+expected stored event. Meeting the separate 100 million stored-events/day target
+requires at least 1,157.41 stored events/second.
+
+## Results
+
+| Profile   | Concurrency | Batch | Accepted |  Stored | Stored/s | Certified stored/day |      p95 | Exact reconciliation | 100M/day target  |
+| --------- | ----------: | ----: | -------: | ------: | -------: | -------------------: | -------: | -------------------- | ---------------- |
+| Lean      |         100 |    50 |  952,600 | 952,600 |    1,582 |          136,654,617 | 4,543 ms | 952,600 / 952,600    | Pass             |
+| Realistic |          75 |    50 |  885,700 | 842,080 |    1,398 |          120,808,947 | 4,247 ms | 842,080 / 842,080    | Pass             |
+| Heavy     |         100 |    20 |  353,980 | 353,980 |      588 |           50,769,480 | 5,787 ms | 353,980 / 353,980    | **Below target** |
+| Feature   |         100 |    15 |  232,170 | 232,170 |      384 |           33,218,356 | 6,226 ms | 232,170 / 232,170    | **Below target** |
+
+The realistic run accepted 43,620 heartbeats, which intentionally update live
+presence without increasing persistent event counts. Its accepted-versus-stored
+difference is therefore exact. The heavy request measured 62,054 bytes and
+sustained 1,828,443 request bytes/second below the 64 KiB HTTP body limit.
+
+## Heavy recertification after byte-accounting optimization
+
+Commit `4460ad34314a6e15c24a0c78aab6e6ce85db5c60` reused each event's exact
+UTF-8 byte measurement for both batch validation and rate limits, removing one
+full-payload serialization from the ingestion transaction. A new isolated
+ten-minute run on 2026-08-22 used the same deployment, site, concurrency, batch
+size, and heavy payload.
+
+| Result               |   Original | Recertified | Change |
+| -------------------- | ---------: | ----------: | -----: |
+| Stored events/s      |        588 |         887 | +50.9% |
+| Certified stored/day | 50,769,480 |  76,623,702 | +50.9% |
+| Request bytes/s      |  1,828,443 |   2,759,875 | +50.9% |
+| p95 latency          |   5,787 ms |    3,670 ms | -36.6% |
+
+The recertification remained below the separate 100 million events/day target,
+but raised the measured heavy-payload ceiling without changing payload shape or
+weakening validation, rate limits, or reconciliation.
+
+```json
+{
+  "label": "heavy-2026-08-22T04:03:22.517Z",
+  "runId": "f3943151-bc71-4806-b609-69faf8bdfb0c",
+  "profile": "heavy",
+  "targetDurationSeconds": 600,
+  "elapsedMs": 601726,
+  "terminationReason": "duration",
+  "durationCompleted": true,
+  "certification": {
+    "requested": true,
+    "passed": true,
+    "minimumDurationSeconds": 600
+  },
+  "concurrency": 100,
+  "batchSize": 20,
+  "sessionPool": 2048,
+  "maximumRequests": null,
+  "requests": { "started": 26682, "completed": 26682, "failures": 0 },
+  "events": {
+    "sent": 533640,
+    "heartbeats": 0,
+    "accepted": 533640,
+    "stored": 533640,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "errors": {},
+  "throughput": {
+    "eventsPerSecond": 887,
+    "storedEventsPerSecond": 887,
+    "estimatedEventsPerDay": 76623702,
+    "estimatedStoredEventsPerDay": 76623702,
+    "certifiedEventsPerDay": 76623702,
+    "certifiedStoredEventsPerDay": 76623702,
+    "requestBytesPerSecond": 2759875
+  },
+  "latencyMs": { "p50": 2035, "p95": 3670, "p99": 4755, "max": 8139 },
+  "reconciliation": {
+    "scope": "run-session-ids",
+    "reliable": true,
+    "expectedSessions": 2048,
+    "requestedSessions": 2048,
+    "foundSessions": 2048,
+    "acceptedEvents": 533640,
+    "acceptedHeartbeats": 0,
+    "expectedStoredEvents": 533640,
+    "storedEvents": 533640,
+    "matches": true
+  }
+}
+```
+
+## Run artifacts
+
+### Lean
+
+```json
+{
+  "label": "lean-2026-08-21T13:31:27.434Z",
+  "runId": "41788de1-964e-4315-8b00-f295e202f2e0",
+  "profile": "lean",
+  "targetDurationSeconds": 600,
+  "elapsedMs": 602282,
+  "terminationReason": "duration",
+  "durationCompleted": true,
+  "certification": {
+    "requested": true,
+    "passed": true,
+    "minimumDurationSeconds": 600
+  },
+  "concurrency": 100,
+  "batchSize": 50,
+  "sessionPool": "unique-per-request",
+  "maximumRequests": null,
+  "requests": { "started": 19052, "completed": 19052, "failures": 0 },
+  "events": {
+    "sent": 952600,
+    "heartbeats": 0,
+    "accepted": 952600,
+    "stored": 952600,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "errors": {},
+  "throughput": {
+    "eventsPerSecond": 1582,
+    "storedEventsPerSecond": 1582,
+    "estimatedEventsPerDay": 136654617,
+    "estimatedStoredEventsPerDay": 136654617,
+    "certifiedEventsPerDay": 136654617,
+    "certifiedStoredEventsPerDay": 136654617,
+    "requestBytesPerSecond": 438830
+  },
+  "latencyMs": { "p50": 2979, "p95": 4543, "p99": 5217, "max": 5628 },
+  "reconciliation": {
+    "scope": "run-session-ids",
+    "reliable": true,
+    "expectedSessions": 19052,
+    "requestedSessions": 19052,
+    "foundSessions": 19052,
+    "acceptedEvents": 952600,
+    "acceptedHeartbeats": 0,
+    "expectedStoredEvents": 952600,
+    "storedEvents": 952600,
+    "matches": true
+  }
+}
+```
+
+### Realistic
+
+```json
+{
+  "label": "realistic-2026-08-21T13:42:07.118Z",
+  "runId": "8a883d9a-537a-4ce5-b300-be77995ba5da",
+  "profile": "realistic",
+  "targetDurationSeconds": 600,
+  "elapsedMs": 602238,
+  "terminationReason": "duration",
+  "durationCompleted": true,
+  "certification": {
+    "requested": true,
+    "passed": true,
+    "minimumDurationSeconds": 600
+  },
+  "concurrency": 75,
+  "batchSize": 50,
+  "sessionPool": 2048,
+  "maximumRequests": null,
+  "requests": { "started": 17714, "completed": 17714, "failures": 0 },
+  "events": {
+    "sent": 885700,
+    "heartbeats": 43620,
+    "accepted": 885700,
+    "stored": 842080,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "errors": {},
+  "throughput": {
+    "eventsPerSecond": 1471,
+    "storedEventsPerSecond": 1398,
+    "estimatedEventsPerDay": 127066887,
+    "estimatedStoredEventsPerDay": 120808947,
+    "certifiedEventsPerDay": 127066887,
+    "certifiedStoredEventsPerDay": 120808947,
+    "requestBytesPerSecond": 445267
+  },
+  "latencyMs": { "p50": 2257, "p95": 4247, "p99": 5122, "max": 6133 },
+  "reconciliation": {
+    "scope": "run-session-ids",
+    "reliable": true,
+    "expectedSessions": 2048,
+    "requestedSessions": 2048,
+    "foundSessions": 2048,
+    "acceptedEvents": 885700,
+    "acceptedHeartbeats": 43620,
+    "expectedStoredEvents": 842080,
+    "storedEvents": 842080,
+    "matches": true
+  }
+}
+```
+
+### Heavy
+
+```json
+{
+  "label": "heavy-2026-08-21T13:52:43.382Z",
+  "runId": "8cd7221e-2e33-4e94-9f9d-13bb4a84c60e",
+  "profile": "heavy",
+  "targetDurationSeconds": 600,
+  "elapsedMs": 602407,
+  "terminationReason": "duration",
+  "durationCompleted": true,
+  "certification": {
+    "requested": true,
+    "passed": true,
+    "minimumDurationSeconds": 600
+  },
+  "concurrency": 100,
+  "batchSize": 20,
+  "sessionPool": 2048,
+  "maximumRequests": null,
+  "requests": { "started": 17699, "completed": 17699, "failures": 0 },
+  "events": {
+    "sent": 353980,
+    "heartbeats": 0,
+    "accepted": 353980,
+    "stored": 353980,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "errors": {},
+  "throughput": {
+    "eventsPerSecond": 588,
+    "storedEventsPerSecond": 588,
+    "estimatedEventsPerDay": 50769480,
+    "estimatedStoredEventsPerDay": 50769480,
+    "certifiedEventsPerDay": 50769480,
+    "certifiedStoredEventsPerDay": 50769480,
+    "requestBytesPerSecond": 1828443
+  },
+  "latencyMs": { "p50": 2968, "p95": 5787, "p99": 6942, "max": 7839 },
+  "reconciliation": {
+    "scope": "run-session-ids",
+    "reliable": true,
+    "expectedSessions": 2048,
+    "requestedSessions": 2048,
+    "foundSessions": 2048,
+    "acceptedEvents": 353980,
+    "acceptedHeartbeats": 0,
+    "expectedStoredEvents": 353980,
+    "storedEvents": 353980,
+    "matches": true
+  }
+}
+```
+
+### Feature
+
+```json
+{
+  "label": "feature-2026-08-21T14:03:20.654Z",
+  "runId": "8e60bc5f-6fff-4791-b02c-0e7340767b55",
+  "profile": "feature",
+  "targetDurationSeconds": 600,
+  "elapsedMs": 603868,
+  "terminationReason": "duration",
+  "durationCompleted": true,
+  "certification": {
+    "requested": true,
+    "passed": true,
+    "minimumDurationSeconds": 600
+  },
+  "concurrency": 100,
+  "batchSize": 15,
+  "sessionPool": 2048,
+  "maximumRequests": null,
+  "requests": { "started": 15478, "completed": 15478, "failures": 0 },
+  "events": {
+    "sent": 232170,
+    "heartbeats": 0,
+    "accepted": 232170,
+    "stored": 232170,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "errors": {},
+  "throughput": {
+    "eventsPerSecond": 384,
+    "storedEventsPerSecond": 384,
+    "estimatedEventsPerDay": 33218356,
+    "estimatedStoredEventsPerDay": 33218356,
+    "certifiedEventsPerDay": 33218356,
+    "certifiedStoredEventsPerDay": 33218356,
+    "requestBytesPerSecond": 119846
+  },
+  "latencyMs": { "p50": 3523, "p95": 6226, "p99": 7722, "max": 8524 },
+  "reconciliation": {
+    "scope": "run-session-ids",
+    "reliable": true,
+    "expectedSessions": 2048,
+    "requestedSessions": 2048,
+    "foundSessions": 2048,
+    "acceptedEvents": 232170,
+    "acceptedHeartbeats": 0,
+    "expectedStoredEvents": 232170,
+    "storedEvents": 232170,
+    "matches": true
+  }
+}
+```
+
+## Conclusions
+
+- Lean and realistic traffic exceeded 100 million stored events/day for a full
+  ten minutes with exact reconciliation and no failures.
+- Heavy and feature traffic completed cleanly but did not meet the 100 million
+  stored-events/day target at the tested payloads and concurrency. The latest
+  heavy ceiling is 76.62 million events/day after recertification.
+- The updated heavy result demonstrates a different ceiling: 2.76 MB/s of
+  near-limit requests. It must not be generalized to minimal or realistic
+  payloads.
+- Trusted financial conversions were not exercised. They require a separate
+  server-side driver because browser traffic cannot cross the financial trust
+  boundary.
+- GeoIP provider behavior was not benchmarked because provider enrichment was
+  disabled on this deployment.

package/docs/benchmarks/2026-08-30-alpha6-recertification.md

@@ -0,0 +1,206 @@
+# Ingest recertification, 2026-08-30
+
+This report recertifies the four workload profiles ahead of `0.1.0-alpha.6`,
+against the same deployment and benchmark site as the
+[2026-08-21 formal certification](2026-08-21-formal-certification.md). It is
+engineering evidence for the tested revisions and deployment state, not a
+production SLA.
+
+## Environment
+
+| Item                | Value                                                            |
+| ------------------- | ---------------------------------------------------------------- |
+| Deployment          | DEV canary `fast-finch-7454` (unchanged since 2026-08-21)        |
+| Benchmark site      | `kx75pnr9wap522n0zmh4nnb4jh8cv110` (unchanged)                   |
+| Lean run revision   | `ba9eff1` (main before Web Vitals)                               |
+| Later run revisions | Web Vitals build; component sources identical to merge `6250526` |
+| Driver              | One local Node.js process                                        |
+| Reconciliation      | Exact run-scoped session IDs, indexed pages of 100               |
+| GeoIP provider      | Disabled                                                         |
+| Health before runs  | `ok: true`                                                       |
+
+Since the 2026-08-21 certification the wire path gained salted daily visitor
+keys, User-Agent bot rejection, client-hint browser detection, the federation
+hardening, and the `vital` ingest branch (present but idle: no profile sends
+vitals). Unlike 2026-08-21, the canary was not fresh: it accumulated the
+battery's own multi-million-event history while the profiles ran, which is the
+honest way most deployments look after a real day.
+
+All certified runs used `--duration 600 --certify`, ran sequentially with at
+least five minutes of cooldown, and had no competing local load. Two runs were
+discarded and repeated because the local DNS resolver failed their first
+connection wave (a client-side condition — the deployment was idle and healthy
+each time); one more was discarded after a mid-run deploy from this same
+machine. Discarded runs are not reported as capacity.
+
+## Results
+
+Meeting the 100 million stored-events/day target requires at least 1,157.41
+stored events/second.
+
+| Profile   | Concurrency | Batch | Sessions |  Accepted |    Stored | Stored/s | Certified stored/day |      p95 | Exact reconciliation  | 100M/day target  |
+| --------- | ----------: | ----: | -------: | --------: | --------: | -------: | -------------------: | -------: | --------------------- | ---------------- |
+| Lean      |         100 |    50 |   unique | 1,247,000 | 1,247,000 |    2,071 |          178,948,552 | 4,167 ms | 1,247,000 / 1,247,000 | Pass             |
+| Realistic |          75 |    50 |    4,096 | 1,039,050 |   988,995 |    1,644 |          142,036,529 | 4,953 ms | 988,995 / 988,995     | Pass             |
+| Heavy     |         100 |    20 |    2,048 |   388,040 |   388,040 |      640 |           55,259,642 | 7,702 ms | 388,040 / 388,040     | **Below target** |
+| Feature   |         100 |    15 |    2,048 |         — |         — |        — |           **failed** |        — | —                     | **Failed**       |
+
+### Lean
+
+Certified by the harness: 24,940 requests, zero failures, duplicates, or
+rejections; exact reconciliation. 178.95M stored events/day is 31% above the
+2026-08-21 result (136.65M) on the same deployment — the byte-measurement reuse
+first recertified for heavy on 2026-08-22 now shows up across the board, and the
+new per-request work (bot check, client hints, salted visitor keys) did not cost
+the hot path its headroom.
+
+### Realistic
+
+The driver's session pool was doubled to 4,096 (default 2,048) and that change
+is part of this row's workload definition. At the current deployment speed the
+2,048-session pool drives individual sessions past the documented 120
+events/session/minute ingest budget, so a prior attempt lost 2 of 20,477
+requests to `429 RATE_LIMITED` — the limiter working as documented, but not a
+clean certification. With 4,096 sessions the run completed 20,781 requests with
+**zero failures** of any kind.
+
+The harness itself did not stamp this run: after the clean 600 s window, its
+first reconciliation query lost a fetch to the local resolver
+(`reconciliation:fetch failed`) and gave up. Reconciliation was re-run
+immediately afterwards with the same `benchmark:reconcileSessionsPage` pages the
+harness uses, with per-page retries:
+
+```json
+{
+  "runId": "3c81abe6-60cd-4b40-828f-cedf03fedd39",
+  "requested": 4096,
+  "found": 4096,
+  "stored": 988995,
+  "expectedStored": 988995
+}
+```
+
+988,995 expected stored events (1,039,050 accepted − 50,055 heartbeats), 988,995
+found — exact. 988,995 stored over the 601.6 s window is 1,644 stored/s, 142.04M
+stored events/day, 17.6% above the 2026-08-21 result (120.81M).
+
+### Heavy
+
+Certified by the harness: 19,402 requests, zero failures, exact reconciliation,
+1,990,190 request bytes/second under the 64 KiB body limit. 55.26M stored
+events/day is 28% below the 2026-08-22 fresh-store recertification (76.62M). The
+payload and validation are unchanged; the store was not. This run followed hours
+of sustained battery load on a canary holding millions of freshly written rows,
+where 2026-08-22 hit a much lighter store. Both numbers stand side by side:
+76.62M remains the fresh-store ceiling, 55.26M is the same workload on a loaded
+store. Neither is overwritten by the other without a matched-state rerun.
+
+### Feature — certification FAILED (reproducible defect)
+
+The feature profile (goals, funnels, and affiliate attribution all matching)
+cannot currently be certified: a steady ~4–5% of requests fail with
+`500 INTERNAL_ERROR` at every concurrency tried (100, 15, and 5), and each
+failed request loses its whole batch — reconciliation of a 60 s smoke found only
+946 of 997 run sessions, so this is data loss under load. Backend logs show OCC
+retry exhaustion on the affiliate daily rollups, always on the same two
+documents — the two `REPORT_ROLLUP_SHARDS` rows for the demo affiliate's day:
+
+```text
+[CONVEX M(ingest:ingestBatch)] Documents read from or written to the
+"affiliateDailyRollups" table changed while this mutation was being run and on
+every subsequent retry. Another call to this mutation changed the document
+with ID "mh735zse9410dt27ftc8t3tv4s8deazf".
+```
+
+The defect postdates alpha.3 (the 2026-08-21 feature run predates the
+generation-1 daily rollups and passed clean at 384 stored/s) and predates the
+Web Vitals change, whose fold path is separate. It is tracked as
+[issue #52](https://github.com/amageweb/amage-rastro/issues/52); the fix needs
+wider rollup sharding, rebalanced feature-report read budgets, and a
+rollout-generation bump. Until it lands, no feature-profile capacity is claimed
+at all — the 2026-08-21 33.22M row is historical evidence for alpha.3 only.
+
+## Certified run JSON
+
+### Lean (`cert-lean-alpha5`, run `862673e3-ff44-492e-a078-5d8c33f70101`)
+
+```json
+{
+  "requests": { "started": 24940, "completed": 24940, "failures": 0 },
+  "events": {
+    "sent": 1247000,
+    "heartbeats": 0,
+    "accepted": 1247000,
+    "stored": 1247000,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "throughput": {
+    "storedEventsPerSecond": 2071,
+    "certifiedStoredEventsPerDay": 178948552,
+    "requestBytesPerSecond": 575501
+  },
+  "latencyMs": { "p50": 2156, "p95": 4167, "p99": 5925, "max": 7189 },
+  "reconciliation": {
+    "expectedStoredEvents": 1247000,
+    "storedEvents": 1247000,
+    "matches": true
+  }
+}
+```
+
+### Heavy (`cert-heavy-alpha5-r2`)
+
+```json
+{
+  "requests": { "started": 19402, "completed": 19402, "failures": 0 },
+  "events": {
+    "sent": 388040,
+    "accepted": 388040,
+    "stored": 388040,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "throughput": {
+    "storedEventsPerSecond": 640,
+    "certifiedStoredEventsPerDay": 55259642,
+    "requestBytesPerSecond": 1990190
+  },
+  "latencyMs": { "p50": 2545, "p95": 7702, "p99": 10325, "max": 11220 },
+  "reconciliation": {
+    "expectedStoredEvents": 388040,
+    "storedEvents": 388040,
+    "matches": true
+  }
+}
+```
+
+### Realistic (`cert-realistic-alpha5-r3-a1`, run `3c81abe6-60cd-4b40-828f-cedf03fedd39`)
+
+```json
+{
+  "requests": { "started": 20781, "completed": 20781, "failures": 0 },
+  "events": {
+    "sent": 1039050,
+    "heartbeats": 50055,
+    "accepted": 1039050,
+    "duplicates": 0,
+    "rejected": 0,
+    "failed": 0
+  },
+  "throughput": {
+    "eventsPerSecond": 1727,
+    "requestBytesPerSecond": 523501
+  },
+  "latencyMs": { "p50": 1790, "p95": 4953, "p99": 7096, "max": 9274 },
+  "reconciliation": "lost to a client-side DNS failure; re-run exactly, see above"
+}
+```
+
+## What still lacks certification
+
+GeoIP with a provider enabled, trusted server-side conversions, vitals-bearing
+traffic (the `vital` ingest branch ran idle here), and the feature profile until
+issue #52 is fixed.