@iann29/rastro 0.1.0-alpha.2 → 0.1.0-alpha.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +325 -63
- package/agent/integration.md +651 -0
- package/agent/manifest.json +183 -0
- package/agent/manifest.schema.json +405 -0
- package/dist/client/federation.d.ts +205 -0
- package/dist/client/federation.d.ts.map +1 -0
- package/dist/client/federation.js +179 -0
- package/dist/client/federation.js.map +1 -0
- package/dist/client/index.d.ts +1600 -8
- package/dist/client/index.d.ts.map +1 -1
- package/dist/client/index.js +210 -2
- package/dist/client/index.js.map +1 -1
- package/dist/component/_generated/api.d.ts +9 -5
- package/dist/component/_generated/api.d.ts.map +1 -1
- package/dist/component/_generated/api.js.map +1 -1
- package/dist/component/_generated/component.d.ts +161 -1
- package/dist/component/_generated/component.d.ts.map +1 -1
- package/dist/component/_generated/server.d.ts +4 -0
- package/dist/component/_generated/server.d.ts.map +1 -1
- package/dist/component/_generated/server.js.map +1 -1
- package/dist/component/cardinality.d.ts +12 -0
- package/dist/component/cardinality.d.ts.map +1 -0
- package/dist/component/cardinality.js +94 -0
- package/dist/component/cardinality.js.map +1 -0
- package/dist/component/constants.d.ts +10 -0
- package/dist/component/constants.d.ts.map +1 -1
- package/dist/component/constants.js +10 -0
- package/dist/component/constants.js.map +1 -1
- package/dist/component/convex.config.d.ts +7 -2
- package/dist/component/convex.config.d.ts.map +1 -1
- package/dist/component/convex.config.js +9 -4
- package/dist/component/convex.config.js.map +1 -1
- package/dist/component/coverage.d.ts +18 -0
- package/dist/component/coverage.d.ts.map +1 -0
- package/dist/component/coverage.js +19 -0
- package/dist/component/coverage.js.map +1 -0
- package/dist/component/diagnostics.d.ts +9 -0
- package/dist/component/diagnostics.d.ts.map +1 -0
- package/dist/component/diagnostics.js +47 -0
- package/dist/component/diagnostics.js.map +1 -0
- package/dist/component/errors.d.ts +1 -1
- package/dist/component/errors.d.ts.map +1 -1
- package/dist/component/errors.js.map +1 -1
- package/dist/component/eventStore.d.ts +11 -9
- package/dist/component/eventStore.d.ts.map +1 -1
- package/dist/component/eventStore.js +55 -227
- package/dist/component/eventStore.js.map +1 -1
- package/dist/component/geo.d.ts +71 -0
- package/dist/component/geo.d.ts.map +1 -0
- package/dist/component/geo.js +610 -0
- package/dist/component/geo.js.map +1 -0
- package/dist/component/http.d.ts.map +1 -1
- package/dist/component/http.js +168 -17
- package/dist/component/http.js.map +1 -1
- package/dist/component/ingest.d.ts.map +1 -1
- package/dist/component/ingest.js +272 -32
- package/dist/component/ingest.js.map +1 -1
- package/dist/component/live.d.ts.map +1 -1
- package/dist/component/live.js +6 -3
- package/dist/component/live.js.map +1 -1
- package/dist/component/reports.d.ts +117 -8
- package/dist/component/reports.d.ts.map +1 -1
- package/dist/component/reports.js +641 -79
- package/dist/component/reports.js.map +1 -1
- package/dist/component/retention.d.ts +75 -1
- package/dist/component/retention.d.ts.map +1 -1
- package/dist/component/retention.js +517 -54
- package/dist/component/retention.js.map +1 -1
- package/dist/component/sanitize.d.ts +4 -1
- package/dist/component/sanitize.d.ts.map +1 -1
- package/dist/component/sanitize.js +9 -4
- package/dist/component/sanitize.js.map +1 -1
- package/dist/component/schema.d.ts +202 -53
- package/dist/component/schema.js +113 -16
- package/dist/component/schema.js.map +1 -1
- package/dist/component/sites.d.ts.map +1 -1
- package/dist/component/sites.js +5 -1
- package/dist/component/sites.js.map +1 -1
- package/dist/component/validators.d.ts +29 -51
- package/dist/component/validators.d.ts.map +1 -1
- package/dist/component/validators.js +11 -12
- package/dist/component/validators.js.map +1 -1
- package/dist/tracker/generated.d.ts +4 -4
- package/dist/tracker/generated.d.ts.map +1 -1
- package/dist/tracker/generated.js +4 -4
- package/dist/tracker/generated.js.map +1 -1
- package/dist/tracker/tracker.js +7 -6
- package/dist/tracker/tracker.js.map +1 -1
- package/dist/tracker.min.js +1 -1
- package/docs/benchmarks/2026-08-20-realistic.md +5 -5
- package/docs/benchmarks/2026-08-21-formal-certification.md +333 -0
- package/docs/federation-setup.md +395 -0
- package/docs/federation.md +258 -0
- package/docs/upgrading.md +130 -0
- package/llms.txt +65 -0
- package/package.json +29 -7
- package/scripts/benchmark-ingest.mjs +81 -32
- package/scripts/generate-federation-keys.mjs +20 -0
- package/src/component/_generated/api.ts +9 -5
- package/src/component/_generated/component.ts +224 -1
- package/src/component/_generated/server.ts +4 -0
- package/src/component/cardinality.ts +116 -0
- package/src/component/constants.ts +10 -0
- package/src/component/convex.config.ts +11 -5
- package/src/component/coverage.ts +25 -0
- package/src/component/diagnostics.ts +64 -0
- package/src/component/errors.ts +2 -1
- package/src/component/eventStore.ts +78 -280
- package/src/component/geo.ts +779 -0
- package/src/component/http.ts +259 -22
- package/src/component/ingest.ts +436 -27
- package/src/component/live.ts +8 -3
- package/src/component/reports.ts +895 -87
- package/src/component/retention.ts +624 -88
- package/src/component/sanitize.ts +15 -4
- package/src/component/schema.ts +125 -16
- package/src/component/sites.ts +5 -1
- package/src/component/validators.ts +12 -13
- package/src/test.ts +0 -2
- package/dist/component/migrations.d.ts +0 -17
- package/dist/component/migrations.d.ts.map +0 -1
- package/dist/component/migrations.js +0 -44
- package/dist/component/migrations.js.map +0 -1
- package/src/component/migrations.ts +0 -52
package/README.md
CHANGED
|
@@ -4,35 +4,79 @@
|
|
|
4
4
|
|
|
5
5
|
Privacy-first, real-time web analytics built as an isolated Convex component.
|
|
6
6
|
|
|
7
|
-
Amage Rastro combines a 1,
|
|
7
|
+
Amage Rastro combines a 1,000-byte gzip browser tracker with reactive traffic
|
|
8
8
|
metrics, a live visitor atlas, journey timelines, trusted payment attribution,
|
|
9
9
|
goals, funnels, affiliates, and multi-site reports. Data and functions stay
|
|
10
10
|
inside your Convex deployment.
|
|
11
11
|
|
|
12
12
|

|
|
13
13
|
|
|
14
|
+
## Project status
|
|
15
|
+
|
|
16
|
+
Amage Rastro is an actively developed alpha with a canonical batched event
|
|
17
|
+
store, protected CI, exact run-scoped benchmark reconciliation, and published
|
|
18
|
+
npm prereleases. The untagged `latest` channel intentionally remains on the
|
|
19
|
+
first public alpha. Federation consumers must statically inspect an
|
|
20
|
+
operator-approved exact registry artifact for every required runtime and type
|
|
21
|
+
export before changing a host; follow the gate in the
|
|
22
|
+
[federation setup runbook](docs/federation-setup.md). Never substitute a mutable
|
|
23
|
+
branch or website-supplied source tarball.
|
|
24
|
+
|
|
25
|
+
Public integration documentation is available at
|
|
26
|
+
[`www.amagerastro.com/docs/`](https://www.amagerastro.com/docs/). Coding agents
|
|
27
|
+
should start with [`llms.txt`](https://www.amagerastro.com/llms.txt) and verify
|
|
28
|
+
the [machine manifest](https://www.amagerastro.com/agent/manifest.json).
|
|
29
|
+
|
|
30
|
+
Current evidence supports more than 120 million stored events/day for realistic
|
|
31
|
+
traffic. Near-limit heavy payloads are certified at 76.62 million/day, and the
|
|
32
|
+
feature-saturated profile at 33.22 million/day. Workload-specific results and
|
|
33
|
+
caveats are reported separately rather than collapsed into one capacity claim.
|
|
34
|
+
|
|
14
35
|
## What ships
|
|
15
36
|
|
|
16
|
-
| Capability | Implementation
|
|
17
|
-
| --------------------------- |
|
|
18
|
-
| Live visitor map | Reactive live sessions with coarse city-level coordinates and TTL expiry
|
|
19
|
-
| Traffic analytics | Sharded hour/day rollups, time series, top pages, sources, countries, devices, browsers, and events
|
|
20
|
-
| Journey replay | Bounded pageview/click/custom-event timelines; no DOM, keystroke, or form-value recording
|
|
21
|
-
| Payment conversion tracking | Trusted server-side ledger with idempotent event IDs and canonical site currency
|
|
22
|
-
| Goals and funnels | Exact pageview/custom-event goals and ordered conversion funnels
|
|
23
|
-
| Affiliates | Referral attribution, trusted revenue, and commission accounting
|
|
24
|
-
| Cross-site analytics | Reports over up to ten authorized site IDs with a preloaded pseudonymous visitor ID
|
|
25
|
-
|
|
|
26
|
-
|
|
|
27
|
-
|
|
|
28
|
-
|
|
29
|
-
|
|
37
|
+
| Capability | Implementation |
|
|
38
|
+
| --------------------------- | ---------------------------------------------------------------------------------------------------- |
|
|
39
|
+
| Live visitor map | Reactive live sessions with coarse city-level coordinates and TTL expiry |
|
|
40
|
+
| Traffic analytics | Sharded hour/day rollups, time series, top pages, sources, countries, devices, browsers, and events |
|
|
41
|
+
| Journey replay | Bounded pageview/click/custom-event timelines; no DOM, keystroke, or form-value recording |
|
|
42
|
+
| Payment conversion tracking | Trusted server-side ledger with idempotent event IDs and canonical site currency |
|
|
43
|
+
| Goals and funnels | Exact pageview/custom-event goals and ordered conversion funnels |
|
|
44
|
+
| Affiliates | Referral attribution, trusted revenue, and commission accounting |
|
|
45
|
+
| Cross-site analytics | Reports over up to ten authorized site IDs with a preloaded pseudonymous visitor ID |
|
|
46
|
+
| Automatic GeoIP | Opt-in native request metadata plus IPinfo/IPWhois fallback; bounded quota and no raw-IP persistence |
|
|
47
|
+
| Auto-captured clicks | Anchors, buttons, outbound links, and explicit `data-rastro-event` controls |
|
|
48
|
+
| Cookieless mode | Session-scoped pseudonymous IDs in `sessionStorage`; no cookies |
|
|
49
|
+
| Tiny tracker | 1,846 bytes raw, 1,000 bytes gzip; the test suite enforces `< 1,024` bytes |
|
|
50
|
+
|
|
51
|
+
## Quickstart
|
|
52
|
+
|
|
53
|
+
### Prerequisites
|
|
54
|
+
|
|
55
|
+
- Convex `^1.43.0` or a compatible Synapse deployment.
|
|
56
|
+
- An existing host authentication provider for administrative and local report
|
|
57
|
+
functions. The browser ingestion route itself is public and origin-checked.
|
|
58
|
+
- Two deployment origins recorded separately:
|
|
59
|
+
|
|
60
|
+
| Purpose | Convex Cloud example | Synapse example |
|
|
61
|
+
| ------------------------------ | ------------------------------ | -------------------------------------- |
|
|
62
|
+
| Functions and reactive queries | `https://project.convex.cloud` | `https://project.synapse.example` |
|
|
63
|
+
| HTTP actions and tracker | `https://project.convex.site` | `https://project.site.synapse.example` |
|
|
64
|
+
|
|
65
|
+
Never append `/rastro/tracker.js` to the functions URL.
|
|
66
|
+
|
|
67
|
+
### 1. Install
|
|
30
68
|
|
|
31
69
|
```bash
|
|
32
|
-
npm install @iann29/rastro
|
|
70
|
+
npm install @iann29/rastro@alpha
|
|
33
71
|
```
|
|
34
72
|
|
|
35
|
-
|
|
73
|
+
The untagged npm `latest` channel intentionally remains on the first public
|
|
74
|
+
alpha. Use `@alpha` until a stable release is promoted.
|
|
75
|
+
|
|
76
|
+
### 2. Mount the component
|
|
77
|
+
|
|
78
|
+
Merge the component into the project's existing configuration; preserve every
|
|
79
|
+
component and environment binding already registered there.
|
|
36
80
|
|
|
37
81
|
```ts
|
|
38
82
|
// convex/convex.config.ts
|
|
@@ -45,14 +89,93 @@ app.use(rastro, { httpPrefix: "/rastro/" });
|
|
|
45
89
|
export default app;
|
|
46
90
|
```
|
|
47
91
|
|
|
48
|
-
This creates isolated component tables and
|
|
92
|
+
This creates isolated component tables and four HTTP routes:
|
|
49
93
|
|
|
50
94
|
- `POST /rastro/events` — browser event ingestion
|
|
51
95
|
- `OPTIONS /rastro/events` — CORS preflight
|
|
52
96
|
- `GET /rastro/tracker.js` — the pre-gzipped tracker
|
|
53
97
|
- `GET /rastro/health` — integration health
|
|
54
98
|
|
|
55
|
-
|
|
99
|
+
Push the component and regenerate host bindings before importing
|
|
100
|
+
`components.rastroAnalytics`:
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
# Convex Cloud development deployment
|
|
104
|
+
npx convex dev
|
|
105
|
+
|
|
106
|
+
# Synapse development deployment
|
|
107
|
+
synapse dev --once
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
For production, use the product's existing release workflow. On an uncoordinated
|
|
111
|
+
Convex project that is `npx convex deploy`; on Synapse use
|
|
112
|
+
`synapse release --execute` when the project has a release manifest, otherwise
|
|
113
|
+
`synapse deploy`. Never deploy before confirming the target.
|
|
114
|
+
|
|
115
|
+
Verify the HTTP origin independently from the functions origin:
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
curl -fsS https://YOUR_HTTP_ORIGIN/rastro/health
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
A successful health response proves that the component HTTP routes are mounted;
|
|
122
|
+
it does not prove that a browser event was accepted.
|
|
123
|
+
|
|
124
|
+
### Automatic GeoIP
|
|
125
|
+
|
|
126
|
+
Rastro uses Convex request metadata as the canonical client IP and can resolve
|
|
127
|
+
coarse city-level coordinates on the first request of each session. GeoIP is
|
|
128
|
+
opt-in because the configured provider receives the request IP. For IPinfo Core:
|
|
129
|
+
|
|
130
|
+
```ts
|
|
131
|
+
// convex/convex.config.ts
|
|
132
|
+
import { defineApp } from "convex/server";
|
|
133
|
+
import { v } from "convex/values";
|
|
134
|
+
import rastro from "@iann29/rastro/convex.config.js";
|
|
135
|
+
|
|
136
|
+
const app = defineApp({ env: { IPINFO_TOKEN: v.string() } });
|
|
137
|
+
app.use(rastro, {
|
|
138
|
+
httpPrefix: "/rastro/",
|
|
139
|
+
env: {
|
|
140
|
+
RASTRO_GEOIP_PROVIDER: "ipinfo",
|
|
141
|
+
RASTRO_GEOIP_TOKEN: app.env.IPINFO_TOKEN,
|
|
142
|
+
RASTRO_GEOIP_DAILY_LIMIT: "1000",
|
|
143
|
+
},
|
|
144
|
+
});
|
|
145
|
+
|
|
146
|
+
export default app;
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
`ipwhois` is also supported. Its token is optional for the documented
|
|
150
|
+
1,000-request/day free endpoint and recommended for production. Bind it as
|
|
151
|
+
`RASTRO_GEOIP_TOKEN` in the same way as the IPinfo example.
|
|
152
|
+
`RASTRO_GEOIP_DAILY_LIMIT` defaults to 1,000 lookups per site per UTC day and
|
|
153
|
+
may be set from `0` through `1000000` to match the provider plan. Admission is
|
|
154
|
+
transactional, so concurrent first-session requests cannot exceed the cap.
|
|
155
|
+
|
|
156
|
+
When Rastro runs behind a reverse proxy that supplies `X-Forwarded-For` or
|
|
157
|
+
Vercel/Cloudflare geographic headers, explicitly bind
|
|
158
|
+
`RASTRO_TRUST_PROXY: "true"`. Leave it unset for direct Convex deployments.
|
|
159
|
+
Rastro uses forwarded data only when this opt-in is enabled and the runtime IP
|
|
160
|
+
identifies a private immediate peer. Caller-supplied proxy or geographic headers
|
|
161
|
+
are ignored when native metadata is absent or public.
|
|
162
|
+
|
|
163
|
+
The lookup has a 1.5-second timeout and fails open: telemetry ingestion
|
|
164
|
+
continues without location when the provider is unavailable. A session reserves
|
|
165
|
+
its single lookup attempt atomically only after its initial event is persisted;
|
|
166
|
+
heartbeats and concurrent duplicate requests do not consume provider quota. The
|
|
167
|
+
raw IP is never passed to a mutation or stored in component tables; only
|
|
168
|
+
sanitized country, city, and coordinates rounded to one decimal place are
|
|
169
|
+
persisted.
|
|
170
|
+
|
|
171
|
+
`GET /rastro/health` reports whether the runtime exposes a usable client IP and
|
|
172
|
+
whether GeoIP is configured and the daily cap, without returning the address or
|
|
173
|
+
token. Deployment administrators can exercise the configured provider through
|
|
174
|
+
the internal `geo:probe` action. If health reports no usable client IP, use
|
|
175
|
+
trusted Vercel/Cloudflare geographic headers with `RASTRO_TRUST_PROXY` or fix
|
|
176
|
+
the deployment's proxy forwarding.
|
|
177
|
+
|
|
178
|
+
### 3. Create a site
|
|
56
179
|
|
|
57
180
|
Components cannot read the host app's authentication state. The host must
|
|
58
181
|
authenticate the caller and pass an opaque owner ID.
|
|
@@ -78,7 +201,7 @@ export const createSite = mutation({
|
|
|
78
201
|
if (!identity) throw new Error("Not authenticated");
|
|
79
202
|
|
|
80
203
|
return await analytics.createSite(ctx, {
|
|
81
|
-
ownerId: identity.
|
|
204
|
+
ownerId: identity.tokenIdentifier,
|
|
82
205
|
name: args.name,
|
|
83
206
|
domains: args.domains,
|
|
84
207
|
currency: args.currency,
|
|
@@ -88,12 +211,30 @@ export const createSite = mutation({
|
|
|
88
211
|
});
|
|
89
212
|
```
|
|
90
213
|
|
|
214
|
+
Call this mutation from the host application's authenticated administration UI
|
|
215
|
+
and persist or display the returned `siteId`:
|
|
216
|
+
|
|
217
|
+
```tsx
|
|
218
|
+
const createSite = useMutation(api.analyticsAdmin.createSite);
|
|
219
|
+
const siteId = await createSite({
|
|
220
|
+
name: "Amage Product",
|
|
221
|
+
domains: ["product.example.com", "localhost"],
|
|
222
|
+
currency: "BRL",
|
|
223
|
+
});
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
Do not call the authenticated example with a bare CLI command: without an
|
|
227
|
+
explicit test identity, `ctx.auth.getUserIdentity()` correctly returns `null`.
|
|
228
|
+
|
|
91
229
|
Allowed domains are checked at ingestion. Exact hosts and wildcard subdomains
|
|
92
230
|
such as `*.example.com` are supported. `Origin` is an abuse signal, not
|
|
93
231
|
authentication; Amage Rastro also enforces per-session and fixed-shard per-site
|
|
94
232
|
event/byte budgets.
|
|
95
233
|
|
|
96
|
-
|
|
234
|
+
Domain entries are hostnames, not host-and-port pairs. Use `localhost` or a full
|
|
235
|
+
origin such as `http://localhost:3000`; both normalize to the `localhost` host.
|
|
236
|
+
|
|
237
|
+
### 4. Add the tracker
|
|
97
238
|
|
|
98
239
|
Use the HTTP-actions URL for your deployment, not the Convex function URL.
|
|
99
240
|
|
|
@@ -125,6 +266,9 @@ serves the script and ingestion from different origins:
|
|
|
125
266
|
|
|
126
267
|
### React
|
|
127
268
|
|
|
269
|
+
React is an optional peer dependency. It is required only when importing the
|
|
270
|
+
`@iann29/rastro/react` entry point; backend-only consumers do not need it.
|
|
271
|
+
|
|
128
272
|
```tsx
|
|
129
273
|
import { useState } from "react";
|
|
130
274
|
import { RastroScript, useRastro } from "@iann29/rastro/react";
|
|
@@ -154,6 +298,26 @@ the same value on authorized sites to connect their journeys. Amage Rastro
|
|
|
154
298
|
intentionally has no mutable post-load `identify` command: changing identity
|
|
155
299
|
after a session starts would make the session ambiguous.
|
|
156
300
|
|
|
301
|
+
### 5. Verify ingestion
|
|
302
|
+
|
|
303
|
+
1. Open an allowed site origin in a fresh browser tab.
|
|
304
|
+
2. In browser developer tools, filter Network requests by `rastro/events`.
|
|
305
|
+
3. Wait for the normal tracker flush, which is debounced by approximately 1.2
|
|
306
|
+
seconds.
|
|
307
|
+
4. Confirm `POST /rastro/events` returns HTTP 200 and a response containing
|
|
308
|
+
`accepted`, `duplicates`, and `rejected` counters.
|
|
309
|
+
5. Run one authenticated local report, such as `analytics:overview`, for the
|
|
310
|
+
returned `siteId` and a complete UTC hour/day range.
|
|
311
|
+
|
|
312
|
+
The tracker intentionally does not interrupt the product UI when transport
|
|
313
|
+
fails. A loaded `tracker.js` or successful `/health` response alone is not proof
|
|
314
|
+
of ingestion. The usual causes of a rejected or missing request are:
|
|
315
|
+
|
|
316
|
+
- using the functions URL instead of the HTTP-actions URL;
|
|
317
|
+
- omitting the exact browser host from the site's `domains`;
|
|
318
|
+
- including query strings or paths in a configured base URL;
|
|
319
|
+
- querying a partial UTC bucket or a site the authenticated user does not own.
|
|
320
|
+
|
|
157
321
|
## Track custom events
|
|
158
322
|
|
|
159
323
|
```js
|
|
@@ -246,12 +410,12 @@ export const {
|
|
|
246
410
|
|
|
247
411
|
for (const siteId of request.siteIds) {
|
|
248
412
|
const site = await analytics.getSite(ctx, siteId);
|
|
249
|
-
if (!site || site.ownerId !== identity.
|
|
413
|
+
if (!site || site.ownerId !== identity.tokenIdentifier) {
|
|
250
414
|
throw new Error("Not authorized for analytics site");
|
|
251
415
|
}
|
|
252
416
|
}
|
|
253
417
|
|
|
254
|
-
if (request.ownerId && request.ownerId !== identity.
|
|
418
|
+
if (request.ownerId && request.ownerId !== identity.tokenIdentifier) {
|
|
255
419
|
throw new Error("Not authorized for analytics owner");
|
|
256
420
|
}
|
|
257
421
|
},
|
|
@@ -262,6 +426,30 @@ Do not copy the unauthenticated demo boundary into production. The included demo
|
|
|
262
426
|
exposes only fixed-scope read reports plus bounded, idempotent seed/live-refresh
|
|
263
427
|
functions.
|
|
264
428
|
|
|
429
|
+
### Federated dashboard backend
|
|
430
|
+
|
|
431
|
+
The reference control plane in `example/` can connect a central dashboard while
|
|
432
|
+
telemetry remains in the customer's deployment. It uses
|
|
433
|
+
`exposeFederatedAnalyticsApi` with issuer and connection-resolver options. The
|
|
434
|
+
helper constructs its authorization boundary internally; it cannot accept a
|
|
435
|
+
permissive host authorizer. Its surface contains a public protocol v1 manifest,
|
|
436
|
+
authenticated redacted connection status, redacted site summaries, and only the
|
|
437
|
+
reactive dashboard report queries.
|
|
438
|
+
|
|
439
|
+
The JWT only identifies the requested connection. Site access remains
|
|
440
|
+
authoritative in a local host record resolved on every query, so revocation is
|
|
441
|
+
immediate and a token cannot grant itself another site. `listSites` accepts no
|
|
442
|
+
site arguments and omits owner, domain, and network configuration. Stable
|
|
443
|
+
manifest names, capabilities, limits, and authorization error codes are exported
|
|
444
|
+
for connector clients. The host must already have Convex authentication and a
|
|
445
|
+
local authoritative grant. Follow
|
|
446
|
+
[`docs/federation-setup.md`](docs/federation-setup.md) for the executable setup
|
|
447
|
+
and verification sequence; use [`docs/federation.md`](docs/federation.md) as the
|
|
448
|
+
protocol reference. The production control plane is available at
|
|
449
|
+
`https://www.amagerastro.com`; before connecting a customer deployment, verify
|
|
450
|
+
the selected package exports the federation helper as shown in the setup
|
|
451
|
+
runbook.
|
|
452
|
+
|
|
265
453
|
## Backend API
|
|
266
454
|
|
|
267
455
|
The `Rastro` class wraps the component boundary for host functions:
|
|
@@ -270,17 +458,46 @@ The `Rastro` class wraps the component boundary for host functions:
|
|
|
270
458
|
- Ingestion: `ingest`, `trackConversion`
|
|
271
459
|
- Reports: `overview`, `liveVisitors`, `listSessions`, `sessionJourney`,
|
|
272
460
|
`listConversions`, `visitorJourney`, `goalsReport`, `funnelsReport`,
|
|
273
|
-
`affiliatesReport`
|
|
461
|
+
`affiliatesReport`, `dataCoverage`
|
|
274
462
|
- Configuration: `upsertGoal`, `listGoals`, `removeGoal`, `upsertFunnel`,
|
|
275
463
|
`listFunnels`, `removeFunnel`, `upsertAffiliate`, `listAffiliates`,
|
|
276
464
|
`removeAffiliate`
|
|
277
|
-
- Retention: `cleanup`
|
|
465
|
+
- Retention: `cleanup`, `setRetentionPolicy`, `disableRetentionPolicy`,
|
|
466
|
+
`runRetentionPolicy`, `retentionStatus`
|
|
278
467
|
|
|
279
468
|
Every public function has argument and return validators. Growing reads are
|
|
280
469
|
indexed and bounded. Component pagination uses scope-bound opaque keyset cursors
|
|
281
470
|
because native `.paginate()` is not supported inside Convex components; page
|
|
282
471
|
cost stays constant instead of increasing with an offset.
|
|
283
472
|
|
|
473
|
+
Overview responses label visitor values as either HLL-estimated unique visitors
|
|
474
|
+
or legacy `bucketOccurrences`; legacy aggregate rows are never presented as
|
|
475
|
+
exact uniques. Breakdown metadata likewise identifies page, source, country,
|
|
476
|
+
device, browser, and event rankings as event-volume counts. Feature status is
|
|
477
|
+
explicit: outbound-link event volume is supported, while session replay, error
|
|
478
|
+
insights, and bot detection are currently unsupported.
|
|
479
|
+
|
|
480
|
+
All report ranges use integer Unix milliseconds with inclusive `from` and `to`
|
|
481
|
+
boundaries. Range metadata reports `boundary: "inclusive"` and
|
|
482
|
+
`timezone: "UTC"`; the site's timezone is presentation metadata and never
|
|
483
|
+
changes report bucket boundaries. Overview queries must contain complete UTC
|
|
484
|
+
hours or days, and funnel and affiliate reports must contain complete UTC days.
|
|
485
|
+
Those aggregate-backed reports fail with `REPORT_INCOMPLETE` instead of silently
|
|
486
|
+
including the rest of a partial bucket. Goal reports retain exact millisecond
|
|
487
|
+
filtering while bounded goal-completion rows are available.
|
|
488
|
+
|
|
489
|
+
Goal, funnel, and affiliate reports use definition-ID keyed daily rollups when
|
|
490
|
+
the low-churn rollout control proves that generation is available. Each loader
|
|
491
|
+
enforces day, row, document, and byte headroom before returning a complete
|
|
492
|
+
result; it never returns a silently truncated report. Ordinary ingestion never
|
|
493
|
+
patches or schedules work against a shared coverage document. `dataCoverage`
|
|
494
|
+
derives availability through bounded indexed reads of source and rollup records,
|
|
495
|
+
while the control document stores only rollout generation and durable retention
|
|
496
|
+
watermarks. Heartbeats can advance session availability but do not claim event
|
|
497
|
+
or aggregate availability. Hourly and daily overview availability are reported
|
|
498
|
+
separately as `overviewHour` and `overviewDay` because their retention
|
|
499
|
+
boundaries use different bucket alignment.
|
|
500
|
+
|
|
284
501
|
## Operational limits
|
|
285
502
|
|
|
286
503
|
- HTTP body: 64 KiB
|
|
@@ -290,17 +507,25 @@ cost stays constant instead of increasing with an offset.
|
|
|
290
507
|
- Primary aggregate shards: 256 hourly and 128 daily, derived from stable
|
|
291
508
|
session identity
|
|
292
509
|
- Funnel and affiliate aggregate shards: 16
|
|
293
|
-
- Site ingress budget: 4,096 deterministic per-minute shards, each capped at
|
|
294
|
-
|
|
510
|
+
- Site ingress budget: 4,096 deterministic per-minute shards, each capped at 300
|
|
511
|
+
events and 512 KiB (1,228,800 admitted events/minute at uniform load)
|
|
295
512
|
- Rate-window storage: one reusable row per active session and site shard
|
|
296
513
|
- Live session timeout: 60 seconds with one bounded five-second expiry sweep per
|
|
297
514
|
site; legacy per-session jobs hand off to that sweep during the cutover
|
|
298
515
|
- Dashboard overview: up to ten sites and bounded report ranges
|
|
299
516
|
- Hourly overview: up to 24 hours; wider ranges use daily buckets
|
|
300
|
-
- Retention cleanup: bounded batches
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
517
|
+
- Retention cleanup: bounded batches. `setRetentionPolicy` automatically starts
|
|
518
|
+
leased, persisted cleanup chains for every policy-managed dataset; interrupted
|
|
519
|
+
attempts resume after lease expiry and stale generations cannot mutate the
|
|
520
|
+
active chain. A completed chain schedules its next cycle after one day and
|
|
521
|
+
recomputes its cutoff from the then-current policy. Policy updates replace the
|
|
522
|
+
generation, and `disableRetentionPolicy` makes every queued attempt stale.
|
|
523
|
+
`runRetentionPolicy` remains available for an explicit bounded run. The
|
|
524
|
+
`events` cleanup kind removes canonical event-batch documents and processes at
|
|
525
|
+
most 32 heavy batches per transaction.
|
|
526
|
+
- Retention cutoffs are bucket-safe: event minutes, hourly aggregates, daily
|
|
527
|
+
aggregates and feature rollups preserve the bucket containing a cutoff and
|
|
528
|
+
return the exact effective `before` boundary they applied.
|
|
304
529
|
|
|
305
530
|
Capacity claims use declared workload profiles. Complete release evidence
|
|
306
531
|
reports ten-minute `lean`, `realistic`, `heavy`, and browser `feature` profiles
|
|
@@ -308,12 +533,52 @@ separately. The browser `feature` profile exercises configured goals, funnels,
|
|
|
308
533
|
and affiliate attribution; trusted conversions require a separate server-side
|
|
309
534
|
driver and certification result.
|
|
310
535
|
|
|
311
|
-
|
|
312
|
-
|
|
536
|
+
### Historical ingest evidence requiring recertification
|
|
537
|
+
|
|
538
|
+
| Profile | Workload | Certified stored/day | p95 latency | 100M/day target |
|
|
539
|
+
| --------- | ------------------------------------------------- | -------------------: | ----------: | --------------- |
|
|
540
|
+
| Lean | Minimal events, maximum batches | 136.65M | 4,543 ms | Pass |
|
|
541
|
+
| Realistic | Reused sessions, browser mix, context, heartbeats | 120.81M | 4,247 ms | Pass |
|
|
542
|
+
| Heavy | 62 KiB requests with bounded custom properties | 76.62M | 3,670 ms | Below target |
|
|
543
|
+
| Feature | Goals, funnels, and affiliate attribution | 33.22M | 6,226 ms | Below target |
|
|
544
|
+
|
|
545
|
+
Every listed run lasted at least ten minutes, completed with zero request
|
|
546
|
+
failures, duplicates, or rejected events, and exactly reconciled the expected
|
|
547
|
+
stored events. Results came from one client process against a non-production
|
|
548
|
+
Synapse DEV deployment; they are engineering evidence, not a production SLA. The
|
|
549
|
+
heavy row is the formal recertification after removing redundant payload
|
|
550
|
+
serialization. GeoIP with a provider enabled and trusted server-side conversions
|
|
551
|
+
remain separate pending certifications.
|
|
552
|
+
|
|
553
|
+
These runs predate the current source-derived coverage and recurring-retention
|
|
554
|
+
changes. They remain historical evidence for the named revisions and profiles,
|
|
555
|
+
not certification of this uncommitted implementation. Recertify the final code
|
|
556
|
+
before presenting any row as current capacity.
|
|
557
|
+
|
|
558
|
+
The latest formal results are recorded in
|
|
559
|
+
[`docs/benchmarks/2026-08-21-formal-certification.md`](docs/benchmarks/2026-08-21-formal-certification.md).
|
|
560
|
+
Earlier development evidence remains available in
|
|
561
|
+
[`docs/benchmarks/2026-08-20-realistic.md`](docs/benchmarks/2026-08-20-realistic.md).
|
|
313
562
|
|
|
314
563
|
From a repository checkout, run a duration-based profile against a confirmed
|
|
315
564
|
non-production deployment and a dedicated idle benchmark site:
|
|
316
565
|
|
|
566
|
+
For a deployment-free CI smoke benchmark of write-heavy ingestion plus reactive
|
|
567
|
+
overview and feature report queries, run:
|
|
568
|
+
|
|
569
|
+
```bash
|
|
570
|
+
npm run benchmark:local
|
|
571
|
+
```
|
|
572
|
+
|
|
573
|
+
The local harness uses `convex-test`, enforces a 12-document ingest transaction
|
|
574
|
+
budget, checks reactive totals before and after a write, and queries proven
|
|
575
|
+
daily rollups with the maximum configured 50 goals, 20 funnels, and 100
|
|
576
|
+
affiliates under deterministic 1,000-document and 4 MiB read ceilings. It also
|
|
577
|
+
drives concurrent ingest calls and asserts that their shared rollout/retention
|
|
578
|
+
control document is unchanged. It fails when its generous CI latency budgets are
|
|
579
|
+
exceeded. This is a deterministic local regression guard, not evidence of live
|
|
580
|
+
deployment throughput or capacity.
|
|
581
|
+
|
|
317
582
|
```bash
|
|
318
583
|
npm run --silent benchmark:ingest -- \
|
|
319
584
|
--url https://example.site/rastro/events \
|
|
@@ -322,7 +587,8 @@ npm run --silent benchmark:ingest -- \
|
|
|
322
587
|
--origin https://your-allowed-origin.example \
|
|
323
588
|
--profile realistic \
|
|
324
589
|
--duration 600 \
|
|
325
|
-
--concurrency
|
|
590
|
+
--concurrency 75 \
|
|
591
|
+
--batch-size 50 \
|
|
326
592
|
--certify
|
|
327
593
|
```
|
|
328
594
|
|
|
@@ -332,12 +598,15 @@ visible in Live Atlas. Only `feature` intentionally matches the demo goal,
|
|
|
332
598
|
funnel, and affiliate definitions; it does not call the trusted conversion API.
|
|
333
599
|
Use `--dry-run` to inspect generated request size without sending traffic. When
|
|
334
600
|
`--convex-url` is supplied, it must identify the same deployment as `--url` and
|
|
335
|
-
expose this repository's `
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
601
|
+
expose this repository's `benchmark:reconcileSessionsPage` query. Reconciliation
|
|
602
|
+
generates exact session IDs from a cryptographic run UUID and reads them through
|
|
603
|
+
bounded index lookups, so unrelated traffic cannot alter the result. The query
|
|
604
|
+
uses the session counters persisted transactionally with event batches; it
|
|
605
|
+
certifies ingest accounting, not raw-event availability after later retention.
|
|
606
|
+
Reports distinguish accepted ingress, heartbeats, and stored events. Shorter
|
|
607
|
+
completed runs report extrapolated daily estimates only. `--certify` requires at
|
|
608
|
+
least ten minutes and successful reconciliation before emitting certified daily
|
|
609
|
+
capacity fields or exiting successfully as a certification.
|
|
341
610
|
|
|
342
611
|
The ingestion transaction folds metric/dimension deltas, session rate counters,
|
|
343
612
|
site-shard counters, session/live state, visitor attribution, and exact funnel
|
|
@@ -350,26 +619,16 @@ explicit `(other)` bucket instead of relabeling old counts.
|
|
|
350
619
|
Click/outbound/browser-conversion telemetry skips goal and funnel work that
|
|
351
620
|
cannot match those event types.
|
|
352
621
|
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
npx convex run --component rastroAnalytics migrations:migrateLegacyEvents \
|
|
359
|
-
'{"dryRun":true,"reset":true}'
|
|
360
|
-
npx convex run --component rastroAnalytics migrations:migrateLegacyEvents '{}'
|
|
361
|
-
npx convex run --component rastroAnalytics migrations:legacyEventsStatus '{}'
|
|
362
|
-
```
|
|
622
|
+
Raw telemetry is stored only in immutable `eventBatches`. Migrated events retain
|
|
623
|
+
their original per-event browser and geo context inside those batches, while new
|
|
624
|
+
events hydrate context from their session. Journey `_id` values are stable
|
|
625
|
+
strings derived from batch and nested-event identity, and session-journey
|
|
626
|
+
cursors use timestamp, creation time, and ID.
|
|
363
627
|
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
bounded canonical batches and retain their original per-event browser and geo
|
|
369
|
-
context. Journey `_id` values remain stable strings for both formats during the
|
|
370
|
-
cutover, and session-journey cursors use timestamp, creation time, and ID. The
|
|
371
|
-
compatibility logic remains isolated in `src/component/eventStore.ts` until a
|
|
372
|
-
later release removes the empty legacy table and its read paths.
|
|
628
|
+
Existing deployments that still contain the retired `events` table must not
|
|
629
|
+
upgrade directly to this cutover. Follow the exact intermediate release and
|
|
630
|
+
verification sequence in [`docs/upgrading.md`](docs/upgrading.md). Fresh
|
|
631
|
+
deployments need no legacy-event migration.
|
|
373
632
|
|
|
374
633
|
## Privacy contract
|
|
375
634
|
|
|
@@ -377,8 +636,10 @@ Amage Rastro's default tracker:
|
|
|
377
636
|
|
|
378
637
|
- sets no cookies;
|
|
379
638
|
- stores a pseudonymous session ID and referral slug in `sessionStorage`;
|
|
380
|
-
-
|
|
381
|
-
-
|
|
639
|
+
- never persists raw visitor IP addresses in component tables;
|
|
640
|
+
- processes an IP only when an edge supplies geography or an operator enables a
|
|
641
|
+
GeoIP provider, which receives that IP under its own privacy terms;
|
|
642
|
+
- stores only country, city, and coordinates rounded to one decimal place;
|
|
382
643
|
- does not record the DOM, screenshots, keystrokes, form values, or arbitrary
|
|
383
644
|
page text;
|
|
384
645
|
- captures only bounded paths, explicit properties, button/anchor targets, and
|
|
@@ -412,14 +673,15 @@ npm test
|
|
|
412
673
|
npm run typecheck
|
|
413
674
|
npm run lint
|
|
414
675
|
npm run build
|
|
415
|
-
|
|
676
|
+
npm exec vite -- build example
|
|
416
677
|
```
|
|
417
678
|
|
|
418
679
|
The test suite covers origin enforcement, event/privacy sanitization,
|
|
419
680
|
idempotency, reusable session/site rate limits, bounded live expiry chains,
|
|
420
681
|
folded aggregate writes, goals, funnels, trusted payments, currency safety,
|
|
421
682
|
affiliate commission, cross-site journeys, keyset pagination beyond 1,000 rows,
|
|
422
|
-
retention,
|
|
683
|
+
retention, request metadata and GeoIP, exact byte boundaries, HTTP routes, React
|
|
684
|
+
integration, and tracker size/behavior.
|
|
423
685
|
|
|
424
686
|
## Synapse demo
|
|
425
687
|
|