@columenlabs/lumen-ops 0.1.80 → 0.1.82

package/README.md

@@ -1,6 +1,6 @@
 # @columenlabs/lumen-ops
 
-Lumen's ops parts for Cloudflare Workers: every data query your app runs becomes one Analytics Engine point, so you can find slow and heavy queries and fix them. The meter wraps your bindings once, so new code is metered without opting in. It writes only each query's shape, never a value. It also meters tool calls (`/tools`) and sends signed webhooks (`/webhooks`).
+Lumen's ops parts for Cloudflare Workers: every request your app answers, every event and every data query becomes one Analytics Engine point, so you can find errors, mismatches and slow or heavy queries and fix them. The meter wraps your bindings once, so new code is metered without opting in. It writes only each query's shape, never a value. It also meters tool calls (`/tools`) and sends signed webhooks (`/webhooks`).
 
 - Docs: https://lumen.columenlabs.com/docs/
 - For agents: https://lumen.columenlabs.com/llms.txt
@@ -57,7 +57,7 @@ Another engine (R2 SQL, an HTTP database) calls `meterQuery({ engine, sql, metho
 
 ## What a point holds
 
-One point per statement (a D1 batch writes one per statement in it). The column order is the contract for reading the dataset back:
+One point per kind of statement per turn: the same statement run many times in one go (a loop of inserts) is one point whose numbers are totals, with `statements` saying how many ran. Divide by it for a mean. The column order is the contract for reading the dataset back:
 
 | Column | Holds |
 |---|---|
@@ -65,18 +65,20 @@ One point per statement (a D1 batch writes one per statement in it). The column
 | `blob1` … `blob5` | Engine, app, kind (`fetch`, `cron`, `queue`, `alarm`, `do`), route (ids folded to `:id`), tool |
 | `blob6` … `blob8` | Op (`select`, `insert`, `update`, `delete`, `ddl`, `pragma`, `other`), fingerprint, the query's shape (up to 1 KB) |
 | `blob9`, `blob10` | `ok` or `error`, method (`all`, `first`, `run`, `raw`, `batch`, `query`, `insert`, `exec`) |
-| `double1` … `double8` | Wall ms, the engine's own ms, rows read, bytes read, rows returned, rows written, response bytes, HTTP status |
+| `double1` … `double10` | Wall ms, the engine's own ms, rows read, bytes read, rows returned, rows written, response bytes (each a total), HTTP status, statements, slowest ms |
 
 The shape has every string and number literal as `?`, `IN` lists folded to `(?...)`, comments and ClickHouse's `FORMAT` dropped, and an insert's rows cut off. The fingerprint is a hash of the shape, one per kind of query.
 
 Read it back with the Analytics Engine SQL API, for example the slowest shapes of the last day:
 
 ```sql
-SELECT blob7 AS fingerprint, any(blob8) AS shape, sum(_sample_interval) AS calls,
-       quantileWeighted(0.95)(double1, _sample_interval) AS p95_ms
+SELECT blob7 AS fingerprint, any(blob8) AS shape,
+       sum(IF(double9 > 0, double9, 1) * _sample_interval) AS calls,
+       sum(double1 * _sample_interval) / sum(IF(double9 > 0, double9, 1) * _sample_interval) AS mean_ms,
+       max(double10) AS slowest_ms
 FROM myapp_queries
 WHERE timestamp > NOW() - INTERVAL '1' DAY
-GROUP BY fingerprint ORDER BY p95_ms DESC LIMIT 20
+GROUP BY fingerprint ORDER BY mean_ms DESC LIMIT 20
 ```
 
 ## Exports
@@ -88,10 +90,83 @@ GROUP BY fingerprint ORDER BY p95_ms DESC LIMIT 20
 | `withQueryCaller(caller, fn)` | Charges every query inside `fn` to `caller` |
 | `meteredD1(db)` | D1, metered: `prepare` and `batch`, with D1's own rows read and written and duration |
 | `meteredClickHouse(binding)` | A ClickHouse HTTP service binding, metered when the answer has been read: wall time to the last byte, ClickHouse's own statistics |
-| `meteredSql(sql, caller)` | A Durable Object's `storage.sql`, metered |
+| `meteredSql(sql, caller?)` | A Durable Object's `storage.sql`, metered; its cursor keeps `toArray`, `one`, `raw`, `next`, iteration and `columnNames` |
 | `meterQuery(point)` | One point by hand, for any other engine |
 | `queryShape(sql)` | The shape, fingerprint and op of a query |
 | `foldRoute(path)` | A path with id-like segments as `:id` |
+| `setQueryRowCap(n)` | The most rows one statement should return; one that returns more is marked `capped` in `blob11` |
+
+## Requests
+
+`@columenlabs/lumen-ops/hono` meters every request a Hono app answers: mount it once, first, and every route the app has or adds later writes one point with its route template, status, outcome code and time. It names the request in scope, so events and queries inside carry its id.
+
+```ts
+import { requestId } from 'hono/request-id';
+import { requestMeter } from '@columenlabs/lumen-ops/hono';
+import { setRequestSink } from '@columenlabs/lumen-ops/requests';
+
+app.use('*', async (c, next) => {
+  setRequestSink(c.env.REQUESTS, { log: true }); // also one JSON line per point in Workers Logs
+  await next();
+});
+app.use('*', requestId());
+app.use('*', requestMeter({
+  app: 'myapp',
+  env: (c) => c.env.DEPLOY_ENV,
+  describe: (c) => ({ surface: 'api', subject: c.get('accountId'), callerKind: 'key' }),
+}));
+```
+
+| Column | Holds |
+|---|---|
+| `index1` | Subject: an opaque id the app makes for who called, or `anon` |
+| `blob1` … `blob7` | App, surface, method, route template, outcome (`ok`, an `error` code from a JSON answer, `exception`, `http_<status>`), caller kind, caller id |
+| `blob8` … `blob13` | Region, env, version, colo, country, request id |
+| `double1` … `double6` | Status, ms, bytes in, bytes out, upstream ms, subrequests |
+
+A path no route matched is written as `*`, so a scanner's junk never lands. Without Hono, wrap the handler in `withRequest(scope, route, fn)` and call `meterRequest(point)` yourself.
+
+## Events
+
+`@columenlabs/lumen-ops/events` writes what an app does that isn't a request it answers: a call to another service, a webhook in or out, a sync's start and end, a sign-in. With the request points they show whether what came in was acted on.
+
+```ts
+import { meterEvent, setEventSink } from '@columenlabs/lumen-ops/events';
+
+setEventSink(env.EVENTS);
+meterEvent({ event: 'bank.call', code: '/accounts/get', ok: res.ok, ms, status: res.status, upstream: true });
+meterEvent({ event: 'sync.done', ok: true, subject: accountId, ref: connectionId, counts: [added, modified, removed] });
+```
+
+| Column | Holds |
+|---|---|
+| `index1` | Subject, or `anon` |
+| `blob1` … `blob10` | App, event, code, outcome, ref, trigger, request id, upstream id, env, version |
+| `double1` … `double6` | ms, status, count1, count2, count3, attempt |
+
+`upstream: true` adds the call's time to the request's `upstream ms`. In a Durable Object no request is in scope: give `setEventSink(binding, { app, env, version })` the defaults, and pass a `requestId` the caller sent if you have one.
+
+## Browser telemetry
+
+`useLumenTelemetry` in `@columenlabs/lumen-ui` sends screens opened, uncaught errors (class and a hash of the stack, never the message), failed requests and Web Vitals in batches to `/api/t`. The Worker checks and writes them:
+
+```ts
+import { meterClientBatch, setClientSink } from '@columenlabs/lumen-ops/client';
+
+setClientSink(env.CLIENT);
+app.post('/api/t', async (c) => {
+  meterClientBatch(await c.req.json().catch(() => null), { app: 'console', subject: c.get('accountId') });
+  return c.body(null, 204);
+});
+```
+
+| Column | Holds |
+|---|---|
+| `index1` | Subject, or `anon` |
+| `blob1` … `blob9` | App, kind (`view`, `error`, `fetch_fail`, `vital`), route template, name, stack fingerprint, request id, version, device, env |
+| `double1`, `double2` | Value (ms or score), status |
+
+A batch that fails the schema is dropped whole; only codes, templates and numbers pass.
 
 ## Tool calls
 

package/dist/client.d.ts

@@ -0,0 +1,41 @@
+/**
+ * The Worker's side of the browser's telemetry (`useLumenTelemetry` in `@columenlabs/lumen-ui`):
+ * checks a batch against a strict schema, so only enums, templates, codes and numbers pass, and
+ * writes one Analytics Engine point per entry. A batch that doesn't pass is dropped whole.
+ */
+import { z } from 'zod';
+import type { PointSink } from './points.js';
+export declare const clientBatch: z.ZodObject<{
+    app: z.ZodString;
+    version: z.ZodOptional<z.ZodString>;
+    points: z.ZodArray<z.ZodObject<{
+        kind: z.ZodEnum<{
+            error: "error";
+            fetch_fail: "fetch_fail";
+            view: "view";
+            vital: "vital";
+        }>;
+        route: z.ZodString;
+        name: z.ZodOptional<z.ZodString>;
+        fingerprint: z.ZodOptional<z.ZodString>;
+        requestId: z.ZodOptional<z.ZodString>;
+        value: z.ZodOptional<z.ZodNumber>;
+        status: z.ZodOptional<z.ZodNumber>;
+        device: z.ZodOptional<z.ZodEnum<{
+            desktop: "desktop";
+            phone: "phone";
+        }>>;
+    }, z.core.$strip>>;
+}, z.core.$strip>;
+export type ClientBatch = z.infer<typeof clientBatch>;
+/** The entry point sets the dataset once per isolate. */
+export declare function setClientSink(s: PointSink | undefined): void;
+/**
+ * Writes a checked batch; says how many points it wrote (0 when the batch didn't pass). Column
+ * order is the contract: index1 subject (or `anon`); blobs app, kind, route, name, fingerprint,
+ * request id, version, device, env; doubles value, status.
+ */
+export declare function meterClientBatch(body: unknown, opts?: {
+    subject?: string;
+    app?: string;
+}): number;

package/dist/client.js

@@ -0,0 +1,70 @@
+/**
+ * The Worker's side of the browser's telemetry (`useLumenTelemetry` in `@columenlabs/lumen-ui`):
+ * checks a batch against a strict schema, so only enums, templates, codes and numbers pass, and
+ * writes one Analytics Engine point per entry. A batch that doesn't pass is dropped whole.
+ */
+import { z } from 'zod';
+import { cap, num } from './fields.js';
+import { currentRequest } from './requests.js';
+const code = z.string().regex(/^[A-Za-z0-9_.:/$-]{1,64}$/);
+const route = z.string().regex(/^[A-Za-z0-9_.:/$-]{0,256}$/);
+const point = z.object({
+    kind: z.enum(['view', 'error', 'fetch_fail', 'vital']),
+    route,
+    name: code.optional(),
+    fingerprint: z
+        .string()
+        .regex(/^[0-9a-f]{1,16}$/)
+        .optional(),
+    requestId: code.optional(),
+    value: z.number().finite().optional(),
+    status: z.number().int().min(0).max(999).optional(),
+    device: z.enum(['phone', 'desktop']).optional(),
+});
+export const clientBatch = z.object({
+    app: code,
+    version: code.optional(),
+    points: z.array(point).max(50),
+});
+let sink;
+/** The entry point sets the dataset once per isolate. */
+export function setClientSink(s) {
+    sink = s;
+}
+/**
+ * Writes a checked batch; says how many points it wrote (0 when the batch didn't pass). Column
+ * order is the contract: index1 subject (or `anon`); blobs app, kind, route, name, fingerprint,
+ * request id, version, device, env; doubles value, status.
+ */
+export function meterClientBatch(body, opts = {}) {
+    const parsed = clientBatch.safeParse(body);
+    if (!parsed.success || !sink)
+        return 0;
+    const b = parsed.data;
+    const env = currentRequest()?.env;
+    // The app named by the server wins: a page can't write as another app.
+    const app = opts.app ?? b.app;
+    for (const p of b.points) {
+        try {
+            sink.writeDataPoint({
+                indexes: [cap(opts.subject || 'anon')],
+                blobs: [
+                    cap(app),
+                    p.kind,
+                    p.route,
+                    p.name ?? '',
+                    p.fingerprint ?? '',
+                    p.requestId ?? '',
+                    b.version ?? '',
+                    p.device ?? '',
+                    cap(env),
+                ],
+                doubles: [num(p.value), num(p.status)],
+            });
+        }
+        catch (err) {
+            console.error('meterClientBatch', err);
+        }
+    }
+    return b.points.length;
+}

package/dist/events.d.ts

@@ -0,0 +1,53 @@
+/**
+ * The event meter: the things an app does that are not a request it answers, one Analytics Engine
+ * point each. A call to another service (`bank.call`, `billing.call`), a webhook in or out, a
+ * sync's start and end, a sign-in. With the request meter they let anyone check that what came in
+ * was acted on: a webhook received and a sync finished, an event paid for and a plan changed.
+ *
+ * A point holds an event name, a code (an upstream's error code or endpoint), counts and ids the
+ * app makes. Never the payload, a token, or a person's name or email.
+ */
+import type { PointSink } from './points.js';
+export interface EventPoint {
+    /** What happened, dotted: `bank.call`, `bank.webhook`, `sync.done`, `webhook.out`, `auth.sign_in`. */
+    event: string;
+    /** The detail as a code: an endpoint path, an upstream's error code, a webhook's type. */
+    code?: string;
+    ok: boolean;
+    /** An opaque id the app makes for whom it happened to, or empty. */
+    subject?: string;
+    /** The thing it happened to, an id the app makes (a connection's, an endpoint's). */
+    ref?: string;
+    /** What started it: schedule, webhook, link, manual, or your own. */
+    trigger?: string;
+    /** The other service's own request id, for its support. */
+    upstreamId?: string;
+    ms?: number;
+    status?: number;
+    /** Up to three counts, named by the event (a sync: added, modified, removed). */
+    counts?: [number?, number?, number?];
+    attempt?: number;
+    /** `true` when the event was a call to another service: its time adds to the request's. */
+    upstream?: boolean;
+    /** Defaults to the request in scope; pass these from a Durable Object or a job. */
+    app?: string;
+    requestId?: string;
+    env?: string;
+    version?: string;
+}
+export interface EventDefaults {
+    app?: string;
+    env?: string;
+    version?: string;
+}
+/**
+ * The entry point (and each Durable Object) sets the dataset once per isolate, with the app, env
+ * and version its points carry when no request is in scope (an object's alarm, a job).
+ */
+export declare function setEventSink(s: PointSink | undefined, d?: EventDefaults): void;
+/**
+ * Column order is the contract with whoever reads the dataset back: index1 subject (or `anon`);
+ * blobs app, event, code, outcome, ref, trigger, request id, upstream id, env, version; doubles
+ * ms, status, count1, count2, count3, attempt.
+ */
+export declare function meterEvent(p: EventPoint): void;

package/dist/events.js

@@ -0,0 +1,66 @@
+import { cap, codeOf, num } from './fields.js';
+import { addUpstream, currentRequest, logLine } from './requests.js';
+let sink;
+let defaults = {};
+/**
+ * The entry point (and each Durable Object) sets the dataset once per isolate, with the app, env
+ * and version its points carry when no request is in scope (an object's alarm, a job).
+ */
+export function setEventSink(s, d = {}) {
+    sink = s;
+    defaults = d;
+}
+/**
+ * Column order is the contract with whoever reads the dataset back: index1 subject (or `anon`);
+ * blobs app, event, code, outcome, ref, trigger, request id, upstream id, env, version; doubles
+ * ms, status, count1, count2, count3, attempt.
+ */
+export function meterEvent(p) {
+    try {
+        const r = currentRequest();
+        if (p.upstream && p.ms !== undefined)
+            addUpstream(p.ms);
+        const fields = {
+            app: p.app ?? r?.app ?? defaults.app,
+            requestId: p.requestId ?? r?.requestId,
+            env: p.env ?? r?.env ?? defaults.env,
+            version: p.version ?? r?.version ?? defaults.version,
+        };
+        const code = codeOf(p.code);
+        sink?.writeDataPoint({
+            indexes: [cap(p.subject || 'anon')],
+            blobs: [
+                cap(fields.app),
+                codeOf(p.event, 'other'),
+                code,
+                p.ok ? 'ok' : 'error',
+                cap(p.ref),
+                cap(p.trigger),
+                cap(fields.requestId),
+                cap(p.upstreamId),
+                cap(fields.env),
+                cap(fields.version),
+            ],
+            doubles: [
+                num(p.ms),
+                num(p.status),
+                num(p.counts?.[0]),
+                num(p.counts?.[1]),
+                num(p.counts?.[2]),
+                num(p.attempt),
+            ],
+        });
+        logLine('event', {
+            id: fields.requestId,
+            event: p.event,
+            code,
+            ok: p.ok,
+            ref: p.ref,
+            ms: p.ms === undefined ? undefined : Math.round(p.ms),
+            status: p.status,
+        });
+    }
+    catch (err) {
+        console.error('meterEvent', err);
+    }
+}

package/dist/fields.d.ts

@@ -0,0 +1,4 @@
+export declare const cap: (s: string | undefined, n?: number) => string;
+export declare const num: (v: unknown) => number;
+/** A code is a short lower-case word with `_`, `.` or `-`, so free text never passes as one. */
+export declare const codeOf: (v: unknown, fallback?: string) => string;

package/dist/fields.js

@@ -0,0 +1,8 @@
+// Field helpers the meters share; not exported from the package.
+export const cap = (s, n = 96) => {
+    const v = s ?? '';
+    return v.length > n ? v.slice(0, n) : v;
+};
+export const num = (v) => (typeof v === 'number' && Number.isFinite(v) ? v : 0);
+/** A code is a short lower-case word with `_`, `.` or `-`, so free text never passes as one. */
+export const codeOf = (v, fallback = '') => typeof v === 'string' && /^[A-Za-z0-9_.:/-]{1,64}$/.test(v) ? v : fallback;

package/dist/hono.d.ts

@@ -0,0 +1,19 @@
+/**
+ * The request meter as Hono middleware: mount it once, first, and every route the app has or adds
+ * later is metered. It names the request (`withRequest`), so events and queries inside carry its
+ * id, then writes one request point after the answer. Put Hono's own `requestId()` before it to
+ * send the id back as `X-Request-Id`.
+ */
+import type { Context, MiddlewareHandler } from 'hono';
+import { type RequestPoint } from './requests.js';
+export interface RequestMeterOptions {
+    /** The app, or the part of it, answering. */
+    app: string;
+    env?: (c: Context) => string | undefined;
+    version?: (c: Context) => string | undefined;
+    /** Which part was asked and who called, read after the answer (auth has run by then). */
+    describe?: (c: Context) => Partial<Pick<RequestPoint, 'surface' | 'subject' | 'callerKind' | 'callerId' | 'region'>>;
+    /** The outcome code of an answer; by default the `error` field of a small JSON error answer. */
+    outcome?: (c: Context) => Promise<string | undefined> | string | undefined;
+}
+export declare function requestMeter(opts: RequestMeterOptions): MiddlewareHandler;

package/dist/hono.js

@@ -0,0 +1,58 @@
+import { routePath } from 'hono/route';
+import { codeOf } from './fields.js';
+import { foldRoute } from './queries.js';
+import { meterRequest, withRequest } from './requests.js';
+const newId = () => crypto.randomUUID().replace(/-/g, '').slice(0, 16);
+const size = (h) => (h ? Number(h) || 0 : 0);
+/** The `error` field of a JSON error answer under 4 KB, if it reads as a code. */
+async function errorCode(c) {
+    const res = c.res;
+    if (res.status < 400 || !res.headers.get('content-type')?.includes('json'))
+        return;
+    if (size(res.headers.get('content-length')) > 4096)
+        return;
+    try {
+        const body = (await res.clone().json());
+        return codeOf(body?.error) || undefined;
+    }
+    catch {
+        return;
+    }
+}
+export function requestMeter(opts) {
+    return async (c, next) => {
+        const t0 = performance.now();
+        const requestId = codeOf(c.var.requestId) || newId();
+        const scope = { app: opts.app, requestId, env: opts.env?.(c), version: opts.version?.(c) };
+        await withRequest(scope, foldRoute(c.req.path), async () => {
+            await next();
+            try {
+                // The template of the deepest route that ran (Hono leaves its index set once the chain
+                // unwinds); `-1` would name the last route that merely matched, a catch-all say. A path
+                // only this middleware matched is `/*`, so junk never lands.
+                const route = routePath(c) || '*';
+                const outcome = c.error
+                    ? 'exception'
+                    : await (opts.outcome ? opts.outcome(c) : errorCode(c));
+                const cf = c.req.raw.cf;
+                meterRequest({
+                    surface: 'other',
+                    ...opts.describe?.(c),
+                    method: c.req.method,
+                    route,
+                    status: c.res.status,
+                    ms: performance.now() - t0,
+                    outcome,
+                    colo: cf?.colo,
+                    country: cf?.country,
+                    bytesIn: size(c.req.header('content-length')),
+                    bytesOut: size(c.res.headers.get('content-length')),
+                    scope,
+                });
+            }
+            catch (err) {
+                console.error('requestMeter', err);
+            }
+        });
+    };
+}

package/dist/index.d.ts

@@ -1,4 +1,6 @@
+export * from './events.js';
 export * from './points.js';
 export * from './queries.js';
+export * from './requests.js';
 export * from './tools.js';
 export * from './webhooks.js';

package/dist/index.js

@@ -1,4 +1,6 @@
+export * from './events.js';
 export * from './points.js';
 export * from './queries.js';
+export * from './requests.js';
 export * from './tools.js';
 export * from './webhooks.js';

package/dist/queries.d.ts

@@ -34,6 +34,11 @@ export interface QueryPoint {
 }
 /** The entry point sets the dataset once per isolate. */
 export declare function setQuerySink(s: PointSink | undefined): void;
+/**
+ * The most rows one statement should return (an app's page size cap); a statement that returns
+ * more is marked `capped`, so a query that dodged paging shows up. 0 turns the check off.
+ */
+export declare function setQueryRowCap(n: number | undefined): void;
 /** How to name the request in scope when no job is (the app's request telemetry). */
 export declare function setQueryCallerOf(f: (() => QueryCaller | undefined) | undefined): void;
 /** Runs `fn` with `caller` as the caller of every query inside (cron runs, queue batches, alarms). */
@@ -52,8 +57,13 @@ export interface QueryShape {
 export declare function queryShape(sql: string): QueryShape;
 /**
  * Column order is the contract with whoever reads the dataset back: index1 engine; blobs
- * engine, app, kind, route, tool, op, fingerprint, sql shape, outcome, method; doubles wall ms,
- * engine ms, rows read, bytes read, rows out, rows written, bytes out, status.
+ * engine, app, kind, route, tool, op, fingerprint, sql shape, outcome, method, cap (`capped` when
+ * it returned more rows than `setQueryRowCap` allows); doubles wall ms, engine ms, rows read,
+ * bytes read, rows out, rows written, bytes out, status, statements, slowest ms.
+ *
+ * The same statement run many times in one turn (a loop of inserts) is one point: doubles 1 to 7
+ * are its totals, `statements` how many ran and `slowest ms` the longest, so a sync that writes
+ * thousands of rows stays a handful of points. Divide a total by `statements` for the mean.
  */
 export declare function meterQuery(p: QueryPoint): void;
 /** The D1 binding, metered: one point per statement, a batch's each with its own meta. */
@@ -68,8 +78,9 @@ export declare function meteredClickHouse<T extends {
     fetch: Fetch;
 }>(binding: T | undefined): T | undefined;
 /**
- * A Durable Object's `storage.sql`, metered for `caller`. Each statement is read to the end at
- * `exec`, so its rows read and written are known; callers read rows only through `toArray`.
+ * A Durable Object's `storage.sql`, metered. Each statement is read to the end at `exec`, so its
+ * rows read and written are known; the cursor it returns reads like the native one. Its queries
+ * are charged to the job in scope (`withQueryCaller`, an alarm say), else to `caller`.
  */
-export declare function meteredSql<T>(sql: T, caller: QueryCaller): T;
+export declare function meteredSql<T>(sql: T, caller?: QueryCaller): T;
 export {};

package/dist/queries.js

@@ -14,12 +14,20 @@
  */
 import { AsyncLocalStorage } from 'node:async_hooks';
 let sink;
+let rowCap = 0;
 let callerOf;
 const scope = new AsyncLocalStorage();
 /** The entry point sets the dataset once per isolate. */
 export function setQuerySink(s) {
     sink = s;
 }
+/**
+ * The most rows one statement should return (an app's page size cap); a statement that returns
+ * more is marked `capped`, so a query that dodged paging shows up. 0 turns the check off.
+ */
+export function setQueryRowCap(n) {
+    rowCap = n && n > 0 ? n : 0;
+}
 /** How to name the request in scope when no job is (the app's request telemetry). */
 export function setQueryCallerOf(f) {
     callerOf = f;
@@ -107,10 +115,32 @@ export function queryShape(sql) {
 }
 const cap = (s, n = 96) => (s.length > n ? s.slice(0, n) : s);
 const num = (v) => (typeof v === 'number' && Number.isFinite(v) ? v : Number(v) || 0);
+const pending = new Map();
+function flush() {
+    const batches = [...pending.values()];
+    pending.clear();
+    for (const b of batches) {
+        try {
+            sink?.writeDataPoint({
+                indexes: [b.index],
+                blobs: b.blobs,
+                doubles: [...b.sums, b.status, b.n, b.max],
+            });
+        }
+        catch (err) {
+            console.error('meterQuery', err);
+        }
+    }
+}
 /**
  * Column order is the contract with whoever reads the dataset back: index1 engine; blobs
- * engine, app, kind, route, tool, op, fingerprint, sql shape, outcome, method; doubles wall ms,
- * engine ms, rows read, bytes read, rows out, rows written, bytes out, status.
+ * engine, app, kind, route, tool, op, fingerprint, sql shape, outcome, method, cap (`capped` when
+ * it returned more rows than `setQueryRowCap` allows); doubles wall ms, engine ms, rows read,
+ * bytes read, rows out, rows written, bytes out, status, statements, slowest ms.
+ *
+ * The same statement run many times in one turn (a loop of inserts) is one point: doubles 1 to 7
+ * are its totals, `statements` how many ran and `slowest ms` the longest, so a sync that writes
+ * thousands of rows stays a handful of points. Divide a total by `statements` for the mean.
  */
 export function meterQuery(p) {
     if (!sink)
@@ -118,30 +148,48 @@ export function meterQuery(p) {
     try {
         const shape = queryShape(p.sql);
         const c = p.caller ?? currentCaller();
-        sink.writeDataPoint({
-            indexes: [p.engine],
-            blobs: [
-                p.engine,
-                cap(c.app),
-                cap(c.kind),
-                cap(c.route),
-                cap(c.tool ?? ''),
-                shape.op,
-                shape.fingerprint,
-                shape.text,
-                p.ok ? 'ok' : 'error',
-                p.method,
-            ],
-            doubles: [
-                p.ms,
-                num(p.engineMs),
-                num(p.rowsRead),
-                num(p.bytesRead),
-                num(p.rowsOut),
-                num(p.rowsWritten),
-                num(p.bytesOut),
-                num(p.status),
-            ],
+        const blobs = [
+            p.engine,
+            cap(c.app),
+            cap(c.kind),
+            cap(c.route),
+            cap(c.tool ?? ''),
+            shape.op,
+            shape.fingerprint,
+            shape.text,
+            p.ok ? 'ok' : 'error',
+            p.method,
+            rowCap && num(p.rowsOut) > rowCap ? 'capped' : '',
+        ];
+        const values = [
+            num(p.ms),
+            num(p.engineMs),
+            num(p.rowsRead),
+            num(p.bytesRead),
+            num(p.rowsOut),
+            num(p.rowsWritten),
+            num(p.bytesOut),
+        ];
+        const key = blobs.filter((_, i) => i !== 7).join('\u0000');
+        const b = pending.get(key);
+        if (b) {
+            values.forEach((v, i) => {
+                b.sums[i] = (b.sums[i] ?? 0) + v;
+            });
+            b.n += 1;
+            b.max = Math.max(b.max, num(p.ms));
+            b.status = num(p.status) || b.status;
+            return;
+        }
+        if (pending.size === 0)
+            queueMicrotask(flush);
+        pending.set(key, {
+            blobs,
+            index: p.engine,
+            sums: values,
+            status: num(p.status),
+            n: 1,
+            max: num(p.ms),
         });
     }
     catch (err) {
@@ -363,11 +411,50 @@ export function meteredClickHouse(binding) {
     });
 }
 /**
- * A Durable Object's `storage.sql`, metered for `caller`. Each statement is read to the end at
- * `exec`, so its rows read and written are known; callers read rows only through `toArray`.
+ * The rows of one statement, read to the end, behind the same cursor a Durable Object's
+ * `storage.sql` hands back: `toArray`, `one`, `raw`, `next`, iteration and `columnNames` all work,
+ * each reading on from where the last stopped.
+ */
+function readCursor(raws, columnNames, rowsRead, rowsWritten) {
+    let at = 0;
+    const row = (r) => Object.fromEntries(columnNames.map((c, i) => [c, r[i]]));
+    const rest = () => {
+        const out = raws.slice(at);
+        at = raws.length;
+        return out;
+    };
+    const cursor = {
+        columnNames,
+        rowsRead,
+        rowsWritten,
+        toArray: () => rest().map(row),
+        one: () => {
+            const left = rest();
+            if (left.length !== 1)
+                throw new Error(`Expected exactly one result from SQL query, but got ${left.length ? 'more than one' : 'no'} results.`);
+            return row(left[0]);
+        },
+        raw: function* () {
+            while (at < raws.length)
+                yield raws[at++];
+        },
+        next: () => at < raws.length
+            ? { done: false, value: row(raws[at++]) }
+            : { done: true, value: undefined },
+        [Symbol.iterator]() {
+            return cursor;
+        },
+    };
+    return cursor;
+}
+/**
+ * A Durable Object's `storage.sql`, metered. Each statement is read to the end at `exec`, so its
+ * rows read and written are known; the cursor it returns reads like the native one. Its queries
+ * are charged to the job in scope (`withQueryCaller`, an alarm say), else to `caller`.
  */
 export function meteredSql(sql, caller) {
     const native = sql;
+    const callerNow = () => scope.getStore() ?? caller ?? currentCaller();
     return new Proxy(native, {
         get(target, prop, receiver) {
             if (prop === 'exec')
@@ -375,7 +462,7 @@ export function meteredSql(sql, caller) {
                     const t0 = performance.now();
                     try {
                         const cursor = target.exec(query, ...bindings);
-                        const rows = cursor.toArray();
+                        const raws = Array.from(cursor.raw());
                         meterQuery({
                             engine: 'do_sqlite',
                             sql: query,
@@ -384,14 +471,10 @@ export function meteredSql(sql, caller) {
                             ms: performance.now() - t0,
                             rowsRead: cursor.rowsRead,
                             rowsWritten: cursor.rowsWritten,
-                            rowsOut: rows.length,
-                            caller,
+                            rowsOut: raws.length,
+                            caller: callerNow(),
                         });
-                        return {
-                            toArray: () => rows,
-                            rowsRead: cursor.rowsRead,
-                            rowsWritten: cursor.rowsWritten,
-                        };
+                        return readCursor(raws, cursor.columnNames, cursor.rowsRead, cursor.rowsWritten);
                     }
                     catch (err) {
                         meterQuery({
@@ -400,7 +483,7 @@ export function meteredSql(sql, caller) {
                             method: 'exec',
                             ok: false,
                             ms: performance.now() - t0,
-                            caller,
+                            caller: callerNow(),
                         });
                         throw err;
                     }

package/dist/requests.d.ts

@@ -0,0 +1,56 @@
+import type { PointSink } from './points.js';
+export interface RequestScope {
+    /** The app, or the part of it, answering. */
+    app: string;
+    /** A short id the app makes per request, sent back as `X-Request-Id`. */
+    requestId: string;
+    /** production, preview, or your own. */
+    env?: string;
+    /** The deploy's version id. */
+    version?: string;
+}
+export interface RequestPoint {
+    /** Which part of the app was asked: console, admin, api, mcp, webhook, site, or your own. */
+    surface: string;
+    method: string;
+    /** The route template (`/v1/accounts/:id`), never the raw path. */
+    route: string;
+    status: number;
+    ms: number;
+    /** `ok`, or a short code (`rate_limited`, `invalid_signature`, `exception`), never a message. */
+    outcome?: string;
+    /** Who called: an opaque id the app makes (a person's or account's id), or empty. */
+    subject?: string;
+    /** How they called: session, key, oauth, admin, webhook, anon, or your own. */
+    callerKind?: string;
+    /** The key's or OAuth client's id, never the key itself. */
+    callerId?: string;
+    region?: string;
+    colo?: string;
+    country?: string;
+    bytesIn?: number;
+    bytesOut?: number;
+    /** Time spent waiting on other services in this request, ms. */
+    upstreamMs?: number;
+    subrequests?: number;
+    /** Defaults to the request in scope. */
+    scope?: RequestScope;
+}
+/** The entry point sets the dataset once per isolate; `log` also prints one JSON line a point. */
+export declare function setRequestSink(s: PointSink | undefined, opts?: {
+    log?: boolean;
+}): void;
+/** Runs `fn` as request `r`: points written inside carry its id, and queries its route. */
+export declare function withRequest<T>(r: RequestScope, route: string, fn: () => T): T;
+/** The request in scope, if any. */
+export declare function currentRequest(): RequestScope | undefined;
+/** Adds a call to another service to the request in scope (the event meter does this itself). */
+export declare function addUpstream(ms: number): void;
+/** Prints the point as one JSON line for the platform's logs, when the sink was set with `log`. */
+export declare function logLine(kind: string, fields: Record<string, unknown>): void;
+/**
+ * Column order is the contract with whoever reads the dataset back: index1 subject (or `anon`);
+ * blobs app, surface, method, route, outcome, caller kind, caller id, region, env, version, colo,
+ * country, request id; doubles status, ms, bytes in, bytes out, upstream ms, subrequests.
+ */
+export declare function meterRequest(p: RequestPoint): void;

package/dist/requests.js

@@ -0,0 +1,102 @@
+/**
+ * The request meter: every request a Worker answers is one Analytics Engine point, with its route
+ * template, status, outcome code, who called and how long it took, so errors, slow routes and a bad
+ * deploy show up without any code in the routes themselves.
+ *
+ * It also holds the request in scope (`currentRequest`), so the event meter and the query meter
+ * charge their points to it and carry the same request id.
+ *
+ * A point holds enums, templates, ids the app makes and numbers only: never a raw path, a query
+ * string, a header, a body or a person's name, email or address.
+ */
+import { AsyncLocalStorage } from 'node:async_hooks';
+import { cap, codeOf, num } from './fields.js';
+import { withQueryCaller } from './queries.js';
+let sink;
+let logLines = false;
+const scope = new AsyncLocalStorage();
+/** The entry point sets the dataset once per isolate; `log` also prints one JSON line a point. */
+export function setRequestSink(s, opts = {}) {
+    sink = s;
+    logLines = Boolean(opts.log);
+}
+/** Runs `fn` as request `r`: points written inside carry its id, and queries its route. */
+export function withRequest(r, route, fn) {
+    return scope.run({ ...r, upstreamMs: 0, subrequests: 0 }, () => withQueryCaller({ app: r.app, kind: 'fetch', route }, fn));
+}
+/** The request in scope, if any. */
+export function currentRequest() {
+    return scope.getStore();
+}
+/** Adds a call to another service to the request in scope (the event meter does this itself). */
+export function addUpstream(ms) {
+    const s = scope.getStore();
+    if (!s)
+        return;
+    s.upstreamMs += ms;
+    s.subrequests += 1;
+}
+/** Prints the point as one JSON line for the platform's logs, when the sink was set with `log`. */
+export function logLine(kind, fields) {
+    if (!logLines)
+        return;
+    try {
+        console.log(JSON.stringify({ lumen: kind, ...fields }));
+    }
+    catch { }
+}
+/**
+ * Column order is the contract with whoever reads the dataset back: index1 subject (or `anon`);
+ * blobs app, surface, method, route, outcome, caller kind, caller id, region, env, version, colo,
+ * country, request id; doubles status, ms, bytes in, bytes out, upstream ms, subrequests.
+ */
+export function meterRequest(p) {
+    if (!sink && !logLines)
+        return;
+    try {
+        const s = p.scope ?? scope.getStore();
+        const live = scope.getStore();
+        const upstreamMs = p.upstreamMs ?? live?.upstreamMs;
+        const subrequests = p.subrequests ?? live?.subrequests;
+        const outcome = codeOf(p.outcome, p.status < 400 ? 'ok' : `http_${p.status}`);
+        sink?.writeDataPoint({
+            indexes: [cap(p.subject || 'anon')],
+            blobs: [
+                cap(s?.app),
+                cap(p.surface),
+                cap(p.method, 8),
+                cap(p.route, 256),
+                outcome,
+                cap(p.callerKind),
+                cap(p.callerId),
+                cap(p.region),
+                cap(s?.env),
+                cap(s?.version),
+                cap(p.colo, 8),
+                cap(p.country, 4),
+                cap(s?.requestId),
+            ],
+            doubles: [
+                num(p.status),
+                num(p.ms),
+                num(p.bytesIn),
+                num(p.bytesOut),
+                num(upstreamMs),
+                num(subrequests),
+            ],
+        });
+        logLine('request', {
+            id: s?.requestId,
+            surface: p.surface,
+            method: p.method,
+            route: p.route,
+            status: p.status,
+            outcome,
+            ms: Math.round(p.ms),
+            caller: p.callerKind,
+        });
+    }
+    catch (err) {
+        console.error('meterRequest', err);
+    }
+}

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "@columenlabs/lumen-ops",
-  "version": "0.1.80",
-  "description": "Lumen's ops parts: every data query and tool call an app runs as one Analytics Engine point (shapes and hashes only), and signed webhooks out.",
+  "version": "0.1.82",
+  "description": "Lumen's ops parts: every request, event, data query and tool call an app runs as one Analytics Engine point (templates, codes, shapes and hashes only), and signed webhooks out.",
   "license": "MIT",
   "type": "module",
   "main": "./dist/index.js",
@@ -11,6 +11,18 @@
       "types": "./dist/index.d.ts",
       "default": "./dist/index.js"
     },
+    "./client": {
+      "types": "./dist/client.d.ts",
+      "default": "./dist/client.js"
+    },
+    "./events": {
+      "types": "./dist/events.d.ts",
+      "default": "./dist/events.js"
+    },
+    "./hono": {
+      "types": "./dist/hono.d.ts",
+      "default": "./dist/hono.js"
+    },
     "./points": {
       "types": "./dist/points.d.ts",
       "default": "./dist/points.js"
@@ -19,6 +31,10 @@
       "types": "./dist/queries.d.ts",
       "default": "./dist/queries.js"
     },
+    "./requests": {
+      "types": "./dist/requests.d.ts",
+      "default": "./dist/requests.js"
+    },
     "./tools": {
       "types": "./dist/tools.d.ts",
       "default": "./dist/tools.js"
@@ -48,6 +64,10 @@
     "registry": "https://registry.npmjs.org/"
   },
   "dependencies": {
-    "standardwebhooks": "1.1.1"
+    "standardwebhooks": "1.1.1",
+    "zod": "^4.3.6"
+  },
+  "peerDependencies": {
+    "hono": ">=4.8.0"
   }
 }