measurr 0.6.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +328 -60
- package/dist/analytics.d.ts +106 -16
- package/dist/dataset.d.ts +65 -4
- package/dist/dialects/clickhouse.d.ts +26 -0
- package/dist/dialects/clickhouse.js +101 -0
- package/dist/handler.d.ts +9 -4
- package/dist/handler.js +54 -8
- package/dist/{index-z5xv3ffk.js → index-8nhqje4s.js} +370 -74
- package/dist/{index-pgccd363.js → index-vjak6aa0.js} +39 -3
- package/dist/index.d.ts +5 -5
- package/dist/index.js +2 -2
- package/dist/metrics.d.ts +152 -35
- package/dist/metrics.js +316 -83
- package/dist/plan.d.ts +10 -1
- package/dist/query.d.ts +170 -11
- package/dist/shapes.d.ts +81 -11
- package/dist/shapes.js +121 -13
- package/dist/testing.d.ts +19 -1
- package/dist/testing.js +45 -5
- package/package.json +11 -4
- package/src/analytics.ts +525 -80
- package/src/dataset.ts +131 -7
- package/src/dialects/clickhouse.ts +147 -0
- package/src/handler.ts +38 -18
- package/src/index.ts +23 -5
- package/src/metrics.ts +646 -144
- package/src/plan.ts +112 -30
- package/src/query.ts +150 -18
- package/src/shapes.ts +293 -29
- package/src/testing.ts +71 -3
- package/dist/index-xf6qpmgy.js +0 -46
package/README.md
CHANGED
|
@@ -7,8 +7,8 @@ returns shaped, typed rows.
|
|
|
7
7
|
|
|
8
8
|
- No SQL text in definitions: datasets are built from typed expression builders, so a
|
|
9
9
|
renamed column, `avg` over a string or `between` on an enum fails typecheck.
|
|
10
|
-
- Database-neutral definitions: dialects plug in through the `Dialect` interface
|
|
11
|
-
|
|
10
|
+
- Database-neutral definitions: dialects plug in through the `Dialect` interface. Postgres
|
|
11
|
+
(`measurr/postgres`) and ClickHouse (`measurr/clickhouse`) ship today.
|
|
12
12
|
- Tenant-scoped by construction: the engine adds the tenant filter to every statement and
|
|
13
13
|
refuses to run without a tenant. A caller can be scoped to one tenant or a list of them;
|
|
14
14
|
platform admins can query every tenant only through an explicit `allTenants` value that no
|
|
@@ -159,8 +159,8 @@ With them:
|
|
|
159
159
|
the field.
|
|
160
160
|
|
|
161
161
|
`AnalyticsResultSchema` is any dataset's result without its keys: rows of `{ key, label }`
|
|
162
|
-
per grouped dimension and a number or null per measure, with totals, `previous?`, `notes
|
|
163
|
-
`truncated`. `analytics.resultSchema(key)` is the same envelope with the dataset's fields, so a
|
|
162
|
+
per grouped dimension and a number or null per measure, with totals, `previous?`, `notes`,
|
|
163
|
+
`truncated` and `fields`. `analytics.resultSchema(key)` is the same envelope with the dataset's fields, so a
|
|
164
164
|
result that passes one passes the other. Use it for a tool's output contract.
|
|
165
165
|
|
|
166
166
|
`analyticsToolGuidance` is a few markdown bullets for a model's instructions: use the tool for
|
|
@@ -176,14 +176,47 @@ ${analyticsToolGuidance}`
|
|
|
176
176
|
```
|
|
177
177
|
|
|
178
178
|
`onQuery(event, ctx)` hears every `query` once it succeeds or fails:
|
|
179
|
-
`{ dataset, durationMs, rows, outcome: 'success' | 'error', cache?, error? }`. `dataset`
|
|
180
|
-
when the input named no known dataset, so free text from a model never becomes a metric
|
|
179
|
+
`{ dataset, metric?, durationMs, rows, outcome: 'success' | 'error', cache?, error? }`. `dataset`
|
|
180
|
+
is null when the input named no known dataset, so free text from a model never becomes a metric
|
|
181
|
+
label. `metric` is the metric the registry ran the query for.
|
|
181
182
|
`cache` is `'hit'` or `'miss'` when a cache is configured. What the hook throws or rejects with is
|
|
182
183
|
swallowed; it cannot fail a query.
|
|
183
184
|
|
|
184
185
|
Each measure in `listDatasets` has a `valueType`: `count` (a whole number, never null),
|
|
185
186
|
`ratio` (0-1, from `share` and `ratio`) or `number` (sums, averages, medians), so a UI knows
|
|
186
|
-
to format a share as a percentage.
|
|
187
|
+
to format a share as a percentage. A resolved measure has the `result` it declares.
|
|
188
|
+
|
|
189
|
+
## Query options
|
|
190
|
+
|
|
191
|
+
```ts
|
|
192
|
+
const result = await analytics.query(
|
|
193
|
+
{ dataset: 'orders', measures: ['orders'], groupBy: ['day', 'status'], limit: 400, totals: false },
|
|
194
|
+
ctx,
|
|
195
|
+
{ signal: request.signal }
|
|
196
|
+
)
|
|
197
|
+
result.fields
|
|
198
|
+
// { groupBy: [{ key: 'day', label: 'Day', kind: 'timeBucket', unit: 'day' },
|
|
199
|
+
// { key: 'status', label: 'Status', kind: 'enum', values: [{ key: 'OPEN', label: 'Open' }, ...] }],
|
|
200
|
+
// measures: [{ key: 'orders', label: 'Orders', valueType: 'count' }] }
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
- `fields` describes every result: its grouped dimensions (kind, a time bucket's `unit`, a
|
|
204
|
+
closed dimension's `values`) and its measures (label, `valueType`), so a chart drawn from a
|
|
205
|
+
model's tool output needs no guessing.
|
|
206
|
+
- `totals: false` skips a grouped query's totals statement, and the result has no `totals`
|
|
207
|
+
(its type says so). The presenters do not need them. It is for app code: `querySchema` does
|
|
208
|
+
not offer it to a model, which would then add up overlapping rows.
|
|
209
|
+
- `createAnalytics({ maxLimit })` raises the highest `limit` `query` accepts, for app code such
|
|
210
|
+
as a long time series. The schema `querySchema` builds keeps 100.
|
|
211
|
+
- `createAnalytics({ concurrency })` caps how many statements run at once on each source, so a
|
|
212
|
+
large dashboard cannot take every pooled connection.
|
|
213
|
+
- `signal` aborts a query: it rejects with the signal's reason, statements waiting for a slot
|
|
214
|
+
never start, and `execute` gets `{ signal }` as its second argument to cancel a running one.
|
|
215
|
+
- `analytics.period(input, ctx)` resolves a period the way `query` does (its local days, start
|
|
216
|
+
and end instants and length, or null for all time), and `analytics.scope(ctx)` returns the
|
|
217
|
+
tenant scope `query` would use, refusing a missing one. Use them when code reads the same rows
|
|
218
|
+
another way (Prisma, another database).
|
|
219
|
+
|
|
187
220
|
|
|
188
221
|
## Caching
|
|
189
222
|
|
|
@@ -209,9 +242,17 @@ createAnalytics({
|
|
|
209
242
|
```
|
|
210
243
|
|
|
211
244
|
A stored value that is not a result of its dataset counts as a miss, and a cache that throws or
|
|
212
|
-
rejects is a miss or a skipped write, never a failed query.
|
|
213
|
-
|
|
214
|
-
|
|
245
|
+
rejects is a miss or a skipped write, never a failed query. The key also hashes the dataset's
|
|
246
|
+
definition (its compiled SQL, labels and values), so a deploy that changes a measure misses the
|
|
247
|
+
cache. Resolver code is not part of it: prefix keys with a build id (as above) if a changed
|
|
248
|
+
resolver's old numbers must not outlive a deploy. A dataset that uses `minutesAgo` is as stale as
|
|
249
|
+
the time to live.
|
|
250
|
+
|
|
251
|
+
With a cache, identical queries that run at the same time (two metrics of one dashboard, two
|
|
252
|
+
people opening it) share one run. The entry names the `metric` a registry ran the query for, so
|
|
253
|
+
`set` can pick a time to live per metric. The key holds the tenant scope and nothing else from
|
|
254
|
+
`ctx`, so a cached or shared result must not depend on anything else in it: a resolver that
|
|
255
|
+
reads the user from `ctx` belongs in an uncached instance.
|
|
215
256
|
|
|
216
257
|
## Chart shapes
|
|
217
258
|
|
|
@@ -234,20 +275,26 @@ toLeaderboard(result, { by: 'region', measure: 'revenue' }) // ranked rows with
|
|
|
234
275
|
```
|
|
235
276
|
|
|
236
277
|
- `toTimeseries` gives every bucket of the period a point; buckets without rows get `fill` (0 by
|
|
237
|
-
default, pass `null` for averages). An
|
|
238
|
-
previous period's points are on its own
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
278
|
+
default, pass `null` for averages). An ungrouped series exists even for a period without rows.
|
|
279
|
+
An all-time result has only the buckets it found. The previous period's points are on its own
|
|
280
|
+
buckets; compare them by position.
|
|
281
|
+
- `seriesKeys` (and `segmentKeys` for distributions) lists keys to draw even when no row has
|
|
282
|
+
them, a zero lane per channel, first and in that order. `'all'` takes a closed dimension's
|
|
283
|
+
values from the result's `fields`.
|
|
284
|
+
- A grouped query returns at most `limit` rows (20 by default, 100 at most, or `maxLimit`), and
|
|
285
|
+
the shape keeps `truncated`. Set a `limit` that covers every bucket and series: 30 days split
|
|
286
|
+
into three series fit in 100, 90 days into eleven do not.
|
|
242
287
|
- `toDistribution`'s `total` is the measure over every row, so many-to-many shares can add up to
|
|
243
|
-
more than 1. `toPivot` sorts numeric keys (weekdays,
|
|
288
|
+
more than 1; it is null for a result without totals. `toPivot` sorts numeric keys (weekdays,
|
|
289
|
+
hours).
|
|
244
290
|
|
|
245
291
|
### Presentation
|
|
246
292
|
|
|
247
|
-
`presentKpi`, `presentTimeseries`, `presentDistribution
|
|
248
|
-
into the wire outputs a dashboard draws (`KpiOutput`,
|
|
249
|
-
`
|
|
250
|
-
value is 0, keys are strings, and the caller picks labels, units and chart types
|
|
293
|
+
`presentKpi`, `presentTimeseries`, `presentDistribution`, `presentLeaderboard`, `presentPivot`
|
|
294
|
+
and `presentTable` turn a result into the wire outputs a dashboard draws (`KpiOutput`,
|
|
295
|
+
`TimeSeriesOutput`, `DistributionOutput`, `LeaderboardOutput`, `PivotOutput`, `TableOutput`). A
|
|
296
|
+
missing value is 0, keys are strings, and the caller picks labels, units and chart types. An
|
|
297
|
+
output whose query was cut at its `limit` has `truncated: true`:
|
|
251
298
|
|
|
252
299
|
```ts
|
|
253
300
|
presentKpi(result, 'paidShare', { label: 'Paid', percent: true })
|
|
@@ -255,32 +302,44 @@ presentKpi(result, 'paidShare', { label: 'Paid', percent: true })
|
|
|
255
302
|
|
|
256
303
|
presentTimeseries(result, {
|
|
257
304
|
bucket: 'day', measure: 'orders', series: 'status',
|
|
305
|
+
seriesKeys: 'all', // a lane for every status, rows or not
|
|
258
306
|
keys: { PAID: 'paid' }, // wire keys, by String(key); 'null' names the empty group
|
|
259
307
|
labels: { paid: 'Settled' }, // labels by wire key
|
|
260
308
|
chartType: 'line'
|
|
261
309
|
})
|
|
262
|
-
// points are { ts: Date, value }: ts is the bucket's local start date as
|
|
310
|
+
// points are { ts: Date, bucket: '2026-10-01', value }: ts is the bucket's local start date as
|
|
311
|
+
// UTC midnight, so format `bucket`, not `ts` in the viewer's timezone
|
|
263
312
|
|
|
264
313
|
presentDistribution(result, { by: 'tag', measure: 'orders', chartType: 'donut', order: ['t2'] })
|
|
265
314
|
// total is what the segments add up to, percent is 0-100; keys order omits go last
|
|
266
315
|
|
|
267
316
|
presentLeaderboard(result, { by: 'region', measure: 'orders', meta: (values) => ({ revenue: values.revenue }) })
|
|
317
|
+
|
|
318
|
+
presentPivot(result, { rows: 'weekday', columns: 'hour', measure: 'orders', totals: true })
|
|
319
|
+
// rows and columns by label; totals are sums of the cells, so leave them off for an average
|
|
320
|
+
|
|
321
|
+
presentTable(result, { columns: ['region', 'orders'], labels: { orders: 'Count' } })
|
|
322
|
+
// a column per grouped dimension (its label; a bucket's date) and per measure, from `fields`
|
|
268
323
|
```
|
|
269
324
|
|
|
270
325
|
`isKpi`, `isTimeSeries`, `isDistribution`, `isTable`, `isLeaderboard` and `isPivot` narrow a
|
|
271
|
-
`MetricOutput`.
|
|
326
|
+
`MetricOutput`. `OUTPUT_KINDS` lists every kind, and `pickRenderer(map, output, fallback)` picks
|
|
327
|
+
a renderer from a map that must name every kind (a new kind fails typecheck) and falls back for a
|
|
328
|
+
kind the client does not know yet. The module is pure, so a browser bundle can import the types
|
|
329
|
+
and guards.
|
|
272
330
|
|
|
273
331
|
## Dashboard metrics
|
|
274
332
|
|
|
275
333
|
`measurr/metrics` names the numbers a dashboard shows: one query and the output it is drawn
|
|
276
|
-
in. The registry runs a batch for a
|
|
334
|
+
in, or a `resolve` function for a number a dataset cannot give. The registry runs a batch for a
|
|
335
|
+
caller and streams one chunk per metric.
|
|
277
336
|
|
|
278
337
|
```ts
|
|
279
338
|
import { createMetricRegistry, metricDefiner } from 'measurr/metrics'
|
|
280
339
|
import { presentKpi, presentTimeseries } from 'measurr/shapes'
|
|
281
340
|
|
|
282
341
|
// Types only, so metric modules need not import the analytics instance.
|
|
283
|
-
export const defineMetric = metricDefiner<(typeof datasets)[number], Role>()
|
|
342
|
+
export const defineMetric = metricDefiner<(typeof datasets)[number], Role, {}, Ctx>()
|
|
284
343
|
|
|
285
344
|
const orderTrend = defineMetric({
|
|
286
345
|
key: 'orders.trend',
|
|
@@ -288,48 +347,123 @@ const orderTrend = defineMetric({
|
|
|
288
347
|
description: 'Orders over time',
|
|
289
348
|
allowedRoles: ['owner', 'member'],
|
|
290
349
|
supportsTimeRange: true, // false for a snapshot: all time, no previous period
|
|
291
|
-
filterSchema: z.object({ status: z.enum(['OPEN', 'PAID']).optional() }),
|
|
350
|
+
filterSchema: z.object({ status: z.enum(['OPEN', 'PAID']).optional() }), // optional
|
|
292
351
|
query: ({ bucket, filters }) => ({
|
|
293
352
|
dataset: 'orders',
|
|
294
353
|
measures: ['orders'],
|
|
295
354
|
groupBy: [bucket],
|
|
296
355
|
filters: filters.status ? [{ dimension: 'status', op: 'in', values: [filters.status] }] : [],
|
|
297
|
-
limit: 100
|
|
356
|
+
limit: 100,
|
|
357
|
+
totals: false // no presenter reads them; saves a statement per period
|
|
298
358
|
}),
|
|
299
359
|
shape: (result, { bucket }) =>
|
|
300
360
|
presentTimeseries(result, { bucket, measure: 'orders', chartType: 'bar' })
|
|
301
361
|
})
|
|
302
362
|
|
|
363
|
+
const myOpenItems = defineMetric({
|
|
364
|
+
key: 'workItems.mine',
|
|
365
|
+
kind: 'kpi',
|
|
366
|
+
description: 'Open items assigned to me',
|
|
367
|
+
supportsTimeRange: false,
|
|
368
|
+
// `ctx` is the caller; the tenant scope applies anyway.
|
|
369
|
+
query: ({ ctx }) => ({
|
|
370
|
+
dataset: 'workItems',
|
|
371
|
+
measures: ['items'],
|
|
372
|
+
filters: [{ dimension: 'assignee', op: 'in', values: [ctx.userId] }]
|
|
373
|
+
}),
|
|
374
|
+
shape: (result) => presentKpi(result, 'items')
|
|
375
|
+
})
|
|
376
|
+
|
|
377
|
+
// Prisma, another database: the registry still decides the period, the previous period and
|
|
378
|
+
// the tenant scope. Filter every read by `tenant`.
|
|
379
|
+
const seatsUsed = defineMetric({
|
|
380
|
+
key: 'billing.seats',
|
|
381
|
+
kind: 'kpi',
|
|
382
|
+
description: 'Seats in use',
|
|
383
|
+
supportsTimeRange: true,
|
|
384
|
+
resolve: async ({ period, tenant, signal }) => ({
|
|
385
|
+
kind: 'kpi',
|
|
386
|
+
value: await countSeats({ organizationIds: tenant, from: period?.start, to: period?.end, signal })
|
|
387
|
+
})
|
|
388
|
+
})
|
|
389
|
+
|
|
303
390
|
export const metrics = createMetricRegistry({
|
|
304
391
|
analytics,
|
|
305
|
-
metrics: [orderTrend,
|
|
306
|
-
canRun: (metric, ctx) =>
|
|
307
|
-
|
|
392
|
+
metrics: [orderTrend, myOpenItems, seatsUsed] as const,
|
|
393
|
+
canRun: (metric, ctx) => (metric.allowedRoles ?? ALL_ROLES).includes(ctx.role),
|
|
394
|
+
onError: (error, { key, code }) => logger.error('metric.failed', { key, code, error })
|
|
308
395
|
})
|
|
309
396
|
|
|
310
397
|
const input = metrics.requestSchema.extend({ refresh: z.boolean().default(false) }).parse(body)
|
|
311
|
-
for await (const chunk of metrics.run(input, ctx)) send(chunk)
|
|
312
|
-
// { key, requestKey, result: { current, previous?, supportsTimeRange }, done }
|
|
313
|
-
// or { key, requestKey, error, done }
|
|
398
|
+
for await (const chunk of metrics.run(input, ctx, { signal: request.signal })) send(chunk)
|
|
399
|
+
// { key, requestKey, result: { current, previous?, supportsTimeRange, period }, done }
|
|
400
|
+
// or { key, requestKey, error, code, done }
|
|
401
|
+
|
|
402
|
+
const { results, errors } = await metrics.collect(input, ctx) // by request key, typed per key
|
|
403
|
+
const seats = await metrics.one('billing.seats', { period: { last: { days: 7 } } }, ctx)
|
|
314
404
|
```
|
|
315
405
|
|
|
316
|
-
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
|
|
406
|
+
- A request names its period like a query, `period: { last: { days: 7 } }` or any other
|
|
407
|
+
`PeriodInput`, counted in `analytics.timezone(ctx)`, so a client that does not know the
|
|
408
|
+
tenant's timezone can still ask for "the last 7 days". Or it sends `from` and `to` as
|
|
409
|
+
instants, counted as the local days they fall on; `from` alone counts up to today. `to`
|
|
410
|
+
without `from`, or `period` with either, rejects with `invalid_period`. Without any, a metric
|
|
411
|
+
counts all time; a snapshot always does.
|
|
412
|
+
- Every result carries the `period` it counted: `{ from, to, timezone }` or `{ all: true,
|
|
413
|
+
timezone }`, so a chart can mark today without a second request.
|
|
414
|
+
- `query` returns the query without `period` or `compareToPrevious`. `compareToPrevious` adds
|
|
415
|
+
the previous period, whose series points are put on the current buckets so a chart overlays
|
|
416
|
+
them. With week or month buckets the two periods can differ by a bucket: points pair from the
|
|
417
|
+
end, and a previous point before the current period's first bucket is dropped.
|
|
418
|
+
- `granularity` (hour to year, inferred from the period's days when omitted: day up to 31
|
|
419
|
+
days, week up to 90, else month) picks `run.bucket`: `day`, `week` or `month`; `hour` draws
|
|
420
|
+
days and `quarter` and `year` draw months, and the output's `granularity` says which. Name
|
|
421
|
+
time-bucket dimensions after them so `groupBy: [bucket]` works.
|
|
422
|
+
- `query` and `shape` get `run.ctx`, and `metricDefiner`'s fourth parameter types it. A filter
|
|
423
|
+
on the caller's own id is part of the query, so the cache key already separates users.
|
|
424
|
+
- A `resolve` metric gets the filters, bucket and granularity, the `period` (local days with
|
|
425
|
+
`start` and `end` instants, or null for all time), the `tenant` from `analytics.scope(ctx)`
|
|
426
|
+
(`allTenants` included), `ctx` and a `signal`. It is called once for the period and, with
|
|
427
|
+
`compareToPrevious`, once for the previous one. Nothing scopes its reads but the code: filter
|
|
428
|
+
every read by `tenant`.
|
|
429
|
+
- A metric fails alone, with a `code` next to its message: `unknown_metric`, `forbidden`
|
|
430
|
+
(`canRun` refused it), `invalid_filters`, `truncated`, `shape_failed`, `internal` (a driver
|
|
431
|
+
error or a throwing `resolve`; the message does not say what), or the engine's codes such as
|
|
432
|
+
`missing_tenant`. `onError(error, { key, requestKey, code, metric }, ctx)` hears every
|
|
433
|
+
failure with what was thrown; `onShapeError` still works and is deprecated.
|
|
434
|
+
- A time series whose query was cut at its `limit` fails with `truncated` instead of drawing
|
|
435
|
+
zeros for the newest buckets. Raise its `limit` (and `maxLimit`) or ask for a shorter period.
|
|
436
|
+
- `filterSchema` is optional; without it a metric takes no filters.
|
|
437
|
+
- `metricDefiner<Datasets, Role, Extra, Ctx>()` takes the app's own definition fields as
|
|
438
|
+
`Extra` (catalog metadata, UI hints for filters); the registry carries them to `canRun` and
|
|
328
439
|
`available` without reading them.
|
|
329
|
-
- Keys and outputs stay literal: a chunk narrows to its metric by `key`, and
|
|
330
|
-
`MetricOutputOf<typeof metrics, 'orders.trend'>`,
|
|
331
|
-
`registry.get(key)` are typed per metric.
|
|
332
|
-
allows, for a catalog.
|
|
440
|
+
- Keys and outputs stay literal: a chunk narrows to its metric by `key`, `collect`'s results and
|
|
441
|
+
`one` are typed per key, and `MetricOutputOf<typeof metrics, 'orders.trend'>`,
|
|
442
|
+
`MetricFiltersOf`, `MetricKeysOfKind` and `registry.get(key)` are typed per metric.
|
|
443
|
+
`registry.available(ctx)` lists what `canRun` allows, for a catalog.
|
|
444
|
+
- A consumer that stops reading `run` (a closed stream) aborts the statements still running;
|
|
445
|
+
`signal` does the same from outside.
|
|
446
|
+
- `metricOutputSchemas` (one per kind), `MetricOutputSchema` and `MetricResultSchema` are Zod
|
|
447
|
+
schemas for an endpoint's output contract or a cached output; `ts` is coerced, so JSON parses
|
|
448
|
+
back.
|
|
449
|
+
|
|
450
|
+
**Access belongs to `ctx`.** `canRun` decides which metrics a caller may run, and `tenant(ctx)`
|
|
451
|
+
decides what any of them can count. Keep both in one discriminated context and never add a
|
|
452
|
+
bypass flag:
|
|
453
|
+
|
|
454
|
+
```ts
|
|
455
|
+
type Ctx =
|
|
456
|
+
| { kind: 'member'; organizationId: OrganizationId; role: Role }
|
|
457
|
+
| { kind: 'staff'; organizationIds: readonly OrganizationId[] | typeof allTenants; role: 'staff' }
|
|
458
|
+
|
|
459
|
+
canRun: (metric, ctx) => (metric.allowedRoles ?? ALL_ROLES).includes(ctx.role),
|
|
460
|
+
tenant: (ctx) => (ctx.kind === 'member' ? ctx.organizationId : ctx.organizationIds)
|
|
461
|
+
```
|
|
462
|
+
|
|
463
|
+
A job that reports for one organization (a monthly PDF, an email) is a caller too: build a
|
|
464
|
+
`ctx` scoped to that organization, the same one a member of it would get, rather than a staff
|
|
465
|
+
context with every tenant. A metric that forgets a scope then counts too little, never another
|
|
466
|
+
customer's rows.
|
|
333
467
|
|
|
334
468
|
## Serving queries
|
|
335
469
|
|
|
@@ -340,7 +474,9 @@ for await (const chunk of metrics.run(input, ctx)) send(chunk)
|
|
|
340
474
|
others; an error other than `AnalyticsError` is reported as `{ code: 'internal' }` without
|
|
341
475
|
its message.
|
|
342
476
|
- `streamBatch(analytics, queries, ctx)` yields `{ key, result }` or `{ key, error }` as each
|
|
343
|
-
query finishes, for a streaming endpoint.
|
|
477
|
+
query finishes, for a streaming endpoint. A consumer that stops reading aborts the statements
|
|
478
|
+
still running.
|
|
479
|
+
- Both take `{ signal }` as a fourth argument; aborting it rejects with the signal's reason.
|
|
344
480
|
- `toHttpError(error)` maps an `AnalyticsError` to `{ status, code, message, data? }`:
|
|
345
481
|
`forbidden` is 403 `FORBIDDEN`, `missing_tenant` 401 `UNAUTHORIZED`, the rest 400
|
|
346
482
|
`BAD_REQUEST` with the code and details. The codes are the names oRPC and tRPC use:
|
|
@@ -355,6 +491,52 @@ try {
|
|
|
355
491
|
}
|
|
356
492
|
```
|
|
357
493
|
|
|
494
|
+
## Resolved measures
|
|
495
|
+
|
|
496
|
+
Some numbers one statement cannot give: sprays per employee when the employees live in another
|
|
497
|
+
database, an average per day of the period, a forecast. Declare such a measure with
|
|
498
|
+
`measure.resolved` and give the code to `createAnalytics`. It stays a measure like any other:
|
|
499
|
+
it is in the catalog and the schema, checked for access, scoped to the tenant and cached.
|
|
500
|
+
|
|
501
|
+
```ts
|
|
502
|
+
const sprays = defineDataset({
|
|
503
|
+
// ...
|
|
504
|
+
measures: {
|
|
505
|
+
sprays: measure.sum(status.col('sprays'), { label: 'Sprays' }),
|
|
506
|
+
spraysPerDay: measure.resolved({ label: 'Sprays per day', result: 'number', uses: ['sprays'] })
|
|
507
|
+
}
|
|
508
|
+
})
|
|
509
|
+
|
|
510
|
+
createAnalytics({
|
|
511
|
+
// ...
|
|
512
|
+
resolvers: {
|
|
513
|
+
sprays: {
|
|
514
|
+
spraysPerDay: ({ rows, totals, period }) => {
|
|
515
|
+
const days = 'all' in period ? null : daysBetween(period.from, period.to)
|
|
516
|
+
const perDay = (value: number | null) =>
|
|
517
|
+
value === null || days === null ? null : value / days
|
|
518
|
+
return { rows: rows.map(({ values }) => perDay(values.sprays)), totals: perDay(totals.sprays) }
|
|
519
|
+
}
|
|
520
|
+
}
|
|
521
|
+
}
|
|
522
|
+
})
|
|
523
|
+
```
|
|
524
|
+
|
|
525
|
+
- `uses` names the measures of the same dataset the resolver reads. The database computes them
|
|
526
|
+
for every row and for the totals, even when the query does not ask for them, and the result
|
|
527
|
+
holds only what it asked for. A measure with no `uses` still gets one value per group.
|
|
528
|
+
- The resolver gets the rows in order (`dimensions` for the grouped keys, `values` for the used
|
|
529
|
+
measures), the `totals`, `groupBy`, the `filters` (open-dimension labels resolved to keys),
|
|
530
|
+
the `period`, the `tenant` scope and `ctx`. It returns one value per row and one for the
|
|
531
|
+
totals. With `compareToPrevious` it is called once per period.
|
|
532
|
+
- `createAnalytics` throws when a resolved measure has no resolver, and when `resolvers` names
|
|
533
|
+
anything else. A wrong number of values fails the query; a value that is not a finite number
|
|
534
|
+
is null, and a `count` result is 0 instead.
|
|
535
|
+
- A resolved measure needs the access of the measures it uses, like a ratio. It cannot be sorted
|
|
536
|
+
by (the database limits the rows before the value exists), and `ratio` and `uses` cannot name
|
|
537
|
+
one.
|
|
538
|
+
- The contract kit leaves resolved measures out: they are app code, so test them with it.
|
|
539
|
+
|
|
358
540
|
## Tenants and platform admins
|
|
359
541
|
|
|
360
542
|
`tenant(ctx)` decides the scope of every statement, including the lookups that resolve filter
|
|
@@ -424,10 +606,10 @@ before any statement runs; a narrower scope is never ignored. `listDatasets(ctx)
|
|
|
424
606
|
| Concept | What it is |
|
|
425
607
|
| --- | --- |
|
|
426
608
|
| Dataset | One fact table: tenant column (or `null`), optional scope, time fields, default period, measures, dimensions, source. |
|
|
427
|
-
| Measure | `count`, `countWhere`, `countDistinct`, `sum`, `avg`, `median`, `share` (one condition over another)
|
|
609
|
+
| Measure | `count`, `countWhere`, `countDistinct`, `sum`, `avg`, `median`, `share` (one condition over another), `ratio` (one measure over another, after aggregation) or `resolved` (computed by the app after the query runs). Optionally restricted with `access`. |
|
|
428
610
|
| Dimension | `enum`, `computed` (closed value sets), `relation` (to-one), `manyToMany`, `number`, `timeBucket` (day, ISO week, month), `timePart` (ISO weekday, hour). Optionally restricted with `access`. |
|
|
429
|
-
| Query | `dataset`, 1-4 `measures`, 0-2 `groupBy`, `filters`, `period`, `time`, `compareToPrevious`, `sort` (by an asked measure or grouped dimension), `limit` (1-100
|
|
430
|
-
| Result | `{ period, rows, totals, previous?, notes, truncated }`; `period` is `{ from, to, timezone }` or `{ all: true, timezone }`. |
|
|
611
|
+
| Query | `dataset`, 1-4 `measures`, 0-2 `groupBy`, `filters`, `period`, `time`, `compareToPrevious`, `sort` (by an asked measure or grouped dimension), `limit` (1-100 or `maxLimit`, default 20), and for app code `totals`. |
|
|
612
|
+
| Result | `{ period, rows, totals, previous?, notes, truncated, fields }`; `period` is `{ from, to, timezone }` or `{ all: true, timezone }`. `previous` is typed as present when the query says `compareToPrevious: true`. |
|
|
431
613
|
|
|
432
614
|
**Operators per dimension kind.** Closed and open dimensions take `in`, `notIn`, `isEmpty`
|
|
433
615
|
and `isNotEmpty`; numeric and time dimensions take `between`, `isEmpty` and `isNotEmpty`. The
|
|
@@ -439,6 +621,18 @@ keys or names (case-insensitive); the engine resolves them before planning. An u
|
|
|
439
621
|
fails with `AnalyticsError('unknown_value')` listing the valid values, so a model can retry
|
|
440
622
|
once.
|
|
441
623
|
|
|
624
|
+
The lookup reads the fact table of the tenant by default, every row it ever had. Give the
|
|
625
|
+
dimension a `lookup` to read the names from their own table instead; the key and the name must
|
|
626
|
+
be columns of it, and `tenantColumn` scopes it (null for a table every tenant shares). The
|
|
627
|
+
valid values are then every entry of that table, not only those with rows.
|
|
628
|
+
|
|
629
|
+
```ts
|
|
630
|
+
team: dimension.relation({
|
|
631
|
+
label: 'Team', key: team.col('id'), name: team.col('name'),
|
|
632
|
+
lookup: { tenantColumn: team.col('organizationId') }
|
|
633
|
+
})
|
|
634
|
+
```
|
|
635
|
+
|
|
442
636
|
**Errors carry typed details.** `isAnalyticsError(error, code?)` narrows by code:
|
|
443
637
|
`unknown_value` details are `{ dimension, unknown, valid }` and `invalid_query` details are the
|
|
444
638
|
schema's issues; the other codes have none.
|
|
@@ -475,7 +669,15 @@ every unit. The generated schema states each dataset's default, so a model knows
|
|
|
475
669
|
|
|
476
670
|
**Totals are a separate, ungrouped statement**, so a many-to-many grouping (an order with
|
|
477
671
|
two tags appears in two rows) never inflates them. Filters on many-to-many dimensions compile
|
|
478
|
-
to `EXISTS`, so they never multiply rows either.
|
|
672
|
+
to `EXISTS`, so they never multiply rows either. A grouped query is therefore two statements
|
|
673
|
+
per period, four with `compareToPrevious`, plus one per open-dimension label lookup; `totals:
|
|
674
|
+
false` drops the totals statements.
|
|
675
|
+
|
|
676
|
+
**Daily rollups cannot see part of a day.** A dataset over a table with one row per day (usage
|
|
677
|
+
per unit per day) counts whole local days. It cannot start a unit at the hour it joined or end a
|
|
678
|
+
period mid-day, so a total read from raw events with exact instants can differ on a unit's first
|
|
679
|
+
day and at the edges of the period. Filter such rows by day (`day >= joinedDay`) and say so in
|
|
680
|
+
the dataset's description, or compute the exact number in a `resolve` metric.
|
|
479
681
|
|
|
480
682
|
**Result types are honest.** `count` measures are `number`; `sum`, `avg` and `median` are
|
|
481
683
|
`number | null`; `share` and `ratio` are a branded `Ratio | null` (0-1). A row has a field only
|
|
@@ -546,6 +748,55 @@ const main: AnalyticsSource = {
|
|
|
546
748
|
}
|
|
547
749
|
```
|
|
548
750
|
|
|
751
|
+
Each statement there is an interactive transaction: five round trips (begin, two `SET`s, the
|
|
752
|
+
query, commit). One round trip does the same when the connection itself is read-only with a
|
|
753
|
+
timeout, for example a role made for analytics:
|
|
754
|
+
|
|
755
|
+
```sql
|
|
756
|
+
CREATE ROLE analytics_reader LOGIN PASSWORD '...';
|
|
757
|
+
GRANT SELECT ON ALL TABLES IN SCHEMA public TO analytics_reader;
|
|
758
|
+
ALTER ROLE analytics_reader SET default_transaction_read_only = on;
|
|
759
|
+
ALTER ROLE analytics_reader SET statement_timeout = '5s';
|
|
760
|
+
```
|
|
761
|
+
|
|
762
|
+
```ts
|
|
763
|
+
const analyticsDb = new PrismaClient({ datasourceUrl: process.env.ANALYTICS_DATABASE_URL })
|
|
764
|
+
const main: AnalyticsSource = {
|
|
765
|
+
dialect: postgresDialect({ timestamps: 'utc' }),
|
|
766
|
+
execute: ({ text, values }) => analyticsDb.$queryRawUnsafe(text, ...values)
|
|
767
|
+
}
|
|
768
|
+
```
|
|
769
|
+
|
|
770
|
+
`concurrency` on `createAnalytics` keeps a dashboard from taking every connection of the pool.
|
|
771
|
+
|
|
772
|
+
**`measurr/clickhouse`** `clickhouseDialect()`: typed `{pN:Type}` parameters, backquoted
|
|
773
|
+
identifiers, `-If` and `-OrNull` aggregates (an empty group is null, as in Postgres), an exact
|
|
774
|
+
interpolated median, `uniqExact` for distinct counts, and buckets with `toTimeZone`. Every
|
|
775
|
+
statement ends with `SETTINGS join_use_nulls = 1, enable_positional_arguments = 1`, so a left
|
|
776
|
+
join without a match gives null (an empty group) instead of an empty string. Pass the values
|
|
777
|
+
with `toQueryParams`:
|
|
778
|
+
|
|
779
|
+
```ts
|
|
780
|
+
import { clickhouseDialect, toQueryParams } from 'measurr/clickhouse'
|
|
781
|
+
|
|
782
|
+
const events: AnalyticsSource = {
|
|
783
|
+
dialect: clickhouseDialect(),
|
|
784
|
+
execute: async (statement) => {
|
|
785
|
+
const result = await clickhouse.query({
|
|
786
|
+
query: statement.text,
|
|
787
|
+
query_params: toQueryParams(statement),
|
|
788
|
+
format: 'JSONEachRow'
|
|
789
|
+
})
|
|
790
|
+
return result.json()
|
|
791
|
+
}
|
|
792
|
+
}
|
|
793
|
+
```
|
|
794
|
+
|
|
795
|
+
ClickHouse cannot run correlated subqueries, so a dataset on it cannot use `exists` or filter
|
|
796
|
+
by a many-to-many dimension (mark it `groupOnly`); `createAnalytics` fails at startup if it
|
|
797
|
+
does. Time columns are `DateTime` or `DateTime64`, which hold instants. Tested against
|
|
798
|
+
ClickHouse 24.
|
|
799
|
+
|
|
549
800
|
**`measurr/postgres`** `postgresDialect({ timestamps })`: `$n` parameters (untyped,
|
|
550
801
|
so Postgres infers them from the column, which keeps enum columns working), double-quoted
|
|
551
802
|
identifiers, `FILTER (WHERE ...)`, `count(DISTINCT ...)`, `percentile_cont`, correlated
|
|
@@ -578,6 +829,17 @@ list are refused with `forbidden`. Every other check runs as usual.
|
|
|
578
829
|
await assertDatasetContract({ dataset: maintenance, source, tenant: allTenants })
|
|
579
830
|
```
|
|
580
831
|
|
|
832
|
+
`assertMetricContract({ registry, ctx })` runs every metric of a registry once over the last 30
|
|
833
|
+
days (or `period`), comparing with the previous period, and fails when one errors, draws
|
|
834
|
+
another kind than it declares, or a metric over a time range has no previous period. Give it a
|
|
835
|
+
`ctx` that `canRun` allows everything for, scoped to a seeded tenant.
|
|
836
|
+
|
|
837
|
+
```ts
|
|
838
|
+
import { assertMetricContract } from 'measurr/testing'
|
|
839
|
+
|
|
840
|
+
await assertMetricContract({ registry: metrics, ctx: seededOwner })
|
|
841
|
+
```
|
|
842
|
+
|
|
581
843
|
## Extending
|
|
582
844
|
|
|
583
845
|
- **A measure or dimension** is one entry in a dataset. Dimensions are groupable and
|
|
@@ -600,14 +862,19 @@ Kept stable for publishing: `createAnalytics` and its options (with `authorize`,
|
|
|
600
862
|
`resultSchema`, `listDatasets`; `AnalyticsResultSchema`, `analyticsToolGuidance`,
|
|
601
863
|
`AnalyticsQueryEvent`; `defineDataset` (with `defaultPeriod` and `tenantColumn: null`),
|
|
602
864
|
`measure.*` and `dimension.*` (with `access`), `table` / `Table`, the expression builders; the
|
|
603
|
-
`Dialect`, `SqlRenderer` and `AnalyticsSource` contracts; `
|
|
604
|
-
`
|
|
865
|
+
`Dialect`, `SqlRenderer` and `AnalyticsSource` contracts; `measure.resolved` with the
|
|
866
|
+
`resolvers` option (`MeasureResolver`, `ResolverInput`, `ResolverOutput`); `postgresDialect`,
|
|
867
|
+
`clickhouseDialect` and `toQueryParams`,
|
|
868
|
+
`checkDatasetContract` / `assertDatasetContract`, `analytics.period` and `analytics.scope`,
|
|
869
|
+
the query options (`totals`, `signal`) and the `maxLimit` and `concurrency` options, result
|
|
870
|
+
`fields`, dimension `lookup`; `measurr/shapes` (`toKpi`, `toTimeseries`,
|
|
605
871
|
`toDistribution`, `toPivot`, `toLeaderboard`) and `measurr/handler` (`queryBatch`,
|
|
606
872
|
`streamBatch`, `toHttpError`); and the types `AnalyticsQuery`, `AnalyticsResult`,
|
|
607
873
|
`ResultPeriod`, `PeriodInput`, `Tenant`, `TenantId`, `AccessKey`, `AnalyticsCache`, `FieldKey`,
|
|
608
874
|
`Ratio`, `AnalyticsError` codes and their details (`isAnalyticsError`). `ExprNode` and
|
|
609
875
|
`SelectStatement` are exported for dialect authors and may grow new node kinds. The
|
|
610
|
-
presenters and wire outputs in `measurr/shapes`, `measurr/metrics`
|
|
876
|
+
presenters and wire outputs in `measurr/shapes`, `measurr/metrics` (with `resolve` metrics and
|
|
877
|
+
the output schemas), `checkMetricContract` / `assertMetricContract` and `analytics.timezone`
|
|
611
878
|
are new and may still change before they join this list.
|
|
612
879
|
|
|
613
880
|
## Install
|
|
@@ -616,7 +883,7 @@ are new and may still change before they join this list.
|
|
|
616
883
|
npm install measurr zod
|
|
617
884
|
```
|
|
618
885
|
|
|
619
|
-
`zod` 4 is a peer dependency. The types need TypeScript
|
|
886
|
+
`zod` 4 is a peer dependency. The types need TypeScript 5.9 or newer.
|
|
620
887
|
|
|
621
888
|
## Development
|
|
622
889
|
|
|
@@ -624,13 +891,14 @@ npm install measurr zod
|
|
|
624
891
|
bun install
|
|
625
892
|
bun run lint # types and Biome
|
|
626
893
|
bun test
|
|
627
|
-
# With a throwaway Postgres (the integration
|
|
628
|
-
ANALYTICS_TEST_DATABASE_URL=postgres://... bun test
|
|
894
|
+
# With a throwaway Postgres and ClickHouse (the integration tests create and drop their own tables):
|
|
895
|
+
ANALYTICS_TEST_DATABASE_URL=postgres://... ANALYTICS_TEST_CLICKHOUSE_URL=http://user:password@localhost:8123 bun test
|
|
629
896
|
bun run build && bun run verify:package
|
|
630
897
|
```
|
|
631
898
|
|
|
632
899
|
The integration test reads `ANALYTICS_TEST_DATABASE_URL`, not `DATABASE_URL`, because Bun
|
|
633
|
-
loads `.env` files and that name may point at a real database. CI runs
|
|
900
|
+
loads `.env` files and that name may point at a real database. CI runs them against Postgres 17
|
|
901
|
+
and ClickHouse 24.
|
|
634
902
|
|
|
635
903
|
Releases are automated with Release Please; see [docs/releasing.md](docs/releasing.md).
|
|
636
904
|
|