measurr 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +84 -7
- package/dist/analytics.d.ts +56 -3
- package/dist/dataset.d.ts +47 -4
- package/dist/dialects/clickhouse.d.ts +26 -0
- package/dist/dialects/clickhouse.js +101 -0
- package/dist/handler.js +2 -2
- package/dist/{index-z5xv3ffk.js → index-g0ah4ehn.js} +112 -10
- package/dist/{index-xf6qpmgy.js → index-psw9s799.js} +1 -1
- package/dist/{index-pgccd363.js → index-rgqsrd72.js} +23 -1
- package/dist/index.d.ts +3 -3
- package/dist/index.js +2 -2
- package/dist/metrics.js +2 -2
- package/dist/plan.d.ts +7 -1
- package/dist/query.d.ts +2 -2
- package/dist/testing.js +5 -4
- package/package.json +9 -2
- package/src/analytics.ts +173 -9
- package/src/dataset.ts +82 -5
- package/src/dialects/clickhouse.ts +147 -0
- package/src/index.ts +14 -3
- package/src/plan.ts +48 -7
- package/src/query.ts +17 -3
- package/src/testing.ts +7 -2
package/README.md
CHANGED
|
@@ -7,8 +7,8 @@ returns shaped, typed rows.
|
|
|
7
7
|
|
|
8
8
|
- No SQL text in definitions: datasets are built from typed expression builders, so a
|
|
9
9
|
renamed column, `avg` over a string or `between` on an enum fails typecheck.
|
|
10
|
-
- Database-neutral definitions: dialects plug in through the `Dialect` interface
|
|
11
|
-
|
|
10
|
+
- Database-neutral definitions: dialects plug in through the `Dialect` interface. Postgres
|
|
11
|
+
(`measurr/postgres`) and ClickHouse (`measurr/clickhouse`) ship today.
|
|
12
12
|
- Tenant-scoped by construction: the engine adds the tenant filter to every statement and
|
|
13
13
|
refuses to run without a tenant. A caller can be scoped to one tenant or a list of them;
|
|
14
14
|
platform admins can query every tenant only through an explicit `allTenants` value that no
|
|
@@ -355,6 +355,52 @@ try {
|
|
|
355
355
|
}
|
|
356
356
|
```
|
|
357
357
|
|
|
358
|
+
## Resolved measures
|
|
359
|
+
|
|
360
|
+
Some numbers one statement cannot give: sprays per employee when the employees live in another
|
|
361
|
+
database, an average per day of the period, a forecast. Declare such a measure with
|
|
362
|
+
`measure.resolved` and give the code to `createAnalytics`. It stays a measure like any other:
|
|
363
|
+
it is in the catalog and the schema, checked for access, scoped to the tenant and cached.
|
|
364
|
+
|
|
365
|
+
```ts
|
|
366
|
+
const sprays = defineDataset({
|
|
367
|
+
// ...
|
|
368
|
+
measures: {
|
|
369
|
+
sprays: measure.sum(status.col('sprays'), { label: 'Sprays' }),
|
|
370
|
+
spraysPerDay: measure.resolved({ label: 'Sprays per day', result: 'number', uses: ['sprays'] })
|
|
371
|
+
}
|
|
372
|
+
})
|
|
373
|
+
|
|
374
|
+
createAnalytics({
|
|
375
|
+
// ...
|
|
376
|
+
resolvers: {
|
|
377
|
+
sprays: {
|
|
378
|
+
spraysPerDay: ({ rows, totals, period }) => {
|
|
379
|
+
const days = 'all' in period ? null : daysBetween(period.from, period.to)
|
|
380
|
+
const perDay = (value: number | null) =>
|
|
381
|
+
value === null || days === null ? null : value / days
|
|
382
|
+
return { rows: rows.map(({ values }) => perDay(values.sprays)), totals: perDay(totals.sprays) }
|
|
383
|
+
}
|
|
384
|
+
}
|
|
385
|
+
}
|
|
386
|
+
})
|
|
387
|
+
```
|
|
388
|
+
|
|
389
|
+
- `uses` names the measures of the same dataset the resolver reads. The database computes them
|
|
390
|
+
for every row and for the totals, even when the query does not ask for them, and the result
|
|
391
|
+
holds only what it asked for. A measure with no `uses` still gets one value per group.
|
|
392
|
+
- The resolver gets the rows in order (`dimensions` for the grouped keys, `values` for the used
|
|
393
|
+
measures), the `totals`, `groupBy`, the `filters` (open-dimension labels resolved to keys),
|
|
394
|
+
the `period`, the `tenant` scope and `ctx`. It returns one value per row and one for the
|
|
395
|
+
totals. With `compareToPrevious` it is called once per period.
|
|
396
|
+
- `createAnalytics` throws when a resolved measure has no resolver, and when `resolvers` names
|
|
397
|
+
anything else. A wrong number of values fails the query; a value that is not a finite number
|
|
398
|
+
is null, and a `count` result is 0 instead.
|
|
399
|
+
- A resolved measure needs the access of the measures it uses, like a ratio. It cannot be sorted
|
|
400
|
+
by (the database limits the rows before the value exists), and `ratio` and `uses` cannot name
|
|
401
|
+
one.
|
|
402
|
+
- The contract kit leaves resolved measures out: they are app code, so test them with it.
|
|
403
|
+
|
|
358
404
|
## Tenants and platform admins
|
|
359
405
|
|
|
360
406
|
`tenant(ctx)` decides the scope of every statement, including the lookups that resolve filter
|
|
@@ -424,7 +470,7 @@ before any statement runs; a narrower scope is never ignored. `listDatasets(ctx)
|
|
|
424
470
|
| Concept | What it is |
|
|
425
471
|
| --- | --- |
|
|
426
472
|
| Dataset | One fact table: tenant column (or `null`), optional scope, time fields, default period, measures, dimensions, source. |
|
|
427
|
-
| Measure | `count`, `countWhere`, `countDistinct`, `sum`, `avg`, `median`, `share` (one condition over another)
|
|
473
|
+
| Measure | `count`, `countWhere`, `countDistinct`, `sum`, `avg`, `median`, `share` (one condition over another), `ratio` (one measure over another, after aggregation) or `resolved` (computed by the app after the query runs). Optionally restricted with `access`. |
|
|
428
474
|
| Dimension | `enum`, `computed` (closed value sets), `relation` (to-one), `manyToMany`, `number`, `timeBucket` (day, ISO week, month), `timePart` (ISO weekday, hour). Optionally restricted with `access`. |
|
|
429
475
|
| Query | `dataset`, 1-4 `measures`, 0-2 `groupBy`, `filters`, `period`, `time`, `compareToPrevious`, `sort` (by an asked measure or grouped dimension), `limit` (1-100, default 20). |
|
|
430
476
|
| Result | `{ period, rows, totals, previous?, notes, truncated }`; `period` is `{ from, to, timezone }` or `{ all: true, timezone }`. |
|
|
@@ -546,6 +592,34 @@ const main: AnalyticsSource = {
|
|
|
546
592
|
}
|
|
547
593
|
```
|
|
548
594
|
|
|
595
|
+
**`measurr/clickhouse`** `clickhouseDialect()`: typed `{pN:Type}` parameters, backquoted
|
|
596
|
+
identifiers, `-If` and `-OrNull` aggregates (an empty group is null, as in Postgres), an exact
|
|
597
|
+
interpolated median, `uniqExact` for distinct counts, and buckets with `toTimeZone`. Every
|
|
598
|
+
statement ends with `SETTINGS join_use_nulls = 1, enable_positional_arguments = 1`, so a left
|
|
599
|
+
join without a match gives null (an empty group) instead of an empty string. Pass the values
|
|
600
|
+
with `toQueryParams`:
|
|
601
|
+
|
|
602
|
+
```ts
|
|
603
|
+
import { clickhouseDialect, toQueryParams } from 'measurr/clickhouse'
|
|
604
|
+
|
|
605
|
+
const events: AnalyticsSource = {
|
|
606
|
+
dialect: clickhouseDialect(),
|
|
607
|
+
execute: async (statement) => {
|
|
608
|
+
const result = await clickhouse.query({
|
|
609
|
+
query: statement.text,
|
|
610
|
+
query_params: toQueryParams(statement),
|
|
611
|
+
format: 'JSONEachRow'
|
|
612
|
+
})
|
|
613
|
+
return result.json()
|
|
614
|
+
}
|
|
615
|
+
}
|
|
616
|
+
```
|
|
617
|
+
|
|
618
|
+
ClickHouse cannot run correlated subqueries, so a dataset on it cannot use `exists` or filter
|
|
619
|
+
by a many-to-many dimension (mark it `groupOnly`); `createAnalytics` fails at startup if it
|
|
620
|
+
does. Time columns are `DateTime` or `DateTime64`, which hold instants. Tested against
|
|
621
|
+
ClickHouse 24.
|
|
622
|
+
|
|
549
623
|
**`measurr/postgres`** `postgresDialect({ timestamps })`: `$n` parameters (untyped,
|
|
550
624
|
so Postgres infers them from the column, which keeps enum columns working), double-quoted
|
|
551
625
|
identifiers, `FILTER (WHERE ...)`, `count(DISTINCT ...)`, `percentile_cont`, correlated
|
|
@@ -600,7 +674,9 @@ Kept stable for publishing: `createAnalytics` and its options (with `authorize`,
|
|
|
600
674
|
`resultSchema`, `listDatasets`; `AnalyticsResultSchema`, `analyticsToolGuidance`,
|
|
601
675
|
`AnalyticsQueryEvent`; `defineDataset` (with `defaultPeriod` and `tenantColumn: null`),
|
|
602
676
|
`measure.*` and `dimension.*` (with `access`), `table` / `Table`, the expression builders; the
|
|
603
|
-
`Dialect`, `SqlRenderer` and `AnalyticsSource` contracts; `
|
|
677
|
+
`Dialect`, `SqlRenderer` and `AnalyticsSource` contracts; `measure.resolved` with the
|
|
678
|
+
`resolvers` option (`MeasureResolver`, `ResolverInput`, `ResolverOutput`); `postgresDialect`,
|
|
679
|
+
`clickhouseDialect` and `toQueryParams`,
|
|
604
680
|
`checkDatasetContract` / `assertDatasetContract`; `measurr/shapes` (`toKpi`, `toTimeseries`,
|
|
605
681
|
`toDistribution`, `toPivot`, `toLeaderboard`) and `measurr/handler` (`queryBatch`,
|
|
606
682
|
`streamBatch`, `toHttpError`); and the types `AnalyticsQuery`, `AnalyticsResult`,
|
|
@@ -624,13 +700,14 @@ npm install measurr zod
|
|
|
624
700
|
bun install
|
|
625
701
|
bun run lint # types and Biome
|
|
626
702
|
bun test
|
|
627
|
-
# With a throwaway Postgres (the integration
|
|
628
|
-
ANALYTICS_TEST_DATABASE_URL=postgres://... bun test
|
|
703
|
+
# With a throwaway Postgres and ClickHouse (the integration tests create and drop their own tables):
|
|
704
|
+
ANALYTICS_TEST_DATABASE_URL=postgres://... ANALYTICS_TEST_CLICKHOUSE_URL=http://user:password@localhost:8123 bun test
|
|
629
705
|
bun run build && bun run verify:package
|
|
630
706
|
```
|
|
631
707
|
|
|
632
708
|
The integration test reads `ANALYTICS_TEST_DATABASE_URL`, not `DATABASE_URL`, because Bun
|
|
633
|
-
loads `.env` files and that name may point at a real database. CI runs
|
|
709
|
+
loads `.env` files and that name may point at a real database. CI runs them against Postgres 17
|
|
710
|
+
and ClickHouse 24.
|
|
634
711
|
|
|
635
712
|
Releases are automated with Release Please; see [docs/releasing.md](docs/releasing.md).
|
|
636
713
|
|
package/dist/analytics.d.ts
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
import { z } from 'zod';
|
|
2
2
|
import type { CompiledStatement, Dialect } from './compile.js';
|
|
3
|
-
import type { AnyDataset, AnyDimension, Dataset } from './dataset.js';
|
|
4
|
-
import { type Tenant } from './plan.js';
|
|
5
|
-
import { type DatasetQuery, type QueryResult, type ResultOf } from './query.js';
|
|
3
|
+
import type { AnyDataset, AnyDimension, Dataset, DimensionValue, MeasureValue, ResolvedMeasure } from './dataset.js';
|
|
4
|
+
import { type NormalizedFilter, type Tenant } from './plan.js';
|
|
5
|
+
import { type DatasetQuery, type QueryResult, type ResultOf, type ResultPeriod } from './query.js';
|
|
6
6
|
export type Row = Record<string, unknown>;
|
|
7
7
|
/** A database: a dialect to compile with and a function that runs the compiled SQL. */
|
|
8
8
|
export type AnalyticsSource = {
|
|
@@ -59,6 +59,51 @@ export type AnalyticsQueryEvent = {
|
|
|
59
59
|
export type FieldKey<D> = D extends Dataset<string, infer M, infer Dm> ? keyof (M & Dm) & string : never;
|
|
60
60
|
/** The access keys a dataset's measures and dimensions are marked with. */
|
|
61
61
|
export type AccessKey<D> = D extends Dataset<string, infer M, infer Dm> ? NonNullable<(M & Dm)[keyof (M & Dm)]['access']> : never;
|
|
62
|
+
/** The resolved measures of a dataset. */
|
|
63
|
+
export type ResolvedKey<D> = D extends Dataset<string, infer M> ? {
|
|
64
|
+
[K in keyof M & string]: M[K] extends {
|
|
65
|
+
kind: 'resolved';
|
|
66
|
+
} ? K : never;
|
|
67
|
+
}[keyof M & string] : never;
|
|
68
|
+
/** The values of the measures resolved measure `K` uses. */
|
|
69
|
+
type UsedValues<D, K> = D extends Dataset<string, infer M> ? M[K & keyof M] extends ResolvedMeasure<MeasureResult, infer U> ? {
|
|
70
|
+
[Used in U & keyof M]: MeasureValue<M[Used]>;
|
|
71
|
+
} : never : never;
|
|
72
|
+
type MeasureResult = ResolvedMeasure['result'];
|
|
73
|
+
/** What a resolver gets for one period of one query. */
|
|
74
|
+
export type ResolverInput<D, K, Ctx> = D extends Dataset<string, infer _M, infer Dm> ? {
|
|
75
|
+
/**
|
|
76
|
+
* The result rows in order: their grouped dimensions and the measures `K` uses. One
|
|
77
|
+
* row when the query has no `groupBy`.
|
|
78
|
+
*/
|
|
79
|
+
rows: readonly {
|
|
80
|
+
dimensions: {
|
|
81
|
+
[G in keyof Dm]?: DimensionValue<Dm[G]>;
|
|
82
|
+
};
|
|
83
|
+
values: UsedValues<D, K>;
|
|
84
|
+
}[];
|
|
85
|
+
/** The used measures over every row of the period, ungrouped. */
|
|
86
|
+
totals: UsedValues<D, K>;
|
|
87
|
+
groupBy: readonly (keyof Dm & string)[];
|
|
88
|
+
/** The query's filters, open-dimension labels already resolved to keys. */
|
|
89
|
+
filters: readonly NormalizedFilter[];
|
|
90
|
+
/** The period being resolved: the query's, or its previous one. */
|
|
91
|
+
period: ResultPeriod;
|
|
92
|
+
tenant: Tenant;
|
|
93
|
+
ctx: Ctx;
|
|
94
|
+
} : never;
|
|
95
|
+
/** One value per row, in the same order, and one for the totals. */
|
|
96
|
+
export type ResolverOutput = {
|
|
97
|
+
rows: readonly (number | null)[];
|
|
98
|
+
totals: number | null;
|
|
99
|
+
};
|
|
100
|
+
export type MeasureResolver<D, K, Ctx> = (input: ResolverInput<D, K, Ctx>) => ResolverOutput | Promise<ResolverOutput>;
|
|
101
|
+
/** The code for every resolved measure, by dataset and measure. */
|
|
102
|
+
export type AnalyticsResolvers<DS extends readonly AnyDataset[], Ctx> = {
|
|
103
|
+
[D in DS[number] as [ResolvedKey<D>] extends [never] ? never : D['key']]: {
|
|
104
|
+
[K in ResolvedKey<D>]: MeasureResolver<D, K, Ctx>;
|
|
105
|
+
};
|
|
106
|
+
};
|
|
62
107
|
export type AnalyticsOptions<DS extends readonly AnyDataset[], Ctx> = {
|
|
63
108
|
datasets: DS;
|
|
64
109
|
/** Databases by name; a dataset's `source` picks one, or the only one is used. */
|
|
@@ -97,6 +142,13 @@ export type AnalyticsOptions<DS extends readonly AnyDataset[], Ctx> = {
|
|
|
97
142
|
* swallowed: a failing cache is a miss, never a failed query.
|
|
98
143
|
*/
|
|
99
144
|
cache?: AnalyticsCache<Ctx, DS[number]['key']> | undefined;
|
|
145
|
+
/**
|
|
146
|
+
* The code for `measure.resolved` measures, by dataset and measure key: called once per
|
|
147
|
+
* period with the result rows and the values of the measures each one uses. Required for
|
|
148
|
+
* every resolved measure (`createAnalytics` throws otherwise). What it throws fails the
|
|
149
|
+
* query; a value that is not a finite number counts as null.
|
|
150
|
+
*/
|
|
151
|
+
resolvers?: NoInfer<AnalyticsResolvers<DS, Ctx>> | undefined;
|
|
100
152
|
};
|
|
101
153
|
/**
|
|
102
154
|
* A store for finished results, such as Redis. Keys name the dataset and hash everything that
|
|
@@ -185,3 +237,4 @@ export type Analytics<DS extends readonly AnyDataset[], Ctx> = {
|
|
|
185
237
|
export type AnalyticsQuery<A> = A extends Analytics<infer DS, never> ? DatasetQuery<DS[number]> : never;
|
|
186
238
|
export type AnalyticsResult<A, Q> = A extends Analytics<infer DS, never> ? ResultOf<DS[number], Q> : never;
|
|
187
239
|
export declare function createAnalytics<const DS extends readonly AnyDataset[], Ctx>(options: AnalyticsOptions<DS, Ctx>): Analytics<DS, Ctx>;
|
|
240
|
+
export {};
|
package/dist/dataset.d.ts
CHANGED
|
@@ -53,7 +53,21 @@ export type RatioMeasure<A extends string = string, B extends string = string> =
|
|
|
53
53
|
readonly numerator: A;
|
|
54
54
|
readonly denominator: B;
|
|
55
55
|
};
|
|
56
|
-
|
|
56
|
+
/**
|
|
57
|
+
* A value the app computes after the query runs, from the measures it `uses` in each row and
|
|
58
|
+
* from anything else it can reach (another database, a forecast). The `resolvers` option of
|
|
59
|
+
* `createAnalytics` supplies the code.
|
|
60
|
+
*/
|
|
61
|
+
export type ResolvedMeasure<R extends MeasureResult = MeasureResult, U extends string = string> = {
|
|
62
|
+
readonly kind: 'resolved';
|
|
63
|
+
readonly result: R;
|
|
64
|
+
readonly label: string;
|
|
65
|
+
readonly description: string | undefined;
|
|
66
|
+
/** A resolved measure also needs the access of the measures it uses. */
|
|
67
|
+
readonly access: string | undefined;
|
|
68
|
+
readonly uses: readonly U[];
|
|
69
|
+
};
|
|
70
|
+
export type AnyMeasure = AggregateMeasure | ShareMeasure | RatioMeasure | ResolvedMeasure;
|
|
57
71
|
/** The TypeScript type of a measure's value in a result row. */
|
|
58
72
|
export type MeasureValue<M> = M extends {
|
|
59
73
|
result: 'count';
|
|
@@ -86,7 +100,23 @@ export declare const measure: {
|
|
|
86
100
|
* measures of the same dataset; `defineDataset` checks them.
|
|
87
101
|
*/
|
|
88
102
|
ratio<const A extends string, const B extends string, Access extends string = never>(numerator: A, denominator: B, options: Described<Access>): RatioMeasure<A, B> & Marked<NoInfer<Access>>;
|
|
103
|
+
resolved: typeof resolved;
|
|
89
104
|
};
|
|
105
|
+
/**
|
|
106
|
+
* A value the app computes after the query runs, for numbers one statement cannot give:
|
|
107
|
+
* another database, a forecast, a per-day average. `uses` names the measures of this dataset
|
|
108
|
+
* the resolver reads from each row; they are fetched even when the query does not ask for
|
|
109
|
+
* them. The code goes in `createAnalytics({ resolvers })`. Queries cannot sort by it, since
|
|
110
|
+
* the database limits the rows before the value exists.
|
|
111
|
+
*/
|
|
112
|
+
declare function resolved<R extends MeasureResult, Access extends string = never>(options: Described<Access> & {
|
|
113
|
+
result: R;
|
|
114
|
+
uses?: undefined;
|
|
115
|
+
}): ResolvedMeasure<R, never> & Marked<NoInfer<Access>>;
|
|
116
|
+
declare function resolved<R extends MeasureResult, const U extends string, Access extends string = never>(options: Described<Access> & {
|
|
117
|
+
result: R;
|
|
118
|
+
uses: readonly U[];
|
|
119
|
+
}): ResolvedMeasure<R, U> & Marked<NoInfer<Access>>;
|
|
90
120
|
/**
|
|
91
121
|
* How a dimension filters:
|
|
92
122
|
* - `closed`: a fixed set of keys (enum, computed) - `in`, `notIn`, `isEmpty`, `isNotEmpty`;
|
|
@@ -274,13 +304,26 @@ export type Dataset<K extends string = string, M extends Record<string, AnyMeasu
|
|
|
274
304
|
readonly dimensions: Dm;
|
|
275
305
|
};
|
|
276
306
|
export type AnyDataset = Dataset<string, any, any, string>;
|
|
277
|
-
/**
|
|
307
|
+
/** Measures the database computes: everything but resolved measures. */
|
|
308
|
+
export type SqlMeasureKey<M> = {
|
|
309
|
+
[Key in keyof M & string]: M[Key] extends {
|
|
310
|
+
kind: 'resolved';
|
|
311
|
+
} ? never : Key;
|
|
312
|
+
}[keyof M & string];
|
|
313
|
+
/**
|
|
314
|
+
* Ratio measures may only name other SQL measures of the same dataset, and resolved measures
|
|
315
|
+
* may only use SQL measures.
|
|
316
|
+
*/
|
|
278
317
|
type CheckMeasures<M> = {
|
|
279
318
|
[Key in keyof M]: M[Key] extends {
|
|
280
319
|
kind: 'ratio';
|
|
281
320
|
} ? {
|
|
282
|
-
numerator: Exclude<
|
|
283
|
-
denominator: Exclude<
|
|
321
|
+
numerator: Exclude<SqlMeasureKey<M>, Key>;
|
|
322
|
+
denominator: Exclude<SqlMeasureKey<M>, Key>;
|
|
323
|
+
} : M[Key] extends {
|
|
324
|
+
kind: 'resolved';
|
|
325
|
+
} ? {
|
|
326
|
+
uses: readonly SqlMeasureKey<M>[];
|
|
284
327
|
} : unknown;
|
|
285
328
|
};
|
|
286
329
|
/** Time dimensions may only name declared time fields. */
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
import { type CompiledStatement, type Dialect } from '../compile.js';
|
|
2
|
+
/**
|
|
3
|
+
* ClickHouse: typed `{pN:Type}` parameters (pass them with `toQueryParams`), backquoted
|
|
4
|
+
* identifiers, `-If` and `-OrNull` aggregates, and buckets in the tenant's timezone. Time
|
|
5
|
+
* columns are `DateTime` or `DateTime64`, which hold an instant, so no timestamp option is
|
|
6
|
+
* needed.
|
|
7
|
+
*
|
|
8
|
+
* Correlated subqueries are not supported, so `exists` and filters on many-to-many dimensions
|
|
9
|
+
* (mark them `groupOnly`) fail when `createAnalytics` compiles the dataset.
|
|
10
|
+
*/
|
|
11
|
+
export declare function clickhouseDialect(): Dialect;
|
|
12
|
+
/**
|
|
13
|
+
* The compiled values as the `query_params` the ClickHouse client takes: `{ p1, p2, ... }`.
|
|
14
|
+
*
|
|
15
|
+
* ```ts
|
|
16
|
+
* execute: async (statement) => {
|
|
17
|
+
* const result = await clickhouse.query({
|
|
18
|
+
* query: statement.text,
|
|
19
|
+
* query_params: toQueryParams(statement),
|
|
20
|
+
* format: 'JSONEachRow'
|
|
21
|
+
* })
|
|
22
|
+
* return result.json()
|
|
23
|
+
* }
|
|
24
|
+
* ```
|
|
25
|
+
*/
|
|
26
|
+
export declare function toQueryParams(statement: CompiledStatement): Record<string, unknown>;
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
import {
|
|
2
|
+
CompileError2,
|
|
3
|
+
decodeNumber2,
|
|
4
|
+
renderStatement2
|
|
5
|
+
} from "../index-nrjdssft.js";
|
|
6
|
+
import {
|
|
7
|
+
walk
|
|
8
|
+
} from "../index-e6k30sfp.js";
|
|
9
|
+
|
|
10
|
+
// src/dialects/clickhouse.ts
|
|
11
|
+
var SETTINGS = "SETTINGS join_use_nulls = 1, enable_positional_arguments = 1";
|
|
12
|
+
function clickhouseDialect() {
|
|
13
|
+
const zone = (expr, timezone) => `toTimeZone(${expr}, '${timezone.replaceAll("'", "\\'")}')`;
|
|
14
|
+
const renderer = {
|
|
15
|
+
name: "clickhouse",
|
|
16
|
+
quote: (identifier) => `\`${identifier.replaceAll("\\", "\\\\").replaceAll("`", "\\`")}\``,
|
|
17
|
+
param(value, index) {
|
|
18
|
+
const { type, bound } = typedParam(value);
|
|
19
|
+
return { placeholder: `{p${index}:${type}}`, value: bound };
|
|
20
|
+
},
|
|
21
|
+
aggregate(fn, arg, where) {
|
|
22
|
+
switch (fn) {
|
|
23
|
+
case "count":
|
|
24
|
+
return where ? `countIf(${where})` : "count()";
|
|
25
|
+
case "countDistinct":
|
|
26
|
+
return where ? `uniqExactIf(${arg}, ${where})` : `uniqExact(${arg})`;
|
|
27
|
+
case "sum":
|
|
28
|
+
case "avg": {
|
|
29
|
+
const args = where ? `${arg}, ${where}` : arg;
|
|
30
|
+
return `${fn}${where ? "If" : ""}OrNull(${args})`;
|
|
31
|
+
}
|
|
32
|
+
case "median":
|
|
33
|
+
return where ? `quantileExactInclusiveIfOrNull(0.5)(${arg}, ${where})` : `quantileExactInclusiveOrNull(0.5)(${arg})`;
|
|
34
|
+
}
|
|
35
|
+
},
|
|
36
|
+
divide: (left, right) => `(toFloat64(${left}) / nullIf(${right}, 0))`,
|
|
37
|
+
toText: (expr) => `toString(${expr})`,
|
|
38
|
+
timeBucket(unit, expr, timezone) {
|
|
39
|
+
const local = zone(expr, timezone);
|
|
40
|
+
switch (unit) {
|
|
41
|
+
case "day":
|
|
42
|
+
return `toString(toDate(${local}))`;
|
|
43
|
+
case "week":
|
|
44
|
+
return `toString(toMonday(${local}))`;
|
|
45
|
+
case "month":
|
|
46
|
+
return `toString(toStartOfMonth(${local}))`;
|
|
47
|
+
}
|
|
48
|
+
},
|
|
49
|
+
timePart: (part, expr, timezone) => part === "isodow" ? `toDayOfWeek(${zone(expr, timezone)})` : `toHour(${zone(expr, timezone)})`
|
|
50
|
+
};
|
|
51
|
+
return {
|
|
52
|
+
name: "clickhouse",
|
|
53
|
+
compile(statement) {
|
|
54
|
+
rejectCorrelatedSubqueries(statement);
|
|
55
|
+
const compiled = renderStatement2(statement, renderer);
|
|
56
|
+
return { ...compiled, text: `${compiled.text}
|
|
57
|
+
${SETTINGS}` };
|
|
58
|
+
},
|
|
59
|
+
decodeNumber: decodeNumber2
|
|
60
|
+
};
|
|
61
|
+
}
|
|
62
|
+
function toQueryParams(statement) {
|
|
63
|
+
return Object.fromEntries(statement.values.map((value, index) => [`p${index + 1}`, value]));
|
|
64
|
+
}
|
|
65
|
+
function typedParam(value) {
|
|
66
|
+
if (value === null)
|
|
67
|
+
return { type: "Nullable(String)", bound: null };
|
|
68
|
+
if (value instanceof Date) {
|
|
69
|
+
const utc = value.toISOString().replace("T", " ").replace("Z", "");
|
|
70
|
+
return { type: "DateTime64(3, 'UTC')", bound: utc };
|
|
71
|
+
}
|
|
72
|
+
switch (typeof value) {
|
|
73
|
+
case "string":
|
|
74
|
+
return { type: "String", bound: value };
|
|
75
|
+
case "boolean":
|
|
76
|
+
return { type: "Bool", bound: value };
|
|
77
|
+
case "number":
|
|
78
|
+
return {
|
|
79
|
+
type: Number.isSafeInteger(value) ? "Int64" : "Float64",
|
|
80
|
+
bound: value
|
|
81
|
+
};
|
|
82
|
+
}
|
|
83
|
+
}
|
|
84
|
+
function rejectCorrelatedSubqueries(statement) {
|
|
85
|
+
const nodes = [
|
|
86
|
+
...statement.select.map((item) => item.node),
|
|
87
|
+
...statement.where,
|
|
88
|
+
...statement.virtualJoins.map((join) => join.on)
|
|
89
|
+
];
|
|
90
|
+
for (const node of nodes) {
|
|
91
|
+
walk(node, (inner) => {
|
|
92
|
+
if (inner.kind === "exists") {
|
|
93
|
+
throw new CompileError2(`ClickHouse cannot run the correlated EXISTS that ${inner.table.name} needs: avoid exists() and mark many-to-many dimensions groupOnly`);
|
|
94
|
+
}
|
|
95
|
+
});
|
|
96
|
+
}
|
|
97
|
+
}
|
|
98
|
+
export {
|
|
99
|
+
clickhouseDialect,
|
|
100
|
+
toQueryParams
|
|
101
|
+
};
|
package/dist/handler.js
CHANGED
|
@@ -108,7 +108,8 @@ function datasetQuerySchema(dataset) {
|
|
|
108
108
|
throw new Error(`Dataset ${dataset.key} has no measures`);
|
|
109
109
|
const groupKeys = nonEmpty(groupable.map(([key]) => key));
|
|
110
110
|
const timeKeys = nonEmpty(Object.keys(dataset.time));
|
|
111
|
-
const
|
|
111
|
+
const sqlMeasureKeys = measures.filter(([, entry]) => entry.kind !== "resolved").map(([key]) => key);
|
|
112
|
+
const sortKeys = nonEmpty([...sqlMeasureKeys, ...groupKeys ?? []]) ?? measureKeys;
|
|
112
113
|
const notes = groupable.filter(([, dim]) => dim.kind === "manyToMany" && dim.note).map(([key, dim]) => `${key}: ${dim.kind === "manyToMany" ? dim.note : ""}`);
|
|
113
114
|
const [firstFilter, ...otherFilters] = filterable.map(([key, dim]) => filterSchema(key, dim));
|
|
114
115
|
return z.object({
|
|
@@ -142,6 +143,14 @@ function datasetQuerySchema(dataset) {
|
|
|
142
143
|
message: `Sort by one of the asked measures or grouped dimensions: ${returned.join(", ")}`
|
|
143
144
|
});
|
|
144
145
|
}
|
|
146
|
+
const sortMeasure = by === undefined ? undefined : dataset.measures[by];
|
|
147
|
+
if (sortMeasure?.kind === "resolved") {
|
|
148
|
+
context.addIssue({
|
|
149
|
+
code: "custom",
|
|
150
|
+
path: ["sort", "by"],
|
|
151
|
+
message: `${by} is computed after the query runs and cannot be sorted by`
|
|
152
|
+
});
|
|
153
|
+
}
|
|
145
154
|
const period = query.period ?? dataset.defaultPeriod;
|
|
146
155
|
if (query.compareToPrevious && "all" in period) {
|
|
147
156
|
context.addIssue({
|
|
@@ -256,7 +265,22 @@ function measureNode(dataset, key) {
|
|
|
256
265
|
left: measureNode(dataset, entry.numerator),
|
|
257
266
|
right: measureNode(dataset, entry.denominator)
|
|
258
267
|
};
|
|
268
|
+
case "resolved":
|
|
269
|
+
throw new Error(`Dataset ${dataset.key}: measure ${key} is resolved by the app, not the database`);
|
|
270
|
+
}
|
|
271
|
+
}
|
|
272
|
+
function sqlMeasures(dataset, keys) {
|
|
273
|
+
const result = new Set;
|
|
274
|
+
for (const key of keys) {
|
|
275
|
+
const entry = measureOf(dataset, key);
|
|
276
|
+
if (entry.kind === "resolved") {
|
|
277
|
+
for (const used of entry.uses)
|
|
278
|
+
result.add(used);
|
|
279
|
+
} else {
|
|
280
|
+
result.add(key);
|
|
281
|
+
}
|
|
259
282
|
}
|
|
283
|
+
return [...result];
|
|
260
284
|
}
|
|
261
285
|
function dimensionNodes(dataset, key, timezone) {
|
|
262
286
|
const entry = dimensionOf(dataset, key);
|
|
@@ -410,12 +434,19 @@ function planQuery(dataset, query, context, grouped) {
|
|
|
410
434
|
select.push({ alias: columnAlias.groupLabel(key), node: nodes.label });
|
|
411
435
|
}
|
|
412
436
|
}
|
|
413
|
-
|
|
437
|
+
const measureKeys = sqlMeasures(dataset, query.measures);
|
|
438
|
+
for (const key of measureKeys) {
|
|
414
439
|
select.push({
|
|
415
440
|
alias: columnAlias.measure(key),
|
|
416
441
|
node: measureNode(dataset, key)
|
|
417
442
|
});
|
|
418
443
|
}
|
|
444
|
+
if (select.length === 0) {
|
|
445
|
+
select.push({
|
|
446
|
+
alias: "row_count",
|
|
447
|
+
node: { kind: "aggregate", fn: "count", arg: null, where: null }
|
|
448
|
+
});
|
|
449
|
+
}
|
|
419
450
|
const where = [
|
|
420
451
|
...baseWhere(dataset, context.tenant, context.period && {
|
|
421
452
|
time: query.time,
|
|
@@ -442,7 +473,10 @@ function orderBy(dataset, query, select) {
|
|
|
442
473
|
const kind = dimensionOf(dataset, key).kind;
|
|
443
474
|
return kind === "timeBucket" || kind === "timePart";
|
|
444
475
|
});
|
|
445
|
-
const sort = query.sort ?? (timeKey ? { by: timeKey, direction: "asc" } : {
|
|
476
|
+
const sort = query.sort ?? (timeKey ? { by: timeKey, direction: "asc" } : {
|
|
477
|
+
by: sqlMeasures(dataset, query.measures)[0] ?? "",
|
|
478
|
+
direction: "desc"
|
|
479
|
+
});
|
|
446
480
|
const sortIndex = sort.by in dataset.measures ? indexOf(columnAlias.measure(sort.by)) : indexOf(columnAlias.groupKey(sort.by));
|
|
447
481
|
if (sortIndex >= 0)
|
|
448
482
|
order.push({ index: sortIndex, direction: sort.direction });
|
|
@@ -546,8 +580,11 @@ function labelOf(dimension, key, rawLabel) {
|
|
|
546
580
|
function shapeMeasures(dataset, measures, raw, dialect) {
|
|
547
581
|
const values = {};
|
|
548
582
|
for (const key of measures) {
|
|
583
|
+
const entry = measureOf(dataset, key);
|
|
584
|
+
if (entry.kind === "resolved")
|
|
585
|
+
continue;
|
|
549
586
|
const value = dialect.decodeNumber(raw[columnAlias.measure(key)]);
|
|
550
|
-
values[key] =
|
|
587
|
+
values[key] = entry.result === "count" ? value ?? 0 : value;
|
|
551
588
|
}
|
|
552
589
|
return values;
|
|
553
590
|
}
|
|
@@ -566,7 +603,7 @@ function shapeRows(dataset, query, rawRows, dialect) {
|
|
|
566
603
|
}
|
|
567
604
|
return { ...row, ...shapeMeasures(dataset, query.measures, raw, dialect) };
|
|
568
605
|
});
|
|
569
|
-
return { rows, truncated };
|
|
606
|
+
return { rows, kept, truncated };
|
|
570
607
|
}
|
|
571
608
|
function resultNotes(dataset, query) {
|
|
572
609
|
const notes = [];
|
|
@@ -613,6 +650,25 @@ function createAnalytics2(options) {
|
|
|
613
650
|
}
|
|
614
651
|
restrictedOf.set(dataset.key, restricted);
|
|
615
652
|
}
|
|
653
|
+
const resolvers = options.resolvers ?? {};
|
|
654
|
+
for (const [datasetKey, entries] of Object.entries(resolvers)) {
|
|
655
|
+
const dataset = byKey.get(datasetKey);
|
|
656
|
+
if (!dataset)
|
|
657
|
+
throw new Error(`Resolvers name unknown dataset ${datasetKey}`);
|
|
658
|
+
for (const key of Object.keys(entries ?? {})) {
|
|
659
|
+
const entry = dataset.measures[key];
|
|
660
|
+
if (entry?.kind !== "resolved") {
|
|
661
|
+
throw new Error(`Resolvers name ${datasetKey}.${key}, which is not a resolved measure`);
|
|
662
|
+
}
|
|
663
|
+
}
|
|
664
|
+
}
|
|
665
|
+
for (const dataset of datasets) {
|
|
666
|
+
for (const [key, entry] of Object.entries(dataset.measures)) {
|
|
667
|
+
if (entry.kind === "resolved" && !resolvers[dataset.key]?.[key]) {
|
|
668
|
+
throw new Error(`Dataset ${dataset.key} resolves ${key} in the app, but createAnalytics has no resolver for it`);
|
|
669
|
+
}
|
|
670
|
+
}
|
|
671
|
+
}
|
|
616
672
|
const now = options.now ?? (() => new Date);
|
|
617
673
|
const querySchemas = new Map;
|
|
618
674
|
const resultSchemas = new Map;
|
|
@@ -733,12 +789,13 @@ function createAnalytics2(options) {
|
|
|
733
789
|
const previous = query.compareToPrevious && period ? previousPeriod(period) : null;
|
|
734
790
|
return { dataset, query, tenant, timezone, period, previous, now: at };
|
|
735
791
|
};
|
|
736
|
-
const runQuery = async (prepared) => {
|
|
792
|
+
const runQuery = async (prepared, ctx) => {
|
|
737
793
|
const { dataset, query, tenant, timezone, period, previous, now } = prepared;
|
|
738
794
|
const source = sourceFor(dataset);
|
|
739
795
|
const { dialect } = source;
|
|
740
796
|
const run = (statement) => source.execute(dialect.compile(statement));
|
|
741
797
|
const resolvedValues = await resolveOpenValues(dataset, query, tenant, now, run, dialect);
|
|
798
|
+
const range = ({ from, to }) => ({ from, to, timezone });
|
|
742
799
|
const runPeriod = async (target) => {
|
|
743
800
|
const statements = periodStatements(dataset, query, {
|
|
744
801
|
tenant,
|
|
@@ -751,12 +808,46 @@ function createAnalytics2(options) {
|
|
|
751
808
|
run(statements.rows),
|
|
752
809
|
statements.totals ? run(statements.totals) : null
|
|
753
810
|
]);
|
|
754
|
-
const { rows, truncated } = shapeRows(dataset, query, rawRows, dialect);
|
|
811
|
+
const { rows, kept, truncated } = shapeRows(dataset, query, rawRows, dialect);
|
|
755
812
|
const totalsRow = (rawTotals ?? rawRows)[0] ?? {};
|
|
756
813
|
const totals = shapeMeasures(dataset, query.measures, totalsRow, dialect);
|
|
814
|
+
for (const key of query.measures) {
|
|
815
|
+
const entry = dataset.measures[key];
|
|
816
|
+
if (entry?.kind !== "resolved")
|
|
817
|
+
continue;
|
|
818
|
+
const used = (raw) => shapeMeasures(dataset, entry.uses, raw, dialect);
|
|
819
|
+
const resolver = resolvers[dataset.key]?.[key];
|
|
820
|
+
if (!resolver)
|
|
821
|
+
throw new Error(`No resolver for ${dataset.key}.${key}`);
|
|
822
|
+
const output = await resolver({
|
|
823
|
+
rows: kept.map((raw, index) => ({
|
|
824
|
+
dimensions: Object.fromEntries(query.groupBy.map((group) => [group, rows[index]?.[group]])),
|
|
825
|
+
values: used(raw)
|
|
826
|
+
})),
|
|
827
|
+
totals: used(totalsRow),
|
|
828
|
+
groupBy: query.groupBy,
|
|
829
|
+
filters: query.filters.map((filter) => {
|
|
830
|
+
const resolved = resolvedValues.get(filter.dimension);
|
|
831
|
+
return resolved && "values" in filter ? { ...filter, values: [...resolved] } : filter;
|
|
832
|
+
}),
|
|
833
|
+
period: target ? range(target) : { all: true, timezone },
|
|
834
|
+
tenant,
|
|
835
|
+
ctx
|
|
836
|
+
});
|
|
837
|
+
if (output.rows.length !== rows.length) {
|
|
838
|
+
throw new Error(`The resolver for ${dataset.key}.${key} returned ${output.rows.length} values for ${rows.length} rows`);
|
|
839
|
+
}
|
|
840
|
+
const value = (raw) => {
|
|
841
|
+
const finite = typeof raw === "number" && Number.isFinite(raw) ? raw : null;
|
|
842
|
+
return entry.result === "count" ? finite ?? 0 : finite;
|
|
843
|
+
};
|
|
844
|
+
rows.forEach((row, index) => {
|
|
845
|
+
row[key] = value(output.rows[index]);
|
|
846
|
+
});
|
|
847
|
+
totals[key] = value(output.totals);
|
|
848
|
+
}
|
|
757
849
|
return { rows, totals, truncated };
|
|
758
850
|
};
|
|
759
|
-
const range = ({ from, to }) => ({ from, to, timezone });
|
|
760
851
|
const [current, before] = await Promise.all([
|
|
761
852
|
runPeriod(period),
|
|
762
853
|
previous && runPeriod(previous).then(({ rows, totals }) => ({
|
|
@@ -814,7 +905,7 @@ function createAnalytics2(options) {
|
|
|
814
905
|
if (key)
|
|
815
906
|
cache = result ? "hit" : "miss";
|
|
816
907
|
if (!result) {
|
|
817
|
-
result = await runQuery(prepared);
|
|
908
|
+
result = await runQuery(prepared, ctx);
|
|
818
909
|
if (key)
|
|
819
910
|
store(key, result, entry);
|
|
820
911
|
}
|
|
@@ -983,7 +1074,18 @@ function restrictedFields(dataset) {
|
|
|
983
1074
|
if (!entry)
|
|
984
1075
|
return [];
|
|
985
1076
|
const own = entry.access === undefined ? [] : [entry.access];
|
|
986
|
-
|
|
1077
|
+
switch (entry.kind) {
|
|
1078
|
+
case "ratio":
|
|
1079
|
+
return [
|
|
1080
|
+
...own,
|
|
1081
|
+
...accessOf(entry.numerator),
|
|
1082
|
+
...accessOf(entry.denominator)
|
|
1083
|
+
];
|
|
1084
|
+
case "resolved":
|
|
1085
|
+
return [...own, ...entry.uses.flatMap(accessOf)];
|
|
1086
|
+
default:
|
|
1087
|
+
return own;
|
|
1088
|
+
}
|
|
987
1089
|
};
|
|
988
1090
|
return [
|
|
989
1091
|
...Object.keys(measures).map((key) => ({
|