@malloydata/cli 0.0.53 → 0.0.54-next.2026.5.1.cb0227a
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli.js +639 -481
- package/dist/index.js +2 -2
- package/dist/skills/log-analysis-to-notebook.md +98 -0
- package/dist/skills/malloy-language-reference.md +528 -0
- package/dist/skills/writing-malloy-with-mcp.md +84 -0
- package/dist/third_party_notices.txt +7923 -6184
- package/package.json +14 -10
package/dist/index.js
CHANGED
|
@@ -1,3 +1,3 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
process.MALLOY_CLI_VERSION="0.0.
|
|
3
|
-
require('./cli.js')
|
|
2
|
+
process.MALLOY_CLI_VERSION="0.0.54-next.2026.5.1.cb0227a";
|
|
3
|
+
require('./cli.js')
|
|
@@ -0,0 +1,98 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: How to append a Malloy query and its interpretation to a .malloynb notebook file. Follow this when the user says "let's log this", "save this", "add this to the notebook", or similar.
|
|
3
|
+
---
|
|
4
|
+
# Logging an Analysis to a .malloynb Notebook
|
|
5
|
+
|
|
6
|
+
## When to use this skill
|
|
7
|
+
|
|
8
|
+
Trigger phrases: "let's log this", "save this to the notebook", "add this to the file", "log that", "add that to the analysis", or any request to persist a query and its results to a notebook.
|
|
9
|
+
|
|
10
|
+
## Step 1 — Identify the target file
|
|
11
|
+
|
|
12
|
+
If the user has not specified a `.malloynb` file in this conversation, ask:
|
|
13
|
+
|
|
14
|
+
> "Which `.malloynb` file should I append this to?"
|
|
15
|
+
|
|
16
|
+
Do not proceed until you have a file path.
|
|
17
|
+
|
|
18
|
+
## Step 2 — Read the file to understand its current state
|
|
19
|
+
|
|
20
|
+
Before appending, read the file so you know:
|
|
21
|
+
- Whether the first cell already contains the experimental flags and import (it should — don't add them again).
|
|
22
|
+
- The existing section structure, so your new section follows the same style.
|
|
23
|
+
|
|
24
|
+
## Step 3 — Append the new section
|
|
25
|
+
|
|
26
|
+
**Always append** to the end of the file. Never rewrite existing content.
|
|
27
|
+
|
|
28
|
+
Each logged analysis consists of these cells in order:
|
|
29
|
+
|
|
30
|
+
### 1. User question cell
|
|
31
|
+
The user's question, verbatim, as a blockquote. Use a backslash before `>` so it renders as a blockquote, not a Malloy cell marker:
|
|
32
|
+
|
|
33
|
+
```
|
|
34
|
+
>>>markdown
|
|
35
|
+
\> What are the top directors by total ratings?
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
### 2. Section header + description cell
|
|
39
|
+
A `##` heading and one sentence describing what the query does:
|
|
40
|
+
|
|
41
|
+
```
|
|
42
|
+
>>>markdown
|
|
43
|
+
## Top Directors by Total Ratings
|
|
44
|
+
Ranked by sum of vote counts (in thousands) across all their films.
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
### 3. Query cell
|
|
48
|
+
The Malloy query **without** an import statement (the first cell in the file already imports the model):
|
|
49
|
+
|
|
50
|
+
```
|
|
51
|
+
>>>malloy
|
|
52
|
+
run: movies -> top_directors
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### 4. Interpretation cell
|
|
56
|
+
Bullet-pointed markdown analysis of the results. Lead with the most interesting finding. Use `**bold**` for names and key numbers:
|
|
57
|
+
|
|
58
|
+
```
|
|
59
|
+
>>>markdown
|
|
60
|
+
- **Christopher Nolan** leads with 18M total votes across 12 films.
|
|
61
|
+
- **Spielberg / Williams** is the most loyal partnership at 85% of films.
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
## Cell format rules
|
|
65
|
+
|
|
66
|
+
- Cell delimiters are `>>>malloy` and `>>>markdown` on their own line — no trailing spaces.
|
|
67
|
+
- No blank lines between the delimiter and the cell content.
|
|
68
|
+
- No blank lines between cells (the next `>>>` starts immediately).
|
|
69
|
+
- Do **not** add `>>>malloy` cells for imports or flags — only for executable queries.
|
|
70
|
+
- Lists and bold in markdown cells render correctly; use them freely for analysis.
|
|
71
|
+
|
|
72
|
+
## Example — what a complete logged section looks like
|
|
73
|
+
|
|
74
|
+
```
|
|
75
|
+
>>>markdown
|
|
76
|
+
\> Can you find composers that work with the top directors?
|
|
77
|
+
>>>markdown
|
|
78
|
+
## Directors and Their Composer Collaborators
|
|
79
|
+
For each top director, shows the composers they work with most frequently.
|
|
80
|
+
>>>malloy
|
|
81
|
+
run: movies -> {
|
|
82
|
+
where: job = 'director'
|
|
83
|
+
group_by: director is name
|
|
84
|
+
aggregate: total_ratings, title_count
|
|
85
|
+
nest: composers is {
|
|
86
|
+
group_by: principals2.people.primaryName
|
|
87
|
+
where: principals2.category = 'composer'
|
|
88
|
+
aggregate: title_count, percent_of_titles
|
|
89
|
+
limit: 5
|
|
90
|
+
}
|
|
91
|
+
order_by: total_ratings desc
|
|
92
|
+
limit: 20
|
|
93
|
+
}
|
|
94
|
+
>>>markdown
|
|
95
|
+
- **Most loyal partnerships** by % of films: Zemeckis/Silvestri (91%), Coen/Burwell (89%), Spielberg/Williams (85%).
|
|
96
|
+
- **Hans Zimmer** is the most in-demand, appearing across six top directors.
|
|
97
|
+
- **Martin Scorsese** is the most eclectic — 31 films, top composer at only 16%.
|
|
98
|
+
```
|
|
@@ -0,0 +1,528 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Malloy language reference — concepts, syntax, compilation model. Load this before writing or reviewing Malloy code.
|
|
3
|
+
---
|
|
4
|
+
# Malloy Language Reference
|
|
5
|
+
|
|
6
|
+
Malloy is a semantic data modeling and query language. It compiles to SQL and runs against existing database engines (DuckDB, BigQuery, Snowflake, PostgreSQL, MySQL, Trino, Presto). It is not a SQL wrapper or abstraction layer — it has its own type system, scoping rules, expression semantics, and compilation pipeline.
|
|
7
|
+
|
|
8
|
+
Malloy is designed around how humans think about data, not how data computations are mechanically accomplished. SQL is oriented around the machine — you specify joins, group-by columns, subqueries, and window functions in terms of what the database needs to do. Malloy is oriented around the analyst — you describe relationships, name computations, and compose questions in terms of what the data means. Malloy bridges the gap between these two by compiling the human-oriented description into correct, efficient SQL.
|
|
9
|
+
|
|
10
|
+
A core design principle is that **most queries are themselves designing a new semantic model.** Formulating a question about data — choosing what to group by, what to aggregate, what to nest — is inherently an act of defining a new way to look at that data. Malloy is built around this idea: the output of every query is not just a result set but a new source with its own schema, and data comprehension is an ongoing iterative process where later stages want not only the data from a previous stage but how that data came into being. This is why query output carries metadata, why queries can be used as sources, and why views and pipelines compose naturally.
|
|
11
|
+
|
|
12
|
+
## Documents and Statements
|
|
13
|
+
|
|
14
|
+
A Malloy file (`.malloy`) is a sequence of statements, optionally separated by semicolons. There are four statement types:
|
|
15
|
+
|
|
16
|
+
- **`import`** — import sources and queries from another `.malloy` file
|
|
17
|
+
- **`source:`** — define a named, reusable data source with its schema and extensions
|
|
18
|
+
- **`query:`** — define a named query (source + view) for reuse
|
|
19
|
+
- **`run:`** — execute a query (the "do it now" statement)
|
|
20
|
+
|
|
21
|
+
```malloy
|
|
22
|
+
import "shared_model.malloy"
|
|
23
|
+
|
|
24
|
+
source: flights is duckdb.table('flights.parquet') extend {
|
|
25
|
+
measure: flight_count is count()
|
|
26
|
+
}
|
|
27
|
+
|
|
28
|
+
query: carrier_summary is flights -> {
|
|
29
|
+
group_by: carrier
|
|
30
|
+
aggregate: flight_count
|
|
31
|
+
}
|
|
32
|
+
|
|
33
|
+
run: carrier_summary
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Comments use `//` or `--` (both are line comments).
|
|
37
|
+
|
|
38
|
+
## Sources
|
|
39
|
+
|
|
40
|
+
A **source** is anything you can hand a SQL database and get a schema back — a table name, a SQL SELECT, or the output of another Malloy query. The columns in that schema become the source's initial fields (all dimensions).
|
|
41
|
+
|
|
42
|
+
```malloy
|
|
43
|
+
source: flights is duckdb.table('flights.parquet')
|
|
44
|
+
source: limited is duckdb.sql("""SELECT * FROM flights LIMIT 100""")
|
|
45
|
+
source: carrier_facts is carrier_summary -- a query used as a source
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
What makes sources central to Malloy is **extension**. The `extend` block lets you layer on dimensions, measures, views, joins, filters, primary keys, field restrictions, and renames. These extensions travel with the source — any query against it gets them for free.
|
|
49
|
+
|
|
50
|
+
```malloy
|
|
51
|
+
source: flights is duckdb.table('flights.parquet') extend {
|
|
52
|
+
primary_key: id
|
|
53
|
+
|
|
54
|
+
dimension: distance_km is distance * 1.609344
|
|
55
|
+
|
|
56
|
+
measure:
|
|
57
|
+
flight_count is count()
|
|
58
|
+
total_distance is sum(distance)
|
|
59
|
+
|
|
60
|
+
join_one: carriers with carrier
|
|
61
|
+
join_one: origin_airport is airports on origin_airport.code = origin
|
|
62
|
+
|
|
63
|
+
where: dep_time > @2001
|
|
64
|
+
|
|
65
|
+
view: by_carrier is {
|
|
66
|
+
group_by: carrier
|
|
67
|
+
aggregate: flight_count, total_distance
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Sources can extend other sources, creating a refinement chain:
|
|
73
|
+
|
|
74
|
+
```malloy
|
|
75
|
+
source: ca_flights is flights extend {
|
|
76
|
+
where: origin.state = 'CA'
|
|
77
|
+
}
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Field access control uses `accept:` (allowlist) or `except:` (denylist) to restrict which inherited columns are visible. Fields can be renamed with `rename: new_name is old_name`.
|
|
81
|
+
|
|
82
|
+
## Joins
|
|
83
|
+
|
|
84
|
+
Joins are declared in the source, not reconstructed in every query. This is a fundamental design difference from SQL: the graph structure of your data is a property of the model.
|
|
85
|
+
|
|
86
|
+
```malloy
|
|
87
|
+
join_one: carriers with carrier -- FK → PK shorthand
|
|
88
|
+
join_one: origin_airport is airports on origin_airport.code = origin -- explicit ON
|
|
89
|
+
join_many: line_items on line_items.order_id = id -- one-to-many
|
|
90
|
+
join_cross: other_table on other_table.key = key -- cross join
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
- `join_one` — the joined source has at most one row per source row (many-to-one or one-to-one)
|
|
94
|
+
- `join_many` — the joined source has potentially many rows per source row
|
|
95
|
+
- `join_cross` — a full cross product
|
|
96
|
+
|
|
97
|
+
The `with` shorthand requires the joined source to have a declared `primary_key`. All joins are left outer by default. There is no right join — Malloy's graph model doesn't need one.
|
|
98
|
+
|
|
99
|
+
When you reference a joined source's fields, you use dot notation: `carriers.nickname`, `origin_airport.state`. This is one of Malloy's most important abstractions: **the access path to nested data is identical regardless of how the nesting is physically stored.** An array of records embedded in a column, a `join_many` to a separate table, a record-typed column — all are navigated with the same dot notation. The SQL required to traverse these different physical arrangements varies wildly (unnesting arrays, LEFT JOINs, correlated subqueries, ARRAY_AGG), but Malloy hides all of that. You think about the logical shape of your data — "flights have carriers, carriers have a nickname" — and write `carriers.nickname`. The compiler figures out what SQL is needed to get there. This means you can restructure your physical schema (normalize a nested array into a separate table, or denormalize a joined table into a record column) without changing any of the Malloy that references that data.
|
|
100
|
+
|
|
101
|
+
## Fields
|
|
102
|
+
|
|
103
|
+
Malloy has four kinds of fields: **dimensions**, **measures**, **views**, and **calculations**.
|
|
104
|
+
|
|
105
|
+
### Dimensions
|
|
106
|
+
|
|
107
|
+
Scalar expressions — they compute a value per row. All columns inherited from a table are dimensions. Computed dimensions reference other dimensions or columns:
|
|
108
|
+
|
|
109
|
+
```malloy
|
|
110
|
+
dimension: full_name is concat(first_name, ' ', last_name)
|
|
111
|
+
dimension: is_long_haul is distance > 1000
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
### Measures
|
|
115
|
+
|
|
116
|
+
Aggregate expressions — they compute a value across a set of rows. A field is a measure when its defining expression contains an aggregate function (`count`, `sum`, `avg`, `min`, `max`):
|
|
117
|
+
|
|
118
|
+
```malloy
|
|
119
|
+
measure:
|
|
120
|
+
flight_count is count()
|
|
121
|
+
total_distance is sum(distance)
|
|
122
|
+
avg_distance is avg(distance)
|
|
123
|
+
pct_delayed is count() { where: dep_delay > 30 } / count()
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
**`count(expr)` counts distinct values.** Unlike SQL's `COUNT(DISTINCT expr)`, Malloy uses `count(expr)` for distinct counting. The `count(distinct expr)` form is a deprecated syntax that will produce an error. Use `count()` for total row count, `count(field)` for distinct values of that field:
|
|
127
|
+
|
|
128
|
+
```malloy
|
|
129
|
+
aggregate:
|
|
130
|
+
total_rows is count() -- all rows
|
|
131
|
+
unique_carriers is count(carrier) -- distinct carriers
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
Measures can be filtered inline with `{ where: ... }`, which is how you build things like "percent of flights delayed" without subqueries.
|
|
135
|
+
|
|
136
|
+
### Views
|
|
137
|
+
|
|
138
|
+
A view is a query saved into the source — a reusable transformation:
|
|
139
|
+
|
|
140
|
+
```malloy
|
|
141
|
+
view: by_carrier is {
|
|
142
|
+
group_by: carrier
|
|
143
|
+
aggregate: flight_count, total_distance
|
|
144
|
+
limit: 10
|
|
145
|
+
}
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
Views can reference other views from the same source as a starting point, and can be extended with `+`.
|
|
149
|
+
|
|
150
|
+
### Calculations
|
|
151
|
+
|
|
152
|
+
Window functions over the grouped result. Calculations can only be defined in a query stage with `calculate:`, never in a source definition, because they depend on the output columns of the query:
|
|
153
|
+
|
|
154
|
+
```malloy
|
|
155
|
+
run: flights -> {
|
|
156
|
+
group_by: carrier
|
|
157
|
+
aggregate: flight_count
|
|
158
|
+
calculate: carrier_rank is rank()
|
|
159
|
+
}
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
## Queries and Views
|
|
163
|
+
|
|
164
|
+
A query pairs a source with a view (the transformation). Everything after the first `->` is the view.
|
|
165
|
+
|
|
166
|
+
```malloy
|
|
167
|
+
run: flights -> {
|
|
168
|
+
group_by: carrier
|
|
169
|
+
aggregate: flight_count
|
|
170
|
+
}
|
|
171
|
+
```
|
|
172
|
+
|
|
173
|
+
### Reduction vs. Projection
|
|
174
|
+
|
|
175
|
+
Each stage of a view performs exactly one of:
|
|
176
|
+
|
|
177
|
+
- **Reduction** — uses `group_by:` and/or `aggregate:` to reduce grain. Analogous to `SELECT ... GROUP BY` in SQL.
|
|
178
|
+
- **Projection** — uses `select:` to pick fields without aggregation. Analogous to `SELECT` without `GROUP BY`.
|
|
179
|
+
|
|
180
|
+
These cannot be mixed in a single stage. A stage with `group_by:` cannot have `select:`, and vice versa.
|
|
181
|
+
|
|
182
|
+
### Source-level definitions vs. query-level operations
|
|
183
|
+
|
|
184
|
+
The same `name is expression` syntax defines fields in both sources and queries:
|
|
185
|
+
|
|
186
|
+
```malloy
|
|
187
|
+
-- In a source (reusable):
|
|
188
|
+
source: flights is ... extend {
|
|
189
|
+
measure: flight_count is count() -- defines a measure in the model
|
|
190
|
+
}
|
|
191
|
+
|
|
192
|
+
-- In a query (ad hoc):
|
|
193
|
+
run: flights -> {
|
|
194
|
+
aggregate: flight_count is count() -- defines the same measure inline
|
|
195
|
+
}
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
When used in a source, `measure:` and `dimension:` are **definition statements** — they add named fields to the source's schema. When used in a query, `group_by:`, `aggregate:`, `select:`, `nest:`, and `calculate:` are **query operations** — they specify what the query does. The field definitions are syntactically identical in both contexts, but the enclosing keyword determines the role:
|
|
199
|
+
|
|
200
|
+
| Source keyword | Query keyword | What it holds |
|
|
201
|
+
|---|---|---|
|
|
202
|
+
| `dimension:` | `group_by:` or `select:` | scalar expressions |
|
|
203
|
+
| `measure:` | `aggregate:` | aggregate expressions |
|
|
204
|
+
| `view:` | `nest:` | sub-queries |
|
|
205
|
+
| _(n/a)_ | `calculate:` | window functions |
|
|
206
|
+
|
|
207
|
+
This is why `measure` and `aggregate` are separate keywords. `measure:` is a *modeling* statement — "this source has a reusable aggregate computation called X." `aggregate:` is a *query* statement — "in this query, include these aggregate values in the output." A query's `aggregate:` can reference a previously defined measure by name, or define one inline. The distinction parallels the separation between defining a dimension in a source and using it via `group_by:` in a query.
|
|
208
|
+
|
|
209
|
+
### Multi-stage Pipelines
|
|
210
|
+
|
|
211
|
+
Stages chain with `->`. Each stage's output becomes the next stage's source:
|
|
212
|
+
|
|
213
|
+
```malloy
|
|
214
|
+
run: flights -> {
|
|
215
|
+
group_by: carrier
|
|
216
|
+
aggregate: flight_count is count()
|
|
217
|
+
} -> {
|
|
218
|
+
where: flight_count > 1000
|
|
219
|
+
select: *
|
|
220
|
+
}
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
### Refinement with `+`
|
|
224
|
+
|
|
225
|
+
The refinement operator `+` merges query operations together. It works both within a view and at the top level on a named query:
|
|
226
|
+
|
|
227
|
+
```malloy
|
|
228
|
+
-- Refining a view within a query:
|
|
229
|
+
run: flights -> by_carrier + { limit: 5 } + { nest: by_destination }
|
|
230
|
+
|
|
231
|
+
-- Refining a named query at the top level:
|
|
232
|
+
run: carrier_summary + { group_by: origin } -- add origin grouping to existing query
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
When a dimension name appears as a bare reference, it expands to `{ group_by: name }`. A measure name expands to `{ aggregate: name }`:
|
|
236
|
+
|
|
237
|
+
```malloy
|
|
238
|
+
run: flights -> carrier + flight_count + { limit: 10 }
|
|
239
|
+
-- equivalent to: flights -> { group_by: carrier; aggregate: flight_count; limit: 10 }
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
For multi-stage queries, refinement semantics get more complex — but for single-stage queries, `+` straightforwardly merges operations into the stage.
|
|
243
|
+
|
|
244
|
+
### Nesting
|
|
245
|
+
|
|
246
|
+
`nest:` embeds an aggregating subquery inside a reduction. Each row of the outer query gets a subtable from the nested query. Nests can nest arbitrarily deep:
|
|
247
|
+
|
|
248
|
+
```malloy
|
|
249
|
+
run: flights -> {
|
|
250
|
+
group_by: carrier
|
|
251
|
+
aggregate: flight_count
|
|
252
|
+
nest: top_routes is {
|
|
253
|
+
group_by: origin, destination
|
|
254
|
+
aggregate: flight_count
|
|
255
|
+
limit: 3
|
|
256
|
+
}
|
|
257
|
+
}
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
### Other query operations
|
|
261
|
+
|
|
262
|
+
- **`where:`** — filter rows (pre-aggregation). Comma-separated filters are ANDed.
|
|
263
|
+
- **`having:`** — filter groups (post-aggregation), like SQL's HAVING.
|
|
264
|
+
- **`limit:`** / **`order_by:`** — limit and sort output.
|
|
265
|
+
- **`extend`** — add fields or joins to a source inline within a query expression.
|
|
266
|
+
|
|
267
|
+
## Aggregate Locality (Symmetric Aggregates)
|
|
268
|
+
|
|
269
|
+
This is one of Malloy's most important features. In SQL, when you join tables and aggregate, you risk double-counting (the "fan trap"). Malloy solves this with **aggregate locality** — you specify *where in the join graph* an aggregation should be computed.
|
|
270
|
+
|
|
271
|
+
```malloy
|
|
272
|
+
run: flights -> {
|
|
273
|
+
aggregate:
|
|
274
|
+
-- avg seats weighted by number of flights (locality: source, i.e. flights)
|
|
275
|
+
avg_seats_per_flight is source.avg(aircraft.aircraft_models.seats)
|
|
276
|
+
-- avg seats per aircraft model (locality: aircraft_models)
|
|
277
|
+
avg_seats_per_model is aircraft.aircraft_models.seats.avg()
|
|
278
|
+
}
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
Three syntactic forms:
|
|
282
|
+
|
|
283
|
+
- `avg(expr)` — aggregate at the current source (implicit locality)
|
|
284
|
+
- `joined_source.avg(expr)` — aggregate at the specified join point (explicit locality)
|
|
285
|
+
- `joined_source.field.avg()` — shorthand for aggregating the field at its parent source
|
|
286
|
+
|
|
287
|
+
For `sum` and `avg` (asymmetric aggregates), when the expression crosses a join boundary, Malloy *requires* explicit locality — it won't silently give you a wrong answer. For `min`, `max`, and `count` (symmetric), locality doesn't change the result, so implicit is always fine.
|
|
288
|
+
|
|
289
|
+
Malloy implements this with a technique called **symmetric aggregates** — it internally de-duplicates rows based on primary keys at the appropriate join level, so aggregations are always mathematically correct regardless of join fan-out.
|
|
290
|
+
|
|
291
|
+
## Ungrouped Aggregates
|
|
292
|
+
|
|
293
|
+
`all()` and `exclude()` allow computing aggregates at different grouping levels within a single query:
|
|
294
|
+
|
|
295
|
+
```malloy
|
|
296
|
+
run: airports -> {
|
|
297
|
+
group_by: state, faa_region
|
|
298
|
+
aggregate:
|
|
299
|
+
airport_count is count()
|
|
300
|
+
total_airports is all(count()) -- ungrouped: total across all rows
|
|
301
|
+
region_airports is all(count(), faa_region) -- grouped only by faa_region
|
|
302
|
+
pct_of_total is count() / all(count())
|
|
303
|
+
}
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
`all(expr)` removes all grouping. `all(expr, dim1, dim2)` keeps only the specified grouping dimensions. `exclude(expr, dim)` removes the specified dimension from grouping.
|
|
307
|
+
|
|
308
|
+
**Important:** `all(expr, dim)` takes the **local alias name** as defined in the query's `group_by:`, not a dotted path. If you want to partition by a joined field, alias it first:
|
|
309
|
+
|
|
310
|
+
```malloy
|
|
311
|
+
-- WRONG: all(count(), director.primaryName) -- dot paths don't work here
|
|
312
|
+
-- RIGHT:
|
|
313
|
+
run: movies -> {
|
|
314
|
+
group_by: director is director.primaryName -- alias it
|
|
315
|
+
aggregate:
|
|
316
|
+
movies is count()
|
|
317
|
+
director_total is all(count(), director) -- reference the alias
|
|
318
|
+
pct is count() / all(count(), director)
|
|
319
|
+
}
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
## Expressions
|
|
323
|
+
|
|
324
|
+
Malloy expressions include arithmetic, comparison, logical operators, function calls, type casts, and several Malloy-specific forms.
|
|
325
|
+
|
|
326
|
+
### Evaluation Spaces
|
|
327
|
+
|
|
328
|
+
Every expression has an evaluation space: **literal**, **constant**, **input**, or **output**. Input expressions reference source columns/dimensions. Output expressions reference the results of the current query stage (used in `calculate:`). Some functions constrain their arguments — e.g., `lag(expr)` requires an output expression, `avg(expr)` requires an input expression.
|
|
329
|
+
|
|
330
|
+
### Application and Partial Comparison
|
|
331
|
+
|
|
332
|
+
The `?` operator applies a condition to a value. Partial comparisons are conditions without a left-hand side:
|
|
333
|
+
|
|
334
|
+
```malloy
|
|
335
|
+
where: state ? 'CA' | 'NY' -- state is 'CA' or 'NY'
|
|
336
|
+
where: distance ? > 500 & < 2000 -- distance between 500 and 2000
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
`|` is alternation (OR), `&` is conjunction (AND) within partials.
|
|
340
|
+
|
|
341
|
+
### Pick Expressions
|
|
342
|
+
|
|
343
|
+
Malloy's equivalent of CASE:
|
|
344
|
+
|
|
345
|
+
```malloy
|
|
346
|
+
dimension: size_bucket is
|
|
347
|
+
pick 'short' when distance < 500
|
|
348
|
+
pick 'medium' when distance < 1500
|
|
349
|
+
else 'long'
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
### Filtered Aggregate Expressions
|
|
353
|
+
|
|
354
|
+
Any aggregate can be filtered inline:
|
|
355
|
+
|
|
356
|
+
```malloy
|
|
357
|
+
measure: ca_flights is count() { where: origin.state = 'CA' }
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
### Type Casting
|
|
361
|
+
|
|
362
|
+
```malloy
|
|
363
|
+
total_distance::string -- Malloy type cast
|
|
364
|
+
name::"VARCHAR(32)" -- database-native type cast
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
### Time Literals and Ranges
|
|
368
|
+
|
|
369
|
+
```malloy
|
|
370
|
+
@2003 -- the year 2003
|
|
371
|
+
@2003-Q2 -- second quarter of 2003
|
|
372
|
+
@2024-01-15 10:30:00 -- timestamp literal
|
|
373
|
+
dep_time ? @2003 to @2005 -- range comparison
|
|
374
|
+
now -- current timestamp
|
|
375
|
+
```
|
|
376
|
+
|
|
377
|
+
## Data Types
|
|
378
|
+
|
|
379
|
+
Malloy's type system: `string`, `number`, `boolean`, `date`, `timestamp`, `timestamptz`, `json`, and `sql native` (for unsupported database types). Compound types: `type[]` for arrays, `{ name :: type, ... }` for records, nesting arbitrarily: `{ x :: number, tags :: string[] }[]`.
|
|
380
|
+
|
|
381
|
+
## Annotations and Tags
|
|
382
|
+
|
|
383
|
+
These are related but distinct concepts.
|
|
384
|
+
|
|
385
|
+
### Annotations
|
|
386
|
+
|
|
387
|
+
Annotations are **text strings** attached to objects during compilation. They are metadata — they never affect query execution or SQL generation. An annotation starts with `#` and continues to end of line:
|
|
388
|
+
|
|
389
|
+
```malloy
|
|
390
|
+
# bar_chart
|
|
391
|
+
view: by_carrier is { ... }
|
|
392
|
+
```
|
|
393
|
+
|
|
394
|
+
- `#` annotations attach to the next object defined below them
|
|
395
|
+
- `##` annotations attach to the model (the file)
|
|
396
|
+
- Block annotations use `#|` ... `|#` for multi-line content (closing delimiter must match the column position of the opener)
|
|
397
|
+
|
|
398
|
+
Annotations distribute over definition lists:
|
|
399
|
+
|
|
400
|
+
```malloy
|
|
401
|
+
# currency
|
|
402
|
+
measure: -- all three measures get the # currency annotation
|
|
403
|
+
revenue is sum(amount)
|
|
404
|
+
# percent -- this measure also gets # percent
|
|
405
|
+
margin is revenue / cost
|
|
406
|
+
cost is sum(amount)
|
|
407
|
+
```
|
|
408
|
+
|
|
409
|
+
### Tags (a use of annotations)
|
|
410
|
+
|
|
411
|
+
Tags are the primary *consumer* of annotation strings. They interpret annotation text using a structured property language (MOTLY). The key distinction: **annotations are the transport mechanism (raw strings attached to objects), tags are the interpretation layer (parsed key-value properties).**
|
|
412
|
+
|
|
413
|
+
Not all annotations are tags. An annotation is just text. Tags are annotations that happen to be written in the tag property language and parsed by an application.
|
|
414
|
+
|
|
415
|
+
### Annotation prefixes (routing)
|
|
416
|
+
|
|
417
|
+
The character(s) immediately after `#` route the annotation to different consumers:
|
|
418
|
+
|
|
419
|
+
- `# ` (hash-space) — renderer tags, parsed by the Malloy VS Code extension for visualization
|
|
420
|
+
- `##!` — compiler directives (e.g., `##! experimental.parameters`)
|
|
421
|
+
- `#"` — reserved for documentation strings
|
|
422
|
+
- `#(appName)` — application-specific tags (e.g., `#(docs) hidden`)
|
|
423
|
+
|
|
424
|
+
```malloy
|
|
425
|
+
# bar_chart size=large -- renderer tag: tells VS Code how to render
|
|
426
|
+
##! experimental.parameters -- compiler tag: enables a feature flag
|
|
427
|
+
#(myApp) priority=high -- custom app tag: ignored by renderer/compiler
|
|
428
|
+
```
|
|
429
|
+
|
|
430
|
+
### Tag property syntax
|
|
431
|
+
|
|
432
|
+
```
|
|
433
|
+
tName -- boolean flag (exists = true)
|
|
434
|
+
tName=value -- set property value
|
|
435
|
+
tName=[a, b, c] -- array value
|
|
436
|
+
tName: { p1=v1 p2=v2 } -- nested properties (replaces)
|
|
437
|
+
tName { p1=v1 } -- nested properties (merges)
|
|
438
|
+
-tName -- delete a property
|
|
439
|
+
tName.sub.path=value -- deep path assignment
|
|
440
|
+
```
|
|
441
|
+
|
|
442
|
+
Values can be unquoted identifiers, quoted strings, numbers, or typed values prefixed with `@` (`@true`, `@false`, `@2024-01-15`).
|
|
443
|
+
|
|
444
|
+
## How a Malloy Query Becomes SQL
|
|
445
|
+
|
|
446
|
+
The compilation pipeline has two phases:
|
|
447
|
+
|
|
448
|
+
### Phase 1: Translation (source code → IR)
|
|
449
|
+
|
|
450
|
+
```
|
|
451
|
+
Malloy source → ANTLR lexer/parser → parse tree → AST builder → AST → IR generator → IR
|
|
452
|
+
```
|
|
453
|
+
|
|
454
|
+
The **Intermediate Representation (IR)** is a plain, serializable data structure (JSON-compatible) that fully describes the semantic model and query. Note that IR is *not* dialect-agnostic — the same Malloy source compiled against different databases can produce different IR, because schema information, type mappings, and available functions vary by backend. It can be cached, transmitted, and reused. Key IR types:
|
|
455
|
+
|
|
456
|
+
- **`SourceDef`** — a source's complete definition: schema, fields, joins, filters
|
|
457
|
+
- **`Query`** — a source paired with a pipeline of operations
|
|
458
|
+
- **`FieldDef`** — definition of any field (dimension, measure, join, calculation)
|
|
459
|
+
- **`Expr`** — expression tree (arithmetic, comparisons, aggregates, function calls, field references)
|
|
460
|
+
|
|
461
|
+
The translator handles all language-level semantics: scoping, name resolution, type checking, evaluation space validation.
|
|
462
|
+
|
|
463
|
+
### Phase 2: Compilation (IR → SQL)
|
|
464
|
+
|
|
465
|
+
```
|
|
466
|
+
IR → query compiler → expression compiler → dialect-specific SQL generator → SQL + metadata
|
|
467
|
+
```
|
|
468
|
+
|
|
469
|
+
The compiler walks the IR query pipeline, translating each stage into SQL constructs (CTEs, subqueries, GROUP BY, window functions). A **Dialect** layer handles database-specific SQL generation.
|
|
470
|
+
|
|
471
|
+
The compiler also produces **metadata** alongside the SQL — structural information needed to interpret the result set (column types, nesting structure, annotation data). This metadata is what allows Malloy renderers to reconstruct nested/hierarchical results from the flat SQL result set and apply visualization tags.
|
|
472
|
+
|
|
473
|
+
### Key architectural consequences
|
|
474
|
+
|
|
475
|
+
- Because the IR is serializable, it can be cached and reused across compilations (though IR is database-specific — the same source compiled against different backends may produce different IR).
|
|
476
|
+
- Because joins are declared in the source (not the query), the compiler knows the full join graph and can compute symmetric aggregates correctly.
|
|
477
|
+
- Because nested queries are first-class, the compiler generates the appropriate SQL (correlated subqueries or ARRAY_AGG patterns depending on dialect) automatically.
|
|
478
|
+
- Because measures are typed as aggregates in the IR, the compiler can validate that they only appear in aggregate context and enforce locality rules.
|
|
479
|
+
|
|
480
|
+
## Where to Go Deeper
|
|
481
|
+
|
|
482
|
+
This document is a conceptual reference — enough to reason about the language and its design, but not exhaustive. Here's where to find more detail.
|
|
483
|
+
|
|
484
|
+
### Language Documentation
|
|
485
|
+
|
|
486
|
+
The full docs live at [https://docs.malloydata.dev](https://docs.malloydata.dev). Key pages by topic:
|
|
487
|
+
|
|
488
|
+
| Topic | URL |
|
|
489
|
+
|---|---|
|
|
490
|
+
| Sources, extensions, joins, primary keys | [documentation/language/source](https://docs.malloydata.dev/documentation/language/source) |
|
|
491
|
+
| Queries, views, reduction vs projection | [documentation/language/query](https://docs.malloydata.dev/documentation/language/query), [views](https://docs.malloydata.dev/documentation/language/views) |
|
|
492
|
+
| Fields: dimensions, measures, views, calculations | [documentation/language/fields](https://docs.malloydata.dev/documentation/language/fields) |
|
|
493
|
+
| Aggregate functions and aggregate locality | [documentation/language/aggregates](https://docs.malloydata.dev/documentation/language/aggregates) |
|
|
494
|
+
| Ungrouped aggregates (`all`, `exclude`) | [documentation/language/ungrouped-aggregates](https://docs.malloydata.dev/documentation/language/ungrouped-aggregates) |
|
|
495
|
+
| Nested views / aggregating subqueries | [documentation/language/nesting](https://docs.malloydata.dev/documentation/language/nesting) |
|
|
496
|
+
| Joins | [documentation/language/join](https://docs.malloydata.dev/documentation/language/join) |
|
|
497
|
+
| Expressions, operators, pick, apply | [documentation/language/expressions](https://docs.malloydata.dev/documentation/language/expressions) |
|
|
498
|
+
| Evaluation spaces (literal, constant, input, output) | [documentation/language/eval_space](https://docs.malloydata.dev/documentation/language/eval_space) |
|
|
499
|
+
| Filters and filter placement | [documentation/language/filters](https://docs.malloydata.dev/documentation/language/filters) |
|
|
500
|
+
| Annotations and tags | [documentation/language/tags](https://docs.malloydata.dev/documentation/language/tags) |
|
|
501
|
+
| Calculations and window functions | [documentation/language/calculations_windows](https://docs.malloydata.dev/documentation/language/calculations_windows) |
|
|
502
|
+
| Data types | [documentation/language/datatypes](https://docs.malloydata.dev/documentation/language/datatypes) |
|
|
503
|
+
| Time operations, ranges, timezones | [documentation/language/timestamp-operations](https://docs.malloydata.dev/documentation/language/timestamp-operations), [time-ranges](https://docs.malloydata.dev/documentation/language/time-ranges), [timezones](https://docs.malloydata.dev/documentation/language/timezones) |
|
|
504
|
+
| Imports | [documentation/language/imports](https://docs.malloydata.dev/documentation/language/imports) |
|
|
505
|
+
| Top-level statements and model structure | [documentation/language/statement](https://docs.malloydata.dev/documentation/language/statement) |
|
|
506
|
+
| Functions reference | [documentation/language/functions](https://docs.malloydata.dev/documentation/language/functions) |
|
|
507
|
+
|
|
508
|
+
### Examples and Patterns
|
|
509
|
+
|
|
510
|
+
The docs site includes worked examples of common analytical patterns at [documentation/patterns](https://docs.malloydata.dev/documentation/patterns/): percent-of-total, year-over-year, cohort analysis, sessionization, moving averages, nested subtotals, and more.
|
|
511
|
+
|
|
512
|
+
End-to-end guides are at [documentation/user_guides](https://docs.malloydata.dev/documentation/user_guides/), including [Malloy by Example](https://docs.malloydata.dev/documentation/user_guides/malloy_by_example) (a comprehensive walkthrough) and a three-part series for SQL users ([part 1](https://docs.malloydata.dev/documentation/user_guides/sql_experts1), [part 2](https://docs.malloydata.dev/documentation/user_guides/sql_experts2), [part 3](https://docs.malloydata.dev/documentation/user_guides/sql_experts3)).
|
|
513
|
+
|
|
514
|
+
### Source Code
|
|
515
|
+
|
|
516
|
+
The Malloy implementation lives at [github.com/malloydata/malloy](https://github.com/malloydata/malloy). Key entry points:
|
|
517
|
+
|
|
518
|
+
| What | Where |
|
|
519
|
+
|---|---|
|
|
520
|
+
| ANTLR grammar (lexer + parser) | `packages/malloy/src/lang/grammar/` |
|
|
521
|
+
| AST node hierarchy | `packages/malloy/src/lang/ast/` |
|
|
522
|
+
| Parse tree → AST builder | `packages/malloy/src/lang/malloy-to-ast.ts` |
|
|
523
|
+
| IR type definitions | `packages/malloy/src/model/malloy_types.ts` |
|
|
524
|
+
| IR → SQL compiler | `packages/malloy/src/model/` |
|
|
525
|
+
| Dialect-specific SQL generation | `packages/malloy/src/dialect/` |
|
|
526
|
+
| Tag/annotation parsing (MOTLY) | `packages/malloy-tag/` |
|
|
527
|
+
| Renderer | `packages/malloy-render/` |
|
|
528
|
+
| Architecture overview | `CONTEXT.md` (root and in each package) |
|