@malloydata/malloyyo 0.2.43 → 0.2.45
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/index.js +29 -7
- package/dist/templates/devcontainer/devcontainer.json +42 -0
- package/dist/templates/skills/malloyyo-auto-update/SKILL.md +5 -106
- package/dist/templates/skills/malloyyo-data-site/SKILL.md +8 -221
- package/package.json +1 -1
- package/dist/templates/skills/malloyyo-data-site/reference/data-to-parquet.md +0 -109
- package/dist/templates/skills/malloyyo-data-site/reference/publish-to-github-pages.md +0 -61
package/dist/index.js
CHANGED
|
@@ -214,6 +214,10 @@ var contentFiles = {
|
|
|
214
214
|
"explore/restricted-queries.md": "# Restricted Query Explanation\n\nThe `query` tool runs your Malloy against a **published model**. You have that\nmodel's entire published surface to work with \u2014 and you can build on it. The\nmodel is an inentionally curated subset of the data available in the\ndatabase.\n\n## You can\n\n- Use everything the model defines: its **sources, dimensions, measures, views,\n joins, and named queries**. `describe_source` shows exactly what's there.\n- **Run a named query and refine it** \u2014\n `run: top_carriers + { where: dep_year = 2024 }`.\n- **Define your own** dimensions, measures, and **your own sources and joins** \u2014\n as long as they are *derived from the model's sources*. You are not limited to\n the author's fields; compose new ones from them.\n- Reference the model's `$NAME` givens and supply values via the `givens` map on\n the `query` call (use `execute: false` to discover which a query needs).\n- Use a model field that was itself defined with raw SQL \u2014 the author vouched\n for the model's own definitions.\n\n## What is \"Restricted\"\n\nIf you see `restricted-construct-forbidden`, the query used something that\nreaches *outside* the published model: pulling in another file (`import`),\nopening a raw connection (`connection.table(...)` / `connection.sql(...)`),\nwriting raw SQL (`name!type(...)` or the `sql_*` functions), declaring new\n`given:`s, or setting `##!` compiler flags.\n\nThe fix is never to work around it \u2014 express the answer in terms of what the\nmodel publishes (define derived sources, joins, dimensions, and measures from\nthe model's sources). If something fundamental is missing, that's feedback for\nthe model's author.\n",
|
|
215
215
|
"language/malloy-language-reference.md": '<!-- Copied from malloy-cli (jrtipton/malloy-cli) skills/malloy-language-reference.md on 2026-06-11.\n Deliberate temporary fork \u2014 converge when the engine is extracted to @malloydata. -->\n---\ndescription: Malloy language reference \u2014 concepts, syntax, compilation model. Load this before writing or reviewing Malloy code.\n---\n# Malloy Language Reference\n\nMalloy is a semantic data modeling and query language. It compiles to SQL and runs against existing database engines (DuckDB, BigQuery, Snowflake, PostgreSQL, MySQL, Trino, Presto). It is not a SQL wrapper or abstraction layer \u2014 it has its own type system, scoping rules, expression semantics, and compilation pipeline.\n\nMalloy is designed around how humans think about data, not how data computations are mechanically accomplished. SQL is oriented around the machine \u2014 you specify joins, group-by columns, subqueries, and window functions in terms of what the database needs to do. Malloy is oriented around the analyst \u2014 you describe relationships, name computations, and compose questions in terms of what the data means. Malloy bridges the gap between these two by compiling the human-oriented description into correct, efficient SQL.\n\nA core design principle is that **most queries are themselves designing a new semantic model.** Formulating a question about data \u2014 choosing what to group by, what to aggregate, what to nest \u2014 is inherently an act of defining a new way to look at that data. Malloy is built around this idea: the output of every query is not just a result set but a new source with its own schema, and data comprehension is an ongoing iterative process where later stages want not only the data from a previous stage but how that data came into being. This is why query output carries metadata, why queries can be used as sources, and why views and pipelines compose naturally.\n\n## Documents and Statements\n\nA Malloy file (`.malloy`) is a sequence of statements, optionally separated by semicolons. There are five statement types:\n\n- **`import`** \u2014 import sources and queries from another `.malloy` file\n- **`source:`** \u2014 define a named, reusable data source with its schema and extensions\n- **`query:`** \u2014 define a named query (source + view) for reuse\n- **`run:`** \u2014 execute a query (the "do it now" statement)\n- **`given:`** \u2014 declare model-level parameters supplied at run time (experimental, see Givens)\n\n```malloy\nimport "shared_model.malloy"\n\nsource: flights is duckdb.table(\'flights.parquet\') extend {\n measure: flight_count is count()\n}\n\nquery: carrier_summary is flights -> {\n group_by: carrier\n aggregate: flight_count\n}\n\nrun: carrier_summary\n```\n\nComments use `//` or `--` (both are line comments).\n\n## Sources\n\nA **source** is anything you can hand a SQL database and get a schema back \u2014 a table name, a SQL SELECT, or the output of another Malloy query. The columns in that schema become the source\'s initial fields (all dimensions).\n\n```malloy\nsource: flights is duckdb.table(\'flights.parquet\')\nsource: limited is duckdb.sql("""SELECT * FROM flights LIMIT 100""")\nsource: carrier_facts is carrier_summary -- a query used as a source\n```\n\nWhat makes sources central to Malloy is **extension**. The `extend` block lets you layer on dimensions, measures, views, joins, filters, primary keys, field restrictions, and renames. These extensions travel with the source \u2014 any query against it gets them for free.\n\n```malloy\nsource: flights is duckdb.table(\'flights.parquet\') extend {\n primary_key: id\n\n dimension: distance_km is distance * 1.609344\n\n measure:\n flight_count is count()\n total_distance is sum(distance)\n\n join_one: carriers with carrier\n join_one: origin_airport is airports on origin_airport.code = origin\n\n where: dep_time > @2001\n\n view: by_carrier is {\n group_by: carrier\n aggregate: flight_count, total_distance\n }\n}\n```\n\nSources can extend other sources, creating a refinement chain:\n\n```malloy\nsource: ca_flights is flights extend {\n where: origin.state = \'CA\'\n}\n```\n\nField access control uses `accept:` (allowlist) or `except:` (denylist) to restrict which inherited columns are visible. Fields can be renamed with `rename: new_name is old_name`.\n\n## Joins\n\nJoins are declared in the source, not reconstructed in every query. This is a fundamental design difference from SQL: the graph structure of your data is a property of the model.\n\n```malloy\njoin_one: carriers with carrier -- FK \u2192 PK shorthand\njoin_one: origin_airport is airports on origin_airport.code = origin -- explicit ON\njoin_many: line_items on line_items.order_id = id -- one-to-many\njoin_cross: other_table on other_table.key = key -- cross join\n```\n\n- `join_one` \u2014 the joined source has at most one row per source row (many-to-one or one-to-one)\n- `join_many` \u2014 the joined source has potentially many rows per source row\n- `join_cross` \u2014 a full cross product\n\nThe `with` shorthand requires the joined source to have a declared `primary_key`. All joins are left outer by default. There is no right join \u2014 Malloy\'s graph model doesn\'t need one.\n\n**Choosing `join_one` vs `join_many`:** Ask "for a single row in the base source, can the joined source match more than one row?" If yes \u2192 `join_many`. If no (or at most one) \u2192 `join_one`. The common mistake is reaching for `join_many` when joining a *lookup or summary table* (e.g., joining an inventory snapshot to a purchase history on a wine key). Even though the joined table may have many rows overall, if each base row resolves to *at most one* joined row, use `join_one`. Use `join_many` only when the join genuinely fans out the base rows \u2014 e.g., joining line items to orders, or notes to a wine catalog.\n\nWhen you reference a joined source\'s fields, you use dot notation: `carriers.nickname`, `origin_airport.state`. This is one of Malloy\'s most important abstractions: **the access path to nested data is identical regardless of how the nesting is physically stored.** An array of records embedded in a column, a `join_many` to a separate table, a record-typed column \u2014 all are navigated with the same dot notation. The SQL required to traverse these different physical arrangements varies wildly (unnesting arrays, LEFT JOINs, correlated subqueries, ARRAY_AGG), but Malloy hides all of that. You think about the logical shape of your data \u2014 "flights have carriers, carriers have a nickname" \u2014 and write `carriers.nickname`. The compiler figures out what SQL is needed to get there. This means you can restructure your physical schema (normalize a nested array into a separate table, or denormalize a joined table into a record column) without changing any of the Malloy that references that data.\n\n## Fields\n\nMalloy has four kinds of fields: **dimensions**, **measures**, **views**, and **calculations**.\n\n### Dimensions\n\nScalar expressions \u2014 they compute a value per row. All columns inherited from a table are dimensions. Computed dimensions reference other dimensions or columns:\n\n```malloy\ndimension: full_name is concat(first_name, \' \', last_name)\ndimension: is_long_haul is distance > 1000\n```\n\n### Measures\n\nAggregate expressions \u2014 they compute a value across a set of rows. A field is a measure when its defining expression contains an aggregate function (`count`, `sum`, `avg`, `min`, `max`):\n\n```malloy\nmeasure:\n flight_count is count()\n total_distance is sum(distance)\n avg_distance is avg(distance)\n pct_delayed is count() { where: dep_delay > 30 } / count()\n```\n\n**`count(expr)` counts distinct values.** Unlike SQL\'s `COUNT(DISTINCT expr)`, Malloy uses `count(expr)` for distinct counting. The `count(distinct expr)` form is a deprecated syntax that will produce an error. Use `count()` for total row count, `count(field)` for distinct values of that field:\n\n```malloy\naggregate:\n total_rows is count() -- all rows\n unique_carriers is count(carrier) -- distinct carriers\n```\n\nMeasures can be filtered inline with `{ where: ... }`, which is how you build things like "percent of flights delayed" without subqueries.\n\n### Views\n\nA view is a query saved into the source \u2014 a reusable transformation:\n\n```malloy\nview: by_carrier is {\n group_by: carrier\n aggregate: flight_count, total_distance\n limit: 10\n}\n```\n\nViews can reference other views from the same source as a starting point, and can be extended with `+`.\n\n### Calculations\n\nWindow functions over the grouped result. Calculations can only be defined in a query stage with `calculate:`, never in a source definition, because they depend on the output columns of the query:\n\n```malloy\nrun: flights -> {\n group_by: carrier\n aggregate: flight_count\n calculate: carrier_rank is rank()\n}\n```\n\n## Queries and Views\n\nA query pairs a source with a view (the transformation). Everything after the first `->` is the view.\n\n```malloy\nrun: flights -> {\n group_by: carrier\n aggregate: flight_count\n}\n```\n\n### Reduction vs. Projection\n\nEach stage of a view performs exactly one of:\n\n- **Reduction** \u2014 uses `group_by:` and/or `aggregate:` to reduce grain. Analogous to `SELECT ... GROUP BY` in SQL.\n- **Projection** \u2014 uses `select:` to pick fields without aggregation. Analogous to `SELECT` without `GROUP BY`.\n\nThese cannot be mixed in a single stage. A stage with `group_by:` cannot have `select:`, and vice versa.\n\n### Source-level definitions vs. query-level operations\n\nThe same `name is expression` syntax defines fields in both sources and queries:\n\n```malloy\n-- In a source (reusable):\nsource: flights is ... extend {\n measure: flight_count is count() -- defines a measure in the model\n}\n\n-- In a query (ad hoc):\nrun: flights -> {\n aggregate: flight_count is count() -- defines the same measure inline\n}\n```\n\nWhen used in a source, `measure:` and `dimension:` are **definition statements** \u2014 they add named fields to the source\'s schema. When used in a query, `group_by:`, `aggregate:`, `select:`, `nest:`, and `calculate:` are **query operations** \u2014 they specify what the query does. The field definitions are syntactically identical in both contexts, but the enclosing keyword determines the role:\n\n| Source keyword | Query keyword | What it holds |\n|---|---|---|\n| `dimension:` | `group_by:` or `select:` | scalar expressions |\n| `measure:` | `aggregate:` | aggregate expressions |\n| `view:` | `nest:` | sub-queries |\n| _(n/a)_ | `calculate:` | window functions |\n\nThis is why `measure` and `aggregate` are separate keywords. `measure:` is a *modeling* statement \u2014 "this source has a reusable aggregate computation called X." `aggregate:` is a *query* statement \u2014 "in this query, include these aggregate values in the output." A query\'s `aggregate:` can reference a previously defined measure by name, or define one inline. The distinction parallels the separation between defining a dimension in a source and using it via `group_by:` in a query.\n\n### Multi-stage Pipelines\n\nStages chain with `->`. Each stage\'s output becomes the next stage\'s source:\n\n```malloy\nrun: flights -> {\n group_by: carrier\n aggregate: flight_count is count()\n} -> {\n where: flight_count > 1000\n select: *\n}\n```\n\n### Refinement with `+`\n\nThe refinement operator `+` merges query operations together. It works both within a view and at the top level on a named query:\n\n```malloy\n-- Refining a view within a query:\nrun: flights -> by_carrier + { limit: 5 } + { nest: by_destination }\n\n-- Refining a named query at the top level:\nrun: carrier_summary + { group_by: origin } -- add origin grouping to existing query\n```\n\nWhen a dimension name appears as a bare reference, it expands to `{ group_by: name }`. A measure name expands to `{ aggregate: name }`:\n\n```malloy\nrun: flights -> carrier + flight_count + { limit: 10 }\n-- equivalent to: flights -> { group_by: carrier; aggregate: flight_count; limit: 10 }\n```\n\nFor multi-stage queries, refinement semantics get more complex \u2014 but for single-stage queries, `+` straightforwardly merges operations into the stage.\n\n### Nesting\n\n`nest:` embeds an aggregating subquery inside a reduction. Each row of the outer query gets a subtable from the nested query. Nests can nest arbitrarily deep:\n\n```malloy\nrun: flights -> {\n group_by: carrier\n aggregate: flight_count\n nest: top_routes is {\n group_by: origin, destination\n aggregate: flight_count\n limit: 3\n }\n}\n```\n\n### Other query operations\n\n- **`where:`** \u2014 filter rows (pre-aggregation). Comma-separated filters are ANDed.\n- **`having:`** \u2014 filter groups (post-aggregation), like SQL\'s HAVING.\n- **`limit:`** / **`order_by:`** \u2014 limit and sort output.\n- **`extend`** \u2014 add fields or joins to a source inline within a query expression.\n\n## Aggregate Locality (Symmetric Aggregates)\n\nThis is one of Malloy\'s most important features. In SQL, when you join tables and aggregate, you risk double-counting (the "fan trap"). Malloy solves this with **aggregate locality** \u2014 you specify *where in the join graph* an aggregation should be computed.\n\n```malloy\nrun: flights -> {\n aggregate:\n -- avg seats weighted by number of flights (locality: source, i.e. flights)\n avg_seats_per_flight is source.avg(aircraft.aircraft_models.seats)\n -- avg seats per aircraft model (locality: aircraft_models)\n avg_seats_per_model is aircraft.aircraft_models.seats.avg()\n}\n```\n\nThree syntactic forms:\n\n- `avg(expr)` \u2014 aggregate at the current source (implicit locality)\n- `joined_source.avg(expr)` \u2014 aggregate at the specified join point (explicit locality)\n- `joined_source.field.avg()` \u2014 shorthand for aggregating the field at its parent source\n\nFor `sum` and `avg` (asymmetric aggregates), when the expression crosses a join boundary, Malloy *requires* explicit locality \u2014 it won\'t silently give you a wrong answer. For `min`, `max`, and `count` (symmetric), locality doesn\'t change the result, so implicit is always fine.\n\nMalloy implements this with a technique called **symmetric aggregates** \u2014 it internally de-duplicates rows based on primary keys at the appropriate join level, so aggregations are always mathematically correct regardless of join fan-out.\n\n## Ungrouped Aggregates\n\n`all()` and `exclude()` allow computing aggregates at different grouping levels within a single query:\n\n```malloy\nrun: airports -> {\n group_by: state, faa_region\n aggregate:\n airport_count is count()\n total_airports is all(count()) -- ungrouped: total across all rows\n region_airports is all(count(), faa_region) -- grouped only by faa_region\n pct_of_total is count() / all(count())\n}\n```\n\n`all(expr)` removes all grouping. `all(expr, dim1, dim2)` keeps only the specified grouping dimensions. `exclude(expr, dim)` removes the specified dimension from grouping.\n\n**Important:** `all(expr, dim)` takes the **local alias name** as defined in the query\'s `group_by:`, not a dotted path. If you want to partition by a joined field, alias it first:\n\n```malloy\n-- WRONG: all(count(), director.primaryName) -- dot paths don\'t work here\n-- RIGHT:\nrun: movies -> {\n group_by: director is director.primaryName -- alias it\n aggregate:\n movies is count()\n director_total is all(count(), director) -- reference the alias\n pct is count() / all(count(), director)\n}\n```\n\n## Expressions\n\nMalloy expressions include arithmetic, comparison, logical operators, function calls, type casts, and several Malloy-specific forms.\n\n### Evaluation Spaces\n\nEvery expression has an evaluation space: **literal**, **constant**, **input**, or **output**. Input expressions reference source columns/dimensions. Output expressions reference the results of the current query stage (used in `calculate:`). Some functions constrain their arguments \u2014 e.g., `lag(expr)` requires an output expression, `avg(expr)` requires an input expression.\n\n### Application and Partial Comparison\n\nThe `?` operator applies a condition to a value. Partial comparisons are conditions without a left-hand side:\n\n```malloy\nwhere: state ? \'CA\' | \'NY\' -- state is \'CA\' or \'NY\'\nwhere: distance ? > 500 & < 2000 -- distance between 500 and 2000\n```\n\n`|` is alternation (OR), `&` is conjunction (AND) within partials.\n\n### Pick Expressions\n\nMalloy\'s equivalent of CASE:\n\n```malloy\ndimension: size_bucket is\n pick \'short\' when distance < 500\n pick \'medium\' when distance < 1500\n else \'long\'\n```\n\n### Filtered Aggregate Expressions\n\nAny aggregate can be filtered inline:\n\n```malloy\nmeasure: ca_flights is count() { where: origin.state = \'CA\' }\n```\n\n### Type Casting\n\n```malloy\ntotal_distance::string -- Malloy type cast\nname::"VARCHAR(32)" -- database-native type cast\n```\n\n### Time Literals and Ranges\n\n```malloy\n@2003 -- the year 2003\n@2003-Q2 -- second quarter of 2003\n@2024-01-15 10:30:00 -- timestamp literal\ndep_time ? @2003 to @2005 -- range comparison\nnow -- current timestamp\n```\n\n## Data Types\n\nMalloy\'s type system: `string`, `number`, `boolean`, `date`, `timestamp`, `timestamptz`, `json`, and `sql native` (for unsupported database types). Compound types: `type[]` for arrays, `{ name :: type, ... }` for records, nesting arbitrarily: `{ x :: number, tags :: string[] }[]`.\n\n## Annotations and Tags\n\nThese are related but distinct concepts.\n\n### Annotations\n\nAnnotations are **text strings** attached to objects during compilation. They are metadata \u2014 they never affect query execution or SQL generation. An annotation starts with `#` and continues to end of line:\n\n```malloy\n# bar_chart\nview: by_carrier is { ... }\n```\n\n- `#` annotations attach to the next object defined below them\n- `##` annotations attach to the model (the file)\n- Block annotations use `#|` ... `|#` for multi-line content (closing delimiter must match the column position of the opener)\n\nAnnotations distribute over definition lists:\n\n```malloy\n# currency\nmeasure: -- all three measures get the # currency annotation\n revenue is sum(amount)\n # percent -- this measure also gets # percent\n margin is revenue / cost\n cost is sum(amount)\n```\n\n### Tags (a use of annotations)\n\nTags are the primary *consumer* of annotation strings. They interpret annotation text using a structured property language (MOTLY). The key distinction: **annotations are the transport mechanism (raw strings attached to objects), tags are the interpretation layer (parsed key-value properties).**\n\nNot all annotations are tags. An annotation is just text. Tags are annotations that happen to be written in the tag property language and parsed by an application.\n\n### Annotation prefixes (routing)\n\nThe character(s) immediately after `#` route the annotation to different consumers:\n\n- `# ` (hash-space) \u2014 renderer tags, parsed by the Malloy VS Code extension for visualization (`# bar_chart`, `# line_chart`, `# shape_map`; `# link { url_template="https://\u2026/$$" }` turns a field\'s cell into a clickable deep link \u2014 `$$` is the cell value, `field=other` links on a different (often `# hidden`) field; `# image { url_template=\u2026 }` renders it as an image)\n- `##!` \u2014 compiler directives (e.g., `##! experimental.parameters`, `##! experimental.givens`)\n- `#"` \u2014 reserved for documentation strings\n- `#(appName)` \u2014 application-specific tags (e.g., `#(docs) hidden`)\n\n```malloy\n# bar_chart size=large -- renderer tag: tells VS Code how to render\n##! experimental.parameters -- compiler tag: enables a feature flag\n#(myApp) priority=high -- custom app tag: ignored by renderer/compiler\n```\n\n### Tag property syntax\n\n```\ntName -- boolean flag (exists = true)\ntName=value -- set property value\ntName=[a, b, c] -- array value\ntName: { p1=v1 p2=v2 } -- nested properties (replaces)\ntName { p1=v1 } -- nested properties (merges)\n-tName -- delete a property\ntName.sub.path=value -- deep path assignment\n```\n\nValues can be unquoted identifiers, quoted strings, numbers, or typed values prefixed with `@` (`@true`, `@false`, `@2024-01-15`).\n\n## Givens (Model-Level Parameters)\n\n**Status: experimental, gated by `##! experimental.givens`.** Naming is provisional.\n\nGivens are values supplied at run time that the model can reference in any expression. The motivating use case is row-level access control \u2014 a model written once with `where: x.tenant_id = $TENANT` and the tenant supplied per API call \u2014 but they also fit configuration values, session context, and any "one compiled model, many invocations with varying context" pattern.\n\nGivens are model-wide: a single namespace, one value per name per compilation. They are *complementary to* source/query parameters (`source: foo(x :: string) is ...`), not a replacement. Use a parameter when you want two differently-bound copies of the same source side-by-side in one model; use a given when you want one value visible everywhere in the compilation.\n\n### Declaration\n\nThe `given:` top-level statement introduces givens, with a name, a type, and an optional default:\n\n```malloy\ngiven:\n TENANT :: string\n MAX_ROWS :: number is 1000\n CUTOFF_DATE :: date is @2024-01-01\n```\n\nType can be any Malloy atomic type or compound type, including `filter<T>`:\n\n```malloy\ngiven:\n ROLE :: string\n ALLOWED_ROLES :: string[]\n SESSION :: { user_id :: string, tenant :: string }\n TENANT_FILTER :: filter<string>\n```\n\nDefaults are expressions over constants and other givens. Annotations attach to given declarations the same way they attach to sources or measures.\n\n### Reference: the `$` sigil\n\nInside any expression, a given is referenced with a leading `$`:\n\n```malloy\nsource: orders_for_user is orders extend {\n where: orders.tenant_id = $TENANT\n}\n\nquery: recent_orders is orders_for_user -> {\n where: order_date >= $CUTOFF_DATE\n limit: $MAX_ROWS\n}\n```\n\n`$` appears *only* at expression references. The other three sites where a given\'s name appears \u2014 declaration, import, and supply (caller side) \u2014 use the bare name, because syntactic position already disambiguates. Givens share the top-level declaration namespace with sources/queries/views, so `source: x is ...` plus `given: x :: string` is a name-conflict error.\n\n### Set membership: `expr in $arrayGiven`\n\nThe RHS of `in` is either a parenthesized list of expressions (`in (1, 2, x, y * 7)`, same as SQL) or a given with an array value (`in $ARR`). A bare array-typed expression \u2014 a dimension, a joined array field, an inline `[a, b, c]` literal \u2014 is *not* legal on the RHS; arrays only reach the RHS via the given form.\n\nWhen a given has array type, `expr in $ARR` tests `expr` against the runtime-bound array; `not in $ARR` is the negation. The left-hand side must match the array\'s element type (`string in $string[]`, `number in $number[]`, etc.); mismatches are translate-time errors. Records and nested arrays are out of scope.\n\n```malloy\ngiven:\n ALLOWED_STATES :: string[]\n URGENT_STATUSES :: string[]\n\nsource: orders extend {\n where: state in $ALLOWED_STATES\n dimension: is_urgent is order_status in $URGENT_STATUSES\n}\n```\n\nAt SQL emit, the array\'s contents land in a generated `IN (...)` clause. Empty or `null` arrays collapse to the obvious result (`IN` \u2192 `FALSE`, `NOT IN` \u2192 `TRUE`). NULL elements inside a non-empty array follow standard SQL `IN` semantics.\n\nTo derive a value from an array \u2014 typically a boolean gate \u2014 *without* the array itself reaching row-position SQL, use an inline given (below).\n\n### Inline givens\n\nAn `inline` given is evaluated at **bind time**, before SQL is emitted: its default expression runs against the resolved given values and reduces to a literal, and that literal is what reaches SQL.\n\n```malloy\ngiven:\n CAPABILITIES :: string[]\n inline CAN_READ_ORDERS :: boolean is \'read_orders\' in $CAPABILITIES\n inline CAN_MUTATE :: boolean\n is \'write_orders\' in $CAPABILITIES or \'admin\' in $CAPABILITIES\n\nsource: orders extend {\n where: $CAN_READ_ORDERS -- SQL sees: WHERE ... AND TRUE (or FALSE)\n}\n```\n\nThis is the **row-level access-control gate** pattern: the host supplies a capability list as a regular given, an inline given derives a boolean from it, and only the boolean \u2014 not the list \u2014 crosses into row-position SQL. The query planner sees a constant predicate.\n\nRules:\n\n- An inline given **must** have a default. `inline FOO :: number` with no `is` clause is a translate-time error.\n- The default may use:\n - Boolean and comparison operators: `and`, `or`, `not`, `=`, `!=`, `<`, `<=`, `>`, `>=`\n - The `in $array` test against another given\n - Literals (string, number, boolean, null, array) and references to other givens\n- The default cannot call SQL functions, reference fields, or use any operator outside that list. Disallowed operators are reported at translate time with the offending operator names.\n- Inline givens are filtered out of `Model.givens` and `PreparedQuery.givens` \u2014 they\'re computed, not supplied \u2014 so introspection-driven UIs don\'t render editors for them. A caller can still shadow one by binding it explicitly (useful in tests).\n- `inline` is a context-sensitive modifier, not a reserved keyword: fields, sources, views, dimensions, and joins can still be named `inline`.\n\n### Imports\n\nGivens behave like every other top-level named thing under import:\n\n- **Bare import** (`import "b.malloy"`) brings B\'s full export surface in, including all of B\'s givens, under their original names.\n- **Selective import** (`import { source1 } from "b.malloy"`) brings in only what\'s listed. To surface a given to your callers, list it: `import { source1, MAX_ROWS } from "b.malloy"`.\n- **Rename** uses the existing `LOCAL is REMOTE` form: `import { CAP is MAX_ROWS } from "b.malloy"`.\n\nSurfacing controls *who can supply a value*, not whether internal references work. An imported source can reference a given the importer didn\'t surface; the reference still resolves internally, and at run time the unsurfaced given relies on its declaration-site default.\n\nA common project convention is a shared `tenant_givens.malloy` (declaring `$TENANT`, `$USER_ROLE`, etc.) that every root file bare-imports on line 1, so the project\'s given contract is visible at the top of any model.\n\n### Satisfiability\n\nA query referencing `$X` is satisfiable if either `$X` is in the model\'s namespace (so a caller can supply a value) or `$X` has a default at its declaration site. Otherwise the query is unsatisfiable and errors. Latent definitions (views, dimensions, measures) that reference `$X` are fine if no query actually invokes them \u2014 satisfiability is a property of running queries.\n\n### Supplying values\n\nValues can be supplied at two layers, which compose (per-query overrides per-runtime):\n\n**Per-runtime** \u2014 bound to a `Runtime`, applied as defaults to every query through it. Two paths:\n\n1. **`givensPath` in `malloy-config.json`** points at a JSON file of `name \u2192 value`:\n ```jsonc\n { "givensPath": "./local-givens.json" }\n // or env-var indirection (resolved at config load):\n { "givensPath": { "env": "GAME_STORE_GIVENS" } }\n ```\n The values file is a flat JSON map, keys are caller-facing surface names:\n ```jsonc\n { "TENANT": "acme", "USER_ROLE": "admin", "CUTOFF_DATE": "2024-01-01" }\n ```\n\n2. **Direct on the Runtime constructor** (for per-request multi-tenant servers, tests, scripts):\n ```typescript\n const runtime = new Runtime({\n config,\n givens: { TENANT: claims.tenant_id, USER_ROLE: claims.role },\n urlReader,\n });\n ```\n Constructor values *merge over* the file at `givensPath` per-key.\n\n**Per-query** \u2014 supplied on a single `.run({ givens: ... })` call:\n```typescript\nawait query.run({ givens: { STATE_FILTER: "CA", LIMIT_OVERRIDE: 50 } })\n```\nAvailable on every compile-or-run entry point (`runtime.loadQuery(...).run(options)`, `preparedQuery.getPreparedResult(options)`, `preparedQuery.getSQL(options)`).\n\nThe resolved per-runtime values are exposed on `runtime.givens` (read-only) for diagnostics.\n\n### Finalized givens (security primitive)\n\nA multi-tenant deployment usually wants `TENANT`/`USER_ROLE`/`REGION` to be runtime-bound and **un-overridable per-query** \u2014 otherwise a downstream endpoint that accidentally accepts user-controlled query params and plumbs them into `.run({ givens: ... })` becomes a tenant-leak vulnerability.\n\n`finalizeGivens` in the config locks names at the API surface:\n\n```jsonc\n{\n "givensPath": { "env": "GAME_STORE_GIVENS" },\n "finalizeGivens": ["TENANT", "USER_ROLE", "REGION"]\n}\n```\n\nFinalize doesn\'t change *what* a given resolves to \u2014 only *who* can supply it. A `.run({ givens: { TENANT: ... } })` for a finalized name throws at API entry (named, not silently dropped). Finalized givens are filtered out of `Model.givens` and `PreparedQuery.givens` so introspection-driven UIs don\'t render editors for locked names.\n\n### JS shapes for supplied values\n\nBoth the JSON values file and per-query `givens` maps accept the same per-type shapes:\n\n| Malloy type | JS |\n|---|---|\n| `string` | string |\n| `number` | `number`, `bigint`, or string (precision escape hatch) |\n| `boolean` | boolean |\n| `date` | ISO date string `"2024-01-15"` |\n| `timestamp` (naive) | ISO string without offset \u2014 *not* a JS `Date` |\n| `timestamptz` | JS `Date` or ISO string with offset (string preferred \u2014 makes TZ choice visible) |\n| `T[]` | JS array |\n| `{ name :: T, ... }` | JS object |\n| `filter<T>` | JS string (Malloy filter expression source) |\n\nNaive timestamp givens reject `Date` because `Date` represents a UTC instant, not a wall-clock value, and `new Date("2001-01-01T00:00:00")` silently picks up the system\'s local TZ. Type mismatches throw at the boundary with a path that points at the offending location (e.g., `givens.SESSION.user_id: expected string, got number`). `null` is legal for any given type.\n\n### Introspection\n\n`Model.givens` and `PreparedQuery.givens` expose, to the host, the supplyable givens \u2014 for whole-model parameter editors and per-query "run this" forms respectively. Each entry carries name, type, default expression (or undefined if the caller must supply), location, and access to declaration-site annotations via `tagParse`/`getTaglines`.\n\n## How a Malloy Query Becomes SQL\n\nThe compilation pipeline has two phases:\n\n### Phase 1: Translation (source code \u2192 IR)\n\n```\nMalloy source \u2192 ANTLR lexer/parser \u2192 parse tree \u2192 AST builder \u2192 AST \u2192 IR generator \u2192 IR\n```\n\nThe **Intermediate Representation (IR)** is a plain, serializable data structure (JSON-compatible) that fully describes the semantic model and query. Note that IR is *not* dialect-agnostic \u2014 the same Malloy source compiled against different databases can produce different IR, because schema information, type mappings, and available functions vary by backend. It can be cached, transmitted, and reused. Key IR types:\n\n- **`SourceDef`** \u2014 a source\'s complete definition: schema, fields, joins, filters\n- **`Query`** \u2014 a source paired with a pipeline of operations\n- **`FieldDef`** \u2014 definition of any field (dimension, measure, join, calculation)\n- **`Expr`** \u2014 expression tree (arithmetic, comparisons, aggregates, function calls, field references)\n\nThe translator handles all language-level semantics: scoping, name resolution, type checking, evaluation space validation.\n\n### Phase 2: Compilation (IR \u2192 SQL)\n\n```\nIR \u2192 query compiler \u2192 expression compiler \u2192 dialect-specific SQL generator \u2192 SQL + metadata\n```\n\nThe compiler walks the IR query pipeline, translating each stage into SQL constructs (CTEs, subqueries, GROUP BY, window functions). A **Dialect** layer handles database-specific SQL generation.\n\nThe compiler also produces **metadata** alongside the SQL \u2014 structural information needed to interpret the result set (column types, nesting structure, annotation data). This metadata is what allows Malloy renderers to reconstruct nested/hierarchical results from the flat SQL result set and apply visualization tags.\n\n### Key architectural consequences\n\n- Because the IR is serializable, it can be cached and reused across compilations (though IR is database-specific \u2014 the same source compiled against different backends may produce different IR).\n- Because joins are declared in the source (not the query), the compiler knows the full join graph and can compute symmetric aggregates correctly.\n- Because nested queries are first-class, the compiler generates the appropriate SQL (correlated subqueries or ARRAY_AGG patterns depending on dialect) automatically.\n- Because measures are typed as aggregates in the IR, the compiler can validate that they only appear in aggregate context and enforce locality rules.\n\n## Where to Go Deeper\n\nThis document is a conceptual reference \u2014 enough to reason about the language and its design, but not exhaustive. Here\'s where to find more detail.\n\n### Language Documentation\n\nThe full docs live at [https://docs.malloydata.dev](https://docs.malloydata.dev). Key pages by topic:\n\n| Topic | URL |\n|---|---|\n| Sources, extensions, joins, primary keys | [documentation/language/source](https://docs.malloydata.dev/documentation/language/source) |\n| Queries, views, reduction vs projection | [documentation/language/query](https://docs.malloydata.dev/documentation/language/query), [views](https://docs.malloydata.dev/documentation/language/views) |\n| Fields: dimensions, measures, views, calculations | [documentation/language/fields](https://docs.malloydata.dev/documentation/language/fields) |\n| Aggregate functions and aggregate locality | [documentation/language/aggregates](https://docs.malloydata.dev/documentation/language/aggregates) |\n| Ungrouped aggregates (`all`, `exclude`) | [documentation/language/ungrouped-aggregates](https://docs.malloydata.dev/documentation/language/ungrouped-aggregates) |\n| Nested views / aggregating subqueries | [documentation/language/nesting](https://docs.malloydata.dev/documentation/language/nesting) |\n| Joins | [documentation/language/join](https://docs.malloydata.dev/documentation/language/join) |\n| Expressions, operators, pick, apply | [documentation/language/expressions](https://docs.malloydata.dev/documentation/language/expressions) |\n| Evaluation spaces (literal, constant, input, output) | [documentation/language/eval_space](https://docs.malloydata.dev/documentation/language/eval_space) |\n| Filters and filter placement | [documentation/language/filters](https://docs.malloydata.dev/documentation/language/filters) |\n| Annotations and tags | [documentation/language/tags](https://docs.malloydata.dev/documentation/language/tags) |\n| Calculations and window functions | [documentation/language/calculations_windows](https://docs.malloydata.dev/documentation/language/calculations_windows) |\n| Data types | [documentation/language/datatypes](https://docs.malloydata.dev/documentation/language/datatypes) |\n| Time operations, ranges, timezones | [documentation/language/timestamp-operations](https://docs.malloydata.dev/documentation/language/timestamp-operations), [time-ranges](https://docs.malloydata.dev/documentation/language/time-ranges), [timezones](https://docs.malloydata.dev/documentation/language/timezones) |\n| Imports | [documentation/language/imports](https://docs.malloydata.dev/documentation/language/imports) |\n| Top-level statements and model structure | [documentation/language/statement](https://docs.malloydata.dev/documentation/language/statement) |\n| Functions reference | [documentation/language/functions](https://docs.malloydata.dev/documentation/language/functions) |\n\n### Examples and Patterns\n\nThe docs site includes worked examples of common analytical patterns at [documentation/patterns](https://docs.malloydata.dev/documentation/patterns/): percent-of-total, year-over-year, cohort analysis, sessionization, moving averages, nested subtotals, and more.\n\nEnd-to-end guides are at [documentation/user_guides](https://docs.malloydata.dev/documentation/user_guides/), including [Malloy by Example](https://docs.malloydata.dev/documentation/user_guides/malloy_by_example) (a comprehensive walkthrough) and a three-part series for SQL users ([part 1](https://docs.malloydata.dev/documentation/user_guides/sql_experts1), [part 2](https://docs.malloydata.dev/documentation/user_guides/sql_experts2), [part 3](https://docs.malloydata.dev/documentation/user_guides/sql_experts3)).\n\n### Source Code\n\nThe Malloy implementation lives at [github.com/malloydata/malloy](https://github.com/malloydata/malloy). Key entry points:\n\n| What | Where |\n|---|---|\n| ANTLR grammar (lexer + parser) | `packages/malloy/src/lang/grammar/` |\n| AST node hierarchy | `packages/malloy/src/lang/ast/` |\n| Parse tree \u2192 AST builder | `packages/malloy/src/lang/malloy-to-ast.ts` |\n| IR type definitions | `packages/malloy/src/model/malloy_types.ts` |\n| IR \u2192 SQL compiler | `packages/malloy/src/model/` |\n| Dialect-specific SQL generation | `packages/malloy/src/dialect/` |\n| Tag/annotation parsing (MOTLY) | `packages/malloy-tag/` |\n| Renderer | `packages/malloy-render/` |\n| Architecture overview | `CONTEXT.md` (root and in each package) |\n',
|
|
216
216
|
"language/pick.md": "---\ndescription: pick expressions \u2014 Malloy's CASE/if-then-else\n---\n\n`pick` is Malloy's equivalent of SQL `CASE WHEN`. Each branch is its own\n`pick` keyword; the `else` clause catches the remainder.\n\nThere are two forms of pick. In the first the `when` expression is any\nboolean expression.\n\n```malloy\n pick 'Female' when upper(first_name) in ('JENNIFER', 'ELIZABETH', 'AMY', 'JESSICA')\n pick 'Male' when upper(first_name) in ('JAMES', 'JOHN', 'ROBERT', 'MICHAEL')\n else 'Unknown'\n```\n\n## Example usage in a query\n\n```malloy\nrun: payments -> {\n group_by: tier is\n pick 'high' when total_amount > 10000\n pick 'medium' when total_amount > 1000\n else 'low'\n aggregate: payment_count is count()\n}\n```\n\n## Common mistakes\n\n- **Every branch needs its own `pick` keyword** \u2014 there is no `when \u2026 then`:\n ```malloy\n -- WRONG:\n pick 'a' when x = 1 'b' when x = 2 else 'c'\n\n -- RIGHT:\n pick 'a' when x = 1\n pick 'b' when x = 2\n else 'c'\n ```\n\n- **`else` is required** when the branches don't cover all cases \u2014 omitting it\n returns `null` for unmatched rows.\n",
|
|
217
|
+
"site/auto-update.md": '# Auto-update a data site weekly\n\nAdd a scheduled GitHub Actions job that rebuilds the data and commits it, so the\npublished Pages site stays current on its own. Use this **after** the site works\n(built with `yo_help("site/data-site")`). If the data never changes, skip this.\n\nThe worked example is `malloydata/malloyyo-imdb` \u2014 fetch and read its\n`scripts/build_data.sh` and `.github/workflows/refresh-data.yml`; adapt, don\'t\ncopy blindly.\n\n## Precondition: one command that rebuilds the data\n\nThe whole thing rests on a single script that turns the source URL(s) into the\ncommitted parquet \u2014 the same script you\'d run by hand. Keep it linear so it\nreads as a recipe:\n\n```bash\n# scripts/build_data.sh\nset -euo pipefail\ncd "$(dirname "$0")/.."\nbash data/get.sh # 1. download source URLs -> data/\nrm -f data/build.duckdb\nmalloy-cli -c malloy-build.json build transform.malloy # 2. transform\necho "ATTACH \'data/build.duckdb\' AS b (READ_ONLY); -- 3. export -> docs/*.parquet\n COPY b.thing TO \'docs/thing.parquet\' (FORMAT parquet)" | malloyyo sql\n```\n\n(If you used Path A \u2014 direct \u2014 the whole script is just the one\n`echo "COPY (FROM read_..(\'https://\u2026\')) TO \'docs/\u2026\'" | malloyyo sql` command.\nNo `data/get.sh`, `malloy-build.json`, `malloy-cli`, or transform needed \u2014 and\nno standalone `duckdb` either, since `malloyyo sql` uses the embedded one.)\n\nGitignore the working files: `data/*.gz`, `data/*.duckdb`, `MANIFESTS/`.\n\n## The workflow\n\n`.github/workflows/refresh-data.yml`:\n\n```yaml\nname: refresh-data\non:\n schedule:\n - cron: "0 6 * * 0" # weekly, Sunday 06:00 UTC\n workflow_dispatch: # ...and on demand from the Actions tab\njobs:\n refresh:\n runs-on: ubuntu-latest\n permissions:\n contents: write # so the job can commit + push\n steps:\n - uses: actions/checkout@v4\n - uses: actions/setup-node@v4\n with: { node-version: "20" }\n # malloyyo carries its own DuckDB (`malloyyo sql`), so no duckdb CLI to install.\n # Add `@malloydata/cli` only for a `#@ persist` transform (malloy-cli build).\n - run: npm install -g @malloydata/malloyyo @malloydata/cli\n - uses: actions/setup-python@v5 # only if an enrichment step needs it\n with: { python-version: "3.12" }\n - name: Build data\n run: bash scripts/build_data.sh\n - name: Commit refreshed data\n run: |\n git config user.name github-actions\n git config user.email github-actions@github.com\n git add docs/*.parquet\n git commit -m "refresh data $(date -u +%F)" || exit 0 # unchanged = clean no-op\n git push\n```\n\n## Hard-won details (these came from a real first run)\n\n- **`git commit \u2026 || exit 0`** \u2014 an unchanged week means nothing new upstream.\n That is success, not a failed job.\n- **Secondary/enrichment steps must be best-effort.** If the build has a step\n that can fail independently (an external API lookup, an optional metadata\n enrichment) and it isn\'t essential to the data, mark it\n `continue-on-error: true`. Otherwise one flaky API call throws away an\n otherwise-good data refresh \u2014 the commit step never runs and the fresh parquet\n is lost. (In `malloyyo-imdb` this bit us: a poster-image lookup with a missing\n secret failed the whole job until it was made non-blocking.)\n- **Secrets** go in repo settings (Settings \u2192 Secrets and variables \u2192 Actions,\n or `gh secret set NAME`), referenced as `${{ secrets.NAME }}`. A step reading\n a secret that isn\'t set gets an empty string \u2014 pair that with best-effort.\n- **No re-bundle needed for data changes.** The dashboards fetch parquet at\n runtime, so committing fresh parquet updates the live site. Only install\n `@malloydata/malloyyo` and run `malloyyo dashboard bundle` in CI if the job\n also needs to regenerate the HTML (i.e. dashboard code changed \u2014 rare for a\n data refresh).\n- **Test before trusting the schedule.** Push, then Actions tab \u2192 the workflow \u2192\n **Run workflow** (that\'s `workflow_dispatch`). Or `gh workflow run\n refresh-data.yml`, then `gh run watch <id> --exit-status`.\n\n## Git growth, and the flatten\n\nEach committed refresh adds the full parquet to history (~its file size per\nrun). That is intentional simplicity, not a leak. When history gets too big,\nflatten it to a single commit by hand \u2014 **not** as part of the weekly job:\n\n```bash\ngit checkout --orphan flat && git add -A && git commit -m "flatten history"\ngit branch -M flat main && git push -f origin main\n```\n\nOnly do this on a repo you solely own (it force-pushes; other clones must\n`git reset --hard origin/main`). Run it as often or as rarely as you like.\nDocument this in the repo\'s README so future-you remembers it\'s available.\n\n## Done when\n\n- The workflow file is on the default branch (so `workflow_dispatch` shows up)\n- A manual run goes green end-to-end and commits fresh `docs/*.parquet`\n- The Pages site shows the new data\n',
|
|
218
|
+
"site/data-site.md": '# Build a public data site from a web data pointer\n\nGoal: given one or more **public data URLs**, produce a **GitHub Pages site** of\ninteractive Malloy dashboards that anyone can open in a browser. The data ends\nup as parquet in `docs/`; the dashboards are static HTML that query it\nclient-side with DuckDB-WASM \u2014 so there is no server, no database to run.\n\nTwo reference repos \u2014 read them, they are the ground truth for structure:\n\n- **`malloydata/malloyyo-babynames`** \u2014 the clean base pattern (one model file,\n a few dashboards, `docs/`). Data is static.\n- **`malloydata/malloyyo-imdb`** \u2014 the same, plus a `transform.malloy` that\n cleans raw source files into parquet, plus a **weekly auto-refresh** (that\n part is `yo_help("site/auto-update")`).\n\nFetch either with `gh api repos/<repo>/git/trees/HEAD?recursive=1` and read the\nfiles you need. Match their layout rather than inventing your own.\n\n## The shape of a finished repo\n\n```\nmalloy-config.json connection(s): duckdb (local), optionally md/gs mirrors\nindex.malloy the EXPORT SURFACE \u2014 only what this file exports is live\n<model>.malloy sources, measures, joins, givens (parameters)\nstorage.malloy sources point at the parquet \u2014 docs-local OR an https URL\n (or gs.malloy / md.malloy) same source names, swappable hosting (step 3)\ndashboards/*.malloy the query behind each dashboard\ndashboards/*.jsx the dashboard layout (grid of charts/tables)\ndocs/ PUBLISHED SITE \u2014 bundled HTML (+ the *.parquet if docs-local)\n *.parquet the data, when hosted docs-local (committed; served by Pages)\n *.html .nojekyll written by `malloyyo dashboard bundle`\n.mcp.json written by `malloyyo init` (author-mode Claude)\n```\n\n## Prerequisites (install once)\n\n```bash\nnpm install -g @malloydata/malloyyo # the `malloyyo` command\n```\n\nThat\'s the whole toolchain for the common case. `malloyyo` has **DuckDB built\nin** \u2014 `malloyyo sql` runs SQL (read a URL, `COPY` to parquet) through it, so you\ndo **not** need a standalone `duckdb` CLI.\n\nAdd `@malloydata/cli` (`malloy-cli`) **only** if you do a heavier transform with\nMalloy `#@ persist` (see Path B in `yo_help("site/data-to-parquet")`):\n\n```bash\nnpm install -g @malloydata/cli # only for #@ persist transforms\n```\n\n## Recipe\n\nWork top to bottom. After each step, prove it before moving on.\n\n### 1. Scaffold\n\n```bash\nmkdir my-site && cd my-site && git init\nmalloyyo init # writes .mcp.json (author-mode Claude) + index.malloy stub\n```\n\nCreate `malloy-config.json` with a local DuckDB connection (this is what reads\nthe parquet):\n\n```json\n{ "connections": { "duckdb": { "is": "duckdb" } } }\n```\n\n### 2. Get the web data into `docs/*.parquet`\n\nThis is the one dataset-specific step. Two paths \u2014 pick the simpler one that\nworks. Full detail and copy-paste commands: **`yo_help("site/data-to-parquet")`**.\n\n- **Direct** (default): `malloyyo sql` reads the URL and writes parquet in one\n shot, through the embedded DuckDB:\n ```bash\n echo "COPY (FROM read_csv_auto(\'https://\u2026/thing.csv\')) TO \'docs/thing.parquet\'" | malloyyo sql\n ```\n Use when the web data is already close to what you want to show.\n- **Transform** (when you need to clean / join / rank / reshape): write a\n `transform.malloy` with `#@ persist` sources and build it with `malloy-cli`,\n then export the tables to `docs/*.parquet`. This is the `malloyyo-imdb`\n pattern.\n\nVerify: `malloyyo sql -e "DESCRIBE SELECT * FROM \'docs/thing.parquet\'"` and a\n`SELECT count(*)`.\n\n### 3. Write the model\n\nA storage file \u2014 one source per parquet file \u2014 that the rest of the model builds\non. **Where the parquet lives is a choice** (the two examples differ here):\n\n- **docs-local** (the `malloyyo-imdb` way) \u2014 parquet committed in `docs/`,\n served same-origin by Pages. Self-contained; git carries the data. Address it\n by project-relative path so the same spelling works locally and published:\n ```malloy\n // storage.malloy\n source: thing_table is duckdb.table(\'docs/thing.parquet\') extend {}\n ```\n- **External URL** (the `malloyyo-babynames` way \u2014 it uses `import "gs.malloy"`)\n \u2014 parquet hosted on GCS / a CDN / any https URL; `docs/` holds only the HTML,\n so git stays small. DuckDB-WASM fetches the URL at runtime:\n ```malloy\n // gs.malloy\n source: thing_table is duckdb.table(\'https://storage.googleapis.com/\u2026/thing.parquet\') extend {}\n ```\n\nKeep the **source names identical** across storage files (as babynames does with\n`gs.malloy` / `md.malloy`) so switching hosting is a one-line import change in\nthe model. Start docs-local (simplest); move to a URL if git growth bites.\n\n`<model>.malloy` \u2014 build real sources on top: primary keys, measures, joins,\nand **givens** (the parameters that become the dashboard\'s filter controls).\n`index.malloy` \u2014 re-export exactly the sources/queries/givens the dashboards\nuse. **Only what `index.malloy` exports is visible** to dashboards, `dashboard\ndev`, and the hosted app.\n\nGive each exported source a `#"` doc string. It\'s what `list_sources` shows the\nhosted app and any MCP client, and it\'s the only place to record the things a\nconsumer can\'t infer \u2014 which source answers which question, and any measure\nwhose meaning is subtle.\n\nFor Malloy modeling and givens specifics, lean on the author MCP rather than\nguessing: the repo\'s `.mcp.json` wires `mcp__malloyyo_author__*`. Call\n`mcp__malloyyo_author__compile` to check files and\n`mcp__malloyyo_author__yo_help` for topics (`develop/working-with-models`).\n\nThree shapes that compile, pass `lint`, and still produce a dashboard that looks\nright and isn\'t:\n\n- **A stage-level `where:` zeroes measures that carry their own filters.**\n `count() { where: status = \'Sold\' }` needs the sold rows to reach it; a\n `where: in_cellar` on the stage removes them first and the measure reads 0\n everywhere. Constrain the output with `having:` instead.\n- **Grouping finer than the fact scatters the counts.** Group by an attribute\n that only applies to some rows and the measures over the others split across\n groups and read near-zero. Pick the grain the question is asked at; if you\n need both, group at the coarser one and `nest:` the finer.\n- **Detail tables default to raw field names and bare numbers.** `exit_date` and\n `829.57`, and a null renders as `\u2205`. Tag them \u2014 `# label="Opened"`,\n `# label="Paid" currency` \u2014 and `coalesce(field, \'\')` where empty is\n meaningful rather than missing.\n\nAggregate locality is NOT one of these: Malloy\'s symmetric aggregates keep\nmeasures honest through a `join_many` fan-out, and it warns (or errors) rather\nthan silently answering when an `avg`/`sum` crosses a join without explicit\nlocality. `notes.rating.avg()` already averages at the notes grain; ask for\n`source.avg(notes.rating)` only when you want the fan-out-weighted number. See\n`yo_help language/aggregate-locality-symmetric-aggregates`.\n\n### 4. Author dashboards and preview live\n\nEach dashboard is **one self-contained `dashboards/<name>.malloy`** \u2014 the query,\nits filtering, and the tags that lay it out. The filename is the dashboard\'s\nname and URL slug.\n\n**A `.jsx` is OPTIONAL and most dashboards don\'t need one.** The default form \u2014\nan inline query tagged `# artifact` + `# dashboard { columns=6 }` \u2014 renders KPI\ntiles and a card grid with no React at all. Reach for a sibling\n`dashboards/<name>.jsx` only when you want bespoke layout or presentation the\ntags can\'t express (a hand-built card, a picker, a Vega chart).\n\nAuthor with the `malloyyo_author` MCP and its `yo_help` topics \u2014 **read these,\ndon\'t guess the API**: `dashboards/authoring` (the dashboard file),\n`dashboards/grid-layout` (colspans), `dashboards/charts` (chart tags and their\nchannel rules), `dashboards/givens-and-controls` (filters),\n`dashboards/custom-components` and `dashboards/vega-charts` (only if you add a\n`.jsx`).\n\nPreview in a browser with live reload:\n\n```bash\nmalloyyo dashboard dev # serves at http://localhost:4173\n```\n\nTwo things about `dashboard dev` worth knowing:\n\n- It runs the queries **server-side**, through the same local DuckDB the CLI\n uses \u2014 the browser only renders results. So it works without DuckDB-WASM or\n its CDN extensions, and it is the only way to see a custom component render\n against real data short of publishing.\n- It holds **one long-lived connection set with a warm schema cache** (a\n deliberate tradeoff \u2014 refetching schemas per compile reads like a hang on a\n warehouse). The file watcher hot-reloads your `.malloy` edits and re-discovers\n `# artifact` tags, but nothing invalidates that cache, so a change to the\n DATA\'s shape \u2014 a new column in a parquet \u2014 needs a restart. The tell is\n `\'<column>\' is not defined` in the browser while the CLI and `lint` are green.\n\nIterate here until the dashboards look right. **Look at every dashboard before\nyou call it done**: `malloyyo lint` compiles the dashboards against the model but\ndoes not render them, so it cannot see a chart that throws at render time, a\ncomponent whose CSS resolved to nothing, or a column of raw field names.\n`malloyyo test` previews what the hosted claude.ai app would see.\n\n### 5. Build the static site\n\n```bash\nmalloyyo dashboard bundle --out docs\n# optional: add analytics + a title\nmalloyyo dashboard bundle --out docs --title "My Site" --analytics G-XXXXXXXXXX\n```\n\n`bundle` writes the HTML, `.nojekyll`, and assets into `docs/`, next to the\nparquet from step 2. The pages fetch `./*.parquet` relative to the site root, so\nthe data must already be in `docs/` \u2014 always run the data step before bundling.\n\n### 6. Publish on GitHub Pages\n\nCommit `docs/` and turn on Pages (Settings \u2192 Pages \u2192 Deploy from a branch \u2192\n`main` / `docs`). Full steps + gotchas: **`yo_help("site/github-pages")`**.\n\n### 7. (Optional) Keep it fresh automatically\n\nIf the data comes from a URL that updates over time and you want the site to\ntrack it, add a weekly GitHub Actions refresh \u2014 see\n**`yo_help("site/auto-update")`** (worked example: `malloydata/malloyyo-imdb`).\n\n## Done when\n\n- `malloyyo sql -e "SELECT count(*) FROM \'docs/<file>.parquet\'"` returns real rows\n- `malloyyo lint` passes\n- **you have LOOKED at every dashboard** in `malloyyo dashboard dev` \u2014 each one\n renders with no console errors, every tile has data, and the labels read like\n English rather than column names. `lint` passing is not this; a dashboard can\n compile and still throw at render time.\n- `docs/` has the bundled `*.html` + `.nojekyll` alongside the parquet\n- the Pages URL loads and the dashboards populate (data fetch succeeds)\n',
|
|
219
|
+
"site/data-to-parquet.md": '# Getting web data into `docs/*.parquet`\n\nThe site queries parquet files (in `docs/`, or at an https URL \u2014 see the model\nstep). This is how you turn a public data URL into that parquet. Pick the\nsimplest path that produces clean data.\n\n`malloyyo sql` runs SQL through malloyyo\'s **embedded DuckDB** \u2014 no standalone\n`duckdb` binary. It takes SQL from `-e`, `-f <file>`, or stdin, and runs against\na connection from `malloy-config.json` (default `duckdb`).\n\n## Path A \u2014 Direct (default)\n\nOne command reads the URL and writes parquet. Best when the web data is already\nclose to what you want to display.\n\n```bash\n# CSV / TSV\necho "COPY (SELECT * FROM read_csv_auto(\'https://example.com/data.csv\'))\n TO \'docs/data.parquet\' (FORMAT parquet)" | malloyyo sql\n\n# TSV, gzipped, tab-delimited, header row (IMDb-style)\necho "COPY (SELECT * FROM read_csv_auto(\'https://example.com/data.tsv.gz\',\n delim=\'\\t\', header=true, all_varchar=true))\n TO \'docs/data.parquet\' (FORMAT parquet)" | malloyyo sql\n\n# JSON / NDJSON\necho "COPY (SELECT * FROM read_json_auto(\'https://example.com/data.json\'))\n TO \'docs/data.parquet\' (FORMAT parquet)" | malloyyo sql\n\n# Already parquet on the web \u2014 just fetch it\ncurl -fsSL -o docs/data.parquet \'https://example.com/data.parquet\'\n```\n\nFor anything longer than a line or two, put the SQL in a file and run it with\n`-f`, so it\'s re-runnable and diffable:\n\n```bash\nmalloyyo sql -f scripts/build.sql # a file of ;-separated statements\n```\n\nNotes:\n- DuckDB autoloads `httpfs` for `https://` reads.\n- Keep only the columns/rows the dashboards need \u2014 `SELECT` the columns and add a\n `WHERE` to drop noise. Smaller parquet = faster page loads.\n- Cast types here if the source is all-strings: `col::INT`, `col::DOUBLE`, etc.\n\n## Path B \u2014 Transform with Malloy (when you need to reshape)\n\nUse when the data needs cleaning, joining across files, ranking, or nesting \u2014\nthe `malloydata/malloyyo-imdb` case. You write the transform once in Malloy;\n`malloy-cli build` materializes it; you export the tables to parquet. This path\nalso needs `@malloydata/cli` (`npm install -g @malloydata/cli`).\n\n**1. A build connection** \u2014 `malloy-build.json` (a DuckDB db just for building):\n\n```json\n{ "connections": { "build": { "is": "duckdb", "databasePath": "data/build.duckdb" } } }\n```\n\n**2. `transform.malloy`** \u2014 read the raw source(s) through the `build`\nconnection and mark each output source with `#@ persist name="\u2026"`:\n\n```malloy\n##! experimental.persistence experimental.virtual_source\n\nsource: raw is build.sql("""\n SELECT * FROM read_csv_auto(\'data/raw.csv.gz\', delim=\'\\t\', all_varchar=true, header=true)\n""")\n\n#@ persist name="thing"\nsource: thing_base is raw -> {\n where: some_count::number > 100\n select: id, name, value is value::number\n calculate: rank is rank() { order_by: value::number desc }\n}\n```\n\nDownload the raw files first (a `data/get.sh` that `wget`s the URLs into `data/`,\ngitignored). Gitignore `data/`, `data/*.duckdb`, and `MANIFESTS/`.\n\n**3. Build, then export the persisted tables to `docs/`** \u2014 the export still\ngoes through `malloyyo sql` (no standalone duckdb needed):\n\n```bash\nrm -f data/build.duckdb\nmalloy-cli -c malloy-build.json build transform.malloy # -> tables in build.duckdb\necho "ATTACH \'data/build.duckdb\' AS b (READ_ONLY);\n COPY b.thing TO \'docs/thing.parquet\' (FORMAT parquet)" | malloyyo sql\n```\n\nThe persist `name=` is the table name inside `build.duckdb`; the `COPY` names the\nserved file (they can differ \u2014 e.g. `thing` \u2192 `docs/mysite_thing.parquet`).\n\nWrap steps 1\u20133 in a single `scripts/build_data.sh` so it is one command, by hand\nand in CI. That\'s exactly what `yo_help("site/auto-update")` automates weekly.\n\n## Either way, verify\n\n```bash\nmalloyyo sql -e "DESCRIBE SELECT * FROM \'docs/thing.parquet\'"\nmalloyyo sql -e "SELECT count(*) FROM \'docs/thing.parquet\'"\n```\n\nThen point a storage source at it (docs-local shown; an https URL works too \u2014\nsee the model step in SKILL.md):\n\n```malloy\nsource: thing_table is duckdb.table(\'docs/thing.parquet\') extend {}\n```\n',
|
|
220
|
+
"site/github-pages.md": '# Publishing the site on GitHub Pages\n\nThe published site is just the `docs/` directory: bundled HTML plus the parquet\nit queries. GitHub Pages serves it directly \u2014 no build step on GitHub\'s side.\n\n## One-time setup\n\n```bash\n# from the repo root, after `malloyyo dashboard bundle --out docs`\ngh repo create <owner>/<name> --public --source=. --remote=origin # or an existing repo\ngit add -A\ngit commit -m "initial data site"\ngit push -u origin main\n```\n\nTurn on Pages, pointing at the `docs/` folder on the default branch:\n\n```bash\ngh api -X POST repos/<owner>/<name>/pages \\\n -f \'source[branch]=main\' -f \'source[path]=/docs\'\n```\n\n(Or in the UI: Settings \u2192 Pages \u2192 Source: **Deploy from a branch** \u2192 Branch\n`main`, folder `/docs`.)\n\nThe site appears at `https://<owner>.github.io/<name>/` within a minute or two.\n\n## Why `docs/` and why `.nojekyll`\n\n- Serving from `/docs` on the main branch keeps the data committed **once**\n (it\'s the same directory you build into), not duplicated on a separate branch.\n- `malloyyo dashboard bundle` writes a `.nojekyll` file so Pages serves the\n bundled assets as-is instead of running them through Jekyll (which would drop\n files beginning with `_`). Keep it committed.\n\n## Updating the site\n\nRe-run the data step and the bundle, then commit `docs/`:\n\n```bash\nbash scripts/build_data.sh # or your Path-A duckdb command \u2192 docs/*.parquet\nmalloyyo dashboard bundle --out docs\ngit add docs && git commit -m "update" && git push\n```\n\nBecause the dashboards fetch the parquet client-side, **committing fresh parquet\nis enough to update the live data** \u2014 you only need to re-`bundle` when you\nchange dashboard code. To automate the data refresh weekly, see the\n`yo_help("site/auto-update")`.\n\n## Gotchas\n\n- **Data must be in `docs/` before you bundle** \u2014 the pages fetch `./*.parquet`\n relative to the site root; `bundle` also reads the parquet to get schemas.\n- **Big parquet grows git history.** Each committed refresh adds the full file.\n Fine for occasional updates; for frequent auto-refresh, see the flatten note\n in `yo_help("site/auto-update")`.\n- **Git LFS does not work with Pages "deploy from a branch"** \u2014 Pages serves the\n LFS pointer text, not the file. Commit parquet as normal git objects.\n- **Private repos**: Pages on private repos needs a paid plan. Use a public repo\n for a public site.\n',
|
|
217
221
|
"writing-malloy-with-mcp.md": "---\ndescription: How to write Malloy over an MCP surface \u2014 the compiler-in-the-loop discipline, common errors, and givens. Tool-agnostic; read once.\n---\n# Writing Malloy over MCP\n\nMalloy is a semantic language: a source already carries measures, dimensions,\nviews, and joins, and the compiler typechecks every query against them. The\nsingle most useful habit is **let the compiler be ground truth** \u2014 don't guess\nsyntax or field names. Read the source's shape first, validate before you run,\nand read the `problems[]` the surface returns.\n\nThe exact tools differ by surface (an explore surface exposes describe + query;\nan authoring surface adds compile/prettify), but the loop is the same:\n\n1. **Read the shape.** Describe the source you're querying \u2014 its measures,\n dimensions, views, and joins. The model usually already defines the\n aggregation you want; reuse it instead of re-deriving it.\n2. **Validate, then run.** Compile/validate the query first (no execution) to\n confirm it typechecks and to see the generated SQL or the givens it needs;\n fix any `problems[]`, then execute to get rows.\n3. **Recover from problems[].** Every failure \u2014 parse, field-not-found,\n aggregate-locality, runtime \u2014 comes back as a uniform `problems[]` with a\n `code` and (when known) a `help_topic`. Pull that topic with `yo_help`.\n\n## Common errors and how to read them\n\n- **Unknown field** \u2014 describe the source and check its dimensions / measures /\n views / joins for what actually exists. A join rendered by `source_ref` is\n described under that name in the same response.\n- **Aggregate locality** \u2014 `sum(joined.x)` across a join needs explicit\n locality: `source.sum(joined.x)` or `joined.x.sum()`.\n- **Mixed reduction / projection** \u2014 one query stage is either\n `group_by:`/`aggregate:` OR `select:`, never both.\n- **Calculation in a source** \u2014 `calculate:` (window functions) lives in\n queries, not in source definitions.\n\n## Givens (`$NAME` parameters)\n\nSome models declare given parameters (`$TENANT`, `$MAX_ROWS`, \u2026). Validate a\nquery with execution off to learn which givens it references (with their types\nand whether a default exists), then supply values keyed by surface name (no\n`$`). A given with a default is optional; one without must be supplied. For the\nper-type value shapes (dates as ISO strings, records as objects, `filter<T>` as\na Malloy filter string, \u2026) pull `yo_help(\"givens\")` \u2014 don't guess; the compiler\nvalidates and points at the offending field.\n\n## Before writing non-trivial Malloy\n\nBrowse the `language/*` topics first (start with `yo_help(\"language/overview\")`).\nThe language has real scoping and typing rules the compiler enforces \u2014 reading\nthe reference beats guessing.\n"
|
|
218
222
|
};
|
|
219
223
|
var prompts = {
|
|
@@ -2881,7 +2885,7 @@ function clearCreds(url6) {
|
|
|
2881
2885
|
}
|
|
2882
2886
|
|
|
2883
2887
|
// package.json
|
|
2884
|
-
var version = "0.2.
|
|
2888
|
+
var version = "0.2.45";
|
|
2885
2889
|
|
|
2886
2890
|
// src/http.ts
|
|
2887
2891
|
var USER_AGENT = `malloyyo/${version}`;
|
|
@@ -4302,13 +4306,15 @@ function scaffoldIndex(root) {
|
|
|
4302
4306
|
note: anyNames ? `wrote index.malloy re-exporting ${models.length} model file(s) \u2014 REVIEW it` : "wrote index.malloy skeleton \u2014 no exports detected, fill them in by hand"
|
|
4303
4307
|
};
|
|
4304
4308
|
}
|
|
4305
|
-
function
|
|
4309
|
+
function templatePath(...parts) {
|
|
4306
4310
|
const distDir = path9.dirname(fileURLToPath2(import.meta.url));
|
|
4307
|
-
|
|
4308
|
-
path9.join(distDir, "templates",
|
|
4309
|
-
path9.join(distDir, "..", "src", "templates",
|
|
4310
|
-
];
|
|
4311
|
-
|
|
4311
|
+
return [
|
|
4312
|
+
path9.join(distDir, "templates", ...parts),
|
|
4313
|
+
path9.join(distDir, "..", "src", "templates", ...parts)
|
|
4314
|
+
].find((p) => fs8.existsSync(p));
|
|
4315
|
+
}
|
|
4316
|
+
function installSkills(root) {
|
|
4317
|
+
const srcSkills = templatePath("skills");
|
|
4312
4318
|
if (!srcSkills) return { wrote: [], skipped: [], note: "no skill templates found \u2014 skipped" };
|
|
4313
4319
|
const destSkills = path9.join(root, ".claude", "skills");
|
|
4314
4320
|
fs8.mkdirSync(destSkills, { recursive: true });
|
|
@@ -4327,6 +4333,20 @@ function installSkills(root) {
|
|
|
4327
4333
|
}
|
|
4328
4334
|
return { wrote, skipped };
|
|
4329
4335
|
}
|
|
4336
|
+
function installDevcontainer(root) {
|
|
4337
|
+
const dest = path9.join(root, ".devcontainer", "devcontainer.json");
|
|
4338
|
+
if (fs8.existsSync(dest)) {
|
|
4339
|
+
return { wrote: false, note: ".devcontainer/devcontainer.json exists \u2014 left as-is" };
|
|
4340
|
+
}
|
|
4341
|
+
const src = templatePath("devcontainer", "devcontainer.json");
|
|
4342
|
+
if (!src) return { wrote: false, note: "no devcontainer template found \u2014 skipped" };
|
|
4343
|
+
fs8.mkdirSync(path9.dirname(dest), { recursive: true });
|
|
4344
|
+
fs8.copyFileSync(src, dest);
|
|
4345
|
+
return {
|
|
4346
|
+
wrote: true,
|
|
4347
|
+
note: "wrote .devcontainer/devcontainer.json \u2014 commit it, then Code \u2192 Codespaces on GitHub"
|
|
4348
|
+
};
|
|
4349
|
+
}
|
|
4330
4350
|
async function initCmd(dir) {
|
|
4331
4351
|
const root = path9.resolve(dir);
|
|
4332
4352
|
if (!fs8.existsSync(root) || !fs8.statSync(root).isDirectory()) {
|
|
@@ -4354,6 +4374,8 @@ async function initCmd(dir) {
|
|
|
4354
4374
|
console.log(`\u2022 skill(s) already present \u2014 left as-is: ${sk.skipped.join(", ")}`);
|
|
4355
4375
|
}
|
|
4356
4376
|
}
|
|
4377
|
+
const dev = installDevcontainer(root);
|
|
4378
|
+
console.log(`${dev.wrote ? "\u2713" : "\u2022"} ${dev.note}`);
|
|
4357
4379
|
const perms = allowAuthorTools(root);
|
|
4358
4380
|
if (perms.note) {
|
|
4359
4381
|
console.log(`\u2022 ${perms.note}`);
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
// Written by `malloyyo init` into a Malloy MODEL repo — this is what makes the
|
|
2
|
+
// repo openable as a GitHub Codespace (or a local Dev Container) with the whole
|
|
3
|
+
// toolchain already installed: Claude Code, the Malloy and Claude VS Code
|
|
4
|
+
// extensions, the `malloyyo` CLI, Node 24, Playwright/Chromium, the Google
|
|
5
|
+
// Cloud CLI. `init` never overwrites an existing one, so local edits survive.
|
|
6
|
+
//
|
|
7
|
+
// The image is prebuilt and published by the malloyyo repository
|
|
8
|
+
// (devcontainer/Dockerfile there), so creating the codespace is a pull rather
|
|
9
|
+
// than a build. The extensions and the dashboard ports are not listed here
|
|
10
|
+
// because the image carries them itself, in a `devcontainer.metadata` label
|
|
11
|
+
// that the Dev Containers tooling merges into this file.
|
|
12
|
+
//
|
|
13
|
+
// See https://github.com/malloydata/malloyyo/blob/main/docs/devcontainer.md
|
|
14
|
+
{
|
|
15
|
+
"name": "Malloy model",
|
|
16
|
+
"image": "ghcr.io/malloydata/malloyyo-devcontainer:latest",
|
|
17
|
+
|
|
18
|
+
// Makes `cd <repo> && claude` open in AUTHOR mode: writes .mcp.json (the
|
|
19
|
+
// `malloyyo mcp --develop` server), pre-approves its tools in
|
|
20
|
+
// .claude/settings.json, and scaffolds index.malloy if the repo has none. It
|
|
21
|
+
// merges rather than clobbers, so re-running it on every rebuild is safe —
|
|
22
|
+
// and once you have committed what it writes, you can drop this line.
|
|
23
|
+
"postCreateCommand": "malloyyo init",
|
|
24
|
+
|
|
25
|
+
// One screen of "what do I type now", printed into each new terminal.
|
|
26
|
+
"postAttachCommand": "malloyyo-devcontainer-info",
|
|
27
|
+
|
|
28
|
+
// Codespaces secrets (repo Settings → Secrets and variables → Codespaces)
|
|
29
|
+
// arrive as environment variables on their own — nothing to declare here.
|
|
30
|
+
// This block is for a LOCAL Dev Container, where the values have to be
|
|
31
|
+
// forwarded from your own shell. Never commit the values themselves.
|
|
32
|
+
//
|
|
33
|
+
// "remoteEnv": {
|
|
34
|
+
// "MALLOYYO_TOKEN": "${localEnv:MALLOYYO_TOKEN}",
|
|
35
|
+
// "ANTHROPIC_API_KEY": "${localEnv:ANTHROPIC_API_KEY}"
|
|
36
|
+
// },
|
|
37
|
+
|
|
38
|
+
"hostRequirements": {
|
|
39
|
+
"cpus": 2,
|
|
40
|
+
"memory": "4gb"
|
|
41
|
+
}
|
|
42
|
+
}
|
|
@@ -5,113 +5,12 @@ description: Keep a malloyyo GitHub-Pages data site current automatically with a
|
|
|
5
5
|
|
|
6
6
|
# Auto-update a data site weekly
|
|
7
7
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
(built with `malloyyo-data-site`). If the data never changes, skip this.
|
|
8
|
+
The procedure ships with the CLI, so it always matches the `malloyyo` you have
|
|
9
|
+
installed rather than whatever was current when this repo was scaffolded:
|
|
11
10
|
|
|
12
|
-
The worked example is `malloydata/malloyyo-imdb` — fetch and read its
|
|
13
|
-
`scripts/build_data.sh` and `.github/workflows/refresh-data.yml`; adapt, don't
|
|
14
|
-
copy blindly.
|
|
15
|
-
|
|
16
|
-
## Precondition: one command that rebuilds the data
|
|
17
|
-
|
|
18
|
-
The whole thing rests on a single script that turns the source URL(s) into the
|
|
19
|
-
committed parquet — the same script you'd run by hand. Keep it linear so it
|
|
20
|
-
reads as a recipe:
|
|
21
|
-
|
|
22
|
-
```bash
|
|
23
|
-
# scripts/build_data.sh
|
|
24
|
-
set -euo pipefail
|
|
25
|
-
cd "$(dirname "$0")/.."
|
|
26
|
-
bash data/get.sh # 1. download source URLs -> data/
|
|
27
|
-
rm -f data/build.duckdb
|
|
28
|
-
malloy-cli -c malloy-build.json build transform.malloy # 2. transform
|
|
29
|
-
echo "ATTACH 'data/build.duckdb' AS b (READ_ONLY); -- 3. export -> docs/*.parquet
|
|
30
|
-
COPY b.thing TO 'docs/thing.parquet' (FORMAT parquet)" | malloyyo sql
|
|
31
|
-
```
|
|
32
|
-
|
|
33
|
-
(If you used Path A — direct — the whole script is just the one
|
|
34
|
-
`echo "COPY (FROM read_..('https://…')) TO 'docs/…'" | malloyyo sql` command.
|
|
35
|
-
No `data/get.sh`, `malloy-build.json`, `malloy-cli`, or transform needed — and
|
|
36
|
-
no standalone `duckdb` either, since `malloyyo sql` uses the embedded one.)
|
|
37
|
-
|
|
38
|
-
Gitignore the working files: `data/*.gz`, `data/*.duckdb`, `MANIFESTS/`.
|
|
39
|
-
|
|
40
|
-
## The workflow
|
|
41
|
-
|
|
42
|
-
`.github/workflows/refresh-data.yml`:
|
|
43
|
-
|
|
44
|
-
```yaml
|
|
45
|
-
name: refresh-data
|
|
46
|
-
on:
|
|
47
|
-
schedule:
|
|
48
|
-
- cron: "0 6 * * 0" # weekly, Sunday 06:00 UTC
|
|
49
|
-
workflow_dispatch: # ...and on demand from the Actions tab
|
|
50
|
-
jobs:
|
|
51
|
-
refresh:
|
|
52
|
-
runs-on: ubuntu-latest
|
|
53
|
-
permissions:
|
|
54
|
-
contents: write # so the job can commit + push
|
|
55
|
-
steps:
|
|
56
|
-
- uses: actions/checkout@v4
|
|
57
|
-
- uses: actions/setup-node@v4
|
|
58
|
-
with: { node-version: "20" }
|
|
59
|
-
# malloyyo carries its own DuckDB (`malloyyo sql`), so no duckdb CLI to install.
|
|
60
|
-
# Add `@malloydata/cli` only for a `#@ persist` transform (malloy-cli build).
|
|
61
|
-
- run: npm install -g @malloydata/malloyyo @malloydata/cli
|
|
62
|
-
- uses: actions/setup-python@v5 # only if an enrichment step needs it
|
|
63
|
-
with: { python-version: "3.12" }
|
|
64
|
-
- name: Build data
|
|
65
|
-
run: bash scripts/build_data.sh
|
|
66
|
-
- name: Commit refreshed data
|
|
67
|
-
run: |
|
|
68
|
-
git config user.name github-actions
|
|
69
|
-
git config user.email github-actions@github.com
|
|
70
|
-
git add docs/*.parquet
|
|
71
|
-
git commit -m "refresh data $(date -u +%F)" || exit 0 # unchanged = clean no-op
|
|
72
|
-
git push
|
|
73
11
|
```
|
|
74
|
-
|
|
75
|
-
## Hard-won details (these came from a real first run)
|
|
76
|
-
|
|
77
|
-
- **`git commit … || exit 0`** — an unchanged week means nothing new upstream.
|
|
78
|
-
That is success, not a failed job.
|
|
79
|
-
- **Secondary/enrichment steps must be best-effort.** If the build has a step
|
|
80
|
-
that can fail independently (an external API lookup, an optional metadata
|
|
81
|
-
enrichment) and it isn't essential to the data, mark it
|
|
82
|
-
`continue-on-error: true`. Otherwise one flaky API call throws away an
|
|
83
|
-
otherwise-good data refresh — the commit step never runs and the fresh parquet
|
|
84
|
-
is lost. (In `malloyyo-imdb` this bit us: a poster-image lookup with a missing
|
|
85
|
-
secret failed the whole job until it was made non-blocking.)
|
|
86
|
-
- **Secrets** go in repo settings (Settings → Secrets and variables → Actions,
|
|
87
|
-
or `gh secret set NAME`), referenced as `${{ secrets.NAME }}`. A step reading
|
|
88
|
-
a secret that isn't set gets an empty string — pair that with best-effort.
|
|
89
|
-
- **No re-bundle needed for data changes.** The dashboards fetch parquet at
|
|
90
|
-
runtime, so committing fresh parquet updates the live site. Only install
|
|
91
|
-
`@malloydata/malloyyo` and run `malloyyo dashboard bundle` in CI if the job
|
|
92
|
-
also needs to regenerate the HTML (i.e. dashboard code changed — rare for a
|
|
93
|
-
data refresh).
|
|
94
|
-
- **Test before trusting the schedule.** Push, then Actions tab → the workflow →
|
|
95
|
-
**Run workflow** (that's `workflow_dispatch`). Or `gh workflow run
|
|
96
|
-
refresh-data.yml`, then `gh run watch <id> --exit-status`.
|
|
97
|
-
|
|
98
|
-
## Git growth, and the flatten
|
|
99
|
-
|
|
100
|
-
Each committed refresh adds the full parquet to history (~its file size per
|
|
101
|
-
run). That is intentional simplicity, not a leak. When history gets too big,
|
|
102
|
-
flatten it to a single commit by hand — **not** as part of the weekly job:
|
|
103
|
-
|
|
104
|
-
```bash
|
|
105
|
-
git checkout --orphan flat && git add -A && git commit -m "flatten history"
|
|
106
|
-
git branch -M flat main && git push -f origin main
|
|
12
|
+
mcp__malloyyo_author__yo_help("site/auto-update")
|
|
107
13
|
```
|
|
108
14
|
|
|
109
|
-
|
|
110
|
-
`
|
|
111
|
-
Document this in the repo's README so future-you remembers it's available.
|
|
112
|
-
|
|
113
|
-
## Done when
|
|
114
|
-
|
|
115
|
-
- The workflow file is on the default branch (so `workflow_dispatch` shows up)
|
|
116
|
-
- A manual run goes green end-to-end and commits fresh `docs/*.parquet`
|
|
117
|
-
- The Pages site shows the new data
|
|
15
|
+
Read it and follow it. Do this **after** the site works — that is
|
|
16
|
+
`site/data-site`. If the data never changes, skip it entirely.
|
|
@@ -5,229 +5,16 @@ description: Turn a public data URL into a browsable, interactive dashboard site
|
|
|
5
5
|
|
|
6
6
|
# Build a public data site from a web data pointer
|
|
7
7
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
up as parquet in `docs/`; the dashboards are static HTML that query it
|
|
11
|
-
client-side with DuckDB-WASM — so there is no server, no database to run.
|
|
8
|
+
The procedure ships with the CLI, so it always matches the `malloyyo` you have
|
|
9
|
+
installed rather than whatever was current when this repo was scaffolded:
|
|
12
10
|
|
|
13
|
-
Two reference repos — read them, they are the ground truth for structure:
|
|
14
|
-
|
|
15
|
-
- **`malloydata/malloyyo-babynames`** — the clean base pattern (one model file,
|
|
16
|
-
a few dashboards, `docs/`). Data is static.
|
|
17
|
-
- **`malloydata/malloyyo-imdb`** — the same, plus a `transform.malloy` that
|
|
18
|
-
cleans raw source files into parquet, plus a **weekly auto-refresh** (that
|
|
19
|
-
part is the separate `malloyyo-auto-update` skill).
|
|
20
|
-
|
|
21
|
-
Fetch either with `gh api repos/<repo>/git/trees/HEAD?recursive=1` and read the
|
|
22
|
-
files you need. Match their layout rather than inventing your own.
|
|
23
|
-
|
|
24
|
-
## The shape of a finished repo
|
|
25
|
-
|
|
26
|
-
```
|
|
27
|
-
malloy-config.json connection(s): duckdb (local), optionally md/gs mirrors
|
|
28
|
-
index.malloy the EXPORT SURFACE — only what this file exports is live
|
|
29
|
-
<model>.malloy sources, measures, joins, givens (parameters)
|
|
30
|
-
storage.malloy sources point at the parquet — docs-local OR an https URL
|
|
31
|
-
(or gs.malloy / md.malloy) same source names, swappable hosting (step 3)
|
|
32
|
-
dashboards/*.malloy the query behind each dashboard
|
|
33
|
-
dashboards/*.jsx the dashboard layout (grid of charts/tables)
|
|
34
|
-
docs/ PUBLISHED SITE — bundled HTML (+ the *.parquet if docs-local)
|
|
35
|
-
*.parquet the data, when hosted docs-local (committed; served by Pages)
|
|
36
|
-
*.html .nojekyll written by `malloyyo dashboard bundle`
|
|
37
|
-
.mcp.json written by `malloyyo init` (author-mode Claude)
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
## Prerequisites (install once)
|
|
41
|
-
|
|
42
|
-
```bash
|
|
43
|
-
npm install -g @malloydata/malloyyo # the `malloyyo` command
|
|
44
|
-
```
|
|
45
|
-
|
|
46
|
-
That's the whole toolchain for the common case. `malloyyo` has **DuckDB built
|
|
47
|
-
in** — `malloyyo sql` runs SQL (read a URL, `COPY` to parquet) through it, so you
|
|
48
|
-
do **not** need a standalone `duckdb` CLI.
|
|
49
|
-
|
|
50
|
-
Add `@malloydata/cli` (`malloy-cli`) **only** if you do a heavier transform with
|
|
51
|
-
Malloy `#@ persist` (see Path B in `reference/data-to-parquet.md`):
|
|
52
|
-
|
|
53
|
-
```bash
|
|
54
|
-
npm install -g @malloydata/cli # only for #@ persist transforms
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
## Recipe
|
|
58
|
-
|
|
59
|
-
Work top to bottom. After each step, prove it before moving on.
|
|
60
|
-
|
|
61
|
-
### 1. Scaffold
|
|
62
|
-
|
|
63
|
-
```bash
|
|
64
|
-
mkdir my-site && cd my-site && git init
|
|
65
|
-
malloyyo init # writes .mcp.json (author-mode Claude) + index.malloy stub
|
|
66
11
|
```
|
|
67
|
-
|
|
68
|
-
Create `malloy-config.json` with a local DuckDB connection (this is what reads
|
|
69
|
-
the parquet):
|
|
70
|
-
|
|
71
|
-
```json
|
|
72
|
-
{ "connections": { "duckdb": { "is": "duckdb" } } }
|
|
12
|
+
mcp__malloyyo_author__yo_help("site/data-site")
|
|
73
13
|
```
|
|
74
14
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
This is the one dataset-specific step. Two paths — pick the simpler one that
|
|
78
|
-
works. Full detail and copy-paste commands: **`reference/data-to-parquet.md`**.
|
|
79
|
-
|
|
80
|
-
- **Direct** (default): `malloyyo sql` reads the URL and writes parquet in one
|
|
81
|
-
shot, through the embedded DuckDB:
|
|
82
|
-
```bash
|
|
83
|
-
echo "COPY (FROM read_csv_auto('https://…/thing.csv')) TO 'docs/thing.parquet'" | malloyyo sql
|
|
84
|
-
```
|
|
85
|
-
Use when the web data is already close to what you want to show.
|
|
86
|
-
- **Transform** (when you need to clean / join / rank / reshape): write a
|
|
87
|
-
`transform.malloy` with `#@ persist` sources and build it with `malloy-cli`,
|
|
88
|
-
then export the tables to `docs/*.parquet`. This is the `malloyyo-imdb`
|
|
89
|
-
pattern.
|
|
90
|
-
|
|
91
|
-
Verify: `malloyyo sql -e "DESCRIBE SELECT * FROM 'docs/thing.parquet'"` and a
|
|
92
|
-
`SELECT count(*)`.
|
|
93
|
-
|
|
94
|
-
### 3. Write the model
|
|
95
|
-
|
|
96
|
-
A storage file — one source per parquet file — that the rest of the model builds
|
|
97
|
-
on. **Where the parquet lives is a choice** (the two examples differ here):
|
|
98
|
-
|
|
99
|
-
- **docs-local** (the `malloyyo-imdb` way) — parquet committed in `docs/`,
|
|
100
|
-
served same-origin by Pages. Self-contained; git carries the data. Address it
|
|
101
|
-
by project-relative path so the same spelling works locally and published:
|
|
102
|
-
```malloy
|
|
103
|
-
// storage.malloy
|
|
104
|
-
source: thing_table is duckdb.table('docs/thing.parquet') extend {}
|
|
105
|
-
```
|
|
106
|
-
- **External URL** (the `malloyyo-babynames` way — it uses `import "gs.malloy"`)
|
|
107
|
-
— parquet hosted on GCS / a CDN / any https URL; `docs/` holds only the HTML,
|
|
108
|
-
so git stays small. DuckDB-WASM fetches the URL at runtime:
|
|
109
|
-
```malloy
|
|
110
|
-
// gs.malloy
|
|
111
|
-
source: thing_table is duckdb.table('https://storage.googleapis.com/…/thing.parquet') extend {}
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
Keep the **source names identical** across storage files (as babynames does with
|
|
115
|
-
`gs.malloy` / `md.malloy`) so switching hosting is a one-line import change in
|
|
116
|
-
the model. Start docs-local (simplest); move to a URL if git growth bites.
|
|
117
|
-
|
|
118
|
-
`<model>.malloy` — build real sources on top: primary keys, measures, joins,
|
|
119
|
-
and **givens** (the parameters that become the dashboard's filter controls).
|
|
120
|
-
`index.malloy` — re-export exactly the sources/queries/givens the dashboards
|
|
121
|
-
use. **Only what `index.malloy` exports is visible** to dashboards, `dashboard
|
|
122
|
-
dev`, and the hosted app.
|
|
123
|
-
|
|
124
|
-
Give each exported source a `#"` doc string. It's what `list_sources` shows the
|
|
125
|
-
hosted app and any MCP client, and it's the only place to record the things a
|
|
126
|
-
consumer can't infer — which source answers which question, and any measure
|
|
127
|
-
whose meaning is subtle.
|
|
128
|
-
|
|
129
|
-
For Malloy modeling and givens specifics, lean on the author MCP rather than
|
|
130
|
-
guessing: the repo's `.mcp.json` wires `mcp__malloyyo_author__*`. Call
|
|
131
|
-
`mcp__malloyyo_author__compile` to check files and
|
|
132
|
-
`mcp__malloyyo_author__yo_help` for topics (`develop/working-with-models`).
|
|
133
|
-
|
|
134
|
-
Three shapes that compile, pass `lint`, and still produce a dashboard that looks
|
|
135
|
-
right and isn't:
|
|
136
|
-
|
|
137
|
-
- **A stage-level `where:` zeroes measures that carry their own filters.**
|
|
138
|
-
`count() { where: status = 'Sold' }` needs the sold rows to reach it; a
|
|
139
|
-
`where: in_cellar` on the stage removes them first and the measure reads 0
|
|
140
|
-
everywhere. Constrain the output with `having:` instead.
|
|
141
|
-
- **Grouping finer than the fact scatters the counts.** Group by an attribute
|
|
142
|
-
that only applies to some rows and the measures over the others split across
|
|
143
|
-
groups and read near-zero. Pick the grain the question is asked at; if you
|
|
144
|
-
need both, group at the coarser one and `nest:` the finer.
|
|
145
|
-
- **Detail tables default to raw field names and bare numbers.** `exit_date` and
|
|
146
|
-
`829.57`, and a null renders as `∅`. Tag them — `# label="Opened"`,
|
|
147
|
-
`# label="Paid" currency` — and `coalesce(field, '')` where empty is
|
|
148
|
-
meaningful rather than missing.
|
|
149
|
-
|
|
150
|
-
Aggregate locality is NOT one of these: Malloy's symmetric aggregates keep
|
|
151
|
-
measures honest through a `join_many` fan-out, and it warns (or errors) rather
|
|
152
|
-
than silently answering when an `avg`/`sum` crosses a join without explicit
|
|
153
|
-
locality. `notes.rating.avg()` already averages at the notes grain; ask for
|
|
154
|
-
`source.avg(notes.rating)` only when you want the fan-out-weighted number. See
|
|
155
|
-
`yo_help language/aggregate-locality-symmetric-aggregates`.
|
|
156
|
-
|
|
157
|
-
### 4. Author dashboards and preview live
|
|
158
|
-
|
|
159
|
-
Each dashboard is **one self-contained `dashboards/<name>.malloy`** — the query,
|
|
160
|
-
its filtering, and the tags that lay it out. The filename is the dashboard's
|
|
161
|
-
name and URL slug.
|
|
162
|
-
|
|
163
|
-
**A `.jsx` is OPTIONAL and most dashboards don't need one.** The default form —
|
|
164
|
-
an inline query tagged `# artifact` + `# dashboard { columns=6 }` — renders KPI
|
|
165
|
-
tiles and a card grid with no React at all. Reach for a sibling
|
|
166
|
-
`dashboards/<name>.jsx` only when you want bespoke layout or presentation the
|
|
167
|
-
tags can't express (a hand-built card, a picker, a Vega chart).
|
|
168
|
-
|
|
169
|
-
Author with the `malloyyo_author` MCP and its `yo_help` topics — **read these,
|
|
170
|
-
don't guess the API**: `dashboards/authoring` (the dashboard file),
|
|
171
|
-
`dashboards/grid-layout` (colspans), `dashboards/charts` (chart tags and their
|
|
172
|
-
channel rules), `dashboards/givens-and-controls` (filters),
|
|
173
|
-
`dashboards/custom-components` and `dashboards/vega-charts` (only if you add a
|
|
174
|
-
`.jsx`).
|
|
175
|
-
|
|
176
|
-
Preview in a browser with live reload:
|
|
177
|
-
|
|
178
|
-
```bash
|
|
179
|
-
malloyyo dashboard dev # serves at http://localhost:4173
|
|
180
|
-
```
|
|
181
|
-
|
|
182
|
-
Two things about `dashboard dev` worth knowing:
|
|
183
|
-
|
|
184
|
-
- It runs the queries **server-side**, through the same local DuckDB the CLI
|
|
185
|
-
uses — the browser only renders results. So it works without DuckDB-WASM or
|
|
186
|
-
its CDN extensions, and it is the only way to see a custom component render
|
|
187
|
-
against real data short of publishing.
|
|
188
|
-
- It holds **one long-lived connection set with a warm schema cache** (a
|
|
189
|
-
deliberate tradeoff — refetching schemas per compile reads like a hang on a
|
|
190
|
-
warehouse). The file watcher hot-reloads your `.malloy` edits and re-discovers
|
|
191
|
-
`# artifact` tags, but nothing invalidates that cache, so a change to the
|
|
192
|
-
DATA's shape — a new column in a parquet — needs a restart. The tell is
|
|
193
|
-
`'<column>' is not defined` in the browser while the CLI and `lint` are green.
|
|
194
|
-
|
|
195
|
-
Iterate here until the dashboards look right. **Look at every dashboard before
|
|
196
|
-
you call it done**: `malloyyo lint` compiles the dashboards against the model but
|
|
197
|
-
does not render them, so it cannot see a chart that throws at render time, a
|
|
198
|
-
component whose CSS resolved to nothing, or a column of raw field names.
|
|
199
|
-
`malloyyo test` previews what the hosted claude.ai app would see.
|
|
200
|
-
|
|
201
|
-
### 5. Build the static site
|
|
202
|
-
|
|
203
|
-
```bash
|
|
204
|
-
malloyyo dashboard bundle --out docs
|
|
205
|
-
# optional: add analytics + a title
|
|
206
|
-
malloyyo dashboard bundle --out docs --title "My Site" --analytics G-XXXXXXXXXX
|
|
207
|
-
```
|
|
208
|
-
|
|
209
|
-
`bundle` writes the HTML, `.nojekyll`, and assets into `docs/`, next to the
|
|
210
|
-
parquet from step 2. The pages fetch `./*.parquet` relative to the site root, so
|
|
211
|
-
the data must already be in `docs/` — always run the data step before bundling.
|
|
212
|
-
|
|
213
|
-
### 6. Publish on GitHub Pages
|
|
214
|
-
|
|
215
|
-
Commit `docs/` and turn on Pages (Settings → Pages → Deploy from a branch →
|
|
216
|
-
`main` / `docs`). Full steps + gotchas: **`reference/publish-to-github-pages.md`**.
|
|
217
|
-
|
|
218
|
-
### 7. (Optional) Keep it fresh automatically
|
|
219
|
-
|
|
220
|
-
If the data comes from a URL that updates over time and you want the site to
|
|
221
|
-
track it, add a weekly GitHub Actions refresh — see the **`malloyyo-auto-update`**
|
|
222
|
-
skill (worked example: `malloydata/malloyyo-imdb`).
|
|
223
|
-
|
|
224
|
-
## Done when
|
|
15
|
+
Read it and follow it. It links on to `site/data-to-parquet`,
|
|
16
|
+
`site/github-pages`, and `site/auto-update` for the steps that need more detail.
|
|
225
17
|
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
renders with no console errors, every tile has data, and the labels read like
|
|
230
|
-
English rather than column names. `lint` passing is not this; a dashboard can
|
|
231
|
-
compile and still throw at render time.
|
|
232
|
-
- `docs/` has the bundled `*.html` + `.nojekyll` alongside the parquet
|
|
233
|
-
- the Pages URL loads and the dashboards populate (data fetch succeeds)
|
|
18
|
+
(No `malloyyo_author` server? `malloyyo init` writes the `.mcp.json` that wires
|
|
19
|
+
it — that is step 1 of the procedure anyway. Outside an MCP client, the same
|
|
20
|
+
text is `malloyyo mcp --develop` → `yo_help`.)
|
package/package.json
CHANGED
|
@@ -1,109 +0,0 @@
|
|
|
1
|
-
# Getting web data into `docs/*.parquet`
|
|
2
|
-
|
|
3
|
-
The site queries parquet files (in `docs/`, or at an https URL — see the model
|
|
4
|
-
step). This is how you turn a public data URL into that parquet. Pick the
|
|
5
|
-
simplest path that produces clean data.
|
|
6
|
-
|
|
7
|
-
`malloyyo sql` runs SQL through malloyyo's **embedded DuckDB** — no standalone
|
|
8
|
-
`duckdb` binary. It takes SQL from `-e`, `-f <file>`, or stdin, and runs against
|
|
9
|
-
a connection from `malloy-config.json` (default `duckdb`).
|
|
10
|
-
|
|
11
|
-
## Path A — Direct (default)
|
|
12
|
-
|
|
13
|
-
One command reads the URL and writes parquet. Best when the web data is already
|
|
14
|
-
close to what you want to display.
|
|
15
|
-
|
|
16
|
-
```bash
|
|
17
|
-
# CSV / TSV
|
|
18
|
-
echo "COPY (SELECT * FROM read_csv_auto('https://example.com/data.csv'))
|
|
19
|
-
TO 'docs/data.parquet' (FORMAT parquet)" | malloyyo sql
|
|
20
|
-
|
|
21
|
-
# TSV, gzipped, tab-delimited, header row (IMDb-style)
|
|
22
|
-
echo "COPY (SELECT * FROM read_csv_auto('https://example.com/data.tsv.gz',
|
|
23
|
-
delim='\t', header=true, all_varchar=true))
|
|
24
|
-
TO 'docs/data.parquet' (FORMAT parquet)" | malloyyo sql
|
|
25
|
-
|
|
26
|
-
# JSON / NDJSON
|
|
27
|
-
echo "COPY (SELECT * FROM read_json_auto('https://example.com/data.json'))
|
|
28
|
-
TO 'docs/data.parquet' (FORMAT parquet)" | malloyyo sql
|
|
29
|
-
|
|
30
|
-
# Already parquet on the web — just fetch it
|
|
31
|
-
curl -fsSL -o docs/data.parquet 'https://example.com/data.parquet'
|
|
32
|
-
```
|
|
33
|
-
|
|
34
|
-
For anything longer than a line or two, put the SQL in a file and run it with
|
|
35
|
-
`-f`, so it's re-runnable and diffable:
|
|
36
|
-
|
|
37
|
-
```bash
|
|
38
|
-
malloyyo sql -f scripts/build.sql # a file of ;-separated statements
|
|
39
|
-
```
|
|
40
|
-
|
|
41
|
-
Notes:
|
|
42
|
-
- DuckDB autoloads `httpfs` for `https://` reads.
|
|
43
|
-
- Keep only the columns/rows the dashboards need — `SELECT` the columns and add a
|
|
44
|
-
`WHERE` to drop noise. Smaller parquet = faster page loads.
|
|
45
|
-
- Cast types here if the source is all-strings: `col::INT`, `col::DOUBLE`, etc.
|
|
46
|
-
|
|
47
|
-
## Path B — Transform with Malloy (when you need to reshape)
|
|
48
|
-
|
|
49
|
-
Use when the data needs cleaning, joining across files, ranking, or nesting —
|
|
50
|
-
the `malloydata/malloyyo-imdb` case. You write the transform once in Malloy;
|
|
51
|
-
`malloy-cli build` materializes it; you export the tables to parquet. This path
|
|
52
|
-
also needs `@malloydata/cli` (`npm install -g @malloydata/cli`).
|
|
53
|
-
|
|
54
|
-
**1. A build connection** — `malloy-build.json` (a DuckDB db just for building):
|
|
55
|
-
|
|
56
|
-
```json
|
|
57
|
-
{ "connections": { "build": { "is": "duckdb", "databasePath": "data/build.duckdb" } } }
|
|
58
|
-
```
|
|
59
|
-
|
|
60
|
-
**2. `transform.malloy`** — read the raw source(s) through the `build`
|
|
61
|
-
connection and mark each output source with `#@ persist name="…"`:
|
|
62
|
-
|
|
63
|
-
```malloy
|
|
64
|
-
##! experimental.persistence experimental.virtual_source
|
|
65
|
-
|
|
66
|
-
source: raw is build.sql("""
|
|
67
|
-
SELECT * FROM read_csv_auto('data/raw.csv.gz', delim='\t', all_varchar=true, header=true)
|
|
68
|
-
""")
|
|
69
|
-
|
|
70
|
-
#@ persist name="thing"
|
|
71
|
-
source: thing_base is raw -> {
|
|
72
|
-
where: some_count::number > 100
|
|
73
|
-
select: id, name, value is value::number
|
|
74
|
-
calculate: rank is rank() { order_by: value::number desc }
|
|
75
|
-
}
|
|
76
|
-
```
|
|
77
|
-
|
|
78
|
-
Download the raw files first (a `data/get.sh` that `wget`s the URLs into `data/`,
|
|
79
|
-
gitignored). Gitignore `data/`, `data/*.duckdb`, and `MANIFESTS/`.
|
|
80
|
-
|
|
81
|
-
**3. Build, then export the persisted tables to `docs/`** — the export still
|
|
82
|
-
goes through `malloyyo sql` (no standalone duckdb needed):
|
|
83
|
-
|
|
84
|
-
```bash
|
|
85
|
-
rm -f data/build.duckdb
|
|
86
|
-
malloy-cli -c malloy-build.json build transform.malloy # -> tables in build.duckdb
|
|
87
|
-
echo "ATTACH 'data/build.duckdb' AS b (READ_ONLY);
|
|
88
|
-
COPY b.thing TO 'docs/thing.parquet' (FORMAT parquet)" | malloyyo sql
|
|
89
|
-
```
|
|
90
|
-
|
|
91
|
-
The persist `name=` is the table name inside `build.duckdb`; the `COPY` names the
|
|
92
|
-
served file (they can differ — e.g. `thing` → `docs/mysite_thing.parquet`).
|
|
93
|
-
|
|
94
|
-
Wrap steps 1–3 in a single `scripts/build_data.sh` so it is one command, by hand
|
|
95
|
-
and in CI. That's exactly what `malloyyo-auto-update` automates weekly.
|
|
96
|
-
|
|
97
|
-
## Either way, verify
|
|
98
|
-
|
|
99
|
-
```bash
|
|
100
|
-
malloyyo sql -e "DESCRIBE SELECT * FROM 'docs/thing.parquet'"
|
|
101
|
-
malloyyo sql -e "SELECT count(*) FROM 'docs/thing.parquet'"
|
|
102
|
-
```
|
|
103
|
-
|
|
104
|
-
Then point a storage source at it (docs-local shown; an https URL works too —
|
|
105
|
-
see the model step in SKILL.md):
|
|
106
|
-
|
|
107
|
-
```malloy
|
|
108
|
-
source: thing_table is duckdb.table('docs/thing.parquet') extend {}
|
|
109
|
-
```
|
|
@@ -1,61 +0,0 @@
|
|
|
1
|
-
# Publishing the site on GitHub Pages
|
|
2
|
-
|
|
3
|
-
The published site is just the `docs/` directory: bundled HTML plus the parquet
|
|
4
|
-
it queries. GitHub Pages serves it directly — no build step on GitHub's side.
|
|
5
|
-
|
|
6
|
-
## One-time setup
|
|
7
|
-
|
|
8
|
-
```bash
|
|
9
|
-
# from the repo root, after `malloyyo dashboard bundle --out docs`
|
|
10
|
-
gh repo create <owner>/<name> --public --source=. --remote=origin # or an existing repo
|
|
11
|
-
git add -A
|
|
12
|
-
git commit -m "initial data site"
|
|
13
|
-
git push -u origin main
|
|
14
|
-
```
|
|
15
|
-
|
|
16
|
-
Turn on Pages, pointing at the `docs/` folder on the default branch:
|
|
17
|
-
|
|
18
|
-
```bash
|
|
19
|
-
gh api -X POST repos/<owner>/<name>/pages \
|
|
20
|
-
-f 'source[branch]=main' -f 'source[path]=/docs'
|
|
21
|
-
```
|
|
22
|
-
|
|
23
|
-
(Or in the UI: Settings → Pages → Source: **Deploy from a branch** → Branch
|
|
24
|
-
`main`, folder `/docs`.)
|
|
25
|
-
|
|
26
|
-
The site appears at `https://<owner>.github.io/<name>/` within a minute or two.
|
|
27
|
-
|
|
28
|
-
## Why `docs/` and why `.nojekyll`
|
|
29
|
-
|
|
30
|
-
- Serving from `/docs` on the main branch keeps the data committed **once**
|
|
31
|
-
(it's the same directory you build into), not duplicated on a separate branch.
|
|
32
|
-
- `malloyyo dashboard bundle` writes a `.nojekyll` file so Pages serves the
|
|
33
|
-
bundled assets as-is instead of running them through Jekyll (which would drop
|
|
34
|
-
files beginning with `_`). Keep it committed.
|
|
35
|
-
|
|
36
|
-
## Updating the site
|
|
37
|
-
|
|
38
|
-
Re-run the data step and the bundle, then commit `docs/`:
|
|
39
|
-
|
|
40
|
-
```bash
|
|
41
|
-
bash scripts/build_data.sh # or your Path-A duckdb command → docs/*.parquet
|
|
42
|
-
malloyyo dashboard bundle --out docs
|
|
43
|
-
git add docs && git commit -m "update" && git push
|
|
44
|
-
```
|
|
45
|
-
|
|
46
|
-
Because the dashboards fetch the parquet client-side, **committing fresh parquet
|
|
47
|
-
is enough to update the live data** — you only need to re-`bundle` when you
|
|
48
|
-
change dashboard code. To automate the data refresh weekly, see the
|
|
49
|
-
`malloyyo-auto-update` skill.
|
|
50
|
-
|
|
51
|
-
## Gotchas
|
|
52
|
-
|
|
53
|
-
- **Data must be in `docs/` before you bundle** — the pages fetch `./*.parquet`
|
|
54
|
-
relative to the site root; `bundle` also reads the parquet to get schemas.
|
|
55
|
-
- **Big parquet grows git history.** Each committed refresh adds the full file.
|
|
56
|
-
Fine for occasional updates; for frequent auto-refresh, see the flatten note
|
|
57
|
-
in `malloyyo-auto-update`.
|
|
58
|
-
- **Git LFS does not work with Pages "deploy from a branch"** — Pages serves the
|
|
59
|
-
LFS pointer text, not the file. Commit parquet as normal git objects.
|
|
60
|
-
- **Private repos**: Pages on private repos needs a paid plan. Use a public repo
|
|
61
|
-
for a public site.
|