@graphenedata/cli 0.0.1 → 0.0.3

package/dist/docs/graphene.md

@@ -1,86 +1,410 @@
 # How to develop in Graphene
 
-Graphene is a framework for building semantic layers and data visualizations in code. Graphene projects are comprised of:
+Graphene is a framework for data analysis, semantic modeling, and data visualization in code. Graphene projects are comprised of:
 - .gsql files that define semantics-enriched tables (aka semantic models)
-- .md files that define data apps (dashboards)
+- .md files that define data apps (aka dashboards)
 
 Graphene also has a CLI that lets you check syntax, run queries, serve data apps, and more.
 
 ## Graphene SQL (GSQL)
 
-### Tables
-Tables have to be declared first before they can be queried. A table in Graphene has the added concept of _semantics_. Semantics are stored expressions and join relationships associated with a table that `select` queries can leverage. This allows query logic to be centralized, reusable, and more easily governed.
+GSQL is comprised of `table` statements that declare tables and `select` statements that query them.
 
-Here's an example:
+### `table` statements
 
-```gsql
+`table` statements manifest tables that already exist in your database. Here's an example of two tables, `orders` and `users`, in GSQL.
+
+```sql
 table orders (
+
+  /* Base columns */
+
   id BIGINT primary_key,
   user_id BIGINT,
   created_at DATETIME,
-  amount FLOAT, -- paid by customer #units=usd
-  cost FLOAT, -- cost of materials #units=usd
+  status STRING, -- One of 'Processing', 'Shipped', 'Complete', 'Cancelled', 'Returned'
+  amount FLOAT, -- Amount paid by customer
+  cost FLOAT, -- Cost of materials
+
+  /* Join relationships */
 
   join_one users on user_id = users.id,
 
-  sum(amount) as revenue,
-  sum(amount - cost) as profit,
+  /* Scalar expressions */
+
+  status in ('Processing', 'Shipped', 'Complete') as revenue_recognized,
+
+  /* Agg expressions */
+
+  sum(case when revenue_recognized then amount else 0 end) as revenue,
+  sum(case when revenue_recognized then cost else 0 end) as cogs,
+  revenue - cogs as profit,
   profit / revenue as profit_margin
-);
+)
 
 table users (
   id BIGINT primary_key,
   name VARCHAR,
   email VARCHAR,
   age INTEGER,
+  country_code VARCHAR,
 
   join_many orders on id = orders.user_id
-);
+)
 ```
 
-Syntax notes
-- `table foo (...)` defines a Graphene table based on the database table `foo`. 
-- The allowed join types are `join_one` and `join_many`. All joins are left outer joins. There is no inner, right, or cross join.
+We can break down a table statement into three parts: [base columns](#base-columns-required), [join relationships](#join-relationships), and [stored expressions](#stored-expressions) (aka dimensions and measures).
+
+#### Base columns (required)
+
+The base column set is simply a reflection of the underlying database table's schema. Similar to `create table` statements in regular SQL DDL, you list each column's name and data type. One column must be designated as the primary key.
+
+#### Join relationships
+
+Join relationships in a `table` statement declare joins that can be used when querying them. This makes query writing easier and more foolproof. See [Using join relationships in queries](#using-join-relationships-in-queries) below for how to use modeled joins in queries.
+
+The other main difference about joins in GSQL vs. regular SQL is that you have to explain if there are many rows in the left table for each row in the right table, or vice versa. This additional bit of information allows Graphene to prevent incorrect aggregation as a result of row duplication (aka fan-out) through joins. See [Safe aggregation in fan-outs](#safe-aggregation-in-fan-outs) for more details.
+
+This information is provided with the two supported join types, `join_one` and `join_many`:
 - `join_one` is used if there are many rows in the **left** table for each row in the **right** table.
 - `join_many` is used if there are many rows in the **right** table for each row in the **left** table.
-- Join names within a table must be unique. Polymorphic relationships (eg., where there are multiple relationships between the same two tables on different keys) are allowed but must be aliased eg. `join_one users as owner on user_id = owner.id` and `join_one users as viewer on user_id = viewer.id`.
-- Comments in tables can provide descriptions as well as metadata (denoted by `#` inside the comment).
 
-Best practices
-- For a given table, only model joins that are directly on that table. Graphene will automatically traverse multi-hop joins when it compiles the collective table space.
-- A join between two tables should be modeled in both the respective `table` statements. This may seem redundant but it offers more flexibility for queries to choose which table to set in the `from` (remember that direction matters since all joins are left joins).
+In the example above with `orders` and `users`, the joins confirm that there are many orders per user, and only one user per order.
+
+Note that all joins in GSQL are left outer joins. There is no inner, right, or cross join.
+
+##### Multiple join relationships between the same two tables
+
+Sometimes there are multiple valid ways to join two tables together. You can model this in Graphene by aliasing the various joins with `as`, just as you would in normal SQL. For example:
+
+```sql
+table projects (
+  ...
+  owner_id BIGINT,
+  viewer_id BIGINT,
+
+  join_one users as project_owner on owner_id = project_owner.id,
+  join_one users as project_viewer on viewer_id = project_viewer.id
+)
+
+table users (
+  ...
+  id BIGINT,
+
+  join_many projects as projects_as_owner on id = projects_as_owner.owner_id,
+  join_many projects as projects_as_viewer on id = projects_as_viewer.viewer_id
+)
+```
+
+##### Best practices for modeling join relationships
+
+- For a given `table` statement, only model joins that are directly on that table. Multi-hop join paths do not need to be written explicitly in order for queries to traverse them.
+- A join between two tables should be modeled in both the respective `table` statements. This may seem redundant but it offers more flexibility for queries to choose which table to set in the `from` (remember that direction matters in queries since all joins are left joins).
+
+#### Stored expressions
+
+**Stored expressions** are GSQL expressions (ie. any arbitrary combination of functions, operators, and column references) that you want to make reusable to queries. Stored expressions are great for canonizing metrics, segments, and other important business definitions.
+
+A stored expression must be given a name via `as`. It can then be referenced by name in queries that use the parent table. See [Using stored expressions in queries](#using-stored-expressions-in-queries) below for how to use stored expressions in queries.
+
+Like expressions in regular SQL, expressions in GSQL are either scalar or aggregative. In BI parlance, these would be called dimensions and measures, respectively.
+
+Expressions can refer to other expressions, as shown below.
 
-### Queries
-Graphene tables can be queried using `select` statements. Here are some example queries on the tables above:
+```sql
+table orders (
+  ...
+
+  /* Scalar expressions */
+
+  status in ('Processing', 'Shipped', 'Complete') as revenue_recognized,
 
+  /* Agg expressions */
+
+  sum(case when revenue_recognized then amount else 0 end) as revenue,
+  sum(case when revenue_recognized then cost else 0 end) as cogs,
+  revenue - cogs as profit, -- even though there are no agg functions here, this is still aggregative as it references other aggregative expressions
+  profit / revenue as profit_margin
+)
 ```
--- top 10 customers by profit
-from orders select
-  users.name, -- notice how we can access the joined table without a join here
-  profit -- this expands into the stored expression defined in the table
+
+
+### `select` statements
+
+`select` is how you write queries in Graphene SQL. It behaves similarly to regular SQL except in the following ways:
+- It can invoke join relationships and stored expressions from `table` statements.
+- It prevents users from accidentally aggregating incorrectly through joins.
+
+These differences are described in the sections below.
+
+#### Using join relationships in queries
+
+If a `table` has join relationships declared in it, a `select` query on that table can leverage that join without needing to write its own join statement. This is helpful for query writers who have not memorized all the correct join keys.
+
+If you recall the model from before:
+
+```sql
+table orders (
+  id BIGINT primary_key,
+  user_id BIGINT,
+  created_at DATETIME,
+  status STRING, -- One of 'Processing', 'Shipped', 'Complete', 'Cancelled', 'Returned'
+  amount FLOAT, -- Amount paid by customer
+  cost FLOAT, -- Cost of materials
+
+  join_one users on user_id = users.id,
+
+  status in ('Processing', 'Shipped', 'Complete') as revenue_recognized,
+
+  sum(case when revenue_recognized then amount else 0 end) as revenue,
+  sum(case when revenue_recognized then cost else 0 end) as cogs,
+  revenue - cogs as profit,
+  profit / revenue as profit_margin
+)
+
+table users (
+  id BIGINT primary_key,
+  name VARCHAR,
+  email VARCHAR,
+  age INTEGER,
+  country_code VARCHAR,
+
+  join_many orders on id = orders.user_id
+)
+```
+
+We can write a query that leverages the modeled join relationship between `orders` and `users`:
+
+
+```sql
+-- Top 10 customers by order count
+select
+  users.name, -- Use the dot operator to traverse the modeled join relationship
+  count(*)
+from orders -- A join statement here is not needed
+group by 1
 order by 2 desc
 limit 10
 ```
 
+##### Multi-hop joins
+
+Sometimes you need to access columns or stored expressions in a table that is two or more joins away from the `from` table. To do this, simply use more dot operators to trace the desired join path. For example, say there is another table added to our project, `countries`:
+
+```sql
+table orders (
+  ...
+
+  join_one users on user_id = users.id
+)
+
+table users (
+  ...
+
+  join_many orders on id = orders.user_id,
+  join_one country on country_code = countries.code
+)
+
+table countries (
+  code VARCHAR primary_key,
+  name VARCHAR,
+  currency VARCHAR,
+  free_shipping BOOLEAN,
+
+  join_many users on code = users.country_code
+)
 ```
--- average age of customers over time
+
+We can write the following query to show the top ten countries by order count:
+
+```sql
+-- Top 10 countries by order count
 select
-  month(date),
-  average(users.age), -- in normal SQL this would fan-out in the join; in Graphene it smartly de-duplicates the fan-out when computing aggregates
-from orders 
+  users.countries.name, -- Orders -> Users -> Countries
+  count(*)
+from orders
+group by 1
+order by 2 desc
+limit 10
 ```
 
-Syntax notes
-- Columns and stored expressions from joined tables can be accessed with the dot operator, eg. `users.age` in the example above. Multiple join hops can be traversed with multiple dots, eg. `users.countries.country_code`.
-- `join_one` and `join_many` work here, too. This is useful if the join you need has not been modeled already.
-- The `from`, `select`, `group by`, and `where` clauses can be written in any order.
-- Expressions in `group by` are implicitly selected, so `from orders select avg(amount) group by user_id` is valid.
-- `group by all` is implied if aggregate and scalar expressions are both present in the `select`. It can be omitted and the query will still effectively execute the `group by all`.
+#### Using stored expressions in queries
+
+A stored expression can be invoked in a query by simply referencing it by name.
+
+Again, using the model from before:
+
+```sql
+table orders (
+  id BIGINT primary_key,
+  user_id BIGINT,
+  created_at DATETIME,
+  status STRING, -- One of 'Processing', 'Shipped', 'Complete', 'Cancelled', 'Returned'
+  amount FLOAT, -- Amount paid by customer
+  cost FLOAT, -- Cost of materials
+
+  join_one users on user_id = users.id,
+
+  status in ('Processing', 'Shipped', 'Complete') as revenue_recognized,
+
+  sum(case when revenue_recognized then amount else 0 end) as revenue,
+  sum(case when revenue_recognized then cost else 0 end) as cogs,
+  revenue - cogs as profit,
+  profit / revenue as profit_margin
+)
+
+table users (
+  id BIGINT primary_key,
+  name VARCHAR,
+  email VARCHAR,
+  age INTEGER,
+  country_code VARCHAR,
+
+  join_many orders on id = orders.user_id
+)
+```
+
+We can count the number of orders that were revenue-recognized vs. not:
+
+```sql
+-- Number of revenue-recognized orders vs. not
+select
+  revenue_recognized, -- Stored expression in orders
+  count(*)
+from orders
+group by 1
+```
+
+This would be equivalent to:
+
+```sql
+select
+  status in ('Processing', 'Shipped', 'Complete') as revenue_recognized,
+  count(*)
+from orders
+group by 1 
+```
+
+You can see that invoking a stored expression is like using a macro: the definition for the stored expression is effectively expanded in-line by Graphene when it runs the query.
+
+This is an important concept to understand when invoking stored expressions that are **aggregative** (ie. contain agg functions). Here's an example.
+
+```sql
+-- Profit by month
+select
+  date_trunc(created_at, month) as month,
+  profit
+from orders
+group by 1
+order by 1 asc
+```
+
+Note that, while `profit` looks like a column here, it is _not_ a column. That's because this query is equivalent to:
+
+```sql
+select
+  date_trunc(created_at, month) as month,
+  sum(case when revenue_recognized then amount else 0 end) - sum(case when revenue_recognized then cost else 0 end) as profit -- Profit is defined as revenue - cogs, which respectively expands out to these two filtered sums
+from orders
+group by 1
+order by 1 asc
+```
+
+For this reason, in a query you would never wrap an aggregative stored expression in a `sum()` or `avg()` or any other agg function for the same reason you would never write `sum(sum(foo))` in SQL. That would throw an error!
+
+#### Safe aggregation in fan-outs
+
+A common and dangerous user error in regular SQL is aggregating data incorrectly after joining tables. This can happen when rows of one table match multiple rows of another, and effectively get duplicated for each match.
+
+For example, after joining `users` to `orders`, your joined result will have some users repeated multiple times if they've made multiple purchases. If you wanted to find the average age of customers over this joined result, simply using an `avg(users.age)` would be _incorrect_, because you would be weighting the average towards users with multiple purchases, rather than taking the true average.
+
+GSQL aims to solve this problem. With the additional information provided via `join_one` and `join_many`, Graphene knows under which scenarios when row dupliation occurs, and will rewrite aggregative expressions in a way that ignores the duplicate rows.
+
+The query `select avg(users.age) from orders` will be rewritten to the following SQL when Graphene queries the underlying database (this is for BigQuery, specifically):
+
+```sql
+SELECT 
+   (CAST((
+    (
+      SUM(DISTINCT
+        (CAST(ROUND(COALESCE(users_0.`age`,0)*(1*1.0), 9) AS NUMERIC) +
+        (cast(cast(concat('0x', substr(to_hex(md5(CAST(users_0.`id` AS STRING))), 1, 15)) as int64) as numeric) * 4294967296 + cast(cast(concat('0x', substr(to_hex(md5(CAST(users_0.`id` AS STRING))), 16, 8)) as int64) as numeric)) * 0.000000001
+      ))
+      -
+       SUM(DISTINCT (cast(cast(concat('0x', substr(to_hex(md5(CAST(users_0.`id` AS STRING))), 1, 15)) as int64) as numeric) * 4294967296 + cast(cast(concat('0x', substr(to_hex(md5(CAST(users_0.`id` AS STRING))), 16, 8)) as int64) as numeric)) * 0.000000001)
+    )/(1*1.0)) AS FLOAT64))/NULLIF(COUNT(DISTINCT CASE WHEN users_0.`age` IS NOT NULL THEN users_0.`id` END),0) as `col_0`
+FROM `bigquery-public-data.thelook_ecommerce.orders` as base
+ LEFT JOIN `bigquery-public-data.thelook_ecommerce.users` AS users_0
+  ON users_0.`id`=base.`user_id`
+```
+
+You don't have to understand this; the point is that GSQL is minimizing the chances that naive users aggregate data incorrectly.
+
+### `table as` statements
+
+You can turn the output of any `select` statement into a table with `table foo as (select ...)`. Here's an example of an additional table `user_facts` added to the two tables from earlier:
+
+```sql
+table orders (
+  id BIGINT primary_key,
+  user_id BIGINT,
+  created_at DATETIME,
+  status STRING, -- One of 'Processing', 'Shipped', 'Complete', 'Cancelled', 'Returned'
+  amount FLOAT, -- Amount paid by customer
+  cost FLOAT, -- Cost of materials
+
+  join_one users on user_id = users.id,
+
+  status in ('Processing', 'Shipped', 'Complete') as revenue_recognized,
+
+  sum(case when revenue_recognized then amount else 0 end) as revenue,
+  sum(case when revenue_recognized then cost else 0 end) as cogs,
+  revenue - cogs as profit,
+  profit / revenue as profit_margin
+)
+
+table users (
+  id BIGINT primary_key,
+  name VARCHAR,
+  email VARCHAR,
+  age INTEGER,
+
+  join_many orders on id = orders.user_id,
+  join_one user_facts on id = user_facts.id,
+
+  /* Scalar expressions */
+
+  user_facts.ltv as ltv,
+  user_facts.lifetime_orders as lifetime_orders
+)
+
+table user_facts as (
+  select
+    id,
+    orders.revenue as ltv,
+    count(orders.id) as lifetime_orders,
+  from users
+  group by id
+)
+```
+
+`table as` statements are conceptually the same as view tables in regular SQL. A few things to note:
+- You cannot yet declare join relationships or stored expressions directly in a `table as` statement. Other tables can declare join relationships to it, though, as shown above.
+- In the example above, the `ltv` and `lifetime_orders` columns from `user_facts` are "hoisted" back into `users` so that they appear as if they are columns from `users`. This is simply a design choice which allows query writers to never need to know about `user_facts`.
+
+### Other miscellaneous details about GSQL
+
+- Trailing commas in `table` statements are optional.
+- Trailing semicolons after `table` and `table as` statements are optional.
+- The clauses in a `select` statement (`select`, `from`, `join`, `group by`, etc.) can be written in any order. They cannot be repeated, however.
+- `group by all` is implied if aggregative and scalar expressions are both present in the `select` clause. This means that `group by` can be omitted and the query will still effectively execute the `group by all`.
+- Expressions in `group by` are implicitly selected, so `from orders select avg(amount) group by user_id` will return two columns.
 - `count` is a reserved word. Do not alias your columns as `count`.
+- Window functions and set operations are not supported.
 
+## Graphene visualizations
 
-## Graphene viz (.md)
-Graphene data apps are written in Markdown with components. Markdown files can contain named GSQL queries in code fences that components can then refer to. Those queries can use any tables defined in .gsql files.
+Graphene data apps are written in Markdown with the addition of special Graphene HTML components. Markdown files can contain named GSQL queries in code fences that components can then refer to. Those queries can use any tables defined in .gsql files.
 
 ````markdown
   # Order analysis
@@ -96,9 +420,14 @@ Graphene data apps are written in Markdown with components. Markdown files can c
   </Row>
 ````
 
-Note that components can also directly refer to Graphene tables in their `data` property; it is not always necessary to prepare data in a code-fenced query. Properties that take column references can also take whole expressions, as shown in the second line chart from the example above.
+Note that components can also directly refer to Graphene tables in their `data` property; it is not always necessary to prepare data in a code-fenced query. Properties that take column references can also take whole GSQL expressions, as shown in the second line chart from the example above.
+
+Best practices
+- If you have multiple time series charts, align their x-axes to have the same range and granularity.
+- Use the same color for a given metric if it is used in multiple charts.
 
 ### Components
+
 The following components are available:
 - [BarChart](./data_apps/components/charts/bar-chart.md)
 - [LineChart](./data_apps/components/charts/line-chart.md)
@@ -111,19 +440,23 @@ The following components are available:
 - [TextInput](./data_apps/components/inputs/text-input.md)
 
 ## Using the Graphene CLI
+
 These are the available commands:
-- `npm run cli check` - Checks the syntax for the entire Graphene project.
-- `npm run cli compile "<GSQL>"` - Shows how GSQL is translated into the underlying database SQL.
-- `npm run cli run "<GSQL>"` - Runs a GSQL query. The tables and semantics defined in all .gsql files in the project are available for the query to use.
-- `npm run cli serve` - Starts (or restarts) the dev server, which allows the user to view their Graphene app on localhost.
-- `npm run cli view <mdPath>` - Captures a screenshot of a given .md file, along with any errors encountered.
+- `npm run graphene check` - Checks the syntax for the entire Graphene project.
+- `npm run graphene compile "<GSQL>"` - Shows how GSQL is translated into the underlying database SQL.
+- `npm run graphene run "<GSQL>"` - Runs a GSQL query. The tables and semantics defined in all .gsql files in the project are available for the query to use.
+- `npm run graphene serve` - Starts (or restarts) the dev server, which allows the user to view their Graphene app on localhost.
+- `npm run graphene view <mdPath>` - Captures a screenshot of a given .md file, along with any errors encountered.
 
 ## AGENT INSTRUCTIONS
+
 Follow these guidelines when working in a Graphene project.
-- Before writing any GSQL queries, run them in the CLI first to make sure that the results make sense.
-- Do not redefine joins or expressions in a GSQL query that already exist in a semantic model. For example, if profit has already been defined as the stored expression `sum(revenue - cost) as profit` on the table `orders`, you can simply use it in a downstream query as `select profit from orders`.
-- Because all joins in Graphene are left outer joins, be mindful about your `from` table selection.
-- When adding a component to a .md file, read the associated documentation page first in /docs/data_apps/components so you understand all the available configurations.
+- When formulating GSQL queries:
+   - First check all available stored expressions to see if there are any you can use. DO NOT redefine important business definitions like `profit` if they've already been modeled!
+   - Run your GSQL queries in the CLI first, _before_ you write them to a file. This way you can reason about the results to make sure they make sense.
 - Do not try to search the web for Graphene-specific info; you will not find anything. All the documentation is in /docs.
-- If you write to a .gsql file, run a syntax check with `npm run cli check`.
-- If you write to a .md file, run a syntax check with `npm run cli check`. Once there are no syntax errors, do a visual check by running `npm run cli view <mdPath>` and looking at the .png it generates. 
+- When writing to a .gsql file, check your code with `npm run graphene check`.
+- When writing to a Graphene .md file:
+  - First read ALL the linked component docs listed in [Components](#components) above.
+  - Check your code with `npm run graphene check`.
+  - Once there are no syntax errors, do a visual check by running `npm run graphene view <mdPath>` and looking at the .png it generates. 

package/dist/ui/internal/telemetry.ts

@@ -6,11 +6,9 @@ let staticErrors: Error[] = []
 let errorProviders: Record<string, ErrorProvider> = {}
 
 window.addEventListener('error', (event) => {
-  console.log('recordedError')
   staticErrors.push(event.error)
 })
 window.addEventListener('unhandledrejection', (event) => {
-  console.log('record unhandled')
   staticErrors.push(event.reason)
 })
 

package/package.json

@@ -3,7 +3,7 @@
   "main": "cli.ts",
   "type": "module",
   "author": "Graphene Systems Inc",
-  "version": "0.0.1",
+  "version": "0.0.3",
   "license": "Elastic-2.0",
   "engines": {
     "node": ">=16"
@@ -25,9 +25,9 @@
   "dependencies": {
     "@duckdb/node-api": "1.3.2-alpha.26",
     "@google-cloud/bigquery": "^8.1.1",
+    "@graphenedata/malloy": "0.0.304",
     "@lezer/common": "^1.2.3",
     "@lezer/lr": "^1.4.2",
-    "@graphenedata/malloy": "0.0.304",
     "@sveltejs/vite-plugin-svelte": "3.1.2",
     "@tidyjs/tidy": "^2.5.2",
     "chalk": "^5.3.0",
@@ -36,7 +36,6 @@
     "cli-table3": "^0.6.3",
     "commander": "^11.0.0",
     "debounce": "^1.2.1",
-    "dompurify": "^3.2.7",
     "echarts": "^5.5.0",
     "fs-extra": "11.2.0",
     "glob": "^11.0.3",
@@ -44,10 +43,7 @@
     "marked": "^16.3.0",
     "mdsvex": "^0.12.6",
     "nanoid": "3.3.8",
-    "rehype-stringify": "^10.0.1",
-    "remark": "^15.0.1",
-    "remark-mdx": "^3.1.1",
-    "remark-rehype": "^11.1.2",
+    "sanitize-html": "^2.17.0",
     "ssf": "^0.11.2",
     "svelte": "4.2.19",
     "unist-util-visit": "4.1.2",
@@ -57,6 +53,7 @@
   "devDependencies": {
     "@types/fs-extra": "^11.0.4",
     "@types/node": "^20.0.0",
+    "@types/sanitize-html": "^2.16.0",
     "@types/ws": "^8.18.1",
     "esbuild": "^0.21.5",
     "vitest": "3.0.5",

package/dist/ui/layout.svelte

@@ -1,3 +0,0 @@
-<main>
-  <slot />
-</main>