@chrismo/superkit 1.0.0

package/docs/tutorials/fork_for_window.md

@@ -0,0 +1,296 @@
+---
+title: "Fork as a Window Function Workaround"
+name: fork-for-window
+description: "Using fork as a workaround for window functions to do per-group selection."
+layout: default
+nav_order: 4
+parent: Tutorials
+superdb_version: "0.3.0"
+last_updated: "2026-02-20"
+---
+
+# Fork as a Window Function Workaround
+
+Window functions like `ROW_NUMBER() OVER (PARTITION BY ...)` are not yet
+available in SuperDB ([brimdata/super#5921][issue]). This tutorial shows how to
+use `fork` to achieve per-group selection — picking the top N items from each
+group.
+
+[issue]: https://github.com/brimdata/super/issues/5921
+
+## The Problem
+
+You have a pool of available EC2 instances spread across availability zones.
+You need to pick instances while maximizing AZ distribution — taking an equal
+number from each zone rather than filling up from one.
+
+```mdtest-input instances.sup
+{id:"i-001",az:"us-east-1a"}
+{id:"i-002",az:"us-east-1a"}
+{id:"i-003",az:"us-east-1a"}
+{id:"i-004",az:"us-east-1b"}
+{id:"i-005",az:"us-east-1c"}
+{id:"i-006",az:"us-east-1c"}
+{id:"i-007",az:"us-east-1c"}
+{id:"i-008",az:"us-east-1c"}
+```
+
+Distribution: 3 in `us-east-1a`, 1 in `us-east-1b`, 4 in `us-east-1c`.
+
+## What You'd Want (Window Functions)
+
+In SQL with window functions, this would be straightforward:
+
+```sql
+SELECT * FROM (
+  SELECT *,
+    ROW_NUMBER() OVER (PARTITION BY az ORDER BY id) as rn
+  FROM instances
+) WHERE rn <= 2
+```
+
+This assigns a row number within each AZ group, then filters to keep only the
+first 2 per group. But SuperDB doesn't support this yet.
+
+## The Fork Approach
+
+`fork` splits the input stream into parallel branches. Each branch receives a
+copy of **all** the input records, processes them independently, and the results
+from every branch are merged back together into a single stream.
+
+Here's the full query — we'll break it down step by step after:
+
+```mdtest-command
+super -s -c "
+  from instances.sup
+  | fork
+    ( where az=='us-east-1a' | head 2 )
+    ( where az=='us-east-1b' | head 2 )
+    ( where az=='us-east-1c' | head 2 )
+  | sort az, id
+"
+```
+```mdtest-output
+{id:"i-001",az:"us-east-1a"}
+{id:"i-002",az:"us-east-1a"}
+{id:"i-004",az:"us-east-1b"}
+{id:"i-005",az:"us-east-1c"}
+{id:"i-006",az:"us-east-1c"}
+```
+
+### Step by Step
+
+**Step 1: `from instances.sup`** — reads all 8 records into the stream:
+
+```mdtest-command
+super -s -c "from instances.sup"
+```
+```mdtest-output
+{id:"i-001",az:"us-east-1a"}
+{id:"i-002",az:"us-east-1a"}
+{id:"i-003",az:"us-east-1a"}
+{id:"i-004",az:"us-east-1b"}
+{id:"i-005",az:"us-east-1c"}
+{id:"i-006",az:"us-east-1c"}
+{id:"i-007",az:"us-east-1c"}
+{id:"i-008",az:"us-east-1c"}
+```
+
+**Step 2: `fork`** — sends all 8 records into each of three branches. Each
+branch sees the full input and processes it independently.
+
+**Branch 1:** `where az=='us-east-1a'` filters to 3 records, then `head 2`
+keeps the first 2:
+
+```mdtest-command
+super -s -c "from instances.sup | where az=='us-east-1a' | head 2"
+```
+```mdtest-output
+{id:"i-001",az:"us-east-1a"}
+{id:"i-002",az:"us-east-1a"}
+```
+
+(i-003 was filtered out by `head 2`)
+
+**Branch 2:** `where az=='us-east-1b'` filters to 1 record, `head 2` returns
+what's available:
+
+```mdtest-command
+super -s -c "from instances.sup | where az=='us-east-1b' | head 2"
+```
+```mdtest-output
+{id:"i-004",az:"us-east-1b"}
+```
+
+Only 1 instance exists in this AZ. `head 2` doesn't error or pad — it just
+returns what's there.
+
+**Branch 3:** `where az=='us-east-1c'` filters to 4 records, `head 2` keeps
+the first 2:
+
+```mdtest-command
+super -s -c "from instances.sup | where az=='us-east-1c' | head 2"
+```
+```mdtest-output
+{id:"i-005",az:"us-east-1c"}
+{id:"i-006",az:"us-east-1c"}
+```
+
+(i-007 and i-008 were filtered out by `head 2`)
+
+**Step 3: implicit combine** — after the fork closes, results from all three
+branches merge back into a single stream of 5 records. Fork branches run in
+parallel and finish in nondeterministic order, so the combined output may be
+interleaved differently on each run. This is why the final `sort` matters.
+
+**Step 4: `sort az, id`** — sorts the combined results for clean, predictable
+output:
+
+```mdtest-command
+super -s -c "
+  from instances.sup
+  | fork
+    ( where az=='us-east-1a' | head 2 )
+    ( where az=='us-east-1b' | head 2 )
+    ( where az=='us-east-1c' | head 2 )
+  | sort az, id
+"
+```
+```mdtest-output
+{id:"i-001",az:"us-east-1a"}
+{id:"i-002",az:"us-east-1a"}
+{id:"i-004",az:"us-east-1b"}
+{id:"i-005",az:"us-east-1c"}
+{id:"i-006",az:"us-east-1c"}
+```
+
+2 from `us-east-1a`, 1 from `us-east-1b` (all it had), 2 from `us-east-1c` —
+as balanced as possible given the available pool.
+
+## Why Not Just Sort and Head?
+
+Without fork, you might try:
+
+```mdtest-command
+super -s -c "from instances.sup | sort az, id | head 5"
+```
+```mdtest-output
+{id:"i-001",az:"us-east-1a"}
+{id:"i-002",az:"us-east-1a"}
+{id:"i-003",az:"us-east-1a"}
+{id:"i-004",az:"us-east-1b"}
+{id:"i-005",az:"us-east-1c"}
+```
+
+All 3 from `us-east-1a`, the 1 from `us-east-1b`, and only 1 from `us-east-1c`.
+That's unbalanced — it fills up from the first AZ alphabetically instead of
+distributing evenly.
+
+## Verifying the Distribution
+
+You can check the balance of your selection by piping through an aggregate:
+
+```mdtest-command
+super -s -c "
+  from instances.sup
+  | fork
+    ( where az=='us-east-1a' | head 2 )
+    ( where az=='us-east-1b' | head 2 )
+    ( where az=='us-east-1c' | head 2 )
+  | aggregate count:=count() by az
+  | sort az
+"
+```
+```mdtest-output
+{az:"us-east-1a",count:2}
+{az:"us-east-1b",count:1}
+{az:"us-east-1c",count:2}
+```
+
+## Alternative: Self-Join for Row Numbering
+
+There's a pure SQL approach that doesn't require fork and works dynamically with
+any number of groups. The idea: for each record, count how many records in the
+same group have an id less than or equal to it. This simulates
+`ROW_NUMBER() OVER (PARTITION BY az ORDER BY id)`.
+
+```mdtest-command
+super -s -c "
+  select a.id, a.az, count(*) as row_num
+  from instances.sup a
+  join instances.sup b on a.az = b.az and b.id <= a.id
+  group by a.id, a.az
+  order by a.az, a.id
+"
+```
+```mdtest-output
+{id:"i-001",az:"us-east-1a",row_num:1}
+{id:"i-002",az:"us-east-1a",row_num:2}
+{id:"i-003",az:"us-east-1a",row_num:3}
+{id:"i-004",az:"us-east-1b",row_num:1}
+{id:"i-005",az:"us-east-1c",row_num:1}
+{id:"i-006",az:"us-east-1c",row_num:2}
+{id:"i-007",az:"us-east-1c",row_num:3}
+{id:"i-008",az:"us-east-1c",row_num:4}
+```
+
+Step by step, for record `i-006` in `us-east-1c`:
+
+1. The self-join matches `i-006` against all `us-east-1c` records with
+   `id <= 'i-006'`: that's `i-005` and `i-006` itself.
+2. `count(*)` = 2, so `row_num` = 2.
+
+Now filter to keep only the first 2 per group:
+
+```mdtest-command
+super -s -c "
+  with ranked as (
+    select a.id, a.az, count(*) as row_num
+    from instances.sup a
+    join instances.sup b on a.az = b.az and b.id <= a.id
+    group by a.id, a.az
+  )
+  select id, az from ranked
+  where row_num <= 2
+  order by az, id
+"
+```
+```mdtest-output
+{id:"i-001",az:"us-east-1a"}
+{id:"i-002",az:"us-east-1a"}
+{id:"i-004",az:"us-east-1b"}
+{id:"i-005",az:"us-east-1c"}
+{id:"i-006",az:"us-east-1c"}
+```
+
+Same result as fork, but no hardcoded AZ names — works with any number of
+groups dynamically.
+
+## Trade-offs
+
+**Fork** is simple and fast (linear scan per branch), but requires hardcoding
+group values. Best when groups are known and stable (like AZs in a region).
+
+**Self-join** is dynamic and handles any number of groups automatically, but
+is O(n^2) per group since every record is joined against all peers with a
+smaller key. Fine for small datasets, potentially slow for large ones.
+
+**With window functions** ([brimdata/super#5921][issue]), the query would be
+both dynamic and efficient — handling any number of groups with a single linear
+pass and supporting sophisticated ranking (e.g., ordering within groups by
+launch time, instance type preference, etc.).
+
+| Approach         | Dynamic groups? | Time complexity  | Notes                                      |
+|------------------|-----------------|------------------|--------------------------------------------|
+| Fork             | No              | O(n) per branch  | Groups must be hardcoded                   |
+| Self-join        | Yes             | O(n^2) per group | Every record joined against its group peers |
+| Window functions | Yes             | O(n log n)       | Sort + single pass (not yet available)     |
+
+For a refresher on what those mean in practice
+([Big O notation](https://en.wikipedia.org/wiki/Big_O_notation)):
+
+| Notation   | Name        | 100 records | 10,000 records | Growth        |
+|------------|-------------|-------------|----------------|---------------|
+| O(n)       | Linear      | 100         | 10,000         | Scales nicely |
+| O(n log n) | Linearithmic| ~664        | ~132,877       | Typical sort  |
+| O(n^2)     | Quadratic   | 10,000      | 100,000,000    | Gets slow fast|

package/docs/tutorials/grok.md

@@ -0,0 +1,166 @@
+---
+title: "grok"
+name: grok
+description: "Tutorial on using the grok function for text parsing in SuperDB."
+layout: default
+nav_order: 5
+parent: Tutorials
+superdb_version: "0.3.0"
+last_updated: "2026-03-28"
+---
+
+# grok
+
+The grok function is a great choice for parsing text, but due to some gaps in
+its documentation and some vague error messages, it can be difficult to use at
+first.
+
+The docs do helpfully encourage building out grok patterns incrementally, but
+without knowing some of grok's gotchyas, this can be discouraging.
+
+Let's demonstrate these starting with this example where we want to extract the
+name out of this string:
+
+```text
+My name is: Muerte!
+```
+
+To start incrementally, I know I want to skip everything up past the colon, and
+then extract the name minus the closing exclamation point. There's probably not
+a predefined pattern that exists for this prefix regex, and/or I'm feeling lazy
+enough right now to not go looking for one, and the regex is pretty simple.
+
+So, I'll define my own pattern in the 3rd arg to grok to handle this:
+```mdtest-command
+super -s -c '
+  values "My name is: Muerte!"
+  | grok("%{NAME_PREFIX}", this, "NAME_PREFIX .*: ")'
+```
+Since there aren't any errors, and no field names assigned, it returns an empty
+record:
+```mdtest-output
+{}
+```
+
+It's a simple regex, it seems like it's accurate, it's hard for me to see what's
+wrong?
+
+The regex is fine, in fact. The real reason this returns an empty record is that
+the capture pattern is **missing a field name** in which to store the value.
+Without a field name, there's nothing to capture into a record field.
+
+We probably made this mistake because we don't really want to capture "My name
+is: " in a field of the record. But, no big deal, we can add one and use the
+cut operator later to remove it.
+
+```mdtest-command
+super -s -c '
+  values "My name is: Muerte!"
+  | grok("%{NAME_PREFIX:prefix}", this, "NAME_PREFIX .*: ")'
+```
+```mdtest-output
+{prefix:"My name is: "}
+```
+
+Success!
+
+For our next incremental step, let's capture the name. That's all that's left.
+```mdtest-command
+super -s -c '
+  values "My name is: Muerte!"
+  | grok("%{NAME_PREFIX:prefix}%{WORD:name}", this, "NAME_PREFIX .*: ")'
+```
+```mdtest-output
+{prefix:"My name is: ",name:"Muerte"}
+```
+    
+Success again! Ok, that wasn't so bad, but - it's a little arduous. It doesn't
+feel like I'm getting to use the power of regex in a straightforward manner.
+
+There are two grok undocumented "hacks" that can make a simple job like this
+even simpler.
+
+First, (as seen already with the unnamed capture pattern in `super`), not all
+capture patterns need a field name as long as _**one**_ of them has a field
+name. So we can reduce our last example to be this:
+
+```mdtest-command
+super -s -c '
+  values "My name is: Muerte!" 
+  | grok("%{NAME_PREFIX}%{WORD:name}", this, "NAME_PREFIX .*: ")'
+```
+```mdtest-output
+{name:"Muerte"}
+```
+
+Second, custom regex patterns can be _inlined_ into the pattern string without
+being a custom named pattern in the 3rd argument at all!
+                   
+```mdtest-command
+super -s -c '
+  values "My name is: Muerte!" 
+  | grok(".*: %{WORD:name}", this)'
+```
+```mdtest-output
+{name:"Muerte"}
+```
+
+Now that feels clean and simple!        
+
+## Pairing grok with infer
+
+Grok extracts fields as strings — even numbers and IPs. The `infer` operator
+can automatically detect and cast these to native types.
+
+Here's a log line parsed with grok — note that everything is a string:
+
+```mdtest-command
+super -s -c '
+  values
+    "192.168.1.1 GET /api/users 200 1234",
+    "10.0.0.5 POST /api/data 404 567"
+  | grok("%{IP:client} %{WORD:method} %{URIPATH:path} %{INT:status} %{INT:bytes}", this)'
+```
+```mdtest-output
+{client:"192.168.1.1",method:"GET",path:"/api/users",status:"200",bytes:"1234"}
+{client:"10.0.0.5",method:"POST",path:"/api/data",status:"404",bytes:"567"}
+```
+
+Add `| infer` and the types get cleaned up automatically:
+
+```mdtest-command
+super -s -c '
+  values
+    "192.168.1.1 GET /api/users 200 1234",
+    "10.0.0.5 POST /api/data 404 567"
+  | grok("%{IP:client} %{WORD:method} %{URIPATH:path} %{INT:status} %{INT:bytes}", this)
+  | infer'
+```
+```mdtest-output
+{client:192.168.1.1,method:"GET",path:"/api/users",status:200,bytes:1234}
+{client:10.0.0.5,method:"POST",path:"/api/data",status:404,bytes:567}
+```
+
+`client` became an `ip` type, `status` and `bytes` became `int64`, while
+`method` and `path` correctly stayed as strings. This means you can now do
+things like `where status >= 400` or `where client in 10.0.0.0/8` without
+manual casting.
+
+## Unit tests in codebase
+
+```mdtest-command
+super -s -c 'values "1", "foo" | grok("%{INT}", this)'
+```
+```mdtest-output
+{}
+error({message:"grok: value does not match pattern",on:"foo"})
+```
+
+## as of versions
+
+```mdtest-command
+super --version
+```
+```mdtest-output
+Version: v0.3.0
+```

package/docs/tutorials/index.md

@@ -0,0 +1,10 @@
+---
+title: Tutorials
+layout: default
+has_children: true
+nav_order: 4
+---
+
+# Tutorials
+
+Step-by-step guides covering common SuperDB patterns and techniques.

package/docs/tutorials/joins.md

@@ -0,0 +1,79 @@
+---
+title: "Joins"
+name: joins
+description: "Examples of outer joins, anti joins, and full outer joins in SuperDB."
+layout: default
+nav_order: 6
+parent: Tutorials
+superdb_version: "0.3.0"
+last_updated: "2026-02-15"
+---
+
+# Joins
+
+## Outer Joins
+
+```mdtest-input za.sup
+{id:1,name:"foo",src:"za"}
+{id:3,name:"qux",src:"za"}
+```
+```mdtest-input zb.sup
+{id:1,name:"foo",src:"zb"}
+{id:2,name:"bar",src:"zb"}
+```
+
+Left Join (Left Only + Inner Joins — where clause required to eliminate inner joins)
+
+`select *` includes columns from both sides. The right table's columns get a
+`_1` suffix to avoid name collisions, and unmatched values are
+`error("missing")`.
+```mdtest-command
+super -s -c "select * from za.sup as za
+             left join zb.sup as zb
+             on za.id=zb.id
+             where is_error(zb.name)"
+```
+```mdtest-output
+{id:3,name:"qux",src:"za",id_1:error("missing"),name_1:error("missing"),src_1:error("missing")}
+```
+
+Right Join (Right Only + Inner Joins — where clause required to eliminate inner joins)
+```mdtest-command
+super -s -c "select * from za.sup as za
+             right join zb.sup as zb
+             on za.id=zb.id
+             where is_error(za.name)"
+```
+```mdtest-output
+{id:error("missing"),name:error("missing"),src:error("missing"),id_1:2,name_1:"bar",src_1:"zb"}
+```
+
+Anti Join (Left Join exclusively — no where clause required)
+```mdtest-command
+super -s -c "select * from za.sup as za
+             anti join zb.sup as zb
+             on za.id=zb.id"
+```
+```mdtest-output
+{id:3,name:"qux",src:"za",id_1:error("missing"),name_1:error("missing"),src_1:error("missing")}
+```
+
+Full Outer (Left Only + Right Only + Inner Joins) - _BUG: Still behaves like Left Join — only returns left-side rows_
+```mdtest-command
+super -s -c "select * from za.sup as za
+             full outer join zb.sup as zb
+             on za.id=zb.id
+             where is_error(za.name) or is_error(zb.name)"
+```
+```mdtest-output
+{id:3,name:"qux",src:"za",id_1:error("missing"),name_1:error("missing"),src_1:error("missing")}
+```
+
+## as of versions
+
+```mdtest-command
+super --version
+```
+```mdtest-output
+Version: v0.2.0
+```

package/docs/tutorials/moar_subqueries.md

@@ -0,0 +1,35 @@
+---
+title: "Moar Subqueries"
+name: moar-subqueries
+description: "Additional subquery patterns including fork and full sub-selects."
+layout: default
+nav_order: 10
+parent: Tutorials
+superdb_version: "0.2.0"
+last_updated: "2026-02-15"
+---
+
+# Moar Subqueries
+
+## Fork
+
+One hassle to this approach is the limit of 2 forks. Nesting forks works, but
+makes constructing this query a bit more difficult.
+
+## Full Sub-Selects
+
+As of 20250815 build, this is much, much slower. I'm guessing it's doing a full
+reload of the data file each time.
+
+```
+select
+(select count(*)
+from './moar_subqueries.sup'
+where win is not null) as total_games,
+...
+```
+
+## All Other SQL-Syntax Subqueries
+
+They all take about the same amount of wall-time, but CPU usage is much higher
+due to re-reading the file each time.