@chrismo/superkit 1.0.0

package/docs/tutorials/subqueries.md

@@ -0,0 +1,236 @@
+---
+title: "Subqueries"
+name: subqueries
+description: "Examples of correlated subqueries and derived table patterns in SuperDB."
+layout: default
+nav_order: 7
+parent: Tutorials
+superdb_version: "0.3.0"
+last_updated: "2026-02-15"
+---
+
+# Subqueries
+
+While there are many different types of subqueries, this document so far is just
+highlighting some common scenarios that may not have obvious implementations in
+superdb.
+
+## Correlated Subqueries
+
+[//]: # (TODO: file versions - phil's versions from Slack - NOT versions - issue #54) 
+
+Let's start with this simple dataset:
+
+```json lines
+{"id":1, "date":"2025-02-27", "foo": 3}
+{"id":2, "date":"2025-02-27", "foo": 2}
+{"id":3, "date":"2025-02-28", "foo": 5}
+{"id":4, "date":"2025-02-28", "foo": 9}
+```
+
+And we want to select the entries with the largest `foo` for each date.
+
+One way to do this in SQL looks like this:
+
+```sql
+select id, date, foo
+from data
+where (date, foo) in
+      (select date, max(foo) as max_foo
+       from data
+       group by date);
+```
+
+Another way, by joining a derived table:
+
+```sql
+select *
+from data d
+       join
+     (select date, max(foo) as max_foo
+      from data
+      group by date) max_foo
+     on d.date = max_foo.date and
+        d.foo = max_foo.max_foo;
+```
+
+In `super` this can be done with piped operators and a Lateral Subquery:
+
+```mdtest-input data.json
+{"id":1, "date":"2025-02-27", "foo": 3}
+{"id":2, "date":"2025-02-27", "foo": 2}
+{"id":3, "date":"2025-02-28", "foo": 5}
+{"id":4, "date":"2025-02-28", "foo": 9}
+```
+
+Here's an example using the piped `where` operator with `unnest ... into`:
+```mdtest-command
+super -s -c '
+  collect(this)
+  | {data: this}
+  | maxes:=[unnest this.data | foo:=max(foo) by date | values {date,foo}]
+  | unnest {this.maxes, this.data} into (
+      where {this.data.date, this.data.foo} in this.maxes
+      | values this.data
+    )' data.json
+```
+```mdtest-output
+{id:1,date:"2025-02-27",foo:3}
+{id:4,date:"2025-02-28",foo:9}
+```
+
+And a simpler example using the piped `join` operator with `from`:
+```mdtest-command
+super -s -c '
+  from data.json
+  | inner join (from data.json
+                | foo:=max(foo) by date
+                | values {date,foo})
+    on {left.date,left.foo}={right.date,right.foo}
+  | values left
+  | sort id'
+```
+```mdtest-output
+{id:1,date:"2025-02-27",foo:3}
+{id:4,date:"2025-02-28",foo:9}
+```
+                                 
+`super` also supports SQL syntax, and these subqueries work[^1]:
+
+```mdtest-command
+super -s -c '
+  select *
+  from "data.json"
+  where foo in (select max(foo), date
+                from "data.json"
+                group by date) '
+```
+```mdtest-output
+{id:1,date:"2025-02-27",foo:3}
+{id:4,date:"2025-02-28",foo:9}
+```
+  
+If we save off the max data to a file first, then we can start to see how this
+could look:
+```mdtest-command
+super -s -c '
+  select max(foo) as max_foo
+  from "data.json"
+  group by date' > max.sup
+
+super -s -c '
+  select l.id, l.date, l.foo
+  from "data.json" l
+    join "max.sup" r
+    on l.foo==r.max_foo'
+```
+```mdtest-output
+{id:1,date:"2025-02-27",foo:3}
+{id:4,date:"2025-02-28",foo:9}
+```
+
+## Subquery with Related Data Join
+
+A more realistic scenario: find the records with the top `score` per date, and
+also pull in user information from a related table.
+
+```mdtest-input scores.json
+{"id":1, "date":"2025-02-27", "score": 3, "user_id": 101}
+{"id":2, "date":"2025-02-27", "score": 2, "user_id": 102}
+{"id":3, "date":"2025-02-28", "score": 5, "user_id": 101}
+{"id":4, "date":"2025-02-28", "score": 9, "user_id": 103}
+```
+
+```mdtest-input users.json
+{"user_id": 101, "name": "Moxie"}
+{"user_id": 102, "name": "Ziggy"}
+{"user_id": 103, "name": "Sprocket"}
+```
+
+First, the basic join returns all records with user names:
+```mdtest-command
+super -s -c '
+  select s.id, s.date, s.score, s.user_id, u.name
+  from "scores.json" s
+    join "users.json" u on s.user_id = u.user_id
+  order by s.id'
+```
+```mdtest-output
+{id:1,date:"2025-02-27",score:3,user_id:101,name:"Moxie"}
+{id:2,date:"2025-02-27",score:2,user_id:102,name:"Ziggy"}
+{id:3,date:"2025-02-28",score:5,user_id:101,name:"Moxie"}
+{id:4,date:"2025-02-28",score:9,user_id:103,name:"Sprocket"}
+```
+
+Filtering to top scores per date using a subquery:
+```mdtest-command
+super -s -c '
+  select *
+  from "scores.json"
+  where score in (select max(score), date
+                  from "scores.json"
+                  group by date)'
+```
+```mdtest-output
+{id:1,date:"2025-02-27",score:3,user_id:101}
+{id:4,date:"2025-02-28",score:9,user_id:103}
+```
+
+The "obvious" SQL approach with tuple comparison returns empty — this is a known
+issue ([#6326](https://github.com/brimdata/super/issues/6326)):
+```mdtest-command
+super -s -c '
+  select s.id, s.date, s.score, s.user_id, u.name
+  from "scores.json" s
+    join "users.json" u on s.user_id = u.user_id
+  where (s.date, s.score) in (
+    select date, max(score)
+    from "scores.json"
+    group by date)'
+```
+```mdtest-output
+```
+
+A derived table approach (subquery in FROM) does work:
+```mdtest-command
+super -s -c '
+  select s.id, s.date, s.score, s.user_id, u.name
+  from "scores.json" s
+    join (
+      select date, max(score) as max_score
+      from "scores.json"
+      group by date
+    ) m on s.date = m.date and s.score = m.max_score
+    join "users.json" u on s.user_id = u.user_id
+  order by s.id'
+```
+```mdtest-output
+{id:1,date:"2025-02-27",score:3,user_id:101,name:"Moxie"}
+{id:4,date:"2025-02-28",score:9,user_id:103,name:"Sprocket"}
+```
+
+The piped approach also works — filter first, then join to get usernames:
+```mdtest-command
+super -s -c '
+  from "scores.json"
+  | where score in (select max(score), date from "scores.json" group by date)
+  | inner join (from "users.json") on left.user_id=right.user_id
+  | select left.id, left.date, left.score, right.name
+  | sort id'
+```
+```mdtest-output
+{id:1,date:"2025-02-27",score:3,name:"Moxie"}
+{id:4,date:"2025-02-28",score:9,name:"Sprocket"}
+```
+
+[^1]: SQL subqueries that reference files re-read the file for each subquery,
+    which increases CPU usage and wall time compared to the piped approach.
+
+# as of versions
+
+```mdtest-command
+super --version
+```
+```mdtest-output
+Version: v0.2.0
+```

package/docs/tutorials/sup_to_bash.md

@@ -0,0 +1,164 @@
+---
+title: "Optimizing Sup Values into Bash Variables"
+name: sup-to-bash
+description: "Optimizing SuperDB output into Bash variables efficiently."
+layout: default
+nav_order: 8
+parent: Tutorials
+superdb_version: "0.3.0"
+last_updated: "2026-02-15"
+---
+
+# Optimizing Sup Values into Bash Variables
+
+[//]: # (TODO: THIS NEEDS THE FULL DOCUMENTATION TREATMENT) 
+
+While you should try to push as much logic into SuperDB commands, inevitably you'll
+need these values in variables in your language of choice.
+
+A simple one of those — languages — is Bash. It can be a common pattern to then do
+something like this:
+
+```bash
+echo '
+  {"InstanceId":"i-05b132aa000f0afa0",
+   "InstanceType":"t4g.teeny",
+   "LaunchTime":"2025-04-01T12:34:56+00:00",
+   "PrivateIpAddress":"10.0.1.2"}
+' > ec2s.json
+
+instance_id=$(super -f line -c "values InstanceId" ec2s.json)
+instance_type=$(super -f line -c "values InstanceType" ec2s.json)
+launch_time=$(super -f line -c "values LaunchTime" ec2s.json)
+private_ip=$(super -f line -c "values PrivateIpAddress" ec2s.json)
+
+echo "$instance_id" : "$instance_type" : "$launch_time" : "$private_ip"
+```
+
+But that just seems ... slow. And repetitive. How can we make this better?
+
+If we can trust that there's no spaces in the data, we can do this:
+
+```bash
+read -r instance_id instance_type launch_time private_ip <<<"$(
+  echo '
+    {"InstanceId":"i-05b132aa000f0afa0",
+     "InstanceType":"t4g.teeny",
+     "LaunchTime":"2025-04-01T12:34:56+00:00",
+     "PrivateIpAddress":"10.0.1.2"}
+  ' |
+    super -f line -c "
+      [InstanceId,InstanceType,LaunchTime,PrivateIpAddress]
+      | join(this, ' ')" - 
+)"
+
+echo "$instance_id" : "$instance_type" : "$launch_time" : "$private_ip"
+```
+
+The IFS env var controls how Bash splits strings and defaults to space, tab, and
+newline. If we need tab-delimited output from super to support spaces in values,
+we can do this:
+
+```bash
+IFS=$'\t' read -r name title <<<"$(
+  echo '{"Name":"David Lloyd George","Title":"Prime Minister"}' |
+    super -f line -c "[Name,Title] | join(this, '\t')" -
+)"
+
+echo "$title" : "$name"
+```
+
+If we want a simpler SuperDB command, just outputting the values as a separate
+string each on its own line will require we handle that with `mapfile` into a
+Bash array. Accessing the Bash array is still fast, and this approach is about
+the equivalent in terms of time.
+
+This version is more verbose on the Bash side of things, and is probably not
+worth the simpler SuperDB command.
+
+```bash
+mapfile -t values <<<"$(
+  echo '
+    {"InstanceId":"i-05b132aa000f0afa0",
+     "InstanceType":"t4g.teeny",
+     "LaunchTime":"2025-04-01T12:34:56+00:00",
+     "PrivateIpAddress":"10.0.1.2"}
+  ' |
+    super -f line -c "values InstanceId,InstanceType,LaunchTime,PrivateIpAddress" -
+)"
+
+instance_id="${values[0]}"
+instance_type="${values[1]}"
+launch_time="${values[2]}"
+private_ip="${values[3]}"
+
+echo "$instance_id" : "$instance_type" : "$launch_time" : "$private_ip"
+```
+                                                         
+Except ...
+
+### Empty Fields and IFS Whitespace
+
+There's a flaw in space or tab-delimited options. If a field being returned is
+*empty*, the multiple delimiters will be collapsed together, causing the reads.
+                                     
+```bash
+IFS=$'\t' read -r a b c <<<"$(
+  echo '{"a":"x","b":"","c":"z"}' |
+    super -f line -c "
+      values [this.a, this.b, this.c]
+      | join(this, '\t')
+    " -
+)"
+
+echo "$a" : "$b" : "$c"
+# => x : z : 
+```
+                                                                      
+"If the value of IFS consists solely of IFS whitespace, any sequence of IFS
+whitespace characters delimits a field, so a field consists of characters that
+are not unquoted IFS whitespace, and null fields result only from quoting.
+
+If IFS contains a non-whitespace character, then any character in the value of
+IFS that is not IFS whitespace, along with any adjacent IFS whitespace
+characters, delimits a field. This means that adjacent non-IFS-whitespace
+delimiters produce a null field. A sequence of IFS whitespace characters also
+delimits a field.
+
+Explicit null arguments ("" or '') are retained and passed to commands as empty
+strings. Unquoted implicit null arguments, resulting from the expansion of
+parameters that have no values, are removed. Expanding a parameter with no value
+within double quotes produces a null field, which is retained and passed to a
+command as an empty string."
+
+-- https://www.gnu.org/software/bash/manual/bash.html#Word-Splitting-1
+
+So, maybe we can put quotes around our values? While it does retain the empty
+field, the quotes aren't stripped out of the values, which makes sense, but
+isn't helpful here.
+
+```bash
+IFS=$'\t' read -r a b c <<<"$(
+  echo '{"a":"x","b":"","c":"z"}' |
+    super -f line -c "
+      values [f'\"{this.a}\"', f'\"{this.b}\"', f'\"{this.c}\"']
+      | join(this, '\t')
+    " -
+)"
+
+echo "$a" : "$b" : "$c"
+# => "x" : "" : "z"
+```
+                   
+We should then be able to use a non-whitespace IFS to split the values, and
+retain empty fields:
+
+```bash
+IFS='|' read -r a b c <<<"$(
+  echo '{"a":"x","b":"","c":"z"}' |
+    super -f line -c "values [a, b, c] | join(this, '|')" -
+)"
+
+echo "$a" : "$b" : "$c"
+# => "x" : "" : "z"
+```

package/docs/tutorials/super_db_update.md

@@ -0,0 +1,34 @@
+---
+title: "Updating Data in a Lake"
+name: super-db-update
+description: "Workarounds for updating data in a SuperDB lake."
+layout: default
+nav_order: 11
+parent: Tutorials
+superdb_version: "0.2.0"
+last_updated: "2026-02-15"
+---
+
+# Updating Data in a Lake
+
+There are plans to eventually support this, captured in this [GitHub Issue
+#4024](https://github.com/brimdata/super/issues/4024). But for now, we'll have
+to fudge it.
+
+All we can do for now are separate `delete` and `load` actions. It's safer to do
+a load-then-delete, in case the delete fails, we'll at least have duplicated
+data, vs. no data at all in the case of failure during a delete-then-load.
+
+Since we have unstructured data, we can attempt to track the ...
+
+load: {id:4,foo:1,ts:time('2025-02-18T01:00:00')}
+load: {id:4,foo:2,ts:time('2025-02-18T02:00:00')}
+delete: -where 'id==4 and ts < 2025-02-18T02:00:00'
+
+if it's typed data
+
+delete: -where 'is(<foo>) ...'
+
+if we need to double-check duplicate data:
+
+'count() by is(<foo>)'

package/docs/tutorials/unnest.md

@@ -0,0 +1,113 @@
+---
+title: "unnest"
+name: unnest
+description: "Guide to the unnest operator including nested unnest...into patterns."
+layout: default
+nav_order: 9
+parent: Tutorials
+superdb_version: "0.3.0"
+last_updated: "2026-02-15"
+---
+
+# unnest
+
+The `unnest` operator has this signature in the super docs:
+
+```
+unnest <expr> [ into ( <query> ) ]
+```
+                                  
+The simple form of unnest is pretty straightforward, to loop over each value in
+the `<expr>` array. In this simple example, we `collect` the values output by
+`seq 1 3` into an array, and then `unnest` each value back out again.
+
+```mdtest-command
+seq 1 3 | super -s -c "
+  collect(this)
+  | unnest this
+" -  
+```
+```mdtest-output
+1
+2
+3
+```
+
+## unnest ... into examples
+
+`unnest ... into` is a bit more complicated, esp. in the case of `<expr>` being
+a two field record. The docs explain it like this:
+
+> If `<expr>` is a record, it must have two fields of the form:
+>
+> `{<first>: <any>, <second>:<array>}`
+>
+> where `<first>` and `<second>` are arbitrary field names, `<any>` is any
+SuperSQL value, and <array> is an array value. In this case, the derived
+sequence has the form:
+> ```
+> {<first>: <any>, <second>:<elem0>}
+> {<first>: <any>, <second>:<elem1>}
+> ...
+> ```
+
+Let's expand on our previous example a bit. We collect the values 1,2,3 into an
+array, as before then package that up as the values of both fields in a new
+record.
+
+```mdtest-command
+seq 1 3 | super -s -c "
+  collect(this)
+  | {foo:this, bar:this}
+  | unnest {foo, bar} into (
+      -- for each value in the bar array, pass that value plus all of data each time
+      values f'foo is {this.foo}, bar is {this.bar}'
+    )
+" -
+```
+```mdtest-output
+"foo is [1,2,3], bar is 1"
+"foo is [1,2,3], bar is 2"
+"foo is [1,2,3], bar is 3"
+```
+
+You can see here, `foo` is the `<any>` value and so is passed in as-is, while
+the inner `bar` value is each individual value in the array.
+
+But what this means is we can actually do a pair of nested `unnest` loops, like
+so: 
+
+```mdtest-command
+seq 1 3 | super -s -c "
+  collect(this)
+  | {foo:this, bar:this}
+  | unnest {foo, bar} into (
+      -- for each value in the bar array, pass that value plus all of data each time
+      unnest {this.bar, this.foo} into (
+        -- now, for each value in the foo array, pass that value plus the single value
+        -- from the outer unnest
+        values f'foo is {this.foo}, bar is {this.bar}'
+      )
+    )
+" -
+```
+```mdtest-output
+"foo is 1, bar is 1"
+"foo is 2, bar is 1"
+"foo is 3, bar is 1"
+"foo is 1, bar is 2"
+"foo is 2, bar is 2"
+"foo is 3, bar is 2"
+"foo is 1, bar is 3"
+"foo is 2, bar is 3"
+"foo is 3, bar is 3"
+```
+
+## as of versions
+
+```mdtest-command
+super --version
+```
+```mdtest-output
+Version: v0.2.0
+```