scout-essentials 1.8.7 → 1.9.0

data/doc/user/CommandLineOptions.md

@@ -0,0 +1,189 @@
+# Command-Line Options
+
+`SOPT` is scout-essentials' option parser: a small registry of inputs,
+shortcuts and descriptions plus a **destructive** consumer that edits
+`ARGV` in place. It is deliberately minimal — no subcommands, no coercion
+beyond booleans, no config-file layer.
+
+## Declaring inputs
+
+Options are declared as a single string. Each entry is one line:
+
+```
+-[short]--[long][*] description
+```
+
+```ruby
+require 'scout-essentials'
+
+SOPT.parse <<~OPT
+  -o--organism* Organism code
+  -t--tissue*   Tissue of origin
+  -d--dry-run   Do not write anything
+OPT
+```
+
+The `*` suffix is the **only** type marker: it marks the option as taking a
+string value. Without it the option is a boolean (`lib/scout/simple_opt/parse.rb:53`).
+It does **not** mean "required".
+
+`SOPT.setup(str)` parses the same grammar from a fuller usage document —
+summary, synopsis (a line starting with `$`), description and options — and
+then immediately calls `SOPT.consume`, so it both registers and consumes
+`ARGV` in one step (`lib/scout/simple_opt/setup.rb`).
+
+The registry lives in module-level accessors (`simple_opt/accessor.rb`):
+`inputs`, `input_types`, `input_shortcuts`, `shortcuts`, `input_descriptions`,
+`input_defaults`.
+
+## Consuming the command line
+
+```ruby
+options = SOPT.consume          # defaults to ARGV, which it MUTATES
+```
+
+`SOPT.consume(args = ARGV)` walks `args` left to right and **deletes**
+every token it recognises:
+
+- a matching `--long` or `-s` is removed and, if the input takes a value,
+  the following token (or `=value`) is removed with it;
+- unknown flags and free-standing words are left alone;
+- `--` stops the whole loop: everything after it is left in `args`.
+
+```ruby
+# probe_02 (tmp/rewrite_C/probe_02_sopt.rb)
+argv = ['-o', 'Human', '--tissue', 'Liver', 'positional', '-d']
+SOPT.consume(argv)
+# => {:organism=>"Human", :tissue=>"Liver", :"dry-run"=>true}
+# argv is now ["positional"]
+
+argv = ['-d', '--', '--not-an-option', '-x']
+# => {:"dry-run"=>true}; argv unchanged from "--" onwards
+```
+
+Because the array is edited in place, `SOPT.consume` is how a program
+separates its own options from the positional arguments it will still read.
+
+### Boolean parsing
+
+Booleans are true unless the value is one of `F`, `false`, `FALSE`, `no`
+(`simple_opt/get.rb:45`):
+
+- `--dry-run=false`, `--dry-run=F` → `false`
+- `--dry-run` → `true`
+- `--dry-run false` → `false`, and Ruby logs a warning telling you to use
+  `=` instead — this convenience swallows the next token, so use `=`.
+
+A word that merely *follows* a boolean flag and is not one of those four is
+**not** eaten: `['--dry-run', 'stray']` leaves `"stray"` in `ARGV`
+(probe_02).
+
+## State and reuse
+
+`SOPT.consume` writes two things:
+
+- the return value (an `IndiferentHash` with symbol keys), which is also
+  stashed as `@@current_options`; `SOPT.current_options = hash` lets you
+  seed it;
+- `SOPT::GOT_OPTIONS`, a module-level hash that is **merged into, never
+  reset**. Every `consume` call accumulates there, which is how
+  sub-commands that each declare their own inputs still end up with a
+  global picture of what was given (probe_02: two separate `consume` calls
+  on disjoint inputs both appear in `GOT_OPTIONS`).
+
+`SOPT.get(opt_str)` is just `parse` followed by `consume(ARGV)`.
+
+### `SOPT.require` — the only enforcement
+
+Nothing about a declared option is required. If you want to fail on a
+missing option, call it yourself:
+
+```ruby
+SOPT.require(options, :organism, :tissue)
+# raises ParameterException: Parameter 'tissue' not given
+```
+
+`ParameterException < ScoutException < StandardError` (probe_02), so plain
+`rescue` works. There is **no** variant that extracts a subset of the
+options — `SOPT.get` always parses a fresh string and consumes the whole
+`ARGV`.
+
+## Help text
+
+`SOPT.doc` renders a man-page-style document:
+
+```text
+myprog(1) -- <summary>
+=========================
+
+## SYNOPSYS
+
+myprog [--organism=<string>] [--tissue=<string>] [--dry-run[=false]]
+
+## OPTIONS
+
+-o,--organism=<string>   Organism code
+...
+```
+
+The header really is `## SYNOPSYS` — the misspelling is in the source
+(`simple_opt/doc.rb:112`) and callers grep for it; do not "fix" it in your
+matching code. `SOPT.usage` prints the doc and calls `exit 0` (probe_07 traps
+`SystemExit` and reports status 0).
+
+`SOPT.input_doc` (used by `doc`) is also the public way to format an
+explicit option list, and `SOPT.input_array_doc` formats
+`[[name, type, description, default, options], ...]` arrays — that is the
+form `Workflow`-level code uses to pass shortcut choices through.
+
+## Shortcuts and `fix_shortcut`
+
+Every declared long name automatically gets a short form: the first letter
+of the long name, if it is free (`simple_opt/doc.rb:33`,
+`fix_shortcut(name[0], name)`). When it is taken, `SOPT.fix_shortcut`
+searches for a free one:
+
+1. an existing shortcut already bound to that exact long name is reused;
+2. if the long name contains `-` or `_`, the initials of its parts
+   (`--max-cpu` → `-m` if free, else the accumulated initials);
+3. if it contains digits, the first letter plus the number;
+4. otherwise it walks forward through the letters.
+
+If no shortcut can be found, `fix_shortcut` returns `nil` and the option
+simply has no short form. **Collisions are silent**: declaring `-a--alpha`
+and `-a--also` yields `{"a"=>"alpha", "al"=>"also"}` — the second entry
+gets a longer shortcut rather than an error (probe_02). Live probe
+(`tmp/rewrite_C/probe_07_sopt_extra.rb`): registering `t` while `-t` is
+bound to `tissue` yields `"th" => "threshold"`; `another_one` gets the
+initials `"ao"`; `alpha2` gets `"a2"`.
+
+`SOPT.delete_inputs(['organism'])` removes an input from `inputs`,
+`input_shortcuts`, `shortcuts`, `input_types`, `input_defaults` and
+`input_descriptions` (`simple_opt/accessor.rb:39`). `input_shortcuts` is the
+reverse map `{'organism'=>'o'}` (probe_07).
+
+`SOPT.reset` clears **only** `shortcuts` and the internal `all` registry;
+`inputs` and the other per-input tables survive (probe_02). Call
+`SOPT.delete_inputs(SOPT.inputs.dup)` if you actually want an empty slate.
+
+## Quirks to design around
+
+- Repeating the same boolean flag just re-sets it to `true`; string
+  options keep the last value (the hash is overwritten).
+- `--opt=value` and `--opt value` are equivalent; `--opt=` (empty value)
+  yields `""`.
+- `-x` unknown: left in `ARGV`, ignored.
+- `--` is not removed from `ARGV` either; it only stops scanning.
+- Options are keyed by their **long** name, symbolised. Shortcuts never
+  appear in the result.
+- `SOPT.consume` returns the current options *and* keeps them in
+  `GOT_OPTIONS`; the two objects are not the same object, and `GOT_OPTIONS`
+  is the one that survives later `SOPT.reset` calls.
+
+## Where to go next
+
+- [StartHere](../StartHere.md)
+- [Logging and Progress](LoggingAndProgress.md) — `Log.warn` used by the
+  boolean heuristic.
+- [Architecture](../developer/Architecture.md) — which repos own the
+  higher-level CLI layers (attribution table).

data/doc/user/Cookbook.md

@@ -0,0 +1,211 @@
+# Cookbook
+
+Short, self-contained recipes that combine the scout-essentials modules.
+Every snippet here was executed; the backing probes are listed under each
+heading and live in `tmp/rewrite_C/` (older probes in `tmp/rewrite_A/`,
+`tmp/rewrite_B/` are referenced by their P-numbers, which map to
+`research/behavior-probes.md`).
+
+The theme of the library: plain objects annotated with provenance, paths
+resolved from a declaration, work cached on disk, streams piped without
+holding everything in memory.
+
+## Get-or-build a file: `claim` + `produce_and_find`
+
+```ruby
+require 'scout-essentials'
+
+module Data
+  extend Resource
+  self.pkgdir = 'cookbook_probe'
+end
+
+Data.claim Data.tmp['list'], :string, "S001\nS002\nS003\n"
+found = Data.tmp['list'].produce_and_find
+Open.read(found)              # => "S001\nS002\nS003\n"
+Data.tmp['list'].produce_and_find   # second call returns the same path, no work
+```
+
+`Resource.claim(path, type, contents, block)` registers *how* a file is
+produced (a type like `:string`/`:proc`, literal contents, or a block); the
+claimed `Path` is annotated and its `produce` materialises it into the
+resource tree (`lib/scout/resource/path.rb:2`,
+`lib/scout/resource/produce.rb`). `produce_and_find` produces when needed and
+returns `self.find`. Probe: `tmp/rewrite_C/probe_03_cookbook.rb`
+(`claim` + first/second call returning the same path, `Open.read` content).
+
+## Cache invalidation with `:update` and `:check`
+
+`Persist.persist` normally returns the cached value untouched. Two options
+change that (`lib/scout/persist.rb`):
+
+```ruby
+a = Persist.persist('expensive', :marshal, :update => false) { "computed-once" }
+b = Persist.persist('expensive', :marshal, :update => false) { "recomputed" }
+a == b                        # => true, block skipped both times
+c = Persist.persist('expensive', :marshal, :update => true)  { "forced-recompute" }
+c                             # => "forced-recompute" — block re-run
+
+d = Persist.persist('dependent', :marshal,
+                    :check => 'tmp/src.txt') { "from-#{Open.read('tmp/src.txt')}" }
+```
+
+`:update => true` always re-runs the block; `:check` names a file whose mtime
+invalidates the entry. Probe: probe_03 (a == b, block not re-run; `:update`
+re-runs; `:check` path resolves).
+
+## Fetch a remote file
+
+```ruby
+require 'scout-essentials'
+
+url  = 'https://example.org/data.tsv'
+Open.wget(url, :auto)         # downloads; cached under Open.remote_cache_dir
+data = Open.wget(url)         # serves from cache on the next call
+
+Open.scp('user@host:/path/file', 'local_copy', :target => 'user@host')
+```
+
+`Open.wget` shells out to `wget` (`lib/scout/open/remote.rb`); `Open.read` on
+an `http(s)://` or `ssh:` URL fetches transparently. Retries and rate limits
+are controlled by `Open.wait(lag, key)`. See
+[RemoteData.md](RemoteData.md) and
+[Working with Files](WorkingWithFiles.md).
+
+## Annotate, serialise, restore
+
+```ruby
+require 'scout-essentials'
+
+module SampleInfo
+  extend Annotation
+  annotation :organism, :tissue
+end
+
+sample = SampleInfo.setup('S001', :organism => 'Human', :tissue => 'Liver')
+sample.annotation_hash        # {:organism=>"Human", :tissue=>"Liver"}
+sample.serialize              # {:organism=>"Human", :tissue=>"Liver",
+                              #  :annotation_types=>[SampleInfo], :annotated_array=>false,
+                              #  :literal=>"S001"}
+
+restored = Annotation.setup('S003', 'SampleInfo',
+                            'organism' => 'Human', 'tissue' => 'Liver')
+restored.tissue               # => "Liver"
+```
+
+`Annotation.setup(obj, "A|B", hash)` is the module-level deserialiser; the
+`"A|B"` string is split on `|` and each name is looked up as a constant. An
+unknown type name is **warned about and skipped** (probe_03 prints
+`Annotation NoSuchAnnotation not defined` on STDERR, then
+`Annotation.setup('S004', 'NoSuchAnnotation', ...)` returns the plain
+un-annotated string). `serialize` produces a plain `Hash` with `:literal`,
+`:annotation_types` (module objects) and `:annotated_array` keys. There is no
+TSV serialisation of annotations in this repo — `Annotation.tsv` belongs to
+scout-gear (see the [attribution table](../developer/Architecture.md)).
+
+## Stream a pipeline without buffering
+
+```ruby
+require 'scout-essentials'
+
+stream = Open.open_pipe do |sin|
+  ['S001', 'S002'].each { |s| sin.write "1\t#{s}\n" }
+  sin.close
+end
+Open.consume_stream(stream)   # => "1\tS001\n1\tS002\n"
+```
+
+`Open.open_pipe` builds an IO from a block and returns it unread; `consume_stream`
+drains it. `tee_stream` splits one stream into two consumers:
+
+```ruby
+Open.write('tmp/in.txt', "1\n2\n3\n")
+main, copy = Open.tee_stream(
+  CMD.cmd('gzip -c', :pipe => true, :in => Open.open('tmp/in.txt'))
+)
+Open.consume_stream(main, true, 'tmp/in.txt.gz')   # writes the file
+copy.join                                          # waits for the copy thread
+File.exist?('tmp/in.txt.gz')                       # => true
+```
+
+Consumers follow the rescue contract for the deliberate-abort signals:
+
+```ruby
+begin
+  data = Open.consume_stream(stream)
+rescue Aborted, AbortedStream
+  Log.warn "aborted mid-stream"
+end
+```
+
+Probe: probe_03 (open_pipe data, tee_stream producing a real gzip file).
+See [Handling Streams](HandlingStreams.md) and the
+[Streaming Model](../developer/StreamingModel.md).
+
+## Progress bars
+
+```ruby
+require 'scout-essentials'
+
+items = (1..100).to_a
+
+Log::ProgressBar.with_obj_bar(items, 100) do |bar|
+  items.each { bar.tick }
+end
+
+Log::ProgressBar.with_bar(20, :desc => 'Counting') do |bar|
+  20.times { bar.tick }
+end
+
+bar = Log::ProgressBar.new_bar(10, :desc => 'Half-way')
+10.times { bar.tick }
+Log::ProgressBar.remove_bar(bar)
+```
+
+The block of `with_obj_bar` receives **only the bar**; the object is never
+yielded back (`log/progress/util.rb:167-170`). The *second* argument selects
+the bar: a String is the description, a Numeric the max, a Hash the options,
+an existing bar object is reused, and `true` guesses the max from the first
+argument via `guess_obj_max` — which returns `nil` even for a plain `Array`
+in a bare scout-essentials process: the first `when TSV` arm raises
+`NameError` (the constant is not defined here) and the surrounding
+`rescue Exception` turns that into `nil` (live check;
+`log/progress/util.rb:101-137`). Pass a Numeric max or a `:max` hash for
+deterministic sizing. There is no `Log.bar` helper; use
+`Log::ProgressBar.new_bar` / `remove_bar`.
+
+## Command-line usage with SOPT
+
+```ruby
+require 'scout-essentials'
+
+SOPT.parse <<~OPT
+  -o--organism* Organism code
+  -t--tissue*   Tissue of origin
+  -d--dry-run   Skip writing
+OPT
+
+argv    = ['-o', 'Human', 'positional', '-d']
+options = SOPT.consume(argv)   # argv is mutated: matched args removed
+argv                          # => ["positional"]
+
+SOPT.require(options, :organism)   # ParameterException when nil
+```
+
+The `*` marks an option as **taking a string value**; without it the option is
+a boolean. Probe: probe_02 (options hash, `argv_left == ["positional"]`,
+`SOPT.require` raising). See
+[Command-Line Options](CommandLineOptions.md).
+
+## Recipe index
+
+| Task | Tool | Page |
+| --- | --- | --- |
+| Resolve a path | `Path.setup`, `Resource` | [Working with Files](WorkingWithFiles.md) |
+| Run a command | `CMD.cmd` | [Running Commands](RunningCommands.md) |
+| Cache a computation | `Persist.persist` | [Caching Results](CachingResults.md) |
+| Build a file on demand | `Resource.claim` + `produce_and_find` | [Producing Resources](ProducingResources.md) |
+| Annotate objects | `Annotation` | [Annotating Data](AnnotatingData.md) |
+| Pipe data | `Open.open_pipe`, `CMD.cmd(:pipe)` | [Handling Streams](HandlingStreams.md) |
+| Log and show progress | `Log`, `Log::ProgressBar` | [Logging and Progress](LoggingAndProgress.md) |
+| Parse options | `SOPT` | [Command-Line Options](CommandLineOptions.md) |

data/doc/user/HandlingStreams.md

@@ -0,0 +1,236 @@
+# Handling Streams
+
+When `CMD.cmd(..., :pipe => true)` or `Open.open_pipe` hands you an IO, that
+object has been extended with `ConcurrentStream`: it carries the producer
+thread(s) and pid(s) that feed it, plus callbacks and an abort protocol. This
+page is the user-facing contract; the internals are in
+[Streaming Model](../developer/StreamingModel.md).
+
+## Anatomy
+
+```ruby
+io = CMD.cmd('grep x', :pipe => true)
+io.threads        # => [input thread, stderr thread, ...]  (producer side)
+io.pids           # => [pid]
+io.callback       # => nil or a Proc run on successful join (see :post)
+io.abort_callback # => nil or a Proc run on abort(exception)
+io.std_err        # => "" (filled by :save_stderr)
+io.log            # => last stderr line (pipe mode, when :log => true)
+io.lock, io.lockfile, io.pair, io.next, io.filename, io.autojoin, io.no_fail
+```
+
+Attributes come from `ConcurrentStream.setup` (concurrent_stream.rb:12-70) and
+`CMD.cmd` sets `:pids`, `:threads`, `:autojoin` (default `no_fail`) and
+`:no_fail` on the returned stream. `exit_status` exists but is *not* reliable
+after a normal read+join — it stays `nil` because only `join_pids` sets it, and
+that method empties `pids` when it runs; see
+[Running Commands](RunningCommands.md) for the probe.
+
+## `close` vs `join` vs abort
+
+- **`join`** is the finishing move: `join_threads`, `join_pids`, raise
+  `stream_exception` if one is set, run `join_callback` (the composed callback),
+  close, release the lock, and mark `joined?`. It never joins `@pair` — the
+  other end of an internal pipe is the producer's business.
+- **`close`** on an `autojoin` stream performs `super` (the plain IO close) and
+  then joins if the stream is at EOF; with `autojoin` off it just closes.
+  Closing early while a producer still writes raises `IOError`/`Errno::EPIPE`
+  in the producer, which the stream converts to an abort — so for an early,
+  deliberate stop use `abort`, not `close`.
+- **`abort(exception)`** is the safe early close: it marks the stream aborted,
+  extends it with the `AbortedStream` marker, runs `abort_callback`, interrupts
+  and joins every producer thread, sends `SIGINT` to every pid, clears the
+  callbacks, **propagates to `@pair`** (aborting the other pipe end), closes and
+  unlocks. It is idempotent (a second call only logs). Threads get
+  `Aborted.new` raised in them, so producer bodies should rescue `Aborted`.
+- **`force_close` does not exist** in this repo. The only reference is a dead
+  `respond_to?` guard inside `Open.grep` (open/util.rb:26); a plain IO has no
+  such method (probe `tmp/rewrite_B/probe_13_force_close.rb`: `IO#respond_to?
+  (:force_close) => false`). The early-close tool is `abort`.
+
+Consumers should follow this rescue contract:
+
+```ruby
+begin
+  data = Open.consume_stream(stream, false, dst)   # or .each / .read
+rescue Aborted, AbortedStream
+  # deliberate stop: log and move on; the stream is already aborted+closed
+rescue ConcurrentStreamProcessFailed => e
+  # a producer failed; e.pid / e.msg available
+end
+```
+
+`AbortedStream` is a *marker module* (`concurrent_stream.rb:4-9`) —
+`AbortedStream.setup(obj, exception)` extends an object so later code can read
+`obj.exception` and recover the real cause; `sensible_write` uses exactly that.
+
+## Callbacks
+
+- `stream.add_callback(&block)` **composes**: it wraps the existing callback so
+  the *new* block runs *after* the old one (probe
+  `tmp/rewrite_B/probe_19_streaming_apis.rb`: `add_callback order:
+  [:first, :second]`). `ConcurrentStream.setup(stream, &block)` also composes in
+  that order, which is why calling setup twice on the same stream is safe.
+- `callback` / `abort_callback` are plain accessors over single chained procs
+  (built by setup when you pass `:callback` / `:abort_callback` options or a
+  block). There is **no `add_abort_callback`** — assign `abort_callback = proc
+  { |exception| ... }` (only the last one wins unless you compose by hand).
+- `join` runs the composed `callback`; `abort` runs `abort_callback` with the
+  exception and then discards both callbacks.
+
+```ruby
+s = Open.open_pipe { |sin| sin.puts 'x' }
+s.add_callback { puts 'a' }
+s.add_callback { puts 'b' }     # runs after 'a'
+s.join                          # prints "a\nb"
+```
+
+## Helpers in `Open`
+
+### `Open.consume_stream(io, in_thread = false, into = nil, into_close = true, &block)`
+
+Pumps the stream to completion and returns the last chunk read. `Path` inputs
+are ignored, closed streams are joined and skipped. With `into` (an IO, or a
+String/Path file path whose parent dirs are created) it writes every chunk
+there; `into_close` (default true) closes `into` when it responds to `close`.
+On `Aborted` or any exception it aborts the source, closes `into`, **removes the
+partial output file** and re-raises. With `in_thread: true` the whole drain runs
+in a new thread that is pushed onto `io.threads`. The block runs after a
+successful drain. Verified: `probe_19_streaming_apis.rb` (`consume_stream
+return: "data"`, `consume_stream into path: "into-file\n"`).
+
+### `Open.sensible_write(path, content, options = {}, &block)`
+
+Atomic write into a lock-protected tmp file followed by a rename. Key
+behaviours for streams:
+
+- When the content is a stream, an `Aborted` raised while copying is
+  **swallowed** (`Log.low "Aborted sensible_write"`), the stream is aborted and
+  the target is deleted; the partial tmp file is always removed in `ensure`.
+- A non-Aborted exception recovers the *original* upstream cause from an
+  `AbortedStream`-marked content (`content.exception`) and re-raises that,
+  deleting the target. See
+  [Streaming Model](../developer/StreamingModel.md) for the marker.
+- After a successful copy the content stream is joined (but not if it is a
+  `Path` or already joined).
+
+Verified: P25/P33 in `research/behavior-probes.md` ("Aborted in
+sensible_write: NOT raised (swallowed); partial tmp left: 0").
+
+### `Open.open_pipe(do_fork = false, close = true, &block)`
+
+Creates a pipe and returns the **read end** (`sout`), with the block executed in
+a producer thread that writes the other end (`sin`).
+
+- **Block arity**: the block always receives `sin` whether it declares a
+  parameter or not (arity 0 blocks simply ignore it). Verified:
+  `probe_20_open_pipe_arity.rb` (`arity-0 block: "arity0\n"`, `arity-1: "w\n"`).
+- **No block** raises `RuntimeError "No block given"`.
+- **Fork mode** (`do_fork: true`) runs the block in a child process instead of
+  a thread: the child purges registered input pipes, closes `sout`, yields,
+  `exit! 0`; the parent closes `sin` and sets the pid on `sout` via
+  `ConcurrentStream.setup(sout, :pids => [pid])` — no threads, no callbacks.
+  `close: false` in the child leaves `sin` open after the block returns
+  (verified: `fork mode: "from-fork\n"`, `fork noclose: "fork-noclose\n"`).
+- **Thread mode** pairs the two ends (`pair`), runs the block through
+  `ConcurrentStream.process_stream` (close+join on exit, abort on error), and
+  registers the thread on both ends. An exception in the block aborts the
+  stream and re-raises at the consumer.
+
+### `Open.pipe`
+
+Takes **no arguments** and returns the raw `[sout, sin]` pair from `IO.pipe`
+(plus registering `sin` in `OPEN_PIPE_IN`). Calling it with a positional
+argument raises `ArgumentError` (probe `probe_19_streaming_apis.rb`:
+`Open.pipe positional: ArgumentError`). There is no multi-command helper here —
+chain commands by feeding one stream into `:in` of the next `CMD.cmd`.
+
+### `Open.tee_stream(stream)` / `tee_stream_thread_multiple(stream, num)`
+
+Returns an **Array** of streams (`num` copies, default 2): the first is the
+"main" copy with `autojoin: true`, the rest have no autojoin. A splitter thread
+reads the source once and writes every chunk to all copies; the main copy's
+callback joins the source and closes the extra write ends, and its
+`abort_callback` propagates an abort to the source and the other copies.
+Verified: P31/P33 and `probe_19_streaming_apis.rb` (`tee_stream count: 2`,
+`tee[0].autojoin => true`, `tee[1].autojoin => nil`).
+
+```ruby
+main, copy = Open.tee_stream(CMD.cmd('gzip -c', :pipe => true, :in => input))
+# write copy to disk while main feeds the next command, then join copy
+```
+
+### `Open.line_monitor_stream(stream, &block)`
+
+Builds a tee, then a monitor thread reads the monitor copy line by line calling
+`block.call(line)` — the block therefore runs **concurrently** with whoever
+consumes the returned stream, not after. Failures in the block abort the monitor
+and are re-raised into the returned stream (`out.raise $!` when supported). The
+returned stream is the second copy, annotated from the source and set up with
+the monitor thread. Verified: P33 and `probe_19_streaming_apis.rb`.
+
+### `Open.read_stream(stream, size)`
+
+Blocking read of exactly `size` bytes (plain `stream.read(missing)` loop,
+`lib/scout/open/stream.rb:401`), raising `ClosedStream` if EOF is reached
+first. Useful for binary framing; `probe_19_streaming_apis.rb` shows
+`read_stream(4) => "0123"`.
+
+### `Open.sort_stream(stream, header_hash: '#', cmd_args: nil, memory: false)`
+
+Streams `header_hash`-prefixed lines straight through, then sorts the rest.
+`memory: false` (default) pipes the remainder into `env LC_ALL=C sort
+<cmd_args>` (`-u` by default when `cmd_args` is nil) and consumes that stream
+into the output; `memory: true` reads the whole remainder, sorts it in Ruby and
+writes it out. Everything runs inside `ConcurrentStream.process_stream`, so the
+source is closed+joined and aborted on error. Verified:
+probe_19_streaming_apis.rb: feeding `"# header\nc\na\nb\n"` returns
+`"# header\na\nb\nc\n"` — the header passes through untouched, the rest is
+sorted.
+
+### `Open.collapse_stream(s, line: nil, sep: "\t", header: nil, compact: false, &block)`
+
+Merges consecutive lines sharing the same first field, joining the other
+columns with `|` (or dropping empty parts when `compact: true`). An optional
+block receives the accumulated column array and its return value becomes the
+row payload. Verified: `probe_19_streaming_apis.rb`.
+
+## `Open.open` block form and `DontClose`
+
+```ruby
+res = Open.open(file) do |io|
+  next io.read if io.is_a?(String)          # IO/StringIO pass straight through
+  raise DontClose.new(io.read)              # payload escapes, io still closes
+end
+```
+
+`Open.open` yields and **always closes and joins** the IO afterwards (the
+`ensure` at open.rb:70-77). Raising `DontClose` with a payload makes the block
+form *return* the payload instead of the IO, while still closing — it is an
+early-return mechanism, not a way to keep the handle. Verified:
+`probe_21_dontclose.rb` (`DontClose returns payload: "payload"`, `closed after
+DontClose: true`). Any other exception aborts, joins and re-raises the stream.
+
+## Progress bars
+
+Pass `:progress_bar` (a `Log::ProgressBar`, `lib/scout/log/progress.rb`; the option key is `:progress_bar` — there is no `:bar` key)
+to `CMD.cmd` and each stderr line ticks it (`bar.process(line)` at cmd.rb:643):
+`probe_24_bar.rb` counts 2 ticks for a 2-line stderr, in both pipe and
+non-pipe mode. With `:log => true` (`CMD.cmd_log`) the stderr text is also
+recorded in `stream.log`. `Log::ProgressBar` itself supports `:process =>
+proc{|elem| elem.length}` so a tick can be weighted per element.
+
+## Where the diagnostics go
+
+Log output (including severity-logged stderr lines) goes to the Log logfile /
+STDERR. **`std_err` is the per-stream capture**, filled by `:save_stderr` — see
+[Running Commands](RunningCommands.md#save-stderr--capture-stderr-instead-of-logging-it).
+There is no "paired stderr stream" object; in pipe mode stderr is drained by a
+thread inside `CMD.cmd`.
+
+## Related
+
+- [Running Commands](RunningCommands.md) — how these streams are produced.
+- [Streaming Model](../developer/StreamingModel.md) — setup, join/abort
+  internals, exception propagation.
+- [Working with Files](WorkingWithFiles.md) — `Open.read/write` on top.