scout-essentials 1.8.7 → 1.9.0

data/doc/user/RunningCommands.md

@@ -0,0 +1,218 @@
+# Running Commands
+
+`CMD` is scout-essentials' subprocess layer: one entry point, `CMD.cmd`, that
+covers shell commands, no-shell command arrays, stdin piping, timeouts, stderr
+handling and external-tool bootstrap. This page documents what the code does
+(`lib/scout/cmd.rb`); for the object `:pipe => true` hands back, see
+[Handling Streams](HandlingStreams.md) and the
+[Streaming Model](../developer/StreamingModel.md).
+
+## Command forms
+
+`CMD.cmd(tool, cmd, options)` accepts three shapes:
+
+```ruby
+CMD.cmd('echo hello').read            # => "hello\n"  (String: run by a shell)
+CMD.cmd(['echo', 'array-form']).read             # => "array-form\n"
+CMD.cmd('echo', 'arg2', '-n' => true, '-r' => true).read
+```
+
+- **String** — `tool` alone, or `tool + ' ' + cmd` when both are given. The
+  string is handed to `Open3.popen3(ENV, cmd)`, so shell features work.
+- **Array** — no shell is involved; `Open3.popen3(ENV, *cmd_array)` is called
+  with the array plus each option as a separate argument. `process_cmd_options_array`
+  turns `'n' => 'val'` into `['n', 'val']` (or `['n=val']` for a key ending in
+  `=`), so quoting is not an issue here.
+- **Hash-only** (`CMD.cmd({'echo' => 'x'})`) is *not* a command form; the Hash is
+  treated as options, `cmd`/`tool` stay nil and the call fails.
+
+Verified: probe `tmp/rewrite_B/probe_15_cmd_forms.rb` (string, string+cmd,
+array, array+string, symbol tool with nil cmd, ProcessFailed on hash-only).
+
+### The `'{opt}'` placeholder
+
+In String form the processed options are substituted for the literal
+placeholder `'{opt}'` **only when it is single-quoted** in the command string:
+
+```ruby
+CMD.cmd("cut -d' ' -f2 '{opt}'", '-n' => true, in: 'one two three').read
+# => "two\n"        options replaced the quoted placeholder
+
+CMD.cmd("echo {opt} a 1", 'x' => 1).read   # => "{opt} a 1\n"
+CMD.cmd("echo '{opt}' a 1", 'x' => 1).read # => "a 1\n"  (note: trailing space)
+```
+
+Unquoted `{opt}` is left untouched — the substitution matches `'\{opt\}'`
+exactly. Without a placeholder the option string is appended to the command.
+Verified: probe `probe_14_cmd_opts.rb` lines 10-11 and the C8 fix in
+`research/doc_audit/RunningCommands.md`.
+
+## Option quoting: `process_cmd_options`
+
+Every option that is not one of the reserved keys below becomes part of the
+command line. `CMD.process_cmd_options(options)` builds that string:
+
+| option/value | result |
+|---|---|
+| `'opt' => true` | `opt` (bare flag, no value) |
+| `'-v' => 'V'` | `-v 'V'` |
+| `'-v=' => 'V'` | `-v='V'` (key ends in `=`: value glued, still quoted) |
+| `'opt' => nil`, `'opt' => false` | dropped |
+| value containing `'` | `\'`-escaped, then wrapped in `'...'` |
+| `:add_option_dashes => true` | prepends `--` to keys not already starting with `-` |
+
+The quoting rule is uniform: the value is always wrapped in single quotes, and
+a value containing an apostrophe has it backslash-escaped first. There is no
+"spaces only" special case.
+
+**Key validation**: an option key that does not match `/^[a-z_0-9\-=.]+$/i`
+raises `Invalid option key` before anything runs.
+
+**Arrays are not expanded.** An Array value is stringified (`to_s`) and quoted
+like any other value — `"#{value}"` produces `n '["a", "b"]'` in String mode
+and `['n', '["a", "b"]']` in array mode. Pass separate calls or build the
+command yourself if you need repeated flags. Verified: `probe_14_cmd_opts.rb`
+lines 4-5 and the cmd-level Array check.
+
+Verified examples: `probe_14_cmd_opts.rb` (dashes, `=`, nil/false, true,
+apostrophes, invalid key, arrays).
+
+## stderr: severities, capture and files
+
+`options[:stderr]` selects how the child's stderr is treated. The default
+(added by `cmd.rb:181`) is **`Log::DEBUG`** — stderr lines are only logged at
+debug severity, i.e. invisible unless you raise `Log.severity` above it.
+
+- `:stderr => true` is normalised to `Log::HIGH` (probe `probe_23_bar_log.rb`).
+- Any other Integer is a `Log` severity: the ladder is
+  `DEBUG=0, LOW=1, MEDIUM=2, HIGH=3, INFO=4, WARN=5, ERROR=6, NONE=7`
+  (`lib/scout/log.rb` `SEVERITY_NAMES`), so `:stderr => Log::MEDIUM` shows
+  stderr as warnings while staying quieter than `HIGH`.
+- Log output itself goes to the Log logfile / STDERR (see the
+  [Streaming Model](../developer/StreamingModel.md) for what per-stream capture
+  means); the per-command stderr *text* is never attached to the stream.
+
+### `:save_stderr` — capture stderr instead of logging it
+
+`ee24c68` extended `:save_stderr` to three shapes. All of them also fill the
+`std_err` attribute (String on non-pipe results, on the returned stream in pipe
+mode), so you can inspect it after the fact:
+
+- **`:save_stderr => true`** — the text is captured into `std_err` and nothing
+  is logged.
+- **`:save_stderr => path`** (String, Scout `Path` or `Pathname`) — CMD opens
+  the path for writing (truncating), **creates missing parent directories**,
+  writes stderr to it **line-buffered** so `tail -f` can follow a running
+  command, and **closes it when the command ends**. `std_err` is populated too.
+- **`:save_stderr => io`** (anything responding to `write`/`<<`) — every chunk
+  is written and flushed, but CMD **never closes it**; closing is the caller's
+  business. `std_err` is populated too.
+
+```ruby
+res  = CMD.cmd('sh -c "echo err >&2; echo out"', :save_stderr => true)
+res2 = CMD.cmd(..., :save_stderr => 'log/cmd.err')   # nested dirs created
+dst  = File.open('err.txt', 'w')
+CMD.cmd(..., :save_stderr => dst)                     # dst stays open
+```
+
+Implementation: `cmd.rb:194-217` (destination setup), the writer thread /
+inline writer, and the `ensure` at `cmd.rb:600-612` that flushes and closes a
+CMD-owned file. Verified by `test/scout/test_cmd_save_stderr.rb` (13 tests,
+incl. live `tail -f` polling) and probes `probe_17_save_stderr.rb`
+(boolean/String/Path/Pathname/IO/StringIO, pipe and non-pipe, truncation) and
+`probe_16_exitstatus.rb` (`std_err` populated in both modes).
+
+## Exit status, `no_fail` and failure
+
+- Non-pipe mode: `CMD.cmd(...)` waits for the child and raises
+  `ProcessFailed` when the exit status is non-zero, unless `:no_fail` (alias
+  `:nofail`) is given.
+- `:no_fail => true` **suppresses** `ProcessFailed`/`ConcurrentStreamProcessFailed`
+  — and `exit_status` then stays `nil`, in pipe mode too. If you need the code,
+  call `join_pids` yourself. Verified: `probe_16_exitstatus.rb`
+  (`pipe read+join exit_status: nil`, `explicit join_pids exit_status: 0`,
+  `non-pipe exit_status: 0`).
+- `exit_status` is only ever set by `ConcurrentStream#join_pids`
+  (`concurrent_stream.rb:127`), which also empties `pids`, so it can only be
+  used once. A stream that is read and joined normally has `exit_status == nil`
+  (probe `probe_26_join_es.rb`: `read+join: es=nil` with `joined?` true; only
+  an explicit early `join_pids` yields `0`). Do not rely on `stream.exit_status`.
+- If the process never starts (bad executable, no such file) `ProcessFailed` is
+  raised immediately — also suppressed by `no_fail`, which then returns `nil`.
+- A failed *producer thread* in pipe mode surfaces as
+  `ConcurrentStreamProcessFailed` when the consumer closes/joins the stream
+  (`probe_13_force_close.rb`).
+
+## Timeout
+
+`CMD::Timeout < ProcessFailed` carries `command` and `timeout` and is raised by
+a watchdog when `:timeout => seconds` elapses (`cmd.rb:310-408`,
+`TIMEOUT_KILL_GRACE = 1.0`). This is the only way to bound a command's runtime.
+
+- **Non-pipe mode**: the watchdog raises in the calling thread.
+- **Pipe mode**: the exception is routed through `ConcurrentStream#abort` — it
+  lands in `stream_exception`, the stream is aborted (killing the process,
+  clearing pids, unblocking a blocked reader) and is re-raised when the consumer
+  reads or joins. It is *not* raised directly in the caller at command start.
+
+## Tool management
+
+There is **no `CMD.add_tool`**. Registration is:
+
+```ruby
+CMD.tool(:samtools, claim, test, cmd, &block)   # internally stored [claim, test, block, cmd]
+CMD.get_tool(:samtools)                          # ensures it is usable; returns the command name
+CMD.versions                                     # => {"samtools" => "1.17", ...}
+CMD.conda('samtools', 'env', 'bioconda')         # conda install fallback
+CMD.bash(cmd)                                    # bash -l login shell, :autojoin => true
+CMD.scan_version_text(text, 'samtools')          # heuristically pull a version string
+CMD.cmd_log('...')                               # run + echo STDOUT/STDERR, returns nil
+CMD.cmd_pid('...')                               # same implementation, also returns nil
+                                                 # (both force :pipe/:log; the pid only shows up
+                                                 #  inside the 'STDOUT [pid]:' header)
+```
+
+`get_tool` runs `test` (or `command -v cmd`), and if that fails produces the
+`claim` Resource (or calls `block`; a Hash result is passed to
+`Resource.install`). It then records a version from `--version`/`-version`/
+`--help` output. Tools are stored in the `TOOLS` IndiferentHash; `versions`
+returns only entries matching `/\d+\./`.
+
+## Options reference (consumed by `CMD.cmd`)
+
+| key | effect |
+|---|---|
+| `:in` | stdin: a String is written by a thread; an IO/StringIO is read; a ConcurrentStream is streamed (and closed unless `:dont_close_in`). Also `:in_pipe` for a pipe-backed writer. Verified: `probe_25_keeping_in.rb` (String, IO, stream, `in_pipe` returns an IO). |
+| `:pipe` | return a ConcurrentStream instead of the text/StringIO |
+| `:stderr` | severity for stderr logging (default `Log::DEBUG`); `true` → `Log::HIGH` |
+| `:save_stderr` | `true` / path / IO — see above |
+| `:no_fail`, `:nofail` | suppress failure raising (both spellings) |
+| `:autojoin` | join the stream when it is closed/read; **`CMD.cmd` sets `:autojoin => no_fail`** |
+| `:no_wait` | alias used to default `autojoin` (`autojoin = no_wait if autojoin.nil?`) |
+| `:timeout` | seconds; watchdog; only runtime bound |
+| `:post` | proc run after the command/stream finishes (teardown, forcing upstream closes) |
+| `:progress_bar` | a `Log::ProgressBar`; stderr lines tick it (`probe_24_bar.rb`: 2 ticks in both pipe and non-pipe mode) |
+| `:log` | defaults to `true` (`log = true if log.nil?`, cmd.rb:224): pipe-mode stderr lines are `Log.log`ged at the chosen `:stderr` severity. `:log => false` silences that logging. Not the same as `CMD.cmd_log` (a separate helper that forces `:pipe`/`:log`). |
+| `:sudo`, `:xvfb` | prefix the command |
+| `:dont_close_in` | keep the `:in` stream open |
+| `:add_option_dashes` | passed through to `process_cmd_options*` |
+
+`:wait`, `:canfail`, `:empty_inputs` and `:separator` are **not** options of
+`CMD.cmd` in this repo — `cmd.rb` deletes none of them, so they would be
+forwarded to `process_cmd_options` and end up as command-line text. `:canfail`
+exists on `Persist` (`persist.rb:133`) and on `Resource` claims, not here.
+Verified by grepping `lib/` for the four names (`probe_25_keeping_in.rb` tail).
+
+## The block is ignored
+
+`CMD.cmd(...) { ... }` accepts a block but **never calls it** — the only block
+invocation inside `cmd.rb` is in `CMD.tool`. Verified live by
+`tmp/docaudit/probe_hs_c15_block.rb`: in both pipe and non-pipe mode the block
+body never runs, while `:post` and `stream.add_callback` do. Use
+`:post => proc{}` or stream callbacks for post-join work.
+
+## Related
+
+- [Handling Streams](HandlingStreams.md) — the returned object's lifecycle.
+- [Streaming Model](../developer/StreamingModel.md) — internals, error paths.
+- [Working with Files](WorkingWithFiles.md) — `Open.grep` on top of `CMD`.

data/doc/user/WorkingWithFiles.md

@@ -0,0 +1,217 @@
+# Working with Files
+
+This guide explains how to read, write, and manage files in scout-essentials:
+the `Open` module for I/O, compression, grepping, atomic writes, locking, and
+remote file handling, plus `TmpFile` for scratch space.
+
+## Reading: `Open.read` and `Open.open`
+
+```ruby
+require 'scout-essentials'
+
+Open.read('VERSION')             # => "1.8.8"     (UTF-8 fixed by default)
+Open.read('VERSION', nofix: true) # => raw bytes as read
+
+Open.open('data.txt') do |f|
+  f.each_line { |line| ... }
+end
+
+Every stream returned by `Open.open` is extended with `Open::NamedStream`,
+whose `filename` attribute records the file it was opened from (useful when
+the stream flows through pipelines):
+
+```ruby
+io = Open.open('data.txt')
+io.filename # => "data.txt"
+```
+
+- `Open.read`/`Open.open` accept a `Path` (or a `String`); a `Path` is
+  resolved with `find` before opening.
+- Both accept options. Notable ones:
+  - `:mode` — open mode (`'r'` default, `'rb'` for binary reads).
+  - `:grep` / `:invert_grep` — filter lines while reading (below).
+  - `:zip` / `:gzip` / `:bgzip` — force a decompressor regardless of
+    extension.
+  - `:noz` — disable automatic decompression and read the raw file.
+
+## Compression
+
+Detection is **extension-only and case-sensitive**. A file is decompressed
+only when its name ends (lower-case) with `.gz`, `.bgz`, `.zip`:
+
+```ruby
+Open.read('file.gz')    # decompressed
+Open.read('file.GZ')    # raw gzip bytes — NOT detected
+Open.read('file.tgz')   # raw gzip bytes — NOT detected
+Open.read('file.tar.gz')# decompressed (ends in .gz)
+Open.read('file.gz.bak')# raw gzip bytes — NOT detected
+Open.gzip?('file.tgz')  # => false
+```
+
+Force decompression explicitly when the name does not cooperate:
+
+```ruby
+Open.open('file.tgz', :gzip => true).read
+```
+
+## Grepping lines
+
+`:grep` and `:invert_grep` select lines using the system `grep`. **With a
+block**, `Open.read` collects the transformed lines into an Array; **without
+a block** it returns the matched lines as a String (and `Open.open` returns
+a grep pipe):
+
+```ruby
+# file f contains: apple / banana / cherry
+Open.read(f, grep: 'an')                 # => "banana\n"   (String)
+Open.read(f, grep: 'an') { |l| l.strip.upcase } # => ["BANANA"]
+Open.read(f, grep: 'an', invert_grep: true)     # => "apple\ncherry\n"
+Open.read(f, grep: %w(apple banana)) { |l| l.strip } # => ["apple", "banana"]
+```
+
+**`invert_grep` has no effect on its own.** `Open.read`/`Open.open` only
+grep when `grep` is truthy (`Open#file_open(file, grep = false, ...)`
+extracts `:grep`/`:invert_grep` but only greps for `grep`); passing
+`invert_grep:` without `grep:` returns the whole file unchanged:
+
+```ruby
+Open.read(f, invert_grep: 'an')          # => "apple\nbanana\ncherry\n" (no-op)
+```
+
+To invert without going through `:grep`, use the low-level helper with its
+third argument:
+
+```ruby
+Open.grep(Open.open(f), 'an', true).read # => "apple\ncherry\n"
+```
+
+`:grep` accepts a String (passed to the system `grep` as a pattern) or an
+Array of Strings (written to a pattern file used with `grep -f`). Arrays are
+matched with `grep -w -F` (whole-word, fixed strings) unless you pass
+`fixed_grep: false`, which switches to plain `grep -f` semantics.
+
+## Writing: `Open.write` and appends
+
+```ruby
+Open.write('out.txt', "content\n")       # creates parent dirs
+Open.write('out.txt') { |f| f.puts "x" } # block form
+```
+
+`Open.write` always overwrites by default (`mode: 'w'`). **To append, pass
+`mode: 'a'`** — there is no public `Open.append` class method:
+
+```ruby
+Open.write('log.txt', "one\n")
+Open.write('log.txt', "two\n", mode: 'a')
+Open.read('log.txt') # => "one\ntwo\n"
+```
+
+On error, `Open.write` removes the partially written file and re-raises.
+
+### Atomic writes: `sensible_write`
+
+`Open.sensible_write(file, content = nil, options = {})` refuses to overwrite
+an existing file unless `:force => true`; it takes a String or a block and
+removes the target when the block raises (`Aborted` is swallowed silently):
+
+```ruby
+Open.sensible_write(path, "v1")
+Open.sensible_write(path, "v2")            # keeps "v1"
+Open.sensible_write(path, "v2", force: true) # overwrites
+```
+
+## File primitives
+
+All of these accept `Path` or `String` and resolve `Path`s via `find`; most
+create parent directories as needed:
+
+| Call | Effect |
+|---|---|
+| `Open.cp(src, dst)` | `cp_r`, creates parent dirs, replaces `dst` |
+| `Open.mv(src, dst)` | two-step move (`.tmp_mv.` intermediate), creates parent dirs |
+| `Open.rm(file)` | removes a file or broken link |
+| `Open.rm_rf(file)` | recursive remove |
+| `Open.mkdir(dir)` | `mkdir -p` if missing |
+| `Open.mkfiledir(file)` | `mkdir -p` the parent of `file` |
+| `Open.touch(file)` | `touch`, creating parent dirs |
+| `Open.ln(src, dst)` | hard link (falls back to `ln_s` via `Open.link`) |
+| `Open.ln_s(src, dst)` | symbolic link |
+| `Open.ln_h(src, dst)` | `ln -L` with `cp -L` fallback |
+| `Open.link(src, dst)` | `Open.ln`, falling back to `Open.ln_s` |
+| `Open.link_dir(src, dst)` | `cp -lr` — copy a directory tree of hard links |
+| `Open.same_file(a, b)` | `File.identical?` |
+| `Open.exists?(f)`, `Open.directory?(f)`, `Open.size(f)`, `Open.ctime(f)`, `Open.mtime(f)` | stats |
+
+```ruby
+src = 'data/src.txt'
+Open.write(src, 'S')
+Open.cp(src, 'data/sub/dst.txt')           # parents created
+Open.mv('data/sub/dst.txt', 'data/m.txt')
+Open.touch('data/t/t.txt')
+Open.ln(src, 'data/hard.txt')              # nlink > 1
+Open.link_dir('data/a', 'data/a2')
+Open.same_file(src, 'data/hard.txt')       # => true
+```
+
+`Open.mtime` has one special case: for a symlink or a file with multiple
+hard links it consults a sibling `.info` file when `Step` is defined (a
+scout-gear concept) and falls back to `Pathname#realpath`; otherwise it is a
+plain `File.mtime`.
+
+## Locking: `Open.lock`
+
+```ruby
+Open.lock('var/cache/persistence/file') do
+  # exclusive access while the block runs
+end
+```
+
+`Open.lock(file, unlock = true, options = {})` uses the vendored `Lockfile`
+implementation. The lock file lives next to the target by default; pass a
+`Lockfile` instance via `options[:lock]` to reuse an existing one. See
+[Caching Results](CachingResults.md) for how `Persist` uses it.
+
+## Streams and `consume_stream`
+
+`Open.open_pipe` returns a `ConcurrentStream` that you can consume lazily;
+`Open.consume_stream(stream)` reads it to the end (and joins it), and
+`Open.sensible_write` accepts a stream, writing it asynchronously. When a
+stream carries a `filename`, `NamedStream` records it so tools can recover
+the original name. See
+[Streaming Model](../developer/StreamingModel.md).
+
+## Remote files
+
+`Open.read`/`Open.open` transparently handle `http(s)://` and `ssh:` URLs by
+shelling out (`Open.wget`, `Open.ssh`), and `Open.sync` / `Open.rsync` copy
+remote trees into the local cache (`Open.remote_cache_dir`, default
+`$HOME/.scout/var/cache/open-remote`). Remote fetching is covered in
+[RemoteData.md](RemoteData.md).
+
+## Scratch files: `TmpFile`
+
+```ruby
+TmpFile.tmp_file('prefix')     # => ".../tmpfiles/tmp-<n>" (a fresh path)
+TmpFile.user_tmp               # => $HOME/tmp/scout
+TmpFile.tmp_for_file('a/b/c')  # => "·a·b·c" (flat, separators replaced with ·)
+TmpFile.with_file(content) do |file| ... end   # writes, yields the path, removes it
+TmpFile.with_dir do |dir| ... end
+```
+
+`TmpFile.tmp_for_file` flattens the whole path into a single file name
+inside the tmpfiles directory (`/` → `·`), it does **not** create a nested
+directory structure; `TmpFile.tmp_for_dir` does not exist. Options may
+append a `[key]` and an `&F[...]` fingerprint, and names are truncated to
+`MAX_FILE_LENGTH` (150).
+
+## Related
+
+- [Caching Results](CachingResults.md) — `Persist` uses `Open.lock` and
+  `sensible_write`.
+- [Path Resolution](../developer/PathResolution.md) — `find`/`follow`, path
+  maps.
+- [Producing Resources](ProducingResources.md) — Resource uses Path for
+  resolution.
+- For streaming internals, see
+  [Streaming Model](../developer/StreamingModel.md).
+- For remote data, see [RemoteData.md](RemoteData.md).