scout-essentials 1.8.7 → 1.9.0

data/research/commands-streaming-analysis.md

@@ -0,0 +1,272 @@
+# Investigation: Command Execution and Logging
+
+**Status:** Non-normative investigation artifact. May be outdated.
+
+## Scope
+CMD and Log.
+
+---
+
+## CMD
+
+### What it is
+A unified wrapper for running external commands with:
+- Tool discovery and auto-installation.
+- Streaming (pipe) and blocking (read-all) modes.
+- stdin piping.
+- stderr logging, progress tracking, and capture.
+- sudo, xvfb, pipe-chaining support.
+- SSH through Open (for remote commands).
+- Option-hash-to-flags conversion.
+
+### Core API
+```ruby
+# Blocking: returns a StringIO-like result
+result = CMD.cmd("ls -la")
+puts result.read
+
+# Streaming (pipe: true): returns a ConcurrentStream
+stream = CMD.cmd("cat huge_file", :pipe => true)
+stream.each { |line| ... }
+
+# With tool discovery
+CMD.cmd(:samtools, "view -bS aln.sam")
+```
+
+### `CMD.cmd(tool, cmd = nil, options = {})`
+- `tool` — may be a Symbol registered via `CMD.tool` (auto-install + discovery)
+  or a plain String command.
+- `cmd` — optional subcommand appended after the tool.
+- `options` — IndiferentHash with special keys:
+  - `:in` — String/IO/StringIO piped to stdin.
+  - `:pipe` — if true, return a ConcurrentStream immediately.
+  - `:stderr` — severity level (Integer) or boolean.
+  - `:progress_bar` — Log::ProgressBar or options hash.
+  - `:sudo` — prepend `sudo`.
+  - `:xvfb` — run under xvfb-run.
+  - `:autojoin` — autojoin the stream on close.
+  - `:no_fail` / `:nofail` — don't raise on failure.
+  - `:log` — toggle logging (default true).
+  - `:save_stderr` — capture stderr text.
+  - `:pipe` — output piped to another command.
+  - `:post` — callback executed after command completion.
+  - Other keys are converted to command-line flags.
+
+### Tool system (`CMD.tool` / `CMD.get_tool`)
+```ruby
+CMD.tool(:samtools, nil, nil, "samtools") { install_samtools }
+```
+- `TOOLS` is a global registry: `{ tool => [claim, test, block, cmd] }`.
+- `get_tool(tool)` checks if the tool exists (via `test` or `command -v`).
+- If not found, it tries:
+  1. `claim.produce` (if a Resource claim is given).
+  2. `block.call` (if a block is given).
+- After installation, version is detected via `scan_version_text`.
+- Tools are cached in `@@init_cmd_tool` to avoid repeated checks.
+
+### Option-to-flags conversion (`process_cmd_options`)
+```ruby
+CMD.cmd("grep", :pattern => "foo", :ignore_case => true)
+# → grep --pattern 'foo' --ignore_case
+```
+- Boolean `true` → flag without value.
+- Boolean `false`/`nil` → flag omitted.
+- String → flag with quoted value.
+- `--key=value` → `--key 'value'`.
+- `--key=` → `--key='value'` (equals form).
+- `:add_option_dashes => true` → adds `--` prefix to keys.
+
+### SSH / remote
+CMD itself does not handle SSH directly. Use `Open.ssh` or `Open.open`/`Open.read`
+with remote paths (detected via `Open.remote?`). However, CMD commands may
+include ssh as a string.
+
+### `CMD.bash`
+```ruby
+CMD.bash(<<~SH)
+  set -e
+  source ~/.bashrc
+  conda activate myenv
+  do_stuff
+SH
+```
+Wraps the string in a login shell.
+
+### `CMD.cmd_log` / `CMD.cmd_pid`
+Variants that tee stdout/stderr to `Log` in real time, useful for long-running
+commands.
+
+---
+
+## Log
+
+### What it is
+A thread-safe, severity-based logging system with colored output, progress
+bars, exception formatting, and support for redirecting output to a file or
+IO.
+
+### Severity levels
+```ruby
+Log::DEBUG = 0
+Log::LOW    = 1
+Log::MEDIUM = 2
+Log::HIGH   = 3
+Log::INFO   = 4
+Log::WARN   = 5
+Log::ERROR  = 6
+Log::NONE   = 7
+```
+Messages are only emitted if `severity >= Log.severity`.
+
+### Configuration
+- `Log.severity = Log::DEBUG` — set threshold (Integer).
+- `SCOUT_LOG` environment variable — DEBUG, LOW, MEDIUM, HIGH, WARN, ERROR, NONE.
+- `~/.scout/etc/log_severity` file — numeric severity.
+- `Log.logfile(path)` — redirect all log output to a file.
+- `Log.nocolor = true` — disable ANSI colors.
+
+### Log methods
+```ruby
+Log.debug("Detailed info")      # DEBUG
+Log.low("Slightly important")    # LOW
+Log.medium("Medium importance")  # MEDIUM
+Log.high("Important")            # HIGH
+Log.info("User-facing info")     # INFO
+Log.warn("Warning")              # WARN
+Log.error("Error")               # ERROR
+Log.exception(e)                 # ERROR + backtrace
+```
+
+### Block form (lazy evaluation)
+```ruby
+Log.debug { "Computed value: #{expensive_computation()}" }
+```
+The block is only evaluated if the severity threshold is met. Use this for
+expensive-to-compute log messages.
+
+### Thread-safety
+- `MUTEX = Mutex.new` guards `log_write` and `log_puts`.
+- Multiple threads logging simultaneously are safe.
+- The ProgressBar system uses `BAR_MUTEX` for its own state.
+
+### Log::ProgressBar
+A sophisticated multi-bar progress tracking system:
+
+```ruby
+Log::ProgressBar.with_bar(1000, :desc => "Processing") do |bar|
+  1000.times do |i|
+    # ... work ...
+    bar.tick
+  end
+end
+```
+
+Features:
+- **Stacked bars** — multiple concurrent bars rendered with vertical stacking
+  using ANSI cursor movement (`up_lines`/`down_lines`).
+- **Throughput estimation** — short-term and long-term rate, ETA calculation.
+- **Auto max-guessing** — `with_obj_bar` infers the max from file size
+  (`wc -l`), TSV/Array/Hash length, etc.
+- **Persistence** — bar state can be saved to/reloaded from a file.
+- **Depth tracking** — nested bars track their position in the stack.
+- **Silencing** — `SILENCED` array hides specific bars from rendering.
+- **Callback chains** — `callback` procs executed on completion.
+
+#### Key methods
+- `Log::ProgressBar.with_bar(max, options) { |bar| ... }` — main entry point.
+- `bar.tick(n = 1)` — increment by n.
+- `bar.pos(n)` — set absolute position.
+- `bar.process(elem)` — call `@process` callback, then tick based on return.
+- `bar.done` — print completion summary.
+- `bar.error` — print error summary.
+- `bar.remove` / `remove_bar` — clean up bar from display.
+- `with_obj_bar(obj, desc_or_max) { |bar| ... }` — auto-infer max from object.
+
+#### Bar removal
+- On normal completion: `bar.done` then `remove_bar`.
+- On exception: `bar.error` then `remove_bar`.
+- `KeepBar` exception — prevents removal for debugging.
+
+### Log::Color
+ANSI color codes for severity levels and general use:
+```ruby
+Log.color(:yellow, "warning text")
+Log.color(Log::INFO, "informational")
+```
+- `nocolor?` — checks if colors are disabled.
+- Colors are defined in `log/color_class.rb` with a mapping from severity → ANSI code.
+- `Log.uncolor(text)` — strips ANSI codes.
+
+### Log::Trap
+```ruby
+Log::Trap.trap('SIGINT') { ... }
+```
+Safely intercepts signals. Stores handlers for restore. Ensures progress bars
+are cleaned up on signal.
+
+### Log::fingerprint
+```ruby
+Log.fingerprint(obj)
+```
+Produces a compact, human-readable representation of an object for logging:
+- String → the string (truncated if long).
+- Array → `[a, b, ...]` (truncated).
+- Hash → `{k => v, ...}` (truncated).
+- Path → the path string.
+- Other → `inspect`.
+
+---
+
+## Cross-module interactions
+
+- **CMD depends on Log** — for stderr logging, debug logging, and progress bars.
+- **CMD depends on ConcurrentStream** — for stream lifecycle.
+- **CMD depends on Open** — for remote operations and `consume_stream`.
+- **Log depends on IndiferentHash** — `process_options` in ProgressBar.
+- **Log::ProgressBar depends on CMD** — `guess_obj_max` uses `CMD.cmd("wc -l")`.
+- **Log depends on Misc** — `format_seconds`, `fixutf8`.
+- **All modules use Log** — it is the foundational logging layer.
+
+---
+
+## Gotchas and warnings
+
+1. **CMD.cmd with `:pipe => true` returns immediately** — the command may
+   still be running when you receive the stream. You must `join` the stream
+   to wait for completion.
+2. **CMD.cmd default stderr level is Log::DEBUG** — by default, stderr is
+   logged at DEBUG severity. Use `:stderr => Log::HIGH` to make it visible at
+   default severity.
+3. **CMD option keys are validated** — `process_cmd_options` raises if an
+   option key contains characters outside `[a-z_0-9\-=.]+`. This prevents
+   shell injection.
+4. **Log.severity is global** — `Log.severity = Log::DEBUG` affects all
+   threads. Use `Log.with_severity(level) { ... }` for scoped severity changes.
+   This is thread-safe via a thread-local pattern.
+5. **Log::ProgressBar bars are global** — the `BARS` array is class-level.
+   Nested calls add bars. Bars from different threads may interleave in
+   display.
+6. **Log.debug block form is not always lazy** — `Log.debug { "..." }`
+   evaluates the block only if severity >= DEBUG, but `Log.debug("..." )`
+   always evaluates the string. For expensive messages, use the block form.
+7. **CMD.cmd missing tool** — if a tool is not found and no install block is
+   given, the command will fail with a `ProcessFailed` exception. Check
+   `CMD.versions` for known tools.
+8. **Log.nocolor detection** — checks `ENV['SCOUT_NO_COLOR']`, output not a
+   tty, and `Log.nocolor` flag. Colors may be unexpectedly disabled in
+   pipelines.
+9. **CMD.cmd option escaping** — single quotes are escaped (`'` → `\'`), but
+   other shell metacharacters are not escaped by default. This is by design
+   (commands may need them), but means user-supplied option values must be
+   sanitized.
+10. **Log.exception NOLOG/NOSTACK** — exception messages containing "NOLOG"
+    or "NOSTACK" will skip logging or backtrace printing. This is an internal
+    convention, not documented.
+11. **CMD.cmd with no_fail:true and pipe:true** — the stream will have
+    `no_fail` set, meaning `join` won't raise on process failure. Check
+    `stream.exit_status` to verify success.
+12. `CMD.cmd("grep", :pattern => "foo")` generates `grep --pattern 'foo'`.
+    The dashes are auto-added via `:add_option_dashes`. To suppress, pass
+    `:add_option_dashes => false`. Check current default: it appears to be
+    falsy by default, so `CMD.cmd("grep", "--pattern" => "foo")` may be
+    needed.

data/research/design-philosophy-analysis.md

@@ -0,0 +1,383 @@
+# Investigation: Design Philosophy and Cross-Cutting Patterns
+
+**Status:** Non-normative investigation artifact. May be outdated.
+
+## Scope
+Cross-cutting design principles that make scout-essentials elegant and
+expressive, identified by analyzing patterns across all modules.
+
+---
+
+## Principle 1: Annotate, don't wrap
+
+**The single most important principle.** When you have a piece of data that
+already has a natural Ruby type (String, Array, Hash, IO), you do not create
+a wrapper class. Instead, you annotate it.
+
+```ruby
+# GOOD — annotate an existing object
+path = Path.setup("data/file.tsv")   # path.class == String
+gene = Gene.setup("BRCA1", :organism, "Hsa")  # gene.class == String
+
+# WRONG — create a wrapper class
+class MyPath < String; end            # breaks String API
+class GeneWrapper; def initialize(g); @g = g; end; end  # loses String-ness
+```
+
+Annotation uses Ruby's singleton-class extension:
+- `extend SomeModule` on the specific object instance.
+- The object's class is unchanged.
+- The annotation is removable via `Annotation.purge`.
+
+This principle is applied to: Path (annotated Strings), NamedArray (annotated
+Arrays), IndiferentHash (annotated Hashes), ConcurrentStream (annotated IOs).
+
+---
+
+## Principle 2: Module composition over inheritance
+
+There are no deep class hierarchies. Behavior is composed via Ruby modules
+mixed into objects at runtime.
+
+```ruby
+# Path = String + Path annotations
+module Path
+  extend Annotation       # becomes an annotation module
+  annotation :pkgdir, :libdir, :path_maps, :map_order, :where, :original
+end
+
+path = Path.setup("data/file.tsv")  # String + Path module
+```
+
+Each concern is a module: Annotation, IndiferentHash, ConcurrentStream,
+Resource, Path. They can be mixed into the same object without conflict.
+
+---
+
+## Principle 3: `setup` is the constructor
+
+Classes are rarely instantiated with `.new`. Instead, `Module.setup(obj, ...)`
+extends an existing object with the module and sets its annotations.
+
+```ruby
+# Annotation modules provide a class-level `setup` method
+Path.setup("data/file.tsv", "mypkg")        # → annotated String
+IndiferentHash.setup({a: 1})                 # → annotated Hash
+ConcurrentStream.setup(io, pids: [123])      # → annotated IO
+Gene.setup("BRCA1", :organism, "Hsa")        # → annotated String
+```
+
+`setup` always returns the same object it received (after extension). It does
+not clone unless the object is frozen. This means `setup` is idempotent and
+non-destructive.
+
+---
+
+## Principle 4: `method_missing` as a fluent builder
+
+When a module needs to provide a dynamic, open-ended API, `method_missing` is
+used to build structures fluently.
+
+```ruby
+# Path uses method_missing to build nested paths
+path = Path.setup("data")
+path.genes.tsv           # → "data/genes/tsv" (annotated Path)
+path.results["run1"]     # → "data/results/run1"
+
+# NamedArray uses method_missing for field accessors
+arr = NamedArray.setup([1, 2, 3], [:a, :b, :c])
+arr.a   # → 1
+arr.b   # → 2
+```
+
+The key insight: `method_missing` is not for error recovery, it's for
+**building an API that mirrors the domain**. Path segments become method
+names; field names become accessors.
+
+---
+
+## Principle 5: Conventions for resource discovery
+
+The framework discovers resources by convention, not by explicit registration:
+
+| Convention | Resolution |
+|---|---|
+| Path maps `{PKGDIR}`, `{LIBDIR}`, `{PWD}`, `{HOME}` | Automatically resolved |
+| `Resource.claim(path, type, &block)` | Declares how to produce a path |
+| `CMD.tool(:name) { ... }` | Registers tool auto-install |
+| `Path#find` | Tries each map in `map_order` |
+| `Resource#produce` | Triggers claim if file missing |
+
+There are no plugin manifests or registration calls. Put a claim in the right
+module and the framework finds it.
+
+---
+
+## Principle 6: The "produced on demand" pattern
+
+Files are not pre-generated. They are produced lazily when first accessed:
+
+```ruby
+module MyData
+  extend Resource
+  self.claim Path.setup("data/file.tsv"), :proc do |path|
+    Open.write(path, generate_data())
+  end
+end
+
+path = MyData.data["file.tsv"]  # accesses via method_missing
+path.read                        # triggers produce → generate_data()
+path.read                        # second read: file exists, no production
+```
+
+This is the Scout equivalent of a build system. The claim is the build rule;
+`produce` is the build trigger; `find` is the output path.
+
+---
+
+## Principle 7: Stream everything
+
+Expensive I/O (command output, file reads, network) is always returned as
+streams, not eagerly-read buffers. Streams are ConcurrentStream-enhanced IOs
+with lifecycle management:
+
+```ruby
+stream = CMD.cmd("cat huge_file", :pipe => true)
+stream.each_line { |line| process(line) }
+stream.join  # wait for command completion
+```
+
+- `pipe: true` → streaming mode.
+- Default → blocking mode (reads all output).
+- Streams carry their PID/threads for proper cleanup.
+- Streams support callbacks for cleanup.
+
+---
+
+## Principle 8: IndiferentHash everywhere
+
+Options and configuration are always IndiferentHash — symbol/string
+indifferent. This eliminates symbol-vs-string bugs:
+
+```ruby
+options = IndiferentHash.setup({})
+options[:model] = "gpt-4"
+options['model']  # → "gpt-4"
+options[:model]   # → "gpt-4"
+```
+
+All option-processing utilities (`process_options`, `add_defaults`, `pull_keys`)
+work on IndiferentHash. When writing code that accepts options, use these
+utilities.
+
+---
+
+## Principle 9: Self-documenting through annotation propagation
+
+When you extract items from annotated collections, they inherit the parent's
+annotations:
+
+```ruby
+list = Gene.setup(["BRCA1", "TP53"], :organism, "Hsa")
+list.first.organism  # → "Hsa" (inherited from parent list)
+list.select { |g| g.length > 4 }.organism  # → "Hsa" (preserved through select)
+```
+
+This is the AnnotatedArray pattern: enumeration methods are overridden to
+propagate annotations. The data carries its metadata with it.
+
+---
+
+## Principle 10: Compact DSLs over verbose configuration
+
+Scout prefers compact, expressive DSLs:
+
+```ruby
+# SOPT compact definition
+SOPT.parse("-t--tool* tool to use
+-d--database* database
+-v--verbose")
+
+# Path maps as a hash of templates
+path_maps = {
+  current: "{PWD}/{TOPLEVEL}/{SUBPATH}",
+  home:    "{HOME}/{TOPLEVEL}/{PKGDIR}/{SUBPATH}"
+}
+```
+
+Configuration is data (hashes, strings), not objects.
+
+---
+
+## Anti-patterns to avoid
+
+### 1. Creating wrapper classes for native types
+```ruby
+# WRONG
+class PathWrapper
+  def initialize(path); @path = path; end
+  def read; File.read(@path); end
+end
+
+# RIGHT — annotate
+path = Path.setup("data/file.tsv")
+path.read  # works because Resource/path.rb adds read
+```
+
+### 2. Eager initialization
+```ruby
+# WRONG
+class Processor
+  def initialize
+    @cache = build_huge_cache()  # expensive, always runs
+  end
+end
+
+# RIGHT — lazy
+def cache
+  @cache ||= build_huge_cache()
+end
+```
+
+### 3. Explicit delegation when method_missing already works
+```ruby
+# WRONG — don't add these to a module that already uses method_missing
+def data; @path.data; end
+def results; @path.results; end
+
+# RIGHT — method_missing handles it
+```
+
+### 4. Not using IndiferentHash for options
+```ruby
+# WRONG — symbol/string fragility
+def foo(options)
+  options[:key] || options['key']  # verbose, error-prone
+end
+
+# RIGHT
+def foo(options)
+  options = IndiferentHash.setup(options) unless IndiferentHash === options
+  options[:key]
+end
+```
+
+### 5. Breaking the annotate-don't-wrap principle for IO
+```ruby
+# WRONG
+class TrackedStream
+  def initialize(io); @io = io; end
+end
+
+# RIGHT — annotate the IO
+ConcurrentStream.setup(io, :pids => [pid], :threads => [thread])
+```
+
+### 6. Forgetting to `join` streams
+```ruby
+# WRONG — orphaned subprocess
+stream = CMD.cmd("long_command", :pipe => true)
+lines = stream.readlines  # may leave process running if exception
+
+# RIGHT
+begin
+  stream = CMD.cmd("long_command", :pipe => true)
+  lines = stream.readlines
+ensure
+  stream.join
+end
+```
+
+### 7. Mutating global path configuration
+```ruby
+# WRONG — affects all paths globally
+Path.path_maps = { ... }  # class-level mutation
+
+# RIGHT — per-path or per-resource
+path = Path.setup("data", path_maps: { ... }, map_order: [...])
+```
+
+---
+
+## Cross-cutting utilities (Misc module)
+
+The `Misc` module is a grab-bag of utilities used across the codebase. It is
+the "stdlib" of scout-essentials. Key categories:
+
+### Format utilities
+- `Misc.format_paragraph(text, size, indent, offset)` — word-wrap text to fit
+  terminal width, preserving code blocks and lists.
+- `Misc.format_definition_list(hash, sep)` — format a hash as `key: value`.
+- `Misc.format_seconds(time)` — `HH:MM:SS` format.
+- `Misc.format_seconds_short(time)` — human-readable short form.
+- `Misc.colors_for(list)` — assign hex colors to unique elements.
+
+### Math utilities
+- `Misc.log2(x)`, `Misc.log10(x)` — cached multiplier versions.
+- `Misc.max`, `Misc.min`, `Misc.mean`, `Misc.stddev` — list statistics.
+- `Misc.zip_fields(lists)` — transpose arrays.
+- `Misc.bin_for_value`, `Misc.bins` — histogram binning.
+
+### Digest utilities
+- `Misc.digest_str(obj)` — deterministic string representation for digesting.
+- `Misc.digest(obj)` — MD5 of `digest_str`.
+- `Misc.file_md5(path)` — MD5 of file content.
+- `Misc.obj_md5(obj)` — MD5 of object representation.
+
+### Filesystem utilities
+- `Misc.in_dir(dir) { ... }` — chdir + yield + restore.
+- `Misc.path_relative_to(basedir, path)` — relative path computation.
+- `Misc.add_libdir(dir)` — add lib directory to LOAD_PATH.
+- `Misc.zip_zones`, `Misc.tar_files` — archive helpers.
+
+### System utilities
+- `Misc.hostname` — cached hostname.
+- `Misc.children(pid)` — child processes.
+- `Misc.pid_alive?(pid)` — check if process is alive.
+
+### Process utilities
+- `Misc.benchmark(repeats) { ... }` — benchmark and log.
+- `Misc.insist(times, sleep) { ... }` — retry with exponential backoff.
+- `Misc.pid_alive?(pid)` — check liveness.
+
+### Matching utilities
+- `Misc.match_value(value, condition)` — fuzzy/comparison matching.
+- `Misc._convert_match_condition(str)` — parse `>1`, `/regex/`, `!value`, etc.
+- `Misc.intersect_sorted_arrays(a1, a2)` — efficient sorted intersection.
+
+---
+
+## Gotchas and warnings
+
+1. **Annotation `setup` on frozen objects** — `setup` dup's frozen objects,
+   which creates a new instance. If you hold a reference to the original
+   frozen object, it won't have the annotation.
+2. **`method_missing` can mask typos** — In Path, calling `path.typo_method`
+   will create a new path `"data/typo_method"` instead of raising. This is by
+   design but can cause subtle bugs.
+   - Mitigation: `path.method_missing` only fires if the method name doesn't
+     start with `to_` and no block is given.
+3. **IndiferentHash default value interaction** — if a hash has a `default`
+   or `default_proc`, the alternate-form lookup is bypassed. This can silently
+   return the default instead of trying the other key form.
+4. **ConcurrentStream callback ordering** — callbacks are chained in LIFO
+   order (most recently added runs first). If ordering matters, document it.
+5. **Persist cache path determinism** — the path is derived from the name +
+   options hash. If the options hash is not deterministic (e.g., contains a
+   Proc or a non-deterministic object), the cache path changes every run.
+6. **Resource produce race condition** — if two processes call `produce`
+   simultaneously on a non-existent resource, both will run the claim. The
+   lock prevents this, but only if both use the same lock path. Verify that
+   `Open.lock` is always called with the canonical lock path.
+7. **Log.severity is global** — changes affect all threads. Use
+   `Log.with_severity(level) { ... }` for scoped changes.
+8. **Misc.digest of a Path** — `Misc.digest` checks if the string is a valid
+   filename and if so, digests the file content. This is usually desired
+   (digesting the data, not the path string), but can be surprising.
+9. **Path.map_order as class variable** — `@@map_order` is shared across all
+   Path instances. Modifying it globally affects all paths. Per-instance
+   `@map_order` can be set during `Path.setup`.
+10. **Annotation propagation may not match expectations** — AnnotatedArray
+    overrides specific enumeration methods (`[]`, `first`, `last`, `select`,
+    `collect`, etc.). Methods not in the list (e.g., `filter_map`, `tally`)
+    will NOT propagate annotations.