scout-essentials 1.7.1 → 1.8.0

data/doc/Annotation.md

@@ -0,0 +1,352 @@
+# Annotation
+
+The Annotation module provides a lightweight system for adding typed, named "annotations" (simple instance variables with accessors) to arbitrary Ruby objects and arrays. It's used by defining annotation modules (modules that `extend Annotation` and declare attributes via `annotation`) and then applying those annotation modules to objects at runtime.
+
+Key features:
+- Define annotation modules with named attributes.
+- Attach annotation modules to objects or arrays (including Strings, Arrays, Hashes, Procs, etc.).
+- Annotated arrays (`AnnotatedArray`) propagate annotations to their items and provide annotation-aware iteration and collection operations.
+- Support for serialization (Marshal) and a `purge` operation that strips annotations from objects (recursively for arrays and hashes).
+
+---
+
+## Creating an annotation module
+
+Define a module and `extend Annotation`. Declare annotation attributes using `annotation :attr1, :attr2`.
+
+Example:
+
+```ruby
+module MyAnnotation
+  extend Annotation
+  annotation :code, :note
+end
+```
+
+When a module is created this way, it gets:
+- an internal `@annotations` list (names of attributes),
+- accessors for declared attributes (e.g. `code`, `code=`),
+- the ability to be used to annotate objects (`MyAnnotation.setup(obj, ...)` or by `obj.extend MyAnnotation`).
+
+---
+
+## Applying annotations
+
+Use `Annotation.setup` or the annotation module's `setup` to attach annotations to objects.
+
+Basic usage:
+
+- Annotation.setup with a single annotation module:
+  ```ruby
+  Annotation.setup(obj, MyAnnotation, code: "X")
+  ```
+  or
+  ```ruby
+  MyAnnotation.setup(obj, code: "X")
+  ```
+
+- Annotation.setup accepts annotation types as:
+  - a Module constant (e.g. `MyAnnotation`),
+  - an Array of modules (`[A, B]`),
+  - a String of module names separated by `|` (will be constantized).
+
+- The `annotation_hash` (values for attributes) can be:
+  - a Hash mapping attribute name => value,
+  - a list of positional values that are zipped with the declared attribute names,
+  - a Hash with string keys (converted to symbols).
+  Examples:
+  ```ruby
+  MyAnnotation.setup(obj, :code)                # sets first attribute to :code (positional)
+  MyAnnotation.setup(obj, code: "some text")    # sets code => "some text"
+  MyAnnotation.setup(obj, "code" => "v")        # string keys are accepted
+  MyAnnotation.setup(obj, code2: :code)         # remap behavior (see examples below)
+  ```
+
+- `Annotation.setup` convenience:
+  ```ruby
+  # Apply one or more annotation modules
+  Annotation.setup(obj, [MyAnnotation, OtherAnnotation], code: "c", code3: "d")
+  ```
+
+Notes on calling `AnnotationModule.setup` (the module-level setup method used by both `Annotation.setup` and `AnnotationModule.setup`):
+- If the target `obj` is frozen, it will be duplicated before extending.
+- If a block is passed or `obj` is `nil` and a block is provided, the block (Proc) itself will be annotated (useful to attach annotations to callbacks).
+- When multiple modules are applied to the same object, their annotations are merged; accessors are available for all annotated attributes.
+
+Examples from tests:
+```ruby
+str = "String"
+MyAnnotation.setup(str, :code)
+# now str responds to `code` and `code` == :code
+```
+
+You can annotate other objects using an annotated object:
+```ruby
+a = MyAnnotation.setup("a", code: "c")
+b = "b"
+a.annotate(b)   # copies the annotation values and types from a to b
+```
+
+---
+
+## Annotated object API
+
+When an object is annotated (extended with annotation modules), it gains these helpers (provided by `Annotation::AnnotatedObject`):
+
+- `annotation_types` -> Array of annotation modules applied to the object.
+- `annotation_hash` -> Hash mapping annotation attribute names (symbols) to values stored on the object.
+- `annotation_info` -> combines `annotation_hash` plus:
+  - `annotation_types` (modules list),
+  - `annotated_array` boolean (true when object is an `AnnotatedArray`).
+- `.serialize` (instance) and `AnnotatedObject.serialize(obj)` (class method) -> returns a purged `annotation_info` merged with a `literal: obj` entry (useful when creating stable representations).
+- `annotation_id` / `id` -> deterministic digest built from the object and its annotation info (uses `Misc.digest` in the framework).
+- `annotate(other)` -> applies all annotation types and attribute values of `self` onto `other`.
+- `purge` -> returns a duplicate of the object with all annotation-related instance variables removed (`@annotations`, `@annotation_types`, `@container`).
+- `make_array` -> returns a new array containing the object, annotated with the same annotation types/values, and extended as an `AnnotatedArray`.
+
+Example:
+```ruby
+s = MyAnnotation.setup("s", code: "C")
+s.annotation_hash   # => { code: "C" }
+s.id                # => some digest
+s2 = "other"
+s.annotate(s2)
+s2.code             # => "C"
+```
+
+---
+
+## Annotated arrays (AnnotatedArray)
+
+`AnnotatedArray` is an array mixin that:
+- stores annotation types/values at the array level,
+- automatically annotates elements when they are accessed or iterated,
+- provides container tracking on items: annotated items get `container` and `container_index` attributes (via `AnnotatedArrayItem`).
+
+To make an array annotation-aware:
+```ruby
+AnnotationModule.setup(ary, code: "C")
+ary.extend AnnotatedArray
+```
+
+Behavior and methods:
+- Element access ([], first, last) returns annotated elements (unless `clean = true` passed to `[]`).
+- `each`, `each_with_index`, `select`, `inject`, `collect` iterate over annotated items (so blocks receive annotated items).
+- `compact`, `uniq`, `flatten`, `reverse`, `sort_by` are overridden to return annotated arrays (the result is annotated and extended with `AnnotatedArray`).
+- `subset(list)` and `remove(list)` return new annotated arrays representing the set intersection/difference.
+- Annotated array items receive `container` (the array) and `container_index` (the index position when produced via iteration/access).
+
+Examples:
+```ruby
+ary = ["x"]
+MyAnnotation.setup(ary, "C")
+ary.extend AnnotatedArray
+
+ary.code          # => "C"        (array-level)
+ary[0].code       # => "C"        (element annotated)
+ary.first.code    # => "C"
+ary.each { |e| puts e.code }  # iterates annotated elements
+```
+
+`AnnotatedArrayItem` helpers:
+- `container` -> reference to the array that annotated the item.
+- `container_index` -> index position supplied by the array when returning that item.
+
+Utility:
+- `AnnotatedArray.is_contained?(obj)` -> true if obj is annotated as an `AnnotatedArrayItem`.
+
+---
+
+## Serialization and Marshal support
+
+Annotations are stored as instance variables on the annotated objects; thus, `Marshal.dump` / `Marshal.load` preserve annotations and attribute values.
+
+Example (from tests):
+```ruby
+a = MyAnnotation.setup("a", code: 'test1', code2: 'test2')
+serialized = Marshal.dump(a)
+a2 = Marshal.load(serialized)
+a2.code   # => 'test1'
+```
+
+Arrays extended with `AnnotatedArray` and annotated likewise survive Marshal roundtrip; loaded arrays still annotate their elements.
+
+---
+
+## Purging annotations
+
+- AnnotatedObject#purge removes annotation instance variables from the object and returns a dup without annotations (`@annotations`, `@annotation_types`, `@container`).
+- Annotation.purge(obj) is recursive:
+  - If obj is nil => returns nil.
+  - If obj is an annotated array => calls the object's purge and then purges each element recursively.
+  - If obj is an Array => returns an Array where each element is purged.
+  - If obj is a Hash => returns a new Hash with purged keys and values.
+  - Otherwise, if object is annotated (`Annotation.is_annotated?(obj)`), returns `obj.purge`, else returns the object itself.
+
+Example:
+```ruby
+ary = ["string"]
+MyAnnotation.setup(ary, "C")
+ary.extend AnnotatedArray
+
+Annotation.is_annotated?(ary)          # => true
+Annotation.is_annotated?(ary.first)    # => true
+
+purged = Annotation.purge(ary)
+Annotation.is_annotated?(purged)       # => false
+Annotation.is_annotated?(purged.first) # => false
+```
+
+---
+
+## Detection helpers
+
+- `Annotation.is_annotated?(obj)` -> true if the object has been annotated (the object has an `@annotation_types` instance variable).
+- `AnnotatedArray.is_contained?(obj)` -> true if object is an `AnnotatedArrayItem`.
+
+---
+
+## Extending and composing annotations
+
+- Multiple annotation modules can be applied to the same object. Their attributes and values are merged on the object.
+- Annotation modules may `include` other annotation modules. When a module including another annotation module is itself extended into an object, the included module's declared attributes are propagated.
+- When a module `extend Annotation`, the `Annotation.extended` hook ensures:
+  - `@annotations` is initialized,
+  - the module includes `Annotation::AnnotatedObject` (so annotated objects get object helpers),
+  - the module extends `Annotation::AnnotationModule` (which implements `annotation`, `setup`, and include/extend integration code).
+
+Example of composing annotations:
+```ruby
+module A
+  extend Annotation
+  annotation :a1
+end
+
+module B
+  extend Annotation
+  annotation :b1
+end
+
+obj = "s"
+Annotation.setup(obj, [A, B], a1: 'one', b1: 'two')
+# obj now responds to a1, b1
+```
+
+---
+
+## Notes and edge cases
+
+- `Annotation.setup(obj, ...)` returns `nil` immediately if `obj.nil?`.
+- If the target object is frozen, the setup will duplicate it before extending.
+- `AnnotationModule.setup` can accept positional values (zipped with the declared attributes) or a hash mapping attribute names to values.
+  - Example: `MyModule.setup(obj, :val_for_first)` sets the first declared attribute to `:val_for_first`.
+  - Example: `MyModule.setup(obj, :a => 1, :b => 2)` sets attributes by name.
+- You can annotate a block/proc by passing a block to `setup` (or passing `nil` as the object and supplying a block). The block (Proc) will be extended with the annotation module.
+- `Annotation.setup` accepts a third argument (`annotation_hash`) or positional values similar to the module-level `setup`. It also accepts `annotation_types` as a string with `|` separated module names, an Array of modules, or a single module.
+
+---
+
+## API quick reference
+
+Annotation module-level:
+- Annotation.setup(obj, annotation_types, annotation_hash_or_positional_values)
+  - obj: object to annotate (String, Array, Array instance, Proc, etc.)
+  - annotation_types: Module, String (module names separated by `|`), or Array of modules
+  - annotation_hash_or_positional_values: Hash or positional values mapped to declared attributes
+  - returns the annotated object (or nil if obj.nil?)
+
+- Annotation.extended(base) (internal hook) — prepares modules that `extend Annotation`.
+- Annotation.is_annotated?(obj) -> boolean
+- Annotation.purge(obj) -> returns object or a purged (annotation-free) copy/structure
+
+Annotation::AnnotationModule (module methods available on modules that `extend Annotation`):
+- annotation(*attrs) -> declare attributes and create accessors
+- annotations -> declared attributes list
+- included(mod) — when the annotation module is included in another module, merges declared attributes
+- extended(obj) — when the annotation module is extended into an object, sets up `@annotations` and registers this module into the object's `annotation_types`
+- setup(obj, *values_or_hash, &block) -> annotate `obj` (or block) with this module and set attribute values
+
+Annotation::AnnotatedObject (instance methods added to annotated objects):
+- annotation_types -> array of modules applied
+- annotation_hash -> Hash of attribute names => values
+- annotation_info -> combines annotation_hash + metadata
+- serialize / AnnotatedObject.serialize(obj) -> purged annotation_info merged with literal
+- annotation_id / id -> digest based id
+- annotate(other) -> copy annotations onto `other`
+- purge -> duplicate object and remove annotation instance variables
+- make_array -> wrap object into annotated array
+
+AnnotatedArray (array-level helpers):
+- extend AnnotatedArray to annotate arrays and propagate annotations to their items
+- annotate_item(obj, position = nil) -> annotate an item and set container/container_index
+- [] (overridden), first, last, each, each_with_index, select, inject, collect, compact, uniq, flatten, reverse, sort_by, subset, remove
+
+---
+
+## Examples (from tests)
+
+Define annotation modules:
+
+```ruby
+module AnnotationClass
+  extend Annotation
+  annotation :code, :code2
+end
+
+module AnnotationClass2
+  extend Annotation
+  annotation :code3, :code4
+end
+```
+
+Annotate a string:
+
+```ruby
+str = "String"
+AnnotationClass.setup(str, :code)    # sets str.code == :code
+AnnotationClass2.setup(str, :c3, :c4)
+# str now includes both annotation modules and has code/code2/code3/code4
+```
+
+Annotate arrays and propagate to elements:
+
+```ruby
+ary = ["string"]
+AnnotationClass.setup(ary, "Annotation String")
+ary.extend AnnotatedArray
+ary.code       # => "Annotation String"
+ary[0].code    # => "Annotation String"
+ary.first.code # => "Annotation String"
+```
+
+Purge annotations:
+
+```ruby
+ary = ["string"]
+AnnotationClass.setup(ary, "C")
+ary.extend AnnotatedArray
+
+purged = Annotation.purge(ary)
+# purged and purged.first are not annotated anymore
+```
+
+Marshal roundtrip preserves annotations:
+
+```ruby
+a = AnnotationClass.setup("a", code: 'test1', code2: 'test2')
+d = Marshal.dump(a)
+a2 = Marshal.load(d)
+a2.code # => 'test1'
+```
+
+Annotating a block:
+
+```ruby
+# annotate the block (proc) itself
+proc_obj = AnnotationClass.setup(nil, code: :c) do
+  puts "hello"
+end
+proc_obj.code # => :c
+```
+
+This document covers the primary use and behaviors of Annotation, the annotation modules that extend it, the AnnotatedObject helpers added to annotated objects, and the AnnotatedArray behaviors. Use the examples above as templates to create, combine, and apply annotations to objects and collections.

data/doc/CMD.md

@@ -0,0 +1,203 @@
+# CMD
+
+CMD provides a convenience layer for running external commands, capturing/streaming their IO, integrating with the framework's ConcurrentStream and Open helpers, and for tool discovery/installation helpers. It wraps Open3.popen3 and adds standard patterns for piping, feeding stdin, logging stderr, auto-joining producer threads/processes, and error handling.
+
+Key features:
+- Run commands (synchronously or as streams) with flexible options.
+- Pipe command output as ConcurrentStream-enabled IO so consumers can read and then join/wait for producers.
+- Feed data into command stdin from String/IO.
+- Collect and log stderr, optionally saving it.
+- Auto-join producer threads/PIDs and surface process failures as exceptions.
+- Tool discovery/installation helpers (TOOLS registry, get_tool, conda, scan version).
+- Convenience helpers: bash, cmd_pid, cmd_log.
+
+---
+
+## Basic usage
+
+- CMD.cmd(command_or_tool, cmd_fragment_or_options = nil, options = {}) -> returns:
+  - When run with `:pipe => true` returns an IO-like stream (ConcurrentStream-enabled) that you can read from; caller should join or let autojoin close/join.
+  - When `:pipe => false` (default) returns a StringIO containing stdout (collected), after waiting for process completion.
+
+Examples:
+```ruby
+# simple capture
+out = CMD.cmd("echo '{opt}' test").read   # => "test\n"
+# with options processed into the command
+out = CMD.cmd("cut", "-f" => 2, "-d" => ' ', :in => "a b").read   # => "b\n"
+
+# pipe mode (stream returned)
+stream = CMD.cmd("tail -f /var/log/syslog", :pipe => true)
+puts stream.read   # streaming consumption
+stream.join        # wait for producers and check exit status
+```
+
+---
+
+## Important options
+
+All options are passed as an options Hash (converted with IndiferentHash), and many are special keys:
+
+- :pipe (boolean) — if true, return a stream you can read from; otherwise CMD returns a StringIO after the process completes.
+- :in — input to feed to the command:
+  - String will be wrapped by StringIO and streamed to process stdin.
+  - IO/StringIO passed will be consumed using `readpartial` in a background thread.
+- :stderr — controls stderr logging/handling:
+  - Integer severity → Log.log writes at that severity.
+  - true → maps to Log::HIGH.
+  - If stderr is enabled, stderr lines are logged as they arrive.
+- :post — callable (proc) run after command finishes (attached as stream callback in pipe mode).
+- :log — boolean to enable logging of stderr to Log (default true in many paths). Passing true/false toggles logging.
+- :no_fail (or :nofail) — if true do not raise on non-zero exit in pipe-mode setup; if omitted errors raise ProcessFailed or ConcurrentStreamProcessFailed.
+- :autojoin — when true, the returned stream will auto-join producers on EOF/close (defaults in many calls to match :no_wait).
+- :no_wait — don't wait for process to finish (used to set autojoin).
+- :xvfb — if true or string, wrap command in xvfb-run with server args (helper for GUI/CMD).
+- :progress_bar / :bar — pass a ProgressBar object to process stderr lines via bar.process.
+- :save_stderr — if true, collect stderr lines into stream.std_err.
+- :dont_close_in — when feeding :in IO, do not close the source IO after streaming to stdin.
+- :log, :autojoin, :no_fail, :pipe, :in etc. are all processed and removed from the command string.
+
+Command option helpers:
+- CMD.process_cmd_options(options_hash) → returns CLI options string:
+  - If `:add_option_dashes` key set, keys without leading dashes are prefixed with `--`.
+  - Values are quoted and single quotes escaped.
+  - Handles boolean flags (true/false/nil).
+
+Examples:
+```ruby
+CMD.process_cmd_options("--user-agent" => "firefox")
+# => "--user-agent 'firefox'"
+
+CMD.process_cmd_options("--user-agent=" => "firefox")
+# => "--user-agent='firefox'"
+
+CMD.process_cmd_options("-q" => true)
+# => "-q"
+```
+
+---
+
+## Streaming mode internals
+
+When `:pipe => true`:
+- CMD uses Open3.popen3 to spawn the process and receives sin (stdin), sout (stdout), serr (stderr), wait_thr.
+- If `:in` is provided and is an IO/StringIO, a background thread writes it into process stdin (unless `dont_close_in`).
+- Stderr is consumed in a background thread (either logged via Log at the provided severity, or passed to ProgressBar if provided, or collected if `save_stderr`).
+- `ConcurrentStream.setup` is called on the returned `sout` with threads and pids plus options like `autojoin` and `no_fail`.
+  - That allows consumers to call `sout.read`, `sout.join`, or rely on `autojoin` to close/join automatically.
+- `sout.callback` can be set to `post` callable to run after successful join.
+
+Error handling:
+- For pipe mode the library will detect non-zero process exit and raise `ConcurrentStreamProcessFailed` on join unless `no_fail` is true.
+- If `:no_fail` is passed true, failures are logged but not raised.
+
+---
+
+## Non-pipe mode internals
+
+When `:pipe` is false (default):
+- CMD still uses Open3.popen3, but it reads all stdout into a StringIO and waits for process completion before returning.
+- Stderr is read in a background thread and optionally logged/collected; after process completion, if process exit is non-zero, a `ProcessFailed` exception is raised (unless `no_fail`).
+- This mode is convenient for quick synchronous captures.
+
+---
+
+## Tool discovery & installation helpers
+
+- CMD.tool(name, claim = nil, test = nil, cmd = nil, &block)
+  - Register tools with metadata: claim (Resource or Path), a test command, install block/command and optional fallback cmd string.
+
+- CMD.get_tool(tool)
+  - Check if tool is available (runs `test` or `cmd --help`); if not, attempts to produce claim or run registered block to install.
+  - Caches result in @@init_cmd_tool to avoid repeated checks.
+  - Attempts to read version by trying `--version`, `-version`, `--help`, etc., and parsing text via `CMD.scan_version_text`.
+
+- CMD.scan_version_text(text, cmd = nil) → returns matched version string or nil.
+  - Heuristics to find version substrings related to the command name.
+
+- CMD.conda(tool, env = nil, channel = 'bioconda')
+  - Convenience to install with conda in either a given env or the login shell.
+
+---
+
+## Convenience wrappers
+
+- CMD.bash(command_string)
+  - Runs the given commands inside `bash -l` (login shell) and returns the resulting stream (pipe) — helpful when you need shell initialization (e.g., conda).
+
+- CMD.cmd_pid(...) / CMD.cmd_log(...)
+  - `cmd_pid` runs a pipe command while streaming stdout to STDERR (or logs) and returns nil; it handles progress bars and returns after join.
+  - `cmd_log` is a thin wrapper around `cmd_pid` that simply returns nil.
+
+---
+
+## Error types
+
+- ProcessFailed — raised for non-zero exit in synchronous mode or when explicitly checked.
+- ConcurrentStreamProcessFailed — raised when pipe-mode join detects failing producer subprocess (non-zero exit) and `no_fail` is not set.
+
+---
+
+## Examples (from tests)
+
+- Basic command capture:
+```ruby
+CMD.cmd("echo '{opt}' test").read        # -> "test\n"
+CMD.cmd("cut", "-f" => 2, "-d" => ' ', :in => "one two").read  # -> "two\n"
+```
+
+- Pipe usage:
+```ruby
+stream = CMD.cmd("echo test", :pipe => true)
+puts stream.read  # "test\n"
+stream.join
+```
+
+- Piped pipeline:
+```ruby
+f = Open.open(file)
+io = CMD.cmd('tail -n 10', :in => f, :pipe => true)
+io2 = CMD.cmd('head -n 10', :in => io, :pipe => true)
+io3 = CMD.cmd('head -n 10', :in => io2, :pipe => true)
+puts io3.read.split("\n").length  # => 10
+```
+
+- Handling errors:
+```ruby
+# Raises ProcessFailed for missing command
+CMD.cmd('fake-command')
+
+# In pipe mode you may get ConcurrentStreamProcessFailed on join or read/join
+CMD.cmd('grep . NONEXISTINGFILE', :pipe => true).join
+```
+
+- Use `:no_fail => true` to suppress exceptions on failure and just log.
+
+---
+
+## Recommendations & patterns
+
+- Prefer `:pipe => true` + ConcurrentStream when you want streaming processing without waiting for full output in memory.
+- Provide `:in` as an IO to stream large inputs into a subprocess.
+- Use `:autojoin => true` to automatically join producers on EOF/close (useful for simple consumers).
+- Register tools via `CMD.tool` and use `CMD.get_tool` to locate or auto-install/produce required tools.
+- Always check or propagate exceptions from `join` for pipe-mode streams to detect failing subprocesses.
+
+---
+
+## Quick API reference
+
+- CMD.cmd(tool_or_cmd, cmd_fragment_or_options = nil, options = {}) => StringIO or ConcurrentStream (when pipe)
+- CMD.process_cmd_options(options_hash) => option string appended to command
+- CMD.setup tool registry:
+  - CMD.tool(name, claim=nil, test=nil, cmd=nil, &block)
+  - CMD.get_tool(name)
+  - CMD.scan_version_text(text, cmd = nil)
+  - CMD.versions -> hash of detected versions
+- CMD.bash(cmd_string) — run in bash -l
+- CMD.cmd_pid / CMD.cmd_log — helpers for logging and running commands that stream stdout to logs
+- CMD.conda(tool, env=nil, channel='bioconda') — convenience installer wrapper
+
+---
+
+CMD centralizes robust process execution patterns needed throughout the framework: streaming, joining, logging, error detection and tool bootstrap. Use its options to control behavior for production-grade command invocation.