scout-essentials 1.7.1 → 1.8.0

data/doc/ConcurrentStream.md

@@ -0,0 +1,163 @@
+# ConcurrentStream
+
+ConcurrentStream is a mixin that augments IO-like stream objects (pipes returned by subprocess wrappers, in-memory streams, etc.) with concurrency-aware lifecycle management: tracking threads and child PIDs that produce/consume the stream, coordinated joining, aborting, callbacks, and safe cleanup. It is used throughout the framework for streams returned by commands (CMD.cmd) and by tee/pipe helpers in Open.
+
+There is also a tiny AbortedStream helper to mark a stream as aborted and attach an exception.
+
+---
+
+## What it does
+
+When a stream is set up with ConcurrentStream.setup, the stream is extended with methods and attributes to:
+- record producer threads and child PIDs (threads/pids),
+- automatically join and check producer exit status,
+- attach callbacks to run once production is complete,
+- abort producers/consumers cleanly (raise exceptions in threads, kill PIDs),
+- support autjoining/auto-closing when the consumer finishes reading,
+- attach a lock object that will be unlocked after join,
+- carry metadata: filename, log, paired stream (pair), next stream in pipeline, and more.
+
+This lets a consumer read from a stream and then reliably wait for producers to finish and detect failures (non-zero exit), or abort the whole pipeline on errors.
+
+---
+
+## Setup
+
+ConcurrentStream.setup(stream, options = {}, &block)
+
+- Extends `stream` with ConcurrentStream methods (unless already extended).
+- Options (recognized):
+  - :threads — thread or array of threads that produce or manage this stream.
+  - :pids — pid or array of child process ids to wait for.
+  - :callback — proc to call after successful join (can also be provided as block).
+  - :abort_callback — proc to call on abort.
+  - :filename — textual name for logging/error messages.
+  - :autojoin — boolean, if true join on close/read EOF and auto-unlock lock.
+  - :lock — Lockfile instance to unlock after join.
+  - :no_fail — boolean; if true treat non-zero child exit as non-fatal.
+  - :pair — paired stream (e.g., the other side of a pipe) so aborts propagate.
+  - :next — next stream in pipeline when teeing/forwarding
+  - :log, :std_err — metadata captured for error messages
+- If a block is given it is appended to the stream callback.
+
+Example:
+```ruby
+ConcurrentStream.setup(io, threads: [t], pids: [pid], autojoin: true, filename: "ls-out")
+```
+
+---
+
+## Important attributes & predicates
+
+The stream object gets attributes:
+- threads, pids — lists of threads and subprocess PIDs to manage.
+- callback, abort_callback — procs to call on success/abort.
+- filename — friendly name used in logs and error messages.
+- joined? — true after join completed.
+- aborted? — true after abort called.
+- autjoin — whether to auto-join on EOF/close.
+- lock — optional Lockfile to unlock after join.
+- stream_exception — exception captured that should be re-raised by readers/joins.
+- no_fail — allow ignoring nonzero child exit.
+
+AbortedStream.setup(obj, exception = nil) can be used to mark a stream aborted and attach exception (helper used internally).
+
+---
+
+## Joining / waiting
+
+- join_threads
+  - Joins registered threads, and if a thread's return value is a Process::Status checks success (unless no_fail). If a thread represented a subprocess and exit status indicates failure, raises ConcurrentStreamProcessFailed.
+
+- join_pids
+  - Waits for PIDs via Process.waitpid and raises on non-zero exit unless no_fail.
+
+- join_callback
+  - Runs `callback` once and clears it.
+
+- join
+  - Calls join_threads, join_pids, raises stored stream_exception if set, runs callback, closes stream (if not closed) and marks joined. Also unlocks `lock` if present. Any exceptions are propagated after unlocking.
+
+---
+
+## Aborting
+
+- abort(exception=nil)
+  - Mark stream aborted, store exception in stream_exception, call abort_callback, abort threads (raise into threads) and kill PIDs with SIGINT (best-effort), clear callbacks, close stream, and unlock lock if held. Also propagate abort to `pair` stream if present.
+
+- abort_threads(exception=nil)
+  - Raises exception (or Aborted) into producer threads and joins them.
+
+- abort_pids
+  - Kills pids with INT.
+
+Use abort to ensure fast cleanup on error and to signal paired streams.
+
+---
+
+## Reading & closing
+
+- read(*args)
+  - Wraps normal `read` with exception capture: on error stores `stream_exception` and re-raises. If `autojoin` is enabled and EOF reached, `close` is called automatically.
+
+- close(*args)
+  - If `autojoin` is true, `close` will try to `join` (ensuring producers have finished) and then close; on exceptions it aborts, joins, and re-raises.
+
+`joined?` and `aborted?` reflect the stream's lifecycle.
+
+---
+
+## Callbacks
+
+- add_callback(&block)
+  - Attaches an additional callback executed after producers finish. Multiple callbacks stack and are executed in order at join time.
+
+- callback and abort_callback may be set in setup — used by caller to run cleanup or post-processing.
+
+---
+
+## Error propagation
+
+- stream_raise_exception(exception)
+  - Stores `stream_exception`, raises it into all producer threads (so they can abort), and calls `abort`.
+
+- When join discovers a failing child process, it raises a ConcurrentStreamProcessFailed. Tests exercise this by running `grep` on a nonexisting file and asserting the exception is raised.
+
+When `no_fail` is set true, non-zero exits or join errors are logged but not raised.
+
+---
+
+## Utilities
+
+- annotate(stream) — copy the current stream's threads/pids/callback/etc. onto another stream (useful when creating derived streams).
+- filename — returns stored filename or a fallback derived from stream.inspect.
+- process_stream(stream, close: true, join: true, message: "...") { ... }
+  - Class method wrapper that sets up the stream and ensures the block runs, then closes and joins the stream as requested. On exceptions it aborts the stream and re-raises.
+
+Example usage (from framework):
+- `CMD.cmd(..., pipe: true, autojoin: true)` returns a ConcurrentStream-enabled IO. Consumer reads from the stream; when EOF reached or read finishes, the stream auto-joins producers and raises errors on non-zero exit.
+
+Test examples:
+```ruby
+# success case
+io = CMD.cmd("ls", pipe: true, autojoin: true)
+io.read
+io.close
+
+# failure case raises ConcurrentStreamProcessFailed
+io = CMD.cmd("grep . NONEXISTINGFILE", pipe: true, autojoin: true)
+io.read   # raises ConcurrentStreamProcessFailed
+```
+
+---
+
+## Typical patterns and recommendations
+
+- When producing streams from background threads or subprocesses, call `ConcurrentStream.setup(stream, threads: t, pids: [pid], autojoin: true, filename: name)` so readers can join/observe failures.
+- When teeing a stream to multiple outputs, register the splitter thread as a producer on each out-stream so each consumer can join independently.
+- Use `abort(exception)` to stop whole pipeline on error and ensure all producers/consumers are signaled and cleaned up.
+- Use `process_stream` helper to wrap a block that processes a stream and ensure it is closed and joined safely.
+
+---
+
+ConcurrentStream centralizes safe management of concurrent IO pipelines: shared producers (threads/PIDs), stream cleanup, callback semantics, and error propagation. It is a core primitive used by Open, CMD, and persistence/teeing helpers to make streaming robust in multi-threaded / multi-process contexts.

data/doc/IndiferentHash.md

@@ -0,0 +1,240 @@
+# IndiferentHash
+
+IndiferentHash provides Hash utilities and a mixin that makes hash access indifferent to String vs Symbol keys. It also includes a set of helper utilities for parsing, merging and transforming option hashes used across the framework.
+
+Two main pieces:
+- IndiferentHash mixin (extend any Hash instance with IndiferentHash to get indifferent access behavior and extra hash helpers).
+- CaseInsensitiveHash mixin (separate mixin to allow case-insensitive string keys).
+
+---
+
+## Quick usage
+
+Make a Hash indifferent (string/symbol interchangeable):
+
+```ruby
+h = { a: 1, "b" => 2 }
+IndiferentHash.setup(h)   # returns h extended with IndiferentHash
+
+h[:a]      # => 1
+h["a"]     # => 1
+h[:b]      # => 2
+h["b"]     # => 2
+```
+
+Make a Hash case-insensitive (string keys compared case-insensitively):
+
+```ruby
+h = { a: 1, "b" => 2 }
+CaseInsensitiveHash.setup(h)
+
+h[:a]      # => 1
+h["A"]     # => 1
+h[:A]      # => 1
+h["B"]     # => 2
+```
+
+---
+
+## IndiferentHash mixin (methods added to a Hash)
+
+Call IndiferentHash.setup(hash) to extend a hash instance.
+
+Behavior highlights:
+- Access by Symbol or String: h[:k] and h["k"] resolve to the same entry when possible.
+- Nested hashes returned from [] are automatically set up with IndiferentHash.
+- Values are stored normally, but deletion and inclusion checks accept Symbol/String interchangeably.
+
+Methods and behaviors:
+
+- IndiferentHash.setup(hash)
+  - Extends the given hash with IndiferentHash and returns it.
+
+- merge(other)
+  - Returns a new IndiferentHash with keys from self merged with other (other wins). The result is an IndiferentHash.
+
+- deep_merge(other)
+  - Recursively merges nested hashes: if both have same key and values are hash-like, merges them deeply (preserving IndiferentHash behavior on nested hashes).
+
+- [](key)
+  - Returns value for key. If not found directly, attempts the alternate form:
+    - If given Symbol/Module, will try the String key.
+    - If given String, will try the Symbol key.
+  - If the value is itself a Hash, it will be extended with IndiferentHash before returning.
+  - Note: behavior pays attention to hash default/default_proc. If a default exists and the key isn't explicitly present in keys, it returns the default without attempting alternative forms.
+
+- []=(key, value)
+  - Deletes any existing matching key (either symbol or string) before setting the new one. This avoids duplicate representations of the same logical key.
+
+- values_at(*keys)
+  - Returns an array of values for the provided keys (indifferent to symbol/string form).
+
+- include?(key)
+  - Returns true if either symbol or string form exists.
+
+- delete(key)
+  - Deletes by key; will try symbol and string form and return deleted value if found.
+
+- clean_version
+  - Produces a plain Ruby hash where keys are strings (stringified keys), preferring the first occurrence.
+
+- slice(*keys)
+  - Returns a new IndiferentHash containing only the requested keys. Accepts symbol or string forms; ensures both forms are considered.
+
+- keys_to_sym! and keys_to_sym
+  - keys_to_sym! converts string keys in-place to symbols (best-effort; rescue on any failed conversion).
+  - keys_to_sym returns a new IndiferentHash with keys converted to symbols.
+
+- prety_print
+  - A convenience wrapper that calls Misc.format_definition_list(self, sep: "\n") (keeps existing behavior/name `prety_print` as in code).
+
+- except(*list)
+  - Returns a hash copy excluding provided keys. Accepts symbol/string forms; returns a result consistent with Hash#except but extended to be indifferent.
+
+Notes:
+- The implementation ensures nested hashes returned from [] or merges are set up with IndiferentHash automatically.
+- Some method names are intentionally spelled as in the implementation (`prety_print`).
+
+---
+
+## CaseInsensitiveHash
+
+A separate mixin to make key lookup case-insensitive (for string keys). Use CaseInsensitiveHash.setup(hash) to extend a hash.
+
+Behavior:
+- On lookup, it first tries the provided key directly. If no value, it converts the key to lowercase string and looks up a precomputed map (original_key_by_downcase) to find the actual stored key. This permits "A" and "a" to refer to the same entry.
+- values_at returns values for provided keys using the case-insensitive lookup.
+
+Example:
+
+```ruby
+h = { a: 1, "b" => 2 }
+CaseInsensitiveHash.setup(h)
+
+h["A"]  # => 1
+h[:A]   # => 1
+h["B"]  # => 2
+```
+
+---
+
+## Options helpers (IndiferentHash::Options functions)
+
+These utilities are useful for parsing and handling option hashes and strings.
+
+- add_defaults(options, defaults = {})
+  - Ensures options is an IndiferentHash, accepts defaults as Hash or string (string gets parsed). Adds defaults only for keys not present in options.
+  - Returns the options (modified/extended).
+
+- process_options(hash, *keys)
+  - Sets up IndiferentHash on hash.
+  - If the last argument is a Hash, it is used as defaults (added first).
+  - If a single key passed, returns and removes that key from hash (prefers symbol then string).
+  - If multiple keys passed, returns array of removed values for each key.
+  - Example: IndiferentHash.process_options(h, :limit) or IndiferentHash.process_options(h, :a, :b, default: 1)
+
+- pull_keys(hash, prefix)
+  - Pulls keys with prefix_... from hash and returns a new IndiferentHash with the suffixes as keys.
+  - Also consumes "#{prefix}_options" if present and merges into result.
+  - Example: given { foo_bar: 1, "foo_x" => 2 }, pull_keys(h, :foo) => { bar: 1, x: 2 } (keys matched as string/symbol appropriately).
+
+- zip2hash(list1, list2)
+  - Zips two lists into a hash (keys from list1, values from list2) and sets it up as IndiferentHash.
+
+- positional2hash(keys, *values)
+  - Converts positional values into a hash keyed by keys.
+  - Supports the common pattern where the last argument is a Hash of extra/defaults. In that case:
+    - Combines given values into a hash, removes nil/empty values, adds defaults from the extras, and prunes keys not in the original keys set.
+  - Example: IndiferentHash.positional2hash([:one,:two], 1, two: 2, extra: 4) => { one: 1, two: 2 }
+
+- array2hash(array, default = nil)
+  - Accepts an array of [key, value] pairs and builds an IndiferentHash.
+  - If value is nil and default provided, uses a dup of default for that key.
+
+- process_to_hash(list) { |list| ... }
+  - Yields list to block, expects a result list; zips original list with returned list into an IndiferentHash.
+
+- hash2string(hash)
+  - Serializes a simple hash into a string representation (sorted by key). Only handles values of certain simple classes; others are omitted. Uses ":" prefix for symbol keys/values in output.
+  - Output format is key=value pairs joined with "#".
+
+- string2hash(string, sep = "#")
+  - Parses the string produced by hash2string (or similar) and converts back into an IndiferentHash. Supports:
+    - :symbol keys/values (leading ":")
+    - quoted strings, integers, floats, booleans, regexps (/.../)
+    - empty values treated as true
+  - Example roundtrip: IndiferentHash.string2hash(IndiferentHash.hash2string(h)) == h (for supported types).
+
+- parse_options(str)
+  - Parses a shell-like option string of key=value pairs, supporting quoted values and comma-separated lists (preserving quoted items with spaces).
+  - Returns an IndiferentHash.
+  - Example: IndiferentHash.parse_options('blueberries=true title="This is a title" list=one,two,"and three"')
+
+- print_options(options)
+  - Serializes an options hash into a space-separated string of key=value pairs; array values become CSV (properly quoted if containing spaces).
+
+---
+
+## Examples (from tests and usage)
+
+Indifferent access:
+
+```ruby
+h = { a: 1, "b" => 2 }
+IndiferentHash.setup(h)
+h[:a]  # => 1
+h["a"] # => 1
+h["b"] # => 2
+h[:b]  # => 2
+```
+
+Deep merge:
+
+```ruby
+o = { h: { a: 1, b: 2 } }
+n = { h: { c: 3 } }
+IndiferentHash.setup(o)
+o2 = o.deep_merge(n)
+o2[:h]["a"]  # => 1
+o2[:h]["c"]  # => 3
+```
+
+Options parsing:
+
+```ruby
+opts = IndiferentHash.parse_options('blueberries=true title="A title" list=one,two,"and three"')
+opts["title"]                # => "A title"
+opts["list"]                 # => ["one", "two", "and three"]
+```
+
+String <-> hash roundtrip:
+
+```ruby
+h = { a: 1, b: :sym, c: true }
+s = IndiferentHash.hash2string(h)
+h2 = IndiferentHash.string2hash(s)
+# h2 should equal h for the supported/simple types
+```
+
+pull_keys example:
+
+```ruby
+h = { "foo_bar" => 1, :foo_baz => 2, "other" => 3 }
+IndiferentHash.setup(h)
+prefixed = IndiferentHash.pull_keys(h, :foo)
+# prefixed => { "bar" => 1, :baz => 2 } (returned as IndiferentHash)
+# and h no longer contains those entries
+```
+
+---
+
+## Implementation notes / caveats
+
+- IndiferentHash.setup extends a single hash instance (not the Hash class). Use it on any hash instance you want to treat indifferently.
+- Nested hashes returned from [] or created by zip/positional helpers get extended automatically with IndiferentHash.
+- The [] lookup has special handling with Hash defaults: if the hash has a default or default_proc
+  and the requested key is not present in keys, it will return the default without trying the alternate symbol/string form.
+- CaseInsensitiveHash is independent of IndiferentHash; you can mix both if needed, but their behaviors are separate.
+- Some helper names are spelled as in the codebase (e.g., prety_print), used for compatibility.
+
+This module covers common patterns required for flexible option handling in the framework: indifferent access, easy merging and defaulting, parsing/printing option strings, and extracting prefixed option subsets.

data/doc/Log.md

@@ -0,0 +1,235 @@
+# Log
+
+The Log module is the framework-wide logging and progress utility. It provides:
+- leveled logging (DEBUG, LOW, MEDIUM, HIGH, INFO, WARN, ERROR, NONE)
+- colored output and utilities for color manipulation and gradients
+- fingerprinting for compact object summaries
+- a rich ProgressBar facility with multi-bar, ETA and history support
+- helpers to trap and ignore stdout/stderr and to direct logs to a logfile
+- convenience debug/inspect helpers
+
+Files / components:
+- log.rb — core logging API, level handling, formatting, logfile control
+- log/color.rb — integration with Term::ANSIColor and concept color mapping
+- log/color_class.rb — Color class for hex color parsing, blending, lightening/darkening
+- log/fingerprint.rb — compact fingerprint representations for many object types
+- log/progress{.rb, /util, /report} — ProgressBar implementation and helpers
+- log/trap.rb — utilities to trap/ignore STDOUT/STDERR
+
+---
+
+## Configuration & Environment
+
+- Log.severity (module attribute): current logging level threshold. Messages below this level are ignored.
+- SEVERITY constants: DEBUG, LOW, MEDIUM, HIGH, INFO, WARN, ERROR, NONE (assigned 0..7).
+  - Use Log.get_level(value) to convert numeric/string/symbol into numeric level.
+- Default severity:
+  - Determined by environment variable `SCOUT_LOG` if set to one of the level names.
+  - Otherwise read from `~/.scout/etc/log_severity` if present, else INFO.
+- Color disabled if `ENV["SCOUT_NOCOLOR"] == 'true'` or SOPT.nocolor set.
+- Log.tty_size returns terminal rows (uses IO.console.winsize or `tput li`), falls back to ENV["TTY_SIZE"] or 80.
+- Log.logfile(file_or_io) — set logfile target:
+  - Passing a String opens the file in append mode, sync=true.
+  - Passing an IO or File sets that as the logfile.
+  - If not set, logging writes to STDERR.
+- Thread-safety: Log.log_write and Log.log_puts synchronize writes via MUTEX.
+
+---
+
+## Coloring & Color utilities
+
+- Log extends Term::ANSIColor and exposes helpers:
+  - Log.color(color, str = nil, reset = false)
+    - color can be:
+      - Symbol naming an ansi color (e.g. :green)
+      - Integer index into SEVERITY_COLOR
+      - Concept color name (Log.CONCEPT_COLORS map keys like :title, :path, :value)
+      - A Color/hex handled by Color class (indirectly via Colorize)
+    - If str is nil, returns only the color control string (unless nocolor true).
+    - If nocolor is true, returns str unchanged.
+  - Log.highlight(str = nil): returns HIGHLIGHT sequence or wraps string.
+  - Log.uncolor(str) — strips ANSI color sequences.
+
+- Color utilities:
+  - Color class (log/color_class.rb) handles hex parsing, lighten/darken/blend and returns hex strings.
+  - Colorize module (log/color.rb) provides:
+    - Color selection by name (Colorize.from_name),
+    - continuous gradient generation (Colorize.continuous),
+    - gradient/rank mapping and distinct color mapping for categorical values,
+    - TSV coloring helpers.
+
+- Concept color map:
+  - Log.CONCEPT_COLORS (IndiferentHash) maps logical concepts to ANSI colors (e.g. :title => magenta).
+
+---
+
+## Basic Logging API
+
+Main functions:
+
+- Log.log(message = nil, severity = MEDIUM, &block)
+  - Adds newline (if missing) and delegates to Log.logn.
+  - Skips if severity < Log.severity.
+
+- Log.logn(message = nil, severity = MEDIUM, &block)
+  - Emits formatted message without appending newline.
+  - Prefix includes timestamp and severity tag inside color.
+  - Uses Log.color to color the line. Messages with severity >= INFO are wrapped in Log.highlight.
+  - Writes via Log.log_write (synchronized).
+
+- Convenience wrappers:
+  - Log.debug(msg), Log.low(msg), Log.medium(msg), Log.high(msg), Log.info(msg), Log.warn(msg), Log.error(msg)
+    - Each calls Log.log with the corresponding severity.
+
+- Log.exception(e)
+  - Nicely logs an exception: formats message and backtrace using Log.fingerprint for very long messages and Log.color_stack for colored backtrace output. Honors environment `SCOUT_ORIGINAL_STACK` for ordering.
+
+- Log.get_level(level)
+  - Accepts Numeric, String (case-insensitive), or Symbol and returns numeric level or 0/nil.
+
+- Log.with_severity(level) { ... }
+  - Temporarily sets Log.severity for the block.
+
+- Log.log_obj_inspect(obj, level, file = $stdout)
+  - Logs caller location and obj.inspect at given level.
+
+- Log.log_obj_fingerprint(obj, level, file = $stdout)
+  - Logs caller location and Log.fingerprint(obj) for compact summary.
+
+- Line/terminal helpers:
+  - Log.up_lines(n), Log.down_lines(n), Log.return_line, Log.clear_line(out = STDOUT)
+
+---
+
+## Fingerprinting
+
+- Log.fingerprint(obj) produces a compact human-readable representation useful in logs:
+  - Strings are truncated with an MD5 snippet if longer than FP_MAX_STRING (150).
+  - Arrays and Hashes are truncated beyond FP_MAX_ARRAY / FP_MAX_HASH.
+  - Special handling for IO/File, Float formatting, Thread names, Symbol, nil/true/false etc.
+  - Used by other logging helpers to keep outputs concise.
+
+---
+
+## Progress bars
+
+The ProgressBar is a full-featured facility for reporting progress from concurrent tasks.
+
+Key pieces:
+- Use Log::ProgressBar.with_bar(max_or_options, options = {}) {|bar| ... } to create a managed bar.
+  - new_bar(max, options) creates a bar; with_bar ensures removal on exit unless KeepBar is raised.
+- ProgressBar instance attributes:
+  - max, ticks, frequency, depth, desc, file, bytes, process, callback, severity
+- Behavior:
+  - bar.init — initialize and print first state.
+  - bar.tick(step = 1) — increment ticks and possibly report (depending on frequency and percent progress).
+  - bar.pos(position) — set bar to a specific position (pos - ticks).
+  - bar.process(elem) — calls `process` callback and interprets return to tick/pos based on type.
+  - bar.percent — computes percent (0..100).
+  - Bars are managed centrally in ProgressBar::BARS with concurrency via BAR_MUTEX.
+  - Bars can be nested (depth management), silenced, removed, persisted via `file` (save/load YAML state).
+  - ProgressBar.report and ProgressBar.report_msg produce formatted output lines including per-second rate, ETA, used time and ticks.
+
+Helpers:
+- ProgressBar.get_obj_bar(obj, bar) — helper to create a meaningful bar given an object (TSV, File, Array, Path, etc.) — guesses max records by inspecting file/TSV length if possible.
+- ProgressBar.with_obj_bar(obj, bar = true) — convenience wrapper around with_bar using a guessed max.
+
+Notes:
+- Progress printing will skip if Log.no_bar is true (set via environment SCOUT_NO_PROGRESS or Log.no_bar=).
+- ProgressBar persistence: if `file` option provided, the bar saves state to YAML.
+
+---
+
+## Trapping / ignoring STDOUT and STDERR
+
+- Log.trap_std(msg = "STDOUT", msge = "STDERR", severity = 0, severity_err = nil) { ... }
+  - Redirects STDOUT/STDERR into pipes; background threads read and call Log.logn on captured lines with provided severity and prefix.
+  - Useful to capture external command output or to consolidate prints into the log.
+
+- Log.trap_stderr(msg = "STDERR", severity = 0) { ... }
+  - Captures only STDERR and logs it.
+
+- Log.ignore_stderr { ... } / Log.ignore_stdout { ... }
+  - Redirects respective stream to /dev/null for the block (silences output). Safe fallback if /dev/null missing.
+
+These functions restore original streams when the block ends even if exceptions occur.
+
+---
+
+## Convenience debug/inspect helpers
+
+Global helper methods (defined outside Log) for quick debugging:
+
+- ppp(message) — pretty print (with color) and file/line location
+- fff(object) — debug printing fingerprint (using Log.debug)
+- ddd(obj, file = $stdout) — Log.log_obj_inspect(obj, :debug)
+- lll(obj, file = $stdout) — low-level inspect wrapper
+- mmm, iii, wwww, eee — wrappers for different severities (medium, info, warn, error)
+- ddf/mmf/llf/iif/wwwf/eef — wrappers calling log_obj_fingerprint at different severities
+- sss(level) { } — temporarily set severity or set it if no block
+- ccc(obj=nil) — conditional debug printing based on $scout_debug_log (used as ad-hoc toggle)
+
+These are small helpers used in tests (e.g., iif :foo writes INFO lines).
+
+---
+
+## Examples
+
+Basic logging:
+```ruby
+Log.severity = Log::DEBUG
+Log.info "Starting task"
+Log.debug { "Expensive debug only evaluated when level allows" }
+Log.error "Something failed"
+```
+
+Exception handling:
+```ruby
+begin
+  raise "boom"
+rescue => e
+  Log.exception(e)
+end
+```
+
+Progress bar:
+```ruby
+Log::ProgressBar.with_bar(100, desc: "Processing") do |bar|
+  100.times do
+    bar.tick
+    # work...
+  end
+end
+```
+
+Trap STDOUT/STDERR block:
+```ruby
+Log.trap_std("OUT", "ERR", Log::INFO, Log::WARN) do
+  system("some_command")
+end
+```
+
+Set logfile:
+```ruby
+Log.logfile("/tmp/mylog.txt")
+Log.info "Wrote to logfile"
+```
+
+Fingerprint:
+```ruby
+s = "a very long string..."
+Log.debug Log.fingerprint(s)   # compact representation for logs
+```
+
+---
+
+## Implementation notes & caveats
+
+- Colors: if nocolor is enabled (env or Log.nocolor), color helpers return raw strings.
+- Log.log_write and Log.log_puts are synchronized to avoid interleaved writes from threads.
+- Log.logn uses caller and Log.last_caller to attempt to find meaningful source location for messages in stack traces.
+- Fingerprint logic truncates long strings and large arrays/hashes to keep logs readable.
+- ProgressBar uses a central registry (BARS) and a mutex for concurrency; nested bars are supported.
+- The Log module depends on utility modules (Misc, IndiferentHash, TSV, Path, etc.) for some features — when used in isolation, those parts may not be available.
+
+This document summarizes the Log module capabilities and usage. Use Log for all script-level diagnostic output, and use ProgressBar for long running operations where periodic status and ETA are useful.