scout-essentials 1.9.1 → 1.9.3

data/doc/user/Cookbook.md

@@ -1,10 +1,7 @@
 # Cookbook
 
 Short, self-contained recipes that combine the scout-essentials modules.
-Every snippet here was executed; the backing probes are listed under each
-heading and live in `tmp/rewrite_C/` (older probes in `tmp/rewrite_A/`,
-`tmp/rewrite_B/` are referenced by their P-numbers, which map to
-`research/behavior-probes.md`).
+Every snippet here was executed against the current `lib/scout/`.
 
 The theme of the library: plain objects annotated with provenance, paths
 resolved from a declaration, work cached on disk, streams piped without
@@ -31,8 +28,8 @@ produced (a type like `:string`/`:proc`, literal contents, or a block); the
 claimed `Path` is annotated and its `produce` materialises it into the
 resource tree (`lib/scout/resource/path.rb:2`,
 `lib/scout/resource/produce.rb`). `produce_and_find` produces when needed and
-returns `self.find`. Probe: `tmp/rewrite_C/probe_03_cookbook.rb`
-(`claim` + first/second call returning the same path, `Open.read` content).
+returns `self.find` (first and second call return the same path, `Open.read`
+yields the claimed content).
 
 ## Cache invalidation with `:update` and `:check`
 
@@ -51,7 +48,7 @@ d = Persist.persist('dependent', :marshal,
 ```
 
 `:update => true` always re-runs the block; `:check` names a file whose mtime
-invalidates the entry. Probe: probe_03 (a == b, block not re-run; `:update`
+invalidates the entry (a == b, block not re-run; `:update`
 re-runs; `:check` path resolves).
 
 ## Fetch a remote file
@@ -95,7 +92,7 @@ restored.tissue               # => "Liver"
 
 `Annotation.setup(obj, "A|B", hash)` is the module-level deserialiser; the
 `"A|B"` string is split on `|` and each name is looked up as a constant. An
-unknown type name is **warned about and skipped** (probe_03 prints
+unknown type name is **warned about and skipped** (the run prints
 `Annotation NoSuchAnnotation not defined` on STDERR, then
 `Annotation.setup('S004', 'NoSuchAnnotation', ...)` returns the plain
 un-annotated string). `serialize` produces a plain `Hash` with `:literal`,
@@ -138,7 +135,7 @@ rescue Aborted, AbortedStream
 end
 ```
 
-Probe: probe_03 (open_pipe data, tee_stream producing a real gzip file).
+(open_pipe data, tee_stream producing a real gzip file).
 See [Handling Streams](HandlingStreams.md) and the
 [Streaming Model](../developer/StreamingModel.md).
 
@@ -193,7 +190,7 @@ SOPT.require(options, :organism)   # ParameterException when nil
 ```
 
 The `*` marks an option as **taking a string value**; without it the option is
-a boolean. Probe: probe_02 (options hash, `argv_left == ["positional"]`,
+a boolean. (options hash, `argv_left == ["positional"]`,
 `SOPT.require` raising). See
 [Command-Line Options](CommandLineOptions.md).
 

data/doc/user/HandlingStreams.md

@@ -24,7 +24,7 @@ Attributes come from `ConcurrentStream.setup` (concurrent_stream.rb:12-70) and
 `:no_fail` on the returned stream. `exit_status` exists but is *not* reliable
 after a normal read+join — it stays `nil` because only `join_pids` sets it, and
 that method empties `pids` when it runs; see
-[Running Commands](RunningCommands.md) for the probe.
+[Running Commands](RunningCommands.md).
 
 ## `close` vs `join` vs abort
 
@@ -45,7 +45,7 @@ that method empties `pids` when it runs; see
   `Aborted.new` raised in them, so producer bodies should rescue `Aborted`.
 - **`force_close` does not exist** in this repo. The only reference is a dead
   `respond_to?` guard inside `Open.grep` (open/util.rb:26); a plain IO has no
-  such method (probe `tmp/rewrite_B/probe_13_force_close.rb`: `IO#respond_to?
+  such method (`IO#respond_to?
   (:force_close) => false`). The early-close tool is `abort`.
 
 Consumers should follow this rescue contract:
@@ -67,9 +67,8 @@ end
 ## Callbacks
 
 - `stream.add_callback(&block)` **composes**: it wraps the existing callback so
-  the *new* block runs *after* the old one (probe
-  `tmp/rewrite_B/probe_19_streaming_apis.rb`: `add_callback order:
-  [:first, :second]`). `ConcurrentStream.setup(stream, &block)` also composes in
+  the *new* block runs *after* the old one (observed order
+  `[:first, :second]`). `ConcurrentStream.setup(stream, &block)` also composes in
   that order, which is why calling setup twice on the same stream is safe.
 - `callback` / `abort_callback` are plain accessors over single chained procs
   (built by setup when you pass `:callback` / `:abort_callback` options or a
@@ -96,8 +95,9 @@ there; `into_close` (default true) closes `into` when it responds to `close`.
 On `Aborted` or any exception it aborts the source, closes `into`, **removes the
 partial output file** and re-raises. With `in_thread: true` the whole drain runs
 in a new thread that is pushed onto `io.threads`. The block runs after a
-successful drain. Verified: `probe_19_streaming_apis.rb` (`consume_stream
-return: "data"`, `consume_stream into path: "into-file\n"`).
+successful drain. (`consume_stream
+return: "data"`, `consume_stream into path: "into-file\n"`)
+.
 
 ### `Open.sensible_write(path, content, options = {}, &block)`
 
@@ -114,24 +114,20 @@ behaviours for streams:
 - After a successful copy the content stream is joined (but not if it is a
   `Path` or already joined).
 
-Verified: P25/P33 in `research/behavior-probes.md` ("Aborted in
-sensible_write: NOT raised (swallowed); partial tmp left: 0").
-
 ### `Open.open_pipe(do_fork = false, close = true, &block)`
 
 Creates a pipe and returns the **read end** (`sout`), with the block executed in
 a producer thread that writes the other end (`sin`).
 
 - **Block arity**: the block always receives `sin` whether it declares a
-  parameter or not (arity 0 blocks simply ignore it). Verified:
-  `probe_20_open_pipe_arity.rb` (`arity-0 block: "arity0\n"`, `arity-1: "w\n"`).
+  parameter or not (arity-0 blocks simply ignore it; `arity-0 block: "arity0\n"`, `arity-1: "w\n"`).
 - **No block** raises `RuntimeError "No block given"`.
 - **Fork mode** (`do_fork: true`) runs the block in a child process instead of
   a thread: the child purges registered input pipes, closes `sout`, yields,
   `exit! 0`; the parent closes `sin` and sets the pid on `sout` via
   `ConcurrentStream.setup(sout, :pids => [pid])` — no threads, no callbacks.
   `close: false` in the child leaves `sin` open after the block returns
-  (verified: `fork mode: "from-fork\n"`, `fork noclose: "fork-noclose\n"`).
+  (same results in fork mode with and without `close`).
 - **Thread mode** pairs the two ends (`pair`), runs the block through
   `ConcurrentStream.process_stream` (close+join on exit, abort on error), and
   registers the thread on both ends. An exception in the block aborts the
@@ -141,8 +137,7 @@ a producer thread that writes the other end (`sin`).
 
 Takes **no arguments** and returns the raw `[sout, sin]` pair from `IO.pipe`
 (plus registering `sin` in `OPEN_PIPE_IN`). Calling it with a positional
-argument raises `ArgumentError` (probe `probe_19_streaming_apis.rb`:
-`Open.pipe positional: ArgumentError`). There is no multi-command helper here —
+argument raises `ArgumentError`. There is no multi-command helper here —
 chain commands by feeding one stream into `:in` of the next `CMD.cmd`.
 
 ### `Open.tee_stream(stream)` / `tee_stream_thread_multiple(stream, num)`
@@ -152,8 +147,9 @@ Returns an **Array** of streams (`num` copies, default 2): the first is the
 reads the source once and writes every chunk to all copies; the main copy's
 callback joins the source and closes the extra write ends, and its
 `abort_callback` propagates an abort to the source and the other copies.
-Verified: P31/P33 and `probe_19_streaming_apis.rb` (`tee_stream count: 2`,
-`tee[0].autojoin => true`, `tee[1].autojoin => nil`).
+(`tee_stream count: 2`,
+`tee[0].autojoin => true`, `tee[1].autojoin => nil`)
+.
 
 ```ruby
 main, copy = Open.tee_stream(CMD.cmd('gzip -c', :pipe => true, :in => input))
@@ -167,14 +163,13 @@ Builds a tee, then a monitor thread reads the monitor copy line by line calling
 consumes the returned stream, not after. Failures in the block abort the monitor
 and are re-raised into the returned stream (`out.raise $!` when supported). The
 returned stream is the second copy, annotated from the source and set up with
-the monitor thread. Verified: P33 and `probe_19_streaming_apis.rb`.
+the monitor thread. 
 
 ### `Open.read_stream(stream, size)`
 
 Blocking read of exactly `size` bytes (plain `stream.read(missing)` loop,
 `lib/scout/open/stream.rb:401`), raising `ClosedStream` if EOF is reached
-first. Useful for binary framing; `probe_19_streaming_apis.rb` shows
-`read_stream(4) => "0123"`.
+first; useful for binary framing: `read_stream(4) => "0123"`.
 
 ### `Open.sort_stream(stream, header_hash: '#', cmd_args: nil, memory: false)`
 
@@ -183,8 +178,7 @@ Streams `header_hash`-prefixed lines straight through, then sorts the rest.
 <cmd_args>` (`-u` by default when `cmd_args` is nil) and consumes that stream
 into the output; `memory: true` reads the whole remainder, sorts it in Ruby and
 writes it out. Everything runs inside `ConcurrentStream.process_stream`, so the
-source is closed+joined and aborted on error. Verified:
-probe_19_streaming_apis.rb: feeding `"# header\nc\na\nb\n"` returns
+source is closed+joined and aborted on error. feeding `"# header\nc\na\nb\n"` returns
 `"# header\na\nb\nc\n"` — the header passes through untouched, the rest is
 sorted.
 
@@ -193,7 +187,7 @@ sorted.
 Merges consecutive lines sharing the same first field, joining the other
 columns with `|` (or dropping empty parts when `compact: true`). An optional
 block receives the accumulated column array and its return value becomes the
-row payload. Verified: `probe_19_streaming_apis.rb`.
+row payload. 
 
 ## `Open.open` block form and `DontClose`
 
@@ -207,16 +201,15 @@ end
 `Open.open` yields and **always closes and joins** the IO afterwards (the
 `ensure` at open.rb:70-77). Raising `DontClose` with a payload makes the block
 form *return* the payload instead of the IO, while still closing — it is an
-early-return mechanism, not a way to keep the handle. Verified:
-`probe_21_dontclose.rb` (`DontClose returns payload: "payload"`, `closed after
-DontClose: true`). Any other exception aborts, joins and re-raises the stream.
+early-return mechanism, not a way to keep the handle (`DontClose returns
+payload: "payload"`, `closed after DontClose: true`). Any other exception
+aborts, joins and re-raises the stream.
 
 ## Progress bars
 
 Pass `:progress_bar` (a `Log::ProgressBar`, `lib/scout/log/progress.rb`; the option key is `:progress_bar` — there is no `:bar` key)
-to `CMD.cmd` and each stderr line ticks it (`bar.process(line)` at cmd.rb:643):
-`probe_24_bar.rb` counts 2 ticks for a 2-line stderr, in both pipe and
-non-pipe mode. With `:log => true` (`CMD.cmd_log`) the stderr text is also
+to `CMD.cmd` and each stderr line ticks it (`bar.process(line)` at cmd.rb:643): 2 ticks for a 2-line stderr, in both
+pipe and non-pipe mode. With `:log => true` (`CMD.cmd_log`) the stderr text is also
 recorded in `stream.log`. `Log::ProgressBar` itself supports `:process =>
 proc{|elem| elem.length}` so a tick can be weighted per element.
 

data/doc/user/LoggingAndProgress.md

@@ -19,8 +19,6 @@ unrecognised value — falls back to `Log.default_severity`, which reads
 `~/.scout/etc/log_severity` if that file exists and is `INFO` otherwise
 (log.rb:23-34).
 
-Verified by `tmp/rewrite_C/probe_04_log_severity.rb` (out04.txt): ladder
-order, `Log.severity == 4` with no env/file, `Log.with_severity(level){}`
 restoring the original value after the block.
 
 Change it at runtime with `Log.severity = Log::DEBUG`, or scope a block with
@@ -38,7 +36,7 @@ Change it at runtime with `Log.severity = Log::DEBUG`, or scope a block with
 - `Log::LAST` is one shared mutable `String` (starts as `"log"`, never frozen)
   used as a protocol between `logn` and progress-bar printing so bars know how
   many lines to move up and which kind of output came last
-  (`progress/report.rb` `print`/`report`). Probe out04.txt confirms
+  (`progress/report.rb` `print`/`report`).
   `class == String`, `frozen? == false`.
 
 ## Message forms
@@ -46,8 +44,8 @@ Change it at runtime with `Log.severity = Log::DEBUG`, or scope a block with
 - `Log.debug "msg"` .. `Log.error "msg"` print when severity allows; the top
   of the ladder is the `Log::NONE` constant, not a method.
 - `Log.debug { "expensive " + build }` — **the block is lazy message
-  evaluation, not a timer**: it is only called if the level passes. Probe
-  (out05.txt): a counter incremented inside a block passed to `Log.debug`
+  evaluation, not a timer**: it is only called if the level passes. A
+  counter incremented inside a block passed to `Log.debug`
   stays at 0 under ERROR severity and reaches 1 under DEBUG severity.
 - `Log.exception(e)` (log.rb:254) prints `BACKTRACE` lines derived from
   `e.backtrace` after fingerprinting messages longer than 1000 chars; it
@@ -55,7 +53,7 @@ Change it at runtime with `Log.severity = Log::DEBUG`, or scope a block with
   it contains `NOSTACK`.
 - `Log.stack(stack)` (log.rb:318) prints a magenta header then the stack
   **reversed** (innermost frame first) unless `SCOUT_ORIGINAL_STACK=true`.
-  Probe (out05.txt): a caller list `a.rb:1, b.rb:2, c.rb:3` prints as
+  A caller list `a.rb:1, b.rb:2, c.rb:3` prints as
   `c.rb:3`, `b.rb:2`, `a.rb:1`.
 
 ## Top-level debug helpers (defined on Object)
@@ -66,7 +64,7 @@ DEBUG. `ddd/lll/mmm/iii/wwww/eee` inspect an object at the matching severity,
 and the `f`-suffixed variants (`ddf`, `llf`, `mmf`, `iif`, `wwwf`, `eef`)
 fingerprint it first. `sss(level[, &blk])` sets or scopes severity. `ccc`
 (log.rb:439) enables `$scout_debug_log` around a block so nested `ccc` calls
-print. Probe out05.txt confirms all five are top-level methods defined inside
+print. All five are top-level methods defined inside
 `lib/scout/log.rb`.
 
 ## Colors
@@ -90,7 +88,7 @@ show first/middle/last elements; hashes longer than 10 fall back to
 keys+values fingerprints; floats get 1/3/6 decimals depending on magnitude.
 
 The signature is `fingerprint(obj)` — **one positional argument**; there are no
-`max_length`/`sep` options (probe_05). Hash rendering separates pairs with a
+`max_length`/`sep` options . Hash rendering separates pairs with a
 space, not a comma: `Log.fingerprint({a: 1, b: 2, c: 3})` gives
 `"{:a=>1 :b=>2 :c=>3}"`.
 
@@ -130,13 +128,13 @@ bytes/process/callback/severity` accessors.
   movement. `report_msg` shows **elapsed time and rate** and, when `max` is
   set, an ETA: `· <rate> per sec. -- <dots> <pct>% <eta> => <elapsed> -
   <ticks> of <max> items · <desc>`; with no `max` the ETA part is replaced
-  by `<ticks> items` (probe_12 shows both forms).
+  by `<ticks> items`; both forms occur.
 - `add_offset` / `remove_offset` / `offset` (util.rb:9-28) indent bars created
   in nested threads.
 - `file:` option: `save` writes the bar state to YAML (`:desc, :last_count,
   :last_percent, :last_time, :max, :start, :ticks`) and `done`/`error` remove
   the file, so a bar given a `file:` **resumes across runs** — the persisted
-  YAML is reloaded into `ticks` (probe_06 round-trips 3 ticks).
+  YAML is reloaded into `ticks` (round-trips 3 ticks).
 
 All bar bookkeeping is guarded by `BAR_MUTEX`; per-bar `tick` state is not
 synchronized and races by design.
@@ -147,7 +145,7 @@ synchronized and races by design.
 `Hash`. In a plain `require 'scout-essentials'` process neither `TSV` nor
 `Step` exists, so the `when TSV` arm raises `NameError`, the rescue turns it
 into `nil`, and **`guess_obj_max` always returns `nil`** — i.e.
-`get_obj_bar(list, true).max == nil` (probe_06). Pass an explicit Numeric or a
+`get_obj_bar(list, true).max == nil` . Pass an explicit Numeric or a
 `:max` hash, or define those constants in the embedding repo, to get a real
 max.
 

data/doc/user/ProducingResources.md

@@ -34,7 +34,6 @@ module MyApp
 end
 ```
 
-
 - `claim(path, type, content = nil, &block)` — `type` is a **mandatory
   positional** argument (`claim path { }` raises `ArgumentError`). Passing
   only the path plus a block is not a valid call.

data/doc/user/RemoteData.md

@@ -5,9 +5,8 @@ Fetching remote files over HTTP(S)/FTP/SSH, the URL cache that backs
 Source: `lib/scout/open/remote.rb`, `lib/scout/open/sync.rb`,
 `lib/scout/resource/sync.rb`.
 
-Executed examples come from `tmp/rewrite_D/probe_07_remote.rb`, which
-starts a throwaway local `python3 -m http.server` to serve real URLs;
-transcripts live in `tmp/rewrite_D/BATCH_D_PROBE_TRANSCRIPTS.txt`.
+Executed examples run against a throwaway local `python3 -m http.server`
+serving real URLs.
 
 For the local side of `Open` see [WorkingWithFiles.md](WorkingWithFiles.md);
 for `Resource` itself see [ProducingResources.md](ProducingResources.md).
@@ -25,8 +24,7 @@ Open.ssh?('http://a')      # => false
 ```
 
 Both are pure regexes (`/^(?:https?|ftp|ssh):\/\//` and `/^ssh:\/\//`). A
-path without a scheme is never remote. Verified by
-`tmp/rewrite_D/probe_07_remote.rb`.
+path without a scheme is never remote.
 
 ## `Open.ssh` — reading over SSH
 
@@ -46,20 +44,19 @@ behaviour is to serve from that cache without touching the network.
 
 ```ruby
 Open.remote_cache_dir            # $HOME/.scout/var/cache/open-remote
-Open.remote_cache_dir = dir      # override (probe sets it to a tmpdir)
+Open.remote_cache_dir = dir      # override (e.g. a tmpdir)
 Open.cache_file(url, options)    # <cache_dir>/<digest>
 ```
 
 The digest (`Open.digest_url`) covers the URL, the `--post-data` value and
 the **sorted lines** of `--post-file`. Different post-data therefore gets a
-different cache entry (probe: three URLs/options => three distinct cache
-paths).
+different cache entry (three URLs/options => three distinct cache paths).
 
 ### No TTL, no expiry
 
 There is no timestamp check and no maximum age. Once a URL is cached, a
 later `Open.wget` of the same URL/options returns the cached bytes even if
-the server has changed. The probe demonstrates it: the file was edited on
+the server has changed: the file can be edited on
 the server three times (`HELLO` → `FIRST` → `SECOND` → `THIRD`) while the
 second `Open.wget` still reported the old cached content.
 
@@ -72,7 +69,7 @@ Open.remove_from_cache(url, options)  # delete the cache entry
 ```
 
 **Exactly `'update'`** re-fetches and re-writes the cache, then *reads it
-back* (probe: after `Open.wget(url, :nocache => 'update')` the returned file
+back*: after `Open.wget(url, :nocache => 'update')` the returned file
 is the cache file again, not the raw pipe). Any other truthy `nocache`
 value takes the other branch and returns the raw wget pipe **without**
 touching the cache. `:force` only bypasses the initial `in_cache?` lookup,
@@ -90,7 +87,7 @@ wrapped in `OpenURLError` with the message
 
 `Open.download(url, file)` runs `wget '<url>' -O '<file>'` via
 `CMD.cmd_log`; on failure it removes the partial output file and re-raises
-the original error. Verified: downloading a live URL produced the current
+the original error: downloading a live URL writes the current
 server bytes (no cache involved).
 
 `Open.scp(source_file, target_file, target:, source:)` first creates the
@@ -123,7 +120,7 @@ directory itself. A local-to-local sync of the identical path logs a warning
 and returns.
 
 `Open.sync(...)` is a literal alias: `Open.sync` and `Open.rsync` are the
-same implementation forwarded with Ruby's `...` (probe: same owner). Use
+same implementation forwarded with Ruby's `...` (same owner). Use
 either name — they are interchangeable.
 
 ## `Resource.sync(path, map, options)`
@@ -143,7 +140,7 @@ Open.wait(lag, key = nil)
 
 A tiny rate limiter backed by the module-level `LAST_TIME` hash. If the
 last call *for that key* was less than `lag` seconds ago, it sleeps the
-remainder. `Open.wget` uses it via `:nice` / `:nice_key`. Probe: two
+remainder. `Open.wget` uses it via `:nice` / `:nice_key`. Two
 `Open.wait(0.3, :k)` calls in a row took 0.3 s wall clock.
 
 Note the process-local `LAST_TIME` map is shared mutable state with no

data/doc/user/RunningCommands.md

@@ -26,7 +26,6 @@ CMD.cmd('echo', 'arg2', '-n' => true, '-r' => true).read
 - **Hash-only** (`CMD.cmd({'echo' => 'x'})`) is *not* a command form; the Hash is
   treated as options, `cmd`/`tool` stay nil and the call fails.
 
-Verified: probe `tmp/rewrite_B/probe_15_cmd_forms.rb` (string, string+cmd,
 array, array+string, symbol tool with nil cmd, ProcessFailed on hash-only).
 
 ### The `'{opt}'` placeholder
@@ -44,8 +43,6 @@ CMD.cmd("echo '{opt}' a 1", 'x' => 1).read # => "a 1\n"  (note: trailing space)
 
 Unquoted `{opt}` is left untouched — the substitution matches `'\{opt\}'`
 exactly. Without a placeholder the option string is appended to the command.
-Verified: probe `probe_14_cmd_opts.rb` lines 10-11 and the C8 fix in
-`research/doc_audit/RunningCommands.md`.
 
 ## Option quoting: `process_cmd_options`
 
@@ -71,10 +68,10 @@ raises `Invalid option key` before anything runs.
 **Arrays are not expanded.** An Array value is stringified (`to_s`) and quoted
 like any other value — `"#{value}"` produces `n '["a", "b"]'` in String mode
 and `['n', '["a", "b"]']` in array mode. Pass separate calls or build the
-command yourself if you need repeated flags. Verified: `probe_14_cmd_opts.rb`
+command yourself if you need repeated flags.
 lines 4-5 and the cmd-level Array check.
 
-Verified examples: `probe_14_cmd_opts.rb` (dashes, `=`, nil/false, true,
+(dashes, `=`, nil/false, true,
 apostrophes, invalid key, arrays).
 
 ## stderr: severities, capture and files
@@ -83,7 +80,7 @@ apostrophes, invalid key, arrays).
 (added by `cmd.rb:181`) is **`Log::DEBUG`** — stderr lines are only logged at
 debug severity, i.e. invisible unless you raise `Log.severity` above it.
 
-- `:stderr => true` is normalised to `Log::HIGH` (probe `probe_23_bar_log.rb`).
+- :stderr => true is normalised to `Log::HIGH`.
 - Any other Integer is a `Log` severity: the ladder is
   `DEBUG=0, LOW=1, MEDIUM=2, HIGH=3, INFO=4, WARN=5, ERROR=6, NONE=7`
   (`lib/scout/log.rb` `SEVERITY_NAMES`), so `:stderr => Log::MEDIUM` shows
@@ -118,9 +115,7 @@ CMD.cmd(..., :save_stderr => dst)                     # dst stays open
 Implementation: `cmd.rb:194-217` (destination setup), the writer thread /
 inline writer, and the `ensure` at `cmd.rb:600-612` that flushes and closes a
 CMD-owned file. Verified by `test/scout/test_cmd_save_stderr.rb` (13 tests,
-incl. live `tail -f` polling) and probes `probe_17_save_stderr.rb`
-(boolean/String/Path/Pathname/IO/StringIO, pipe and non-pipe, truncation) and
-`probe_16_exitstatus.rb` (`std_err` populated in both modes).
+incl. live `tail -f` polling).
 
 ## Exit status, `no_fail` and failure
 
@@ -129,19 +124,19 @@ incl. live `tail -f` polling) and probes `probe_17_save_stderr.rb`
   `:nofail`) is given.
 - `:no_fail => true` **suppresses** `ProcessFailed`/`ConcurrentStreamProcessFailed`
   — and `exit_status` then stays `nil`, in pipe mode too. If you need the code,
-  call `join_pids` yourself. Verified: `probe_16_exitstatus.rb`
+  call `join_pids` yourself.
   (`pipe read+join exit_status: nil`, `explicit join_pids exit_status: 0`,
   `non-pipe exit_status: 0`).
 - `exit_status` is only ever set by `ConcurrentStream#join_pids`
   (`concurrent_stream.rb:127`), which also empties `pids`, so it can only be
   used once. A stream that is read and joined normally has `exit_status == nil`
-  (probe `probe_26_join_es.rb`: `read+join: es=nil` with `joined?` true; only
+  (`read+join: es=nil` with `joined?` true; only
   an explicit early `join_pids` yields `0`). Do not rely on `stream.exit_status`.
 - If the process never starts (bad executable, no such file) `ProcessFailed` is
   raised immediately — also suppressed by `no_fail`, which then returns `nil`.
 - A failed *producer thread* in pipe mode surfaces as
   `ConcurrentStreamProcessFailed` when the consumer closes/joins the stream
-  (`probe_13_force_close.rb`).
+ 
 
 ## Timeout
 
@@ -182,7 +177,8 @@ returns only entries matching `/\d+\./`.
 
 | key | effect |
 |---|---|
-| `:in` | stdin: a String is written by a thread; an IO/StringIO is read; a ConcurrentStream is streamed (and closed unless `:dont_close_in`). Also `:in_pipe` for a pipe-backed writer. Verified: `probe_25_keeping_in.rb` (String, IO, stream, `in_pipe` returns an IO). |
+| `:in` | stdin: a String is written by a thread; an IO/StringIO is read; a ConcurrentStream is streamed (and closed unless `:dont_close_in`). Also `:in_pipe` for a pipe-backed writer. (String, IO, stream, `in_pipe` returns an IO)
+. |
 | `:pipe` | return a ConcurrentStream instead of the text/StringIO |
 | `:stderr` | severity for stderr logging (default `Log::DEBUG`); `true` → `Log::HIGH` |
 | `:save_stderr` | `true` / path / IO — see above |
@@ -191,7 +187,7 @@ returns only entries matching `/\d+\./`.
 | `:no_wait` | alias used to default `autojoin` (`autojoin = no_wait if autojoin.nil?`) |
 | `:timeout` | seconds; watchdog; only runtime bound |
 | `:post` | proc run after the command/stream finishes (teardown, forcing upstream closes) |
-| `:progress_bar` | a `Log::ProgressBar`; stderr lines tick it (`probe_24_bar.rb`: 2 ticks in both pipe and non-pipe mode) |
+| `:progress_bar` | a `Log::ProgressBar`; stderr lines tick it (2 ticks for a 2-line stderr, pipe and non-pipe) |
 | `:log` | defaults to `true` (`log = true if log.nil?`, cmd.rb:224): pipe-mode stderr lines are `Log.log`ged at the chosen `:stderr` severity. `:log => false` silences that logging. Not the same as `CMD.cmd_log` (a separate helper that forces `:pipe`/`:log`). |
 | `:sudo`, `:xvfb` | prefix the command |
 | `:dont_close_in` | keep the `:in` stream open |
@@ -201,13 +197,12 @@ returns only entries matching `/\d+\./`.
 `CMD.cmd` in this repo — `cmd.rb` deletes none of them, so they would be
 forwarded to `process_cmd_options` and end up as command-line text. `:canfail`
 exists on `Persist` (`persist.rb:133`) and on `Resource` claims, not here.
-Verified by grepping `lib/` for the four names (`probe_25_keeping_in.rb` tail).
+Verified by grepping `lib/` for the four names.
 
 ## The block is ignored
 
 `CMD.cmd(...) { ... }` accepts a block but **never calls it** — the only block
-invocation inside `cmd.rb` is in `CMD.tool`. Verified live by
-`tmp/docaudit/probe_hs_c15_block.rb`: in both pipe and non-pipe mode the block
+invocation inside `cmd.rb` is in `CMD.tool`. In both pipe and non-pipe mode the block
 body never runs, while `:post` and `stream.add_callback` do. Use
 `:post => proc{}` or stream callbacks for post-join work.
 

data/doc/user/WorkingWithFiles.md

@@ -48,12 +48,20 @@ Open.read('file.gz.bak')# raw gzip bytes — NOT detected
 Open.gzip?('file.tgz')  # => false
 ```
 
-Force decompression explicitly when the name does not cooperate:
+To decompress content that does not carry a recognised name, run
+`Open.gunzip` yourself — it is `zcat` behind a `CMD.cmd` pipe:
 
 ```ruby
-Open.open('file.tgz', :gzip => true).read
+Open.read('file.tgz', :gzip => false, :noz => true)  # raw bytes, if that is what you want
+stream = Open.open('file.tgz', :noz => true)         # raw gzip bytes as a stream
+Open.read(Open.gunzip(stream))                       # decompressed
 ```
 
+Note that the historical `:gzip => true` option is **broken as a
+decompression force**: it is forwarded to `CMD.cmd('zcat', …)` as a
+command-line flag, the command fails, and — because `:no_fail` is set —
+you get an empty string instead of an error.
+
 ## Grepping lines
 
 `:grep` and `:invert_grep` select lines using the system `grep`. **With a

data/lib/scout/config.rb

@@ -129,6 +129,7 @@ module Scout::Config
       end
     end
 
+
     value = priorities.empty? ? default : priorities.collect{|p| p }.sort_by{|p,v| p}.first.last.first
     value = false if value == 'false'
 

data/lib/scout/indiferent_hash.rb

@@ -79,10 +79,16 @@ module IndiferentHash
     case key
     when Symbol, Module
       v = super(key) 
-      v.nil? ? super(key.to_s) : v
+      vn = super(key.to_s) 
+      v.nil? ? vn : v
     when String
       v = super(key)
-      v.nil? ? super(key.to_sym) : v
+      vn = begin
+             super(key.to_sym) 
+           rescue
+             nil
+           end
+      v.nil? ? vn : v
     else
       super(key)
     end

data/lib/scout/log/fingerprint.rb

@@ -1,7 +1,7 @@
 require 'digest/md5'
 module Log
   FP_MAX_STRING = 150
-  FP_MAX_ARRAY = 20
+  FP_MAX_ARRAY = 25
   FP_MAX_HASH = 10
 
   def self.truncate_string(string, max=FP_MAX_STRING)

data/lib/scout/named_array.rb

@@ -142,6 +142,10 @@ module NamedArray
       new
     end
   end
+  
+  def zip_fields
+    NamedArray.zip_fields(self).collect{|v| self.annotate(v) }
+  end
 
   def self.add_zipped(source, new)
     source.zip(new).each do |s,n|

data/lib/scout/open/final.rb

@@ -147,7 +147,7 @@ module Open
         info_file = file + '.info'
         if File.exist?(info_file) && defined?(Step)
           done = Persist.memory([info_file, File.mtime(info_file)], persist: true ) do
-            Persist.load(info_file, Step::SERIALIZER)[:done]
+            Step.load_info(info_file)[:done]
           end
           return done if done
         end

data/lib/scout/open/stream.rb

@@ -16,14 +16,14 @@ module Open
     end
   end
 
-  def self.consume_stream(io, in_thread = false, into = nil, into_close = true, &block)
+  def self.consume_stream(io, in_thread = false, into = nil, into_close = true, clean = true, &block)
     return if Path === io
     return unless io.respond_to? :read
 
-    if io.respond_to? :closed? and io.closed?
-      io.join if io.respond_to? :join
-      return
-    end
+    #if io.respond_to? :closed? and io.closed?
+    #  io.join if io.respond_to? :join
+    #  return
+    #end
 
     if in_thread
       consumer_thread = Thread.new(Thread.current) do |parent|
@@ -54,10 +54,13 @@ module Open
 
         into_close = false unless into.respond_to? :close
 
-        while c = io.read(BLOCK_SIZE)
-          into << c if into
-          last_c = c if c
-          break if io.closed?
+        begin
+          while c = io.readpartial(BLOCK_SIZE)
+            into << c if into
+            last_c = c if c
+            break if io.closed?
+          end
+        rescue EOFError, IOError
         end
 
         io.join if io.respond_to? :join
@@ -72,7 +75,7 @@ module Open
         Log.low "Consume stream Aborted #{Log.fingerprint io} into #{into_path || into}"
         io.abort $! if io.respond_to? :abort
         into.close if into.respond_to?(:closed?) && ! into.closed?
-        FileUtils.rm into_path if into_path and File.exist?(into_path)
+        FileUtils.rm into_path if into_path and clean and File.exist?(into_path)
       rescue Exception
         Thread.current["exception"] = true
         Log.low "Consume stream Exception reading #{Log.fingerprint io} into #{into_path || into} - #{$!.message}"
@@ -80,9 +83,7 @@ module Open
         io.abort exception if io.respond_to? :abort
         into.close if into.respond_to?(:closed?) && ! into.closed?
         into_path = into if into_path.nil? && String === into
-        if into_path and File.exist?(into_path)
-          FileUtils.rm into_path
-        end
+        FileUtils.rm into_path if into_path and clean and File.exist?(into_path)
         raise exception
       end
     end

data/lib/scout/path/util.rb

@@ -42,7 +42,7 @@ module Path
 
       post_fix = "--#{filename.length}--#{Misc.digest(filename)[0..4]}" + extension
 
-      filename = filename[0..(length - post_fix.length - 1)] << post_fix
+      filename = Annotation.purge(filename.to_s)[0..(length - post_fix.length - 1)] << post_fix
     else
       filename
     end