scout-essentials 1.8.8 → 1.9.1

data/doc/developer/CoreUtilities.md

@@ -0,0 +1,265 @@
+# Core Utilities
+
+The small modules everything else in the stack is built on:
+`IndiferentHash`, `NamedArray`, the `Misc.format` family, `TmpFile` and
+`Hook`. Each is tiny, each is load-bearing. Sources:
+`lib/scout/indiferent_hash.rb` + `lib/scout/indiferent_hash/{options,serialize,case_insensitive}.rb`,
+`lib/scout/named_array.rb`, `lib/scout/misc/format.rb`,
+`lib/scout/tmpfile.rb`, `lib/scout/misc/hook.rb`.
+
+## IndiferentHash
+
+`IndiferentHash.setup(hash)` does not create a class; it extends a plain
+`Hash` with the module, so an IndiferentHash IS an Array/Hash of its base
+type and every `Hash` operation still works. Reads and writes are
+indifferent between String and Symbol keys.
+
+Verified by `tmp/rewrite_D/probe_06_indiferent_misc.rb`:
+
+```ruby
+h = IndiferentHash.setup({ 'a' => 1 })
+h[:b] = 2
+h[:a]      # => 1
+h['b']     # => 2
+h.keys     # => ["a", "b"]      (first-written form is kept)
+```
+
+- **`slice`** — `{'a'=>1,'b'=>2}.slice(:a)` => `{"a"=>1}`
+- **`merge` keeps existing keys, and keeps the FIRST key's form**:
+  `{'a'=>1,'b'=>2}.merge('a'=>9,'c'=>3).keys` => `["b","a","c"]`; with a
+  duplicate `{'k1'=>1, :k1=>2}` both keys survive and `[:k1]` returns `2`.
+- **`clean_version`** — a fresh copy without the module's bookkeeping:
+  `{'a'=>1,'b'=>2}`.
+- **`dig`** — module-level `IndiferentHash.dig(h, :x, :y)` works
+  symbol-first on a String-keyed hash => `1`.
+- `add_defaults(options, defaults)` only fills missing keys:
+  `add_defaults({:a=>1}, {:b=>2})` => `{:a=>1, :b=>2}`.
+
+### The destructive pair
+
+`process_options(hash, *keys)` **destroys** the keys it extracts — it
+`delete`s them (verified: `process_options(opts, :a, :b)` leaves
+`{:extra=>3}`). `pull_keys(hash, :prefix)` returns the sub-hash under that
+prefix key AND removes it from the original
+(`pull_keys({persist: true, x: 1}, :persist)` => `{:persist=>true}`, hash
+left with `{:x=>1}`).
+
+### `string2hash` / `parse_options`
+
+`string2hash(string, sep="#")` splits on `sep`, then per pair applies, in
+order: bare key => `true`; `:sym` values; `/regex/`; quoted strings;
+`Integer`; `Float`; `"true"`; else the raw String
+(`lib/scout/indiferent_hash/options.rb:96-114`). Note `"false"` is *not*
+in the coercion list — see the asymmetry below.
+
+`parse_options(str)` scans `key=value` pairs on whitespace
+(`/\w+=(\"[^\"]*\"|[^\s\"]+)/`), strips surrounding quotes, and splits a
+value containing a comma into an Array. Its coercion ladder is shorter than
+`string2hash`'s: `true`/`false` are **not** special-cased at all, so both
+survive as Strings (verified `tmp/rewrite_D/probe_20_parse_options_false.rb`:
+`parse_options('a=false')` => `{"a"=>"false"}`,
+`parse_options('a=1,b=2')` => `{"a"=>["1", "b=2"]}` — the comma split eats
+the second pair — and `parse_options('a="x y"')` => `{"a"=>"x y"}`).
+`print_options(options)` is the inverse, re-quoting values with spaces.
+
+**Known asymmetry (bug candidate).** In `string2hash`, the `== "false"`
+branch (`options.rb:110`) is dead code for the plain case: `a=false` goes
+through `options[key] = value` and the String survives. The only way the
+branch is reached is when `false` is the *value* of a pair that the default
+`#` separator has already split off — and the executed result is still
+`"false"` in every probed form (`tmp/rewrite_D/probe_21_s2h_false_traced.rb`).
+`parse_options` has no boolean branches at all. `Scout::Config.get`, by
+contrast, turns a stored `'false'` into the boolean `false` (see
+[Configuration.md](Configuration.md)). Verified by
+`tmp/rewrite_D/probe_19_false_coercion.rb` and `probe_20_parse_options_false.rb`:
+
+```text
+string2hash('a=true')            => {"a"=>true}
+string2hash('a=false')           => {"a"=>"false"}
+string2hash('a=false#b=1')       => {"a"=>"false", "b"=>1}
+parse_options('a=true')          => {"a"=>true}
+parse_options('a=false')         => {"a"=>"false"}
+Scout::Config.get('fc')          => false     (set from the String 'false')
+```
+
+So a boolean-looking option that arrives through a string carries no
+guarantee of being a boolean: check the specific path, and treat values
+coming out of `parse_options` as Strings unless you coerce them yourself.
+(`string2hash` also expects `#`-separated pairs; `'a=false b=1'` is a
+single pair because the separator never appears — the value is the literal
+`"false b=1"`.)
+
+### `serializable`
+
+`IndiferentHash.serializable(obj)` (`serialize.rb`) returns a deep copy of
+Hash/Array structures. Arrays longer than 100 are truncated: first 70 +
+`'...'` + last 30 + the marker `"TRUNCATED only 100 out of N shown"`
+(probe_06: 250-element array => 102 elements, marker last). Use it for
+logging fingerprints, not for round-tripping data.
+
+### CaseInsensitiveHash
+
+`lib/scout/indiferent_hash/case_insensitive.rb` — **not auto-loaded**;
+`require 'scout/indiferent_hash/case_insensitive'` first. Only `[]` and
+`values_at` are overridden; writes are plain `Hash` writes. Reads are
+downcase-mapped against the *first* case seen for each key, so a Symbol
+key can never be found by a String lookup (Symbol has no meaningful
+downcase mapping in `downcase_keys`). Verified by
+`tmp/rewrite_D/probe_24_cihash_writes.rb`:
+
+```text
+ci = CaseInsensitiveHash.setup({"Key" => 1})
+ci["Other"] = 5   keys => ["Key", "Other"]
+ci["other"], ci["OTHER"]   => 5, 5          (case-insensitive read)
+ci[:third]  = 7   keys => ["Key", "Other", :third]
+ci["third"]     => nil                     (String lookup misses a Symbol key)
+ci[:third]      => 7
+```
+
+Treat it as a read-side convenience for String-keyed hashes; it is not an
+indifferent-access container.
+
+## NamedArray
+
+`require 'scout/named_array'` (also **not** auto-required by the gem root).
+`NamedArray.setup(array, fields, key)` extends an ordinary Array — it is
+the `Annotation` module applied to Arrays, **not** a String type, and it
+does **not** extend `AnnotatedArray` (that is for annotated Strings; see
+[AnnotationSystem.md](AnnotationSystem.md)).
+
+```ruby
+na = NamedArray.setup([1,2,3], [:first, :second, :third], 'mykey')
+na.class                 # => Array
+na.first                 # => 1   (Array#first shadows nothing here)
+na[:first]               # => 1
+na['first']              # => 1
+na.second                # => 2   (method_missing field accessor)
+na.respond_to?(:second)  # => false  (fields are not real methods)
+na.zip([4,5,6])          # => [[1,4],[2,5],[3,6]]  (Array#zip wins)
+na.count                 # => 3
+```
+
+So `count`/`first`/`last`/`zip` are Array's, and any field that happens to
+be named like an Array method is unreachable through the accessor. The
+`key` is exposed via `all_fields` (`[key, fields].compact.flatten`).
+
+## Misc.format family
+
+All from `lib/scout/misc/format.rb`, verified by probe_06/probe_15:
+
+| call | result |
+|---|---|
+| `Misc.snake_case('FooBar')` | `"foo_bar"` |
+| `Misc.camel_case('foo_bar')` | `"FooBar"` |
+| `Misc.humanize('foo_bar')` | `"Foo bar"` |
+| `Misc.human_number(1234567)` | `"1.2M"` |
+| `Misc.format_paragraph(text, 30)` | re-wraps to the given width |
+| `Misc.format_definition_list([[dt, dd]])` | aligned dt/dd block |
+
+### `Misc.timespan` takes ONE unit per token
+
+`timespan(str, default="s")` (`lib/scout/misc/format.rb:279`) supports the
+unit tokens `s sec m min '' ' h d w mo y`, `HH:MM[:SS]` clock strings, and a
+leading `-` for negatives. The parser is `str.scan(/(\d+)(\w*)/)` and `\w*`
+is greedy, so **each number may be followed by only one unit token**.
+`"1h30m"` is scanned as the single pair `["1", "h30m"]`; `"h30m"` is not in
+the token table, so the product becomes `1 * nil` and raises
+`TypeError: nil can't be coerced into Integer`. Verified by probe_06 and
+`tmp/rewrite_D/probe_16_timespan_exact.rb`:
+
+```text
+Misc.timespan('1h')    => 3600
+Misc.timespan('1d')    => 86400
+Misc.timespan('2w')    => 1209600
+Misc.timespan('3mo')   => 8035200
+Misc.timespan('1y')    => 31536000
+Misc.timespan('1:30')  => 90    (HH:MM clock form)
+Misc.timespan('-1h')   => -3600
+Misc.timespan('1h30m') => TypeError: nil can't be coerced into Integer
+Misc.timespan('1x')    => TypeError: nil can't be coerced into Integer
+Misc.timespan('2')     => 2   (bare number uses the default unit, seconds)
+```
+
+So: one number, one unit, per token; combine durations with `HH:MM:SS` or
+add the seconds yourself.
+
+## Misc.digest and `file_md5`
+
+`Misc.digest(obj)` produces a 32-char MD5 hex (`Misc.digest('x')` =>
+`"9dd4e461268c8034f5c8564e155c67a6"`). `Misc.file_md5(path)` hashes the
+file **contents**; when the digest falls back to a plain `Misc.digest` on a
+path string, it hashes the **path string**, not the content — a missing
+`/nope/x` still yields a digest (`14470bb0...`) instead of raising
+(probe_06). Check `File.exist?` yourself if that distinction matters.
+
+## `Misc.insist`
+
+See [ErrorHandling.md](ErrorHandling.md) for the retry protocol with
+`TryAgain`/`StopInsist`/`Aborted`.
+
+## Hook
+
+`lib/scout/misc/hook.rb` defines a top-level `Hook` module — **not
+auto-required** (verify with `defined?(Hook)` => nil after
+`require 'scout-essentials'`; it becomes a constant only after
+`require 'scout/misc/hook'`). It exposes `Hook.extended`, `Hook.apply` and
+`hook_method` (probe_06). `Hook.apply(hook_class, base_class)` redefines the
+methods the two classes share on `base_class`, aliasing the originals as
+`orig_<name>` and dispatching first to the registered hooks, honouring an
+optional `claim(*args)` predicate on each hook. It is the mechanism behind
+tool registration in `CMD::TOOLS`-style setups.
+
+## TmpFile
+
+Root: `TmpFile.tmpdir` => `$HOME/tmp/scout/tmpfiles` (here
+`/home/mvazque2/tmp/scout/tmpfiles` — a machine-specific example value),
+overridable with `TmpFile.tmpdir=`;
+`TmpFile.user_tmp('sub')` => `$HOME/tmp/scout/sub` (probe_05/probe_15).
+
+### Naming conventions (`tmp_for_file`, `lib/scout/tmpfile.rb:98-118`)
+
+| piece | meaning |
+|---|---|
+| `·` (U+00B7) | each `/` in the source path |
+| `PREFIX:` | `:prefix` option |
+| `[key]` | `:key` option |
+| `&F[match=...]` | `:filters` option (the "other options" hash) |
+| `:md5` tail | digest of the remaining options |
+| `MAX_FILE_LENGTH = 150` | names longer than this are truncated |
+
+Verified shapes (probe_05, re-run as probe_13 for gate 2; `...` is
+`TmpFile.tmpdir`, here `~/tmp/scout/tmpfiles`):
+
+```text
+tmp_for_file('/a/b/c')                        => .../·a·b·c
+tmp_for_file('/a/b/c', :prefix => 'P')        => .../P:·a·b·c
+tmp_for_file('/a/b/c', :key => 'k')           => .../·a·b·c[k]
+tmp_for_file('/a/b/c', {}, :filters => {:m=>1}) => .../·a·b·c&F[m=c4ca4238a0b923820dcc509a6f75849b]:4db87253e0818c624c185ac939aa99c1
+tmp_for_file('/a/b/c', {}, :other => 2)       => .../·a·b·c:9e984fda99565456fdde6f77833b61b4
+```
+
+The `:filters` form embeds `Misc.digest(value)` (here the digest of `1`) in
+the name and, because `other_options` is then non-empty, still gets the
+`:md5` tail of the whole options hash (tmpfile.rb:112-115, 129).
+
+**`nil` is not `#{}`**: `tmp_for_file('/a/b/c', nil, ...)` raises
+`NoMethodError: super: no superclass method 'include?' for nil` from
+`process_options` — pass an explicit empty Hash as the second argument when
+you use `other_options` (probe_13).
+
+### `with_file` leaks on raise
+
+`TmpFile.with_file(content, erase, options)` writes the temp file, yields
+it, then `Open.rm_rf tmpfile if Open.exist?(tmpfile) && erase` — there is
+**no `ensure`**. If the block raises, the temp file stays on disk
+(probe_05: file still present after `raise "boom"`; removed normally when
+the block succeeds). Add your own `begin/ensure` around `with_file` when the
+block can fail. See [ErrorHandling.md](ErrorHandling.md).
+
+## Related pages
+
+- [AnnotationSystem.md](AnnotationSystem.md) — the `Annotation` module
+  `NamedArray` builds on.
+- [ErrorHandling.md](ErrorHandling.md) — `Misc.insist`, cleanup limits.
+- [PathResolution.md](PathResolution.md) — how `tmpfiles` fits into the
+  `tmp` map.

data/doc/developer/DesignPrinciples.md

@@ -0,0 +1,129 @@
+# Design Principles
+
+The handful of conventions that hold across `lib/scout`. Everything below is
+observable in the source of this repo; nothing here is aspirational.
+
+Everything below is backed by `tmp/rewrite_C/probe_11_design.rb`
+(probe_11), which reproduces each claim from a clean `require`.
+
+## Composition by annotation
+
+Rather than defining wrapper classes, scout-essentials **annotates ordinary
+Ruby objects**. `Path`, `Resource`, `Persist` and `NamedArray` are modules
+that are `extend`ed into Strings, Arrays or other modules; the receiver keeps
+its class and gains accessors:
+
+```ruby
+Path.setup('some/dir/file.txt')     # a String with path machinery
+SampleInfo.setup('S003', organism: 'Human')   # a String with metadata
+```
+
+`AnnotatedObject`-backed metadata, `annotation_types`, `purge`, and the
+`setup`/`annotate` round-trip are documented in
+[Annotation System](AnnotationSystem.md).
+
+This is *the* extension mechanism of the gem: when you need to attach data or
+behaviour to a value that may arrive from outside your code, you define an
+annotation module instead of a wrapper class.
+
+## Modules, not classes
+
+The public namespaces (`Path`, `Open`, `CMD`, `Persist`, `Misc`, `Log`,
+`Resource`, `SOPT`, `IndiferentHash`, `TmpFile`, `Annotation`) are modules
+whose methods are module-functions or class methods. Subclassing one of them
+is not an option — `Persist` is a `Module`, so `class X < Persist` raises
+`TypeError: superclass must be an instance of Class` (probe_11). The reuse
+pattern is `extend` (`resource/path.rb`, `annotation/annotated_object.rb`).
+
+## Options hashes over positional flags
+
+Public methods take a trailing `options = {}` and read named keys with
+defaults, rather than growing positional booleans:
+
+```ruby
+Open.sensible_write(path, content, :mode => 'w', :replicates => 3)
+Persist.persist(name, :marshal, :check => [...], :update => true) { ... }
+CMD.cmd('grep foo', :pipe => true, :in => input)
+```
+
+Consequences for callers: options are permissive (unknown keys are ignored),
+they are normalised inside helpers (`IndiferentHash.process_options`,
+`IndiferentHash.add_defaults`), and they compose when one helper forwards its
+options to another. See [Path Resolution](PathResolution.md) and
+[Persistence and Resources](PersistenceAndResources.md).
+
+## Exception-borne control flow
+
+Signals are ordinary exception objects derived from `Exception`, **not**
+`StandardError` (`lib/scout/exceptions.rb`):
+
+```ruby
+class StopInsist  < Exception; end   # misc/insist.rb: stop retrying, re-raise last error
+class DontClose    < Exception; end   # open.rb: keep the payload, skip closing
+class DontPersist  < Exception; end   # persist.rb: do not delete a partial cache
+class KeepLocked   < DontPersist; end # open/lock.rb: leave the lockfile alone
+class KeepBar      < Exception; end   # log/progress/util.rb: keep the bar
+```
+
+(`lib/scout/exceptions.rb` also defines the *ordinary* half of the ladder —
+`ScoutException < StandardError` with `ParameterException`,
+`MissingParameterException` and `ResourceNotFound`; `Aborted`,
+`ProcessFailed`, `TryAgain`, `ClosedStream` … — those are exactly the ones a
+bare `rescue` *does* catch.)
+
+The point is deliberate: a bare `rescue => e` (which rescues
+`StandardError`) **does not intercept them** — probe_11 raises each in turn
+inside `begin ... rescue => e` and shows the signal escapes the bare
+`rescue`. A producer that wants the caller to keep the stream it was handed
+raises `DontClose` (carrying the result in `.payload`), and `Open.open`'s
+block form rescues it by name (`lib/scout/open.rb:66-67`) to return that
+payload without closing; `Persist` likewise checks `DontPersist === e`
+before deleting a half-written cache (`lib/scout/persist.rb:130`). Callers
+must name the class explicitly.
+
+Practical rules:
+
+- in code that closes streams or removes temp files, `rescue DontClose`,
+  `rescue DontPersist`, `rescue KeepLocked` by name;
+- never convert a signal into `StandardError` (e.g. `raise e` inside a
+  `rescue StandardError`) — you would swallow it for the outer caller;
+- aborting a `CMD` pipe surfaces as `Aborted` (a `StandardError`) or as the
+  `AbortedStream` **module** that is `extend`ed onto the stream object —
+  these are what streaming consumers should rescue
+  ([Streaming Model](StreamingModel.md); probe_11 shows
+  `AbortedStream` is a Module, not a Class).
+
+## Atomic writes
+
+Anything that produces a file other users may read concurrently goes through
+`Open.sensible_write(path, content, options)`:
+
+- content is written to a temporary name generated next to the target
+  (`TmpFile.tmp_for_file(path, :dir => Open.sensible_write_dir)`) and guarded
+  by its own lockfile,
+- the real file appears via `Open.mv` inside `Misc.insist`
+  (`stream.rb:137-139`), then is touched,
+- on abort or any other exception the temporary is removed and the target is
+  removed if it appeared (`stream.rb:148-161`); the original exception is
+  re-raised.
+
+`Persist.persist` (`persist.rb:108`), the `Persist.save` drivers
+(`persist/serialize.rb:105,119`) and `Resource#produce`
+(`resource/produce.rb:97,106`) all funnel through it. See
+[Persistence and Resources](PersistenceAndResources.md).
+
+## Release semantics you must not assume
+
+Cleanup is explicit, not automatic. `TmpFile.with_file` removes the temporary
+file only when the block ends normally — there is no `ensure` around the
+`yield`, so an exception leaves the file behind (probe_11; verified against
+`lib/scout/tmpfile.rb:71-73`). Likewise stream closing depends on the signal classes
+above, and progress bars rely on `Log::ProgressBar.remove_bar` being called.
+When correctness matters, put the cleanup in an `ensure` or use
+`Open.consume_stream`.
+
+## Related
+
+- [Architecture](Architecture.md) — module map and attribution of the Scout
+  features that live in other repos.
+- [Streaming Model](StreamingModel.md) — signal classes in the pipe pipeline.

data/doc/developer/ErrorHandling.md

@@ -0,0 +1,203 @@
+# Error Handling
+
+Everything that can be raised by `scout-essentials`, and the two very
+different jobs exceptions do in this codebase: real failures vs.
+control-flow signals. Source: `lib/scout/exceptions.rb` (78 lines, whole
+file), `lib/scout/cmd.rb:22-31` (`CMD::Timeout`),
+`lib/scout/concurrent_stream.rb:2-8` (`AbortedStream` module),
+`lib/scout/open/stream.rb` (`sensible_write`).
+
+## The taxonomy
+
+Verified by `tmp/rewrite_D/probe_14_exceptions_full.rb`:
+
+```text
+ScoutDeprecated                  parent=StandardError
+ScoutException                   parent=StandardError
+FieldNotFoundError               parent=StandardError
+TryAgain                         parent=StandardError
+StopInsist                       parent=Exception        <- NOT StandardError
+Aborted                          parent=StandardError
+ParameterException               parent=ScoutException
+MissingParameterException        parent=ParameterException
+ProcessFailed                    parent=StandardError
+ConcurrentStreamProcessFailed    parent=ProcessFailed
+OpenURLError                     parent=StandardError
+DontClose                        parent=Exception        <- NOT StandardError
+DontPersist                      parent=Exception        <- NOT StandardError
+KeepLocked                       parent=DontPersist      <- NOT StandardError
+KeepBar                          parent=Exception        <- NOT StandardError
+LockInterrupted                  parent=TryAgain
+ClosedStream                     parent=StandardError
+ResourceNotFound                 parent=ScoutException
+CMD::Timeout                     parent=ProcessFailed
+```
+
+Plus one non-class member: `AbortedStream` is a **module**
+(`lib/scout/concurrent_stream.rb:2-8`) that gets extended onto a stream;
+`AbortedStream#exception` carries the original upstream exception. It is
+not in the raise/catch taxonomy at all.
+
+## Two families, and why it matters
+
+Failures derive from `StandardError` and behave normally. Control-flow
+signals derive straight from `Exception`, so a bare `rescue =>` (which
+means `rescue StandardError`) will **not** see them — verified
+`probe_14`:
+
+```ruby
+raise DontClose.new("payload")
+# rescue =>      -> does NOT catch
+# rescue Exception -> catches
+```
+
+Probe results:
+
+```text
+DontClose  caught by 'rescue =>' => NOT caught; needs 'rescue Exception'
+KeepLocked caught by 'rescue =>' => NOT caught; needs 'rescue Exception'
+KeepBar    caught by 'rescue =>' => NOT caught; needs 'rescue Exception'
+DontPersist caught by 'rescue =>' => NOT caught; needs 'rescue Exception'
+```
+
+`Aborted` and `TryAgain`/`LockInterrupted` ARE `StandardError`, so they
+travel through ordinary `rescue =>` handlers — which is exactly what the
+`Aborted` protocol relies on (below).
+
+Each signal has a payload slot, and `StopInsist` carries the inner
+exception it wants re-raised (`StopInsist#exception`,
+`lib/scout/exceptions.rb:10-15`).
+
+## ProcessFailed family
+
+`ProcessFailed.new(pid, msg)` builds its own message
+(`lib/scout/exceptions.rb:22-37`):
+
+```text
+ProcessFailed.new(1234,'custom msg').message => "Process 1234 failed - custom msg"
+ProcessFailed.new(nil,'custom msg').message   => "Failed to run custom msg"
+```
+
+`pid` and `msg` are exposed as accessors.
+
+`ConcurrentStreamProcessFailed < ProcessFailed`
+(`lib/scout/exceptions.rb:40-47`) takes `(pid, msg, concurrent_stream)` and
+exposes `concurrent_stream`; note the constructor has a real quirk — it
+reads `@concurrent_stream` (still `nil`) instead of assigning the argument,
+so the accessor always stays `nil`. The message is whatever `msg` was
+passed at the raise site (`lib/scout/concurrent_stream.rb:113`); it is
+raised from `ConcurrentStream#join_threads` when a thread's value is a
+`Process::Status` that did not succeed (unless `no_fail`).
+
+`CMD::Timeout < ProcessFailed` (`lib/scout/cmd.rb:22-31`) adds `command`
+and `timeout` readers and composes the message through `super` — verified:
+
+```text
+CMD::Timeout message => "Process 12 failed - command 'sleep 3 ' exceeded timeout of 0.1 seconds"
+  command => "sleep 3 " timeout => 0.1
+```
+
+## The Aborted protocol
+
+`Aborted` means "this stream is dead, stop consuming it". It is a
+`StandardError`, so it propagates through normal rescues;
+`Open.sensible_write` has a dedicated `rescue Aborted` arm
+(`lib/scout/open/stream.rb:150-154`) that:
+
+1. logs `Aborted sensible_write -- <path>`,
+2. calls `content.abort` if the content responds to it,
+3. **deletes the partial output** (`Open.rm path if File.exist? path`).
+
+For the generic arm (`rescue Exception`, `lib/scout/open/stream.rb:155-163`)
+the same cleanup happens, plus one extra move: when the content is an
+`AbortedStream` carrying an `exception`, **that original exception is what
+gets re-raised**, not the `Exception` raised in this frame:
+
+```ruby
+exception = (AbortedStream === content and content.exception) ? content.exception : $!
+```
+
+So the caller of `sensible_write` sees the root cause that made the
+producer abort, rather than a secondary wrapper. Verified by
+`research/behavior-probes.md` P25/P33 and `tmp/rewrite_B/probe_12_swallow.rb`
+(`exists after Aborted => false`).
+
+The `ensure` always removes the temp file and unlocks a held
+`Lockfile` (`lib/scout/open/stream.rb:165-171`).
+
+## `Misc.insist` and the retry loop
+
+`Misc.insist` (`lib/scout/misc/insist.rb`) retries the block while the
+code inside raises `TryAgain`; `StopInsist` and `Aborted` break out.
+Verified `tmp/rewrite_B/probe_01` / `probe_10_misc.rb` and
+`tmp/rewrite_D/probe_04_exceptions.rb`:
+
+```ruby
+tries = 0
+Misc.insist do
+  tries += 1
+  raise TryAgain unless tries == 3
+  :ok
+end
+# => :ok, after 3 tries
+
+Misc.insist do
+  raise StopInsist.new(ArgumentError.new("inner"))
+end
+# => raises ArgumentError (the inner exception is recovered)
+
+Misc.insist do
+  raise Aborted
+end
+# => raises Aborted, no retry
+```
+
+`LockInterrupted < TryAgain` is the bridge between the two worlds: when
+`Open.lock` fails to acquire, it raises `LockInterrupted`
+(`lib/scout/open/lock.rb:43`), which an outer `Misc.insist` treats as
+"try again".
+
+## `canfail` / `no_fail` semantics
+
+- `Persist.persist(..., :canfail => true)` — exceptions from the block are
+  caught; the persist file is removed and `nil` is returned
+  (`lib/scout/persist.rb:63-100`; verified `tmp/rewrite_D/probe_03_locks.rb`:
+  `persist canfail => nil`, `file removed: true`).
+- `ConcurrentStream` `no_fail: true` — a non-success
+  `Process::Status` from a joined thread is logged at low level instead of
+  raising `ConcurrentStreamProcessFailed`
+  (`lib/scout/concurrent_stream.rb:113`; also `research/behavior-probes.md`
+  P26).
+- `CMD`'s `no_fail` similarly suppresses both `ProcessFailed` from a thread
+  join and exceptions during join.
+
+The distinction matters when composing: a `no_fail` stream that dies
+produces no exception at join time, so downstream persistence will
+happily write empty output unless the producer itself is aborted.
+
+## Cleanup guarantees, and their real limits
+
+`Open.sensible_write` is the model of a real guarantee: temp file + lock +
+`ensure` that removes both. But cleanup is not uniform across the gem:
+
+- **`TmpFile.with_file` leaks the temp file when the block raises** — there
+  is no `ensure` around the `yield`
+  (`lib/scout/tmpfile.rb:34-48`; verified
+  `tmp/rewrite_D/probe_05_tmpfile_leak.rb`: the created tmp file is still
+  present after the block raises `RuntimeError`). Wrap your own `begin/ensure`
+  around `with_file` if the block can fail.
+- `Open.sensible_write` deletes the *target* on failure but only the
+  *temp* file, never other files written by a streaming producer.
+- `Misc.insist` retries without undoing side effects already produced by
+  earlier attempts.
+
+## Related pages
+
+- [StreamingModel.md](StreamingModel.md) — where `Aborted` is raised,
+  consumed, and swallowed.
+- [LockingAndConcurrency.md](LockingAndConcurrency.md) — `KeepLocked`,
+  `LockInterrupted`, and the lock lifecycle.
+- [PersistenceAndResources.md](PersistenceAndResources.md) — `DontPersist`,
+  `canfail`.
+- [LoggingAndProgress.md](../user/LoggingAndProgress.md) — `KeepBar` and
+  progress-bar lifecycle.