scout-essentials 1.8.8 → 1.9.1

data/doc/developer/StreamingModel.md

@@ -0,0 +1,236 @@
+# Streaming Model
+
+This page documents the streaming model implemented in
+`lib/scout/concurrent_stream.rb` and `lib/scout/open/stream.rb` from the inside:
+how a stream is constructed, how the two lifecycle callbacks compose, how abort
+and join interact, and how errors travel. The user-facing surface is
+[Handling Streams](../user/HandlingStreams.md); the producer side is
+[Running Commands](../user/RunningCommands.md).
+
+## The core idea
+
+A "stream" here is an ordinary IO object (a pipe read end from a subprocess, a
+`StringIO`, a `File`) extended with the `ConcurrentStream` **module**. The
+module does not replace the IO; it adds bookkeeping about who is producing the
+data and what must happen when the data is finished:
+
+- `threads` / `pids` — the producers feeding this end,
+- `callback` / `abort_callback` — what to run on success / on abort,
+- `std_err`, `log`, `exit_status`, `filename`, `lock`, `next`, `pair`.
+
+Everything hangs off `ConcurrentStream.setup`, and most of the model is about
+what setup, join and abort do with that state.
+
+## `ConcurrentStream.setup` is idempotent and accumulative
+
+```ruby
+def self.setup(stream, options = {}, &block)   # concurrent_stream.rb:12
+```
+
+`setup` extends the object unless it already is a `ConcurrentStream`, then:
+
+- `threads ||= []`, `pids ||= []`, and new values are **concatenated** onto the
+  existing arrays — a second `setup` adds producers, it does not replace them
+  (probe `tmp/rewrite_B/probe_22_cs_semantics.rb`: `idempotent setup: cs? true
+  threads=1 pids=[4]`, `after 2nd threads arg: 2`).
+- `std_err` is **reset to `""`** on every setup.
+- `autojoin`, `no_fail`, `next`, `pair`, `filename`, `lock` are only assigned
+  when the corresponding option is non-nil, so an existing value survives.
+- **The block becomes a callback**: a block passed to `setup` is treated as
+  `callback` and *composed* with any existing callback, so the new block runs
+  after the old one (`probe_22`: `setup-block callbacks order: [:block, :block2]`).
+
+This is why `Open.open_pipe` (thread mode) can call `setup` on both ends and
+still compose a user callback, and why `add_callback` composes the same way.
+
+## Callbacks: `callback`, `add_callback`, `abort_callback`
+
+There are two callback slots, each holding a single (possibly chained) Proc:
+
+- `callback` (a.k.a. the join callback) — run by `join_callback`, which is
+  called from `join` and is skipped if the stream is already joined; it is
+  cleared (`@callback = nil`) after running so it fires once.
+- `abort_callback` — run by `abort(exception)` with the abort exception, then
+  discarded together with `callback`.
+
+`ConcurrentStream#add_callback(&block)` wraps the current callback so the new
+block runs **after** the old one — the same composition `setup` uses:
+
+```ruby
+stream.add_callback { cleanup_a }
+stream.add_callback { cleanup_b }   # join runs a, then b
+```
+
+Verified in `probe_19_streaming_apis.rb` (`add_callback order: [:first,
+:second]`) and `probe_22_cs_semantics.rb`. There is no `add_abort_callback`;
+`abort_callback = proc { |exception| ... }` replaces the slot (only `setup`
+with an `:abort_callback` option composes it).
+
+## `join`: threads, pids, callbacks, close — never the pair
+
+```ruby
+def join        # concurrent_stream.rb:146
+  join_threads; join_pids
+  raise stream_exception if stream_exception
+  join_callback
+  close unless closed?
+ensure
+  @joined = true
+  lock.unlock if lock && lock.locked?
+  raise stream_exception if stream_exception
+end
+```
+
+- `join_threads` waits for each producer thread (skipping the current thread).
+  If a thread returned a failed `Process::Status` it raises
+  `ConcurrentStreamProcessFailed`; a thread that died with an exception is
+  routed through `stream_raise_exception` unless `no_fail`.
+- `join_pids` `Process.waitpid`s each pid, records `self.exit_status`, raises
+  `ConcurrentStreamProcessFailed` for a bad status unless `no_fail`, and
+  **empties `@pids`** — so `exit_status` is only ever accurate right here.
+- `join` never touches `@pair`. The pair is the *other end of the same pipe*
+  (see below) and is joined/aborted through `abort` propagation, not through
+  `join`.
+
+`close` (concurrent_stream.rb:230) is a thin wrapper: with `autojoin` it closes
+and then joins when the stream is at EOF or already closed, converting a close
+failure into `abort` + `join` + `stream_raise_exception`; without `autojoin` it
+just closes, swallowing `IOError`.
+
+## `no_fail`
+
+`no_fail` silences every join-time failure path:
+
+- a producer thread's failed status does not raise
+  `ConcurrentStreamProcessFailed`,
+- a dead producer thread is logged at `Log.low` ("Not failing on exception
+  joining thread") instead of being escalated,
+- `join_pids` does not raise for a non-zero exit status.
+
+`CMD.cmd` propagates `:no_fail` to the stream and **defaults `autojoin` to
+`no_fail`** (`cmd.rb:219`, `:autojoin => no_fail`), which is why a `no_fail`
+pipe that is simply read does not blow up at close time.
+
+## `abort`: idempotent, and it propagates to `pair`
+
+```ruby
+def abort(exception = nil)   # concurrent_stream.rb:204
+```
+
+`abort` is the deliberate early-stop path. It:
+
+1. records `stream_exception ||= exception`,
+2. marks the object with `AbortedStream.setup(self, exception)` and sets
+   `@aborted = true`; a second call only logs (`Already aborted stream`) and
+   returns — **idempotent** (probe `probe_22_cs_semantics.rb`:
+   `abort idempotent: aborted? true`, `second abort: no raise, aborted? true`),
+3. runs `abort_callback` with the exception,
+4. `abort_threads`: raises `Aborted` (or the given exception) in each producer
+   thread and joins them — this is what unblocks a producer stuck writing to a
+   pipe nobody reads,
+5. `abort_pids`: sends `SIGINT` to each pid,
+6. clears both callbacks,
+7. **propagates to `@pair`** if it responds to `abort` and is not already
+   aborted — killing one end of an `Open.open_pipe` pair takes down the other,
+8. closes and unlocks in `ensure`.
+
+Producers are expected to `rescue Aborted` and finish quietly; `Open.grep`,
+`consume_stream` and `sensible_write` all do.
+
+## `stream_raise_exception`: the failure amplifier
+
+```ruby
+def stream_raise_exception(exception)   # concurrent_stream.rb:281
+  self.stream_exception = exception
+  threads.each { |thread| thread.raise exception }
+  self.abort
+end
+```
+
+This is how one failing producer takes the whole stream down: the exception is
+stored in `stream_exception` (so a later `join`/`read` re-raises it even if the
+current frame recovers), it is raised in every producer thread, and the stream
+is aborted. `join_threads` and `join_pids` call it, and `ConcurrentStreamProcessFailed`
+carries the offending `pid` plus the stream. Verified: `probe_22_cs_semantics.rb`
+(`stream_raise_exception raised: demo`, `stream_exception set: #<Aborted: demo>`,
+`no_fail join of failed: no raise`).
+
+## `ConcurrentStream.process_stream`
+
+```ruby
+def self.process_stream(stream, close: true, join: true, message: "process_stream",
+                        **kwargs, &block)
+```
+
+The standard producer wrapper (concurrent_stream.rb:286): it sets the stream up
+with `kwargs`, runs the block, and in an `ensure` closes and joins the stream as
+requested. `Aborted` and any other exception are logged, the stream is aborted
+with the exception, and the exception is re-raised. `Open.open_pipe` (thread
+mode) and `Open.sort_stream` are built on it. Verified: `probe_22_cs_semantics.rb`
+(`process_stream: "z\ny\nx\n" src joined=true src closed=true`).
+
+## `AbortedStream` and recovering the original cause
+
+`AbortedStream` is not an exception class: it is a **marker module** with an
+`exception` accessor (`concurrent_stream.rb:4-9`). `abort` marks the stream with
+it so downstream code can tell "this stream was aborted" apart from "this IO is
+just closed", and can ask what the real cause was:
+
+```ruby
+content.abort(my_error)          # stream.exception == my_error
+...
+exception = (AbortedStream === content and content.exception) ? content.exception : $!
+```
+
+That exact snippet is what `Open.sensible_write` uses: when copying a stream
+into its tmp file raises, it recovers the original upstream exception from the
+marker and re-raises *that* (deleting the target and tmp file), while a plain
+`Aborted` is swallowed and cleaned up. Verified by P25/P33 and the
+`probe_22`/`probe_13` sequences in `research/behavior-probes.md`.
+
+## `pair` means pipe ends, not stdout/stderr
+
+`ConcurrentStream#pair` is set by `Open.open_pipe` (thread mode) on both ends of
+the pipe it just created: `setup(sin, :pair => sout)` and
+`setup(sout, :pair => sin)` (open/stream.rb:230-231). It exists so `abort` on
+one end can propagate to the other. **It has nothing to do with "a paired
+stdout/stderr stream"** — there is no such object in this repo.
+
+Likewise, in `CMD.cmd` pipe mode the child's **stderr is drained by a thread**
+(an `err_thread` registered on the returned stream when a severity or
+`save_stderr` asks for it), not by a second stream. Per-stream diagnostics live
+in `std_err` (filled by `:save_stderr`) and `log` (last stderr line); log
+messages themselves go to the Log logfile / STDERR.
+
+## `next`
+
+`next` is a forward link to the stream that logically follows this one
+(`setup` assigns it from `:next`). It lets a helper hand you a derived stream
+while keeping a pointer at the original, so bookkeeping (`filename`, locks,
+`annotate`) can be traced back along the chain. `ConcurrentStream#annotate`
+copies `threads`, `pids`, `callback`, `abort_callback`, `filename`, `autojoin`
+and `lock` onto another stream — used by `Open.line_monitor_stream` and
+annotation-style code.
+
+## Reading between the lines
+
+`read` (concurrent_stream.rb:246) wraps `IO#read` so that an exception is
+recorded in `stream_exception`, the stream aborted and the recorded exception
+re-raised; when `autojoin` is set it polls `eof?` and closes/joins once the
+producer is done. This is what makes a `:pipe => true` result behave like a
+plain IO for `.read`/`.each` while still cleaning up its producers.
+
+## Where this is used
+
+- `CMD.cmd` — `:pipe => true` produces these streams
+  ([Running Commands](../user/RunningCommands.md)).
+- `Open.open_pipe`, `Open.tee_stream`, `Open.sort_stream`,
+  `Open.line_monitor_stream`, `Open.collapse_stream`, `Open.consume_stream`,
+  `Open.sensible_write` — the helpers documented in
+  [Handling Streams](../user/HandlingStreams.md).
+- `Open.grep` composes `CMD.cmd` + `:post`; its `force_close` call is a dead
+  `respond_to?` guard (no such method exists in this repo).
+- For the module dependency graph, see
+  [Architecture](Architecture.md#modules-and-dependencies); for cross-repo
+  concepts (TSV, Step, Workflow, HPC) see the
+  [attribution table](Architecture.md#ecosystem-boundaries-and-attribution).

data/doc/user/AnnotatingData.md

@@ -0,0 +1,202 @@
+# Annotating Data
+
+Annotations attach named metadata — an organism, a provenance URL, a
+"kind" of file — to ordinary Ruby objects (Strings, Arrays, Hashes) without
+changing their class or wrapping them. This page is the user-facing view of the
+system; internals and design notes are in
+[Annotation System](../developer/AnnotationSystem.md).
+
+Every example below was run against the current `lib/scout/` (probes
+P36–P42 in `research/behavior-probes.md` plus
+`tmp/rewrite_C/probe_01..04.rb` and `probe_09_annotated_array.rb`).
+
+## Defining an annotation
+
+```ruby
+module SampleInfo
+  extend Annotation
+  annotation :organism, :tissue, :donor
+end
+
+sample = SampleInfo.setup('S003', organism: 'Human', tissue: 'Liver')
+
+sample                 # => "S003"      (still a String)
+sample.organism        # => "Human"
+sample.tissue          # => "Liver"
+sample.is_a?(String)   # => true
+sample.is_a?(SampleInfo) # => true      (the module really was extended in)
+```
+
+`require 'scout-essentials'` is enough: `Annotation` arrives via
+`scout/path` (`lib/scout/path.rb:1`).
+
+## The object you get back
+
+`setup` **extends the object you pass, in place**, and returns that same
+object — the call is an annotation, not a conversion:
+
+```ruby
+name = 'S003'
+annotated = SampleInfo.setup(name, organism: 'Human')
+annotated.equal?(name)   # => true
+```
+
+Two consequences, both live-probed (probe_02):
+
+- a **frozen** object is `dup`ed first, so `setup` returns a different,
+  unfrozen copy — always use the return value;
+- `dup` of an annotated object **loses the annotations** (`SampleInfo ===
+  sample.dup => false`); `clone` keeps them (`SampleInfo === sample.clone =>
+  true` — `clone` copies the singleton class, `dup` does not). To copy
+  metadata onto a `dup` explicitly:
+
+```ruby
+copy = SampleInfo.setup(sample.dup, sample.annotation_hash)
+# or: sample.annotate(other_object)
+```
+
+`obj.dup` on an `Integer`-like target is different: `setup` rescues the
+`TypeError: can't define singleton` and hands back the plain, un-annotated
+object — nothing raises.
+
+## Introspection
+
+```ruby
+sample.annotation_types      # => [SampleInfo]      (module objects, not names)
+sample.annotation_types.include?(SampleInfo)   # => true
+
+SampleInfo.annotations       # => [:organism, :tissue, :donor]   (module state)
+
+sample.annotation_hash       # => {:organism=>"Human", :tissue=>"Liver"}
+sample.annotation_info       # => {..same.., :annotation_types=>[SampleInfo],
+                             #     :annotated_array=>false}
+sample.annotation_id         # => "860c73f490edb8115064663b7f579d73"
+Annotation.is_annotated?(sample)  # => true
+```
+
+Note where each thing lives: `annotation_types` is on the **object**; the
+declared attribute list is read from the **module** with `.annotations`
+(there is no `ANNOTATIONS` constant).
+
+### Round-trip
+
+`annotation_hash` plus `Annotation.setup` is the serialisation pair:
+
+```ruby
+info = sample.annotation_info
+Annotation.setup('S003', 'SampleInfo', sample.annotation_hash)
+```
+
+The type argument may be a `"A|B"` String of module names (unknown names are
+only `Log.warn`ed and skipped, probe_01) or an Array of module objects.
+
+`#serialize` produces the plain Hash (`annotation_info` merged with
+`:literal`) consumed by the `:annotation` persistence driver. There is **no
+TSV serialisation of annotations in this repo** — `Annotation.tsv` /
+`Annotation.load_tsv` live in scout-gear (see the attribution table in
+[Architecture](../developer/Architecture.md)).
+
+Nested values you store are left alone: hashes stay plain `Hash`, they are
+**not** converted to `IndiferentHash`, and `deep_indifferent` does not exist
+in this gem.
+
+## NamedArray — field names over Array positions
+
+`NamedArray` (`lib/scout/named_array.rb`) is a separate annotation module
+that gives an Array named fields. It needs its **explicit**
+`require 'scout/named_array'` — `scout-essentials.rb` does not load it
+(probe_03):
+
+```ruby
+require 'scout/named_array'
+
+row = NamedArray.setup(%w[S003 Human Liver], %w[id organism tissue])
+row.organism    # => "Human"   (method_missing access)
+row[:organism]  # => "Human"
+row['tissue']   # => "Liver"
+```
+
+Signatures (probe_04): `NamedArray.setup(array, names, *rest)` — the names are
+a positional Array, **not** a `key:` keyword.
+
+### Access is via `method_missing`
+
+Field accessors are *not* real methods, so they do not participate in the
+usual introspection:
+
+```ruby
+row.respond_to?(:organism)          # => false
+row.methods.include?(:organism)     # => false
+row.organism                        # => "Human"   (still works)
+```
+
+`:[]` accepts a Symbol or a String and resolves through the field list
+(`identify_name`), so a field named like an Array method can still be reached
+positionally by name.
+
+### Array methods shadow field names
+
+If a field is called `first`, `last`, `count`, `zip`, `sample` … the real
+`Array` method wins (probe_10):
+
+```ruby
+row2 = NamedArray.setup(%w[a b c], %w[first second third])
+row2.first       # => "a"      (Array#first, not the field)
+row2.first(2)    # => ["a", "b"]
+
+row3 = NamedArray.setup(%w[S001 S002 S003], %w[values count])
+row3.values      # => "S001"   (field: Array has no #values)
+row3.count       # => 3        (Array#count wins, not the field)
+```
+
+Note `values` — unlike `first`/`count` — is **not** an `Array` method, so the
+field accessor still works; `count` is real and wins. Prefer field names that
+are not `Array`/`Enumerable` verbs, or use `row[:name]` for ambiguous ones.
+
+`to_hash` (only available on `NamedArray`) returns an `IndiferentHash` of
+field → value.
+
+Watch out for `id` — `annotation_id` (aliased `id`) is defined by the
+annotation system itself, so a field named `id` collides and the digest is
+returned instead (probe_10).
+
+## `AnnotatedArray` — elements inherit the container's annotations
+
+Annotate the *container*, then `extend AnnotatedArray`, and every element
+handed out by `[]`, `first`, `last`, `each`, `collect`, `select`, `compact`,
+`uniq`, `flatten`, `reverse`, `sort_by`, `subset`, `remove` is re-annotated
+(probe_09):
+
+```ruby
+samples = SampleInfo.setup(%w[S001 S002 S003], organism: 'Human')
+samples.extend AnnotatedArray
+
+samples[0].organism    # => "Human"
+samples.each { |s| s.organism }
+samples.collect { |s| s.length }   # [4, 4, 4], elements annotated
+samples.select { |s| s != 'S002' } # re-annotated array
+```
+
+Not overridden — **annotations are dropped** (probe_09, method owners are
+`Array`/`Enumerable`): `map`, `zip`, `+`, `filter_map`, `flat_map`,
+`each_slice`, `values_at`. `zip` in particular does not propagate annotations
+to the *other* operand's elements.
+
+`[index]` re-annotates; `[index, true]` (the clean second argument) returns the
+raw element with no annotations (`lib/scout/annotation/array.rb:21-25`).
+
+Two hard limits:
+
+- elements must be extendable: an Array of `Integer`s raises `TypeError: can't
+  define singleton`;
+- `#make_array` does **not** annotate the elements — it wraps `self` in a
+  one-element annotated Array (see
+  [Annotation System](../developer/AnnotationSystem.md)).
+
+## Related
+
+- [Annotation System](../developer/AnnotationSystem.md) — internals, limits,
+  serialisation drivers.
+- [Path Resolution](../developer/PathResolution.md) — `Path` is an annotated
+  module.
+- [Caching Results](CachingResults.md) — the `:annotation` persistence type.

data/doc/user/CachingResults.md

@@ -0,0 +1,183 @@
+# Caching Results
+
+This guide explains how to cache computation results in scout-essentials
+using the `Persist` module: the `persist` pattern, serialization types,
+staleness invalidation, in-memory caching, and locking.
+
+## The persist pattern
+
+`Persist.persist` runs a block once and reuses the cached result on later
+calls:
+
+```ruby
+require 'scout-essentials'
+
+value = Persist.persist('result', :string) do
+  "expensive computation"
+end
+
+value # => "expensive computation"  (first call: block runs)
+value # => "expensive computation"  (second call: loaded from disk)
+```
+
+The signature is `Persist.persist(name, type = :serializer, options = {}, &block)`:
+
+- `name` — a string used to build the cache file name (a path is also
+  accepted; its `filename` is used). Unless you give `:path`/`:persist_path`,
+  the cache lands under `Persist.cache_dir`, which returns the relative
+  `Path`/String `var/cache/persistence`.
+- `type` — a serialization type (table below); `:serializer` is the default
+  and resolves to `:json` (`Persist::SERIALIZER == :json`).
+- `options` — a plain `Hash` of options. `persist_path` takes an options
+  hash, and a **second positional argument is NOT supported**: the option key
+  is `:persist_path` (or `:path`), never a positional argument.
+
+```ruby
+Persist.persistence_path('result')                 # => "var/cache/persistence/result"
+Persist.persistence_path('result', key: 'X')       # => "var/cache/persistence/result[X]"
+Persist.persistence_path('result', :marshal)       # TypeError — no positional type
+Persist.persistence_path('result', dir: tmp_dir)   # honoured via :dir
+```
+
+**Options:**
+
+| Option | Meaning |
+|---|---|
+| `:persist_path` / `:path` | Exact file to use as cache (String or Path) |
+| `:persist` | `false` bypasses persistence entirely and just returns `yield` |
+| `:no_load` | `true` returns the cache file itself instead of its contents |
+| `:update` | Force recomputation. A `true` recomputes; a `Time`/number recomputes when the cache is older; a `Path` uses that file's `mtime` |
+| `:check` | Array of dependency paths; the cache is invalidated when any is newer |
+| `:canfail` | Swallow errors, returning `nil` (or the file with `:no_load`) instead of raising |
+| `:data` | Passed as the block argument when the block has arity 1 |
+| `:tee_copies` | Number of extra stream copies when the block returns a stream |
+| `:lockfile` | Use a specific lock file instead of the default |
+
+If the block has arity 1, it receives the cache file (a `Path`/String), and
+its return value is loaded from disk only if it is `nil`:
+
+```ruby
+Persist.persist('result', :text, persist_path: path) do |file|
+  Open.write(file, "written by block\n")
+end # => loads the file
+```
+
+## Serialization types
+
+`Persist.serialize` / `Persist.deserialize` / `Persist.save` / `Persist.load`
+understand these types (`type` is `nil, :string, :text, :integer, :float,
+:boolean, :file, :path, :select, :folder, :binary, :array, :yaml, :json,
+:marshal, :annotation, :serializer`, plus `:stream` on load):
+
+| Type | Saved as | Loaded as |
+|---|---|---|
+| `nil`, `:text` | `to_s` | String, exactly as written (no stripping) |
+| `:string` | `to_s` | String, stripped |
+| `:integer`, `:float` | `to_s` | `Integer` / `Float` |
+| `:boolean` | `to_s` | `true` only if in `TRUE_STRINGS` (`"true"`, `"yes"`, `"y"`, `"t"`, `"on"` and their case variants, `"1"`) |
+| `:file`, `:folder`, `:select` | `to_s` | Stripped String (a `:file` entry starting with `"./"` is resolved relative to the cache file's directory) |
+| `:path` | `to_s` | `Path.setup(...)` |
+| `:binary` | bytes (encoding forced to ASCII-8BIT) | bytes read with `mode: 'rb'` |
+| `:array` | elements joined with `"\n"` | Array of lines |
+| `:yaml` | `to_yaml` | Loaded with `Open.yaml` (`YAML.unsafe_load` — `Open.yaml` is defined in the persist layer, `lib/scout/persist/open.rb:11`), so hashes/arrays round-trip. Note `:yaml_array` goes through the per-line `deserialize` path (`YAML.parse`) and yields `Psych::Nodes::Document` objects — use `:json_array` for array round-trips |
+| `:json` | `to_json` | `JSON.parse` |
+| `:marshal` | `Marshal.dump` | `Marshal.load` |
+| `:annotation` | `Annotation.tsv(content, :all).to_s` | `Annotation.load_tsv(TSV.open(...))` (part of the scout-gear ecosystem, not usable standalone) |
+| `:serializer` | alias for `:json` | alias for `:json` |
+| `:stream` (load only) | — | `Open.open(file)` |
+| `:file_array` (load only) | — | Array of files, `"./"`-relative entries resolved |
+
+Any type can take an `_array` suffix (`:yaml_array`, `:path_array`, …) when
+saving or loading: elements are serialized individually and joined/split on
+newlines. **No type suffix is ever appended to cache file names** — two
+different types sharing a `name` collide on the same cache file. An unknown
+type raises `RuntimeError: Persist does not know <type>`.
+
+## Cache location and lock location
+
+```ruby
+Persist.cache_dir    # => var/cache/persistence   (a relative String path)
+Persist.cache_dir = '/some/other/dir'
+Persist.lock_dir     # => $HOME/.scout/tmp/persist_locks  (an absolute String)
+Persist.lock_dir = '/some/other/locks'
+```
+
+Locks live under `Persist.lock_dir` and are named after the cache file plus
+`.persist`: `<cache-file>.persist`. They are `Open.lock` lockfiles (see
+[Working with Files](WorkingWithFiles.md)); there is **no** `Persist.lock` —
+`Persist.persist` uses `Open.lock` internally.
+
+## Staleness: `:update` and `:check`
+
+`:check` lists dependencies; when any of them is newer than the cache, the
+cache is recomputed. `:update` forces recomputation, optionally guarded by a
+`Time` (numeric age in seconds) or a `Path` (an mtime to compare against):
+
+```ruby
+cache = Path.setup('var/cache/persistence/example')
+
+Persist.persist('example', :string, persist_path: cache, check: [input]) do
+  "computed"
+end
+# input updated -> "computed" re-runs
+
+Persist.persist('example', :string, persist_path: cache, update: 60) { "x" }
+# re-runs only if the cache is older than 60 seconds
+
+Persist.persist('example', :string, persist_path: cache, update: dep_path) { "x" }
+# re-runs only if dep_path's mtime is newer than the cache's
+```
+
+`:check` and `:update` (and the `update` computation itself) depend on
+`Open.mtime` and on `file.outdated?(check)`, which are `Path` methods; give
+`persist` a `Path` (`persist_path:` a `Path.setup(...)`), not a plain
+`String`, when you rely on them.
+
+## Memory caching
+
+Type `:memory` stores the block result in a process-global hash
+(`Persist::MEMORY_CACHE`) instead of a file; a custom repo hash can be passed
+with `:memory:` / `:repo:`:
+
+```ruby
+Persist.persist('m', :memory) { [1, 2] } # => [1, 2]
+
+Persist.memory('m2', key: 'K') { "in-memory-value" }
+# => "in-memory-value"; the key is composed into the entry name
+```
+
+`Persist.memory(name, options, &block)` is the helper for the common
+`[name, key]` case.
+
+## Streams and `KeepLocked`
+
+When the block returns an `IO`/`StringIO`, `persist` does not block: it
+tee's the stream so the caller consumes one copy while a background thread
+writes another to the cache file. The returned stream keeps the persist lock
+until it is joined (`KeepLocked`); join the stream (or let `autojoin` run)
+before relying on the cache file.
+
+```ruby
+stream = Open.open_pipe { |sin| 10.times { |i| sin.puts "row#{i}" } }
+res = Persist.persist('rows', :string, persist_path: path) { stream }
+res # => a ConcurrentStream (IO); the cache file is complete once res.join
+```
+
+## Error handling
+
+If the block raises, `persist` removes the partial cache file (unless the
+exception is a `DontPersist`) and re-raises — **unless** `:canfail` is set,
+in which case it returns `nil` (or the file with `:no_load`):
+
+```ruby
+Persist.persist('failing', :string, canfail: true) { raise "boom" } # => nil
+```
+
+## Related
+
+- [Working with Files](WorkingWithFiles.md) — `Open.lock`, `sensible_write`,
+  and I/O that `Persist` builds on.
+- [Producing Resources](ProducingResources.md) — Resource composes with
+  Persist.
+- For internal implementation details, see
+  [Persistence and Resources](../developer/PersistenceAndResources.md).