scout-essentials 1.8.8 → 1.9.1

data/doc/developer/LockingAndConcurrency.md

@@ -0,0 +1,157 @@
+# Locking and Concurrency
+
+How the Scout stack serialises access to files: a vendored `Lockfile`
+implementation, three distinct lock namespaces, and the process/thread model
+the locks are meant to protect.
+
+## The vendored Lockfile
+
+`lib/scout/open/lock/lockfile.rb` is Ara T. Howard's `lockfile` library,
+version **2.1.8**, vendored and modified, and guarded so a second load is a
+no-op:
+
+```ruby
+unless defined?(Lockfile)
+  ... (whole class)
+end
+```
+
+`Open.init_lock` (called when the file is loaded) overrides the library
+defaults, so the effective settings here are **not** the upstream ones
+(probe_08):
+
+| setting | library default | effective here |
+|---|---|---|
+| `max_age` | 3600 | **30** |
+| `refresh` | 8 | **2** |
+| `suspend` | 16 | **4** |
+| `Lockfile.version` | 2.1.8 | 2.1.8 |
+
+So a lock older than 30 s is considered stale and may be stolen, the holder
+re-touches (`FileUtils.touch`) the lock file every 2 s, and a waiter between
+attempts sleeps 4 s.
+
+## What a lock looks like on disk
+
+Acquiring `<file>` creates `<file>.lock` and a hidden sibling. The payload
+written into the lock is four lines (probe_08):
+
+```text
+/tmp/…/data.txt.lock contents:
+  host: turbo
+  pid: 6
+  ppid: 3
+  time: 2026-08-22 00:58:55.751779
+```
+
+(`dump_lock_id`, `lib/scout/open/lock/lockfile.rb:504-507`: `host`, `pid`,
+`ppid`, `time`).
+
+The `.lock` sibling is created by `tmpnam`/`create_tmplock` as a dot-prefixed
+temporary file in the same directory and then **hard-linked** to
+`<file>.lock`; the temp name embeds the pid, which is how the sweep can
+recognise locks left behind by dead processes and remove them. Once the
+original temp name is unlinked, `nlink` on the surviving `.lock` is 1.
+
+## `Open.lock` and `LockInterrupted`
+
+```ruby
+Open.lock(file, options = {}) { ... }       # block form: unlock after
+Open.lock(file, true, options)              # legacy 2nd arg = unlock
+Open.lock(file, false, options = {})        # do not unlock afterwards
+```
+
+`options[:lock]` accepts a `Lockfile` instance (reuse an existing lock), a
+`Path`/`String` (an alternative lock file location), or `false` (no locking
+at all, and no unlock). Without a block the lock is acquired and the
+`Lockfile` object returned. If another holder releases the lock while we are
+waiting, `Open.lock` raises `LockInterrupted` (< `TryAgain`, so it is a
+`StandardError`) — callers retry the whole operation rather than proceeding
+unlock-guarded.
+
+Unlock failures are caught and logged (`Exception unlocking: <path>`), never
+raised over the block's own result.
+
+## The three lock namespaces
+
+Each subsystem locks in its own directory so the names never collide
+(probe_02/probe_03):
+
+| who | directory | name shape |
+|---|---|---|
+| `Persist` | `Persist.lock_dir` = `tmp/persist_locks`.find → `$HOME/.scout/tmp/persist_locks` | `<persistence_path>.persist` suffix |
+| `Resource#lock_dir` | `$HOME/.scout/tmp/produce_locks` | `TmpFile` digest of the resource path |
+| `Open.sensible_write` | `$HOME/.scout/tmp/sensible_write_locks` | digest of the output path |
+
+Verified strings:
+
+```text
+Persist.lock_dir                => $HOME/.scout/tmp/persist_locks
+persist lock file               => .../persist_locks/probeD3.persist
+Resource#lock_dir               => $HOME/.scout/tmp/produce_locks
+produce lock file               => .../produce_locks/·home·mvazque2·.scout·etc·probeD2
+sensible_write lock dir         => $HOME/.scout/tmp/sensible_write_locks
+```
+
+(The `·home·mvazque2·...` name is literal `TmpFile.tmp_for_file` output —
+each `/` in the probe path was flattened to `·` — and is kept verbatim; the
+home-directory component is a machine-specific example value.)
+
+(See [PersistenceAndResources.md](PersistenceAndResources.md) for how
+`persist` uses its lock, and [StreamingModel.md](StreamingModel.md) for
+`sensible_write`.)
+
+## KeepLocked — streaming persistence
+
+`Persist.persist(..., :persist_type/:type)` can return a stream that must
+stay locked while the consumer reads it. Raising `KeepLocked.new(res)` from
+inside the persist block leaves the lock held; the caller gets the stream
+back. probe_03:
+
+```ruby
+res = Persist.persist("probeD3", :text, :persist => false) do
+  raise KeepLocked.new("payload")
+end
+res                       # => "payload"
+File.exist?(lock_path)    # => true   (lock still held)
+```
+
+`KeepLocked < DontPersist < Exception` — it is deliberately outside
+`StandardError` so that generic `rescue =>` blocks do not swallow it.
+
+## The fork/thread model
+
+- **Fork.** `CMD.cmd(:pipe => true)` returns an `IO` whose writer is a forked
+  child; `Open.open_pipe` in fork mode reuses the same machinery, and
+  `ScoutRake.run` forks per task. Child pids are registered on the
+  `ConcurrentStream` (`stream.pids`) and reaped by `abort_pids`/`join`.
+- **Threads.** Stream consumers are ordinary Ruby threads, registered with
+  `stream.threads`. `ConcurrentStream.join` runs the registered `@callback`
+  chain exactly once (`join_callback`) and then joins the threads.
+- **Cross-process.** That is what the locks above are for.
+
+## Thread-safety caveats
+
+Verified by `tmp/rewrite_D/probe_09_threads_fork.rb`:
+
+- `Log::LAST` is a shared top-level `String` — a write from one thread is
+  visible to the next with no synchronisation.
+- `Scout::Config` holds **no mutex**; concurrent `set`/`get`/`with_config`
+  is not atomic. See [Configuration.md](Configuration.md).
+- `Persist` likewise has no lock on its class-level state.
+- `ConcurrentStream` callbacks fire at `join`, in registration order, once
+  each; they are not protected against being added from two threads at once.
+- `Open.wait`'s `LAST_TIME` hash (see [RemoteData.md](../user/RemoteData.md))
+  is unsynchronised shared state.
+
+If you need real parallel writers on the same key, take the relevant lock
+namespace yourself, or serialise at the call site.
+
+## Related pages
+
+- [ErrorHandling.md](ErrorHandling.md) — `LockInterrupted`, `KeepLocked` and
+  the other control-flow exceptions.
+- [PersistenceAndResources.md](PersistenceAndResources.md) — what the
+  persist lock actually guards.
+- [StreamingModel.md](StreamingModel.md) — `sensible_write`, forked pipes and
+  `ConcurrentStream`.

data/doc/developer/PathResolution.md

@@ -0,0 +1,200 @@
+# Path Resolution
+
+`Path` is a `String` subclass carrying resource metadata. This page explains
+how an *unlocated* logical name such as `data/config.yaml` is turned into a
+real filesystem location, how to configure the search, and how `find`,
+`follow`, `identify`, and friends behave.
+
+## Anatomy of a `Path`
+
+A `Path` is a plain `String` plus annotations (`:pkgdir`, `:libdir`,
+`:path_maps`, `:map_order`, `:where`, `:original`), added by the `Annotation`
+mechanism. Because it is a String, it can be joined, split, and used with
+core `File` methods; annotations travel along when you `join` or use `/`:
+
+```ruby
+require 'scout-essentials'
+
+p = Path.setup('data/config.yaml')
+p.pkgdir    # => "scout" (Path.default_pkgdir)
+p.located?  # => false ('data/config.yaml' is relative, not './x' or '/x')
+p.to_s      # => "data/config.yaml"
+
+p.join(:a)          # => "data/config.yaml/a"
+p.join(:a, :b)      # => "data/config.yaml/b/a"  (b goes first)
+p / :a              # => "data/config.yaml/a"
+p._toplevel         # => "data"   (first path segment)
+p._subpath          # => "config.yaml" (the rest)
+p.data.samples      # => "data/config.yaml/data/samples"  (method_missing
+                    #    appends a segment, right to left)
+```
+
+`[]` and `/` are both aliases of `join`. **There is no `Path#[]=` and no
+class-level `Path.map_order=`** — use `Path.add_path`,
+`Path.prepend_path`, `Path.append_path`, or the per-instance
+`path_maps`/`map_order` annotations instead.
+
+## Location: `find`
+
+`find` resolves an unlocated path by trying every map in `map_order` until
+one produces an existing file (or a `.gz`/`.bgz`/`.zip` alternative).
+`find` **never returns nil**:
+
+- If the path is already `located?` (starts with `/`, `./`, or `~/`) and
+  exists, it returns the expanded path.
+- If it is `located?` but missing, it tries the compressed alternatives and
+  otherwise **returns itself**.
+- If it is unlocated, it walks `map_order`; on total failure it returns
+  `follow(:default)` — the default location where the file *would* be.
+
+```ruby
+p = Path.setup('data/config.yaml')
+p.find             # tries :current, :user, :home, ... then :default
+p.find(:user)      # force one specific map
+p.find(:all)       # == find_all
+p.exists?          # find then File.exist?
+```
+
+The returned `Path` is annotated with `where` (the map that matched) and
+`original` (a copy of the unlocated path) so you can trace how a file was
+resolved:
+
+```ruby
+found = p.find
+found.where     # e.g. :user
+found.original  # the original unlocated Path
+```
+
+## The default map order
+
+The built-in maps and their order (13 entries) are:
+
+| # | Map | Template |
+|---|---|---|
+| 1 | `:current` | `{PWD}/{TOPLEVEL}/{SUBPATH}` |
+| 2 | `:user` | `{HOME}/.{PKGDIR}/{TOPLEVEL}/{SUBPATH}` |
+| 3 | `:home` | `{HOME}/{TOPLEVEL}/{PKGDIR}/{SUBPATH}` |
+| 4 | `:local` | `/usr/local/{TOPLEVEL}/{PKGDIR}/{SUBPATH}` |
+| 5 | `:global` | `/{TOPLEVEL}/{PKGDIR}/{SUBPATH}` |
+| 6 | `:usr` | `/usr/{TOPLEVEL}/{PKGDIR}/{SUBPATH}` |
+| 7 | `:scout_essentials_lib` | `<gem libdir>/{TOPLEVEL}/{SUBPATH}` |
+| 8 | `:lib` | `{LIBDIR}/{TOPLEVEL}/{SUBPATH}` |
+| 9 | `:fast` | `/fast/{TOPLEVEL}/{PKGDIR}/{SUBPATH}` |
+| 10 | `:cache` | `/cache/{TOPLEVEL}/{PKGDIR}/{SUBPATH}` |
+| 11 | `:bulk` | `/bulk/{TOPLEVEL}/{PKGDIR}/{SUBPATH}` |
+| 12 | `:default` | `{PWD}/{TOPLEVEL}/{SUBPATH}` |
+| 13 | `:tmp` | `/tmp/{PKGDIR}/{TOPLEVEL}/{SUBPATH}` |
+
+A per-instance `map_order` is recomputed lazily as
+`(Path.map_order & available_maps) + (remaining maps, in reverse key order)`.
+This is why a new map registered with `Path.add_path` (which clears the
+class-level order) or `p.add_path` (which clears the instance order) ends up
+at the *end* of the effective order, after the built-ins:
+
+```ruby
+p = Path.setup('data/config.yaml')
+p.add_path(:onlymine, '/x/{SUBPATH}')
+p.map_order # => [:current, :user, ..., :tmp, :onlymine]
+```
+
+`*_lib` maps (e.g. `:scout_essentials_lib`) are regular entries built from
+the gem's own libdir at load time; `{LIBDIR}` in a template resolves to the
+`libdir` annotation or, failing that, the directory of the calling library.
+
+## `follow`: applying a map without searching
+
+`follow(map)` applies one template, no matter whether the target exists:
+
+```ruby
+Path.setup('data/config.yaml').follow(:user)
+# => "$HOME/.scout/data/config.yaml"
+```
+
+Placeholders available in templates: `{PWD}`, `{HOME}`, `{PKGDIR}`,
+`{RESOURCE}`, `{TOPLEVEL}`, `{SUBPATH}`, `{BASENAME}`, `{PATH}`, `{LIBDIR}`,
+`{MAPNAME}`, `{REMOVE}` (deletes itself and the following slash). A template
+without any placeholder gets `{PATH}` appended, so the *whole path* is used
+verbatim. A map value may be another map name (a `Symbol`), which is
+dereferenced until a String is found. When `map_name` is an unknown String,
+`follow` builds `<map_name>/{TOPLEVEL}/{SUBPATH}` on the fly — this is how
+`Scout.etc` (`"etc"`) resolves to `$HOME/.scout/etc`:
+
+```ruby
+Scout.etc['path_maps'].find  # => "$HOME/.scout/etc/path_maps"
+```
+
+## Configuring the search
+
+```ruby
+Path.add_path(:mymap, '/my/{TOPLEVEL}/{SUBPATH}')   # effective in map_order
+Path.prepend_path(:first, '/first/{TOPLEVEL}/{SUBPATH}')
+Path.append_path(:last, '/last/{TOPLEVEL}/{SUBPATH}')
+
+p = Path.setup('data/config.yaml')
+p.add_path(:onlymine, '/x/{SUBPATH}')  # per-instance; recomputes map_order
+p.path_maps                            # a dup of Path.path_maps
+p.map_order                            # instance order, :onlymine included
+```
+
+`Path.load_path_maps(filename)` reads a YAML mapping of `where => location`
+and registers each with `add_path`; at boot,
+`Scout.etc['path_maps']` (i.e. `$HOME/.scout/etc/path_maps`) is loaded this
+way, so users can add search locations without code changes.
+
+## Finding every candidate: `find_all` / `glob_all`
+
+```ruby
+Path.setup('data/config.yaml').find_all
+# => every location in map_order where the file exists (uniqued)
+
+Path.setup('data/*').glob_all
+# => Path#glob over each map result; annotated with original/where
+```
+
+`glob` on a `located?` path calls `Dir.glob` directly; on an unlocated path
+it delegates to `glob_all`.
+
+## Reversing the process: `identify` and `relocate`
+
+`Resource.identify(path)` maps a located path back to an unlocated one by
+matching each map template as a regexp (dropping `:current`); the shortest
+candidate wins, and `$HOME` is folded back to `~`. `Resource.relocate(path)`
+returns the existing path if it exists, otherwise identifies and re-finds it.
+
+```ruby
+Resource.identify(File.join(ENV['HOME'], '.scout', 'data', 'config'))
+# => "data/config"
+Resource.relocate(File.join(ENV['HOME'], '.scout', 'data', 'config'))
+# => re-resolved through find
+```
+
+## Digest names and `etc`/`tmp` helpers
+
+`Path#digest_str` produces a stable digest for a file or directory (used for
+caching); for a directory with more than 10 files it uses a count plus an
+MD5 of the file list, otherwise the MD5 of each file. See
+[Caching Results](../user/CachingResults.md).
+
+Resource helpers: `Scout.etc`, `Scout.tmp`, `Scout.share`, ... are
+`Path#method_missing` segment builders over `Scout`'s own path (`Scout` is
+itself a Resource with `pkgdir 'scout'`), so they produce unlocated
+sub-paths that `find`/`follow(:user)` resolve under `$HOME/.scout`:
+
+```ruby
+Scout.etc                # => "etc"           (unlocated, pkgdir Scout)
+Scout.etc.find           # => "$HOME/.scout/etc"
+Scout.etc['path_maps'].find # => "$HOME/.scout/etc/path_maps"
+```
+
+`Scout.etc` is not a statically defined method: it resolves through
+`Resource#method_missing` (resource.rb:69) into `Path#method_missing`
+(path.rb:45) segment building, the same mechanism as any other segment
+(`Scout.tmp`, `Scout.share`, ...).
+
+## Related
+
+- [Producing Resources](../user/ProducingResources.md) — claims and produce
+  on top of `find`.
+- [Working with Files](../user/WorkingWithFiles.md) — `Open` I/O that
+  consumes `Path`s.
+- [Architecture](Architecture.md) for the module dependency graph.

data/doc/developer/PersistenceAndResources.md

@@ -0,0 +1,119 @@
+# Persistence and Resources
+
+This page documents the developer contracts behind two mechanisms that both
+write files: **`Persist`** (caching the result of a computation) and
+**`Resource#produce`** (materializing a declared resource). Both are
+lock-protected and rely on `Open.sensible_write` for atomic output.
+
+## `Persist.persist`
+
+```ruby
+Persist.persist(name, type, options = {}, &block)
+```
+
+`Persist.persist` runs `block` and stores its serialized result under a
+cache path, unless the cache is already valid, in which case the stored
+value is deserialized and returned without running the block.
+
+- `name` — usually a cache path; `options[:persist_path]` (or `:path`)
+  overrides it.
+- `type` — a serialization type; see the table in
+  [Caching Results](../user/CachingResults.md).
+- With no block the call is a lookup.
+- `path_hash`/`:path` options name the cache file; `key:` appends
+  `[key]` to it.
+- `:check` — a file, an Array of files or a Proc; when any of them is newer
+  than the cache (`Path#outdated?`) the cache is recomputed. `:check`
+  requires the cache path itself to be a `Path` (a plain `String` cache
+  path raises `NoMethodError`).
+- `:update` — `true` (always recompute), a `Time` or a `Numeric` number of
+  seconds (recompute when the cache file is older), or a `Path` whose mtime
+  is compared with the cache's.
+- `:memory` type + `MEMORY_CACHE` — in-process memoization with the same
+  API; `Persist.memory("key", key: k) { ... }` is the shorthand.
+- `KeepLocked` — when passed, the persist lock is held across the whole
+  read/serialize cycle, preventing readers from seeing a half-written cache
+  file.
+- `:canfail` — on failure (of the computation, not of the lookup) the error
+  is swallowed and `nil` is returned instead of raising.
+- **Error path**: unless `DontPersist` is raised, a failing block leaves no
+  partial cache file behind — the target is removed.
+- `no_load: true` (or a `TrueClass` value) returns the cache **path**
+  instead of the content.
+
+The serialization drivers are not constants: they are entries in the
+accessor hashes `Persist.save_drivers` and `Persist.load_drivers`, both
+keyed by type symbol. You can add a type by inserting a Proc into both
+hashes. **No serialization suffix is ever appended to the filename**: two
+calls with different types and the same `name`/`persist_path` collide on
+the same file.
+
+## Lock namespaces
+
+Three lock directories exist, and they are **not** interchangeable:
+
+| Lock dir | Owner | Naming of the lock file |
+|----------|-------|------------------------|
+| `Persist.lock_dir` (`tmp/persist_locks`) | `Persist.persist` | derived from the cache path |
+| `Resource.default_lock_dir` (`tmp/produce_locks`) | `Resource#produce` | `TmpFile.tmp_for_file(final_path, dir: lock_dir)` |
+| `Open.sensible_write_lock_dir` (`tmp/sensible_write_locks`) | `Open.sensible_write` | derived from the target path |
+
+The locking primitive is `Open.lock(filename, &block)` on the vendored
+`Lockfile`. **`Persist.lock` does not exist.**
+
+## `Resource` and `Path`: one mechanism
+
+`Path` is a String subclass extended with `Annotation`; `Resource` is a
+module extended with `Annotation` and `extend`ed by resource packages.
+Because both are annotation-based, a `Path` produced from a resource module
+carries the module itself as its `pkgdir` annotation, and `Path#produce`
+delegates to `pkgdir.produce(self, force)`:
+
+```ruby
+module MyApp
+  extend Resource
+  annotation :pkgdir
+  self.pkgdir = 'myapp'
+end
+```
+
+- `claim(path, type, content = nil, &block)` — `type` is mandatory and
+  positional; valid types are `:string`, `:url`, `:proc`, `:rake`,
+  `:install`. `:csv` raises `"TSV/CSV Not implemented yet"` when produced.
+- A `:proc` claim of arity 0 is called with no arguments; arity 1 receives
+  the final path. The return value is dispatched: `String`/`IO`/`StringIO`
+  → written; `Array` → joined with newlines; `TSV`/`TSV::Dumper` → dumper
+  stream (scout-gear only; **a `nil` return hits the `when TSV` branch
+  first** and raises `NameError` without scout-gear); `nil` → nothing
+  written.
+- Producing an unclaimed path falls through to `.gz`/`.bgz` variants when
+  those are claimed — asymmetrically (plain → `.gz` works, `.gz` → plain
+  does not).
+- `@produced` on the `Path` latches tri-state: `true` after a successful
+  produce, `false` after `ResourceNotFound`, the exception after a failure;
+  later calls return/raise that value instead of retrying, unless `force`.
+- `Path#read`/`#open`/`#list` produce first (`#list` uses
+  `produce_and_find('list')`); `Path#write` does not.
+- `Resource.sync(path, map, options)` — module-level helper (no `Path#sync`)
+  that re-materializes resource files found elsewhere into a local map
+  directory.
+
+## Rake and software
+
+- `rake_dirs` directories make a package "have rake": `has_rake?(path)` /
+  `rake_for(path)` (longest prefix match). Production runs the rake task
+  through `ScoutRake`, which forks; a `Don't know how to build task`
+  message triggers a walk-up retry, then `ResourceNotFound`.
+- `:install` claims run `Resource.install(name, software_dir, options)`,
+  sourcing `share/software/install_helpers` and honoring spec keys such as
+  `:git`, `:src`, `:url`, `:jar`, `:extra`. `Resource.set_software_env`
+  exports the installed binaries (run once at load time on the default
+  `software` dir).
+
+## Related
+
+- [Caching Results](../user/CachingResults.md) — user-facing guide to
+  `Persist`.
+- [Producing Resources](../user/ProducingResources.md) — user-facing guide
+  to claims and `produce`.
+- [Architecture](Architecture.md) — module graph and lock overview.