scout-essentials 1.8.8 → 1.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2ccc931c0ac9a43124ba25f56b1d1e940ee2a00504c37d530df92949bd8a104e
-  data.tar.gz: 4fab60400647db1a7ceb761c7e59c5900e22630afacbb2516467483f07dff402
+  metadata.gz: 526dfca4b8e68e08295052cb58f86b37b346857f0eb80ada50406b2683921de6
+  data.tar.gz: b2babc599d88a5e429fbcc1aad07e3f7d62483c3a73c1b56492a5847f7238319
 SHA512:
-  metadata.gz: 172b38d8cb18f249f322429df932379c71b4b5067cc92ac26bf26e93c9fba486da6a0e0e51873387d79e0fdc46f6697fc999eaaf6fb93daa99e29b405b19680e
-  data.tar.gz: 49ae1f7eca956c1fffd420efc34b2c2b700aaf3f66a780ccb34acabc02f4d598420df64bb5552a6b759fc0451229e77c4c56cd11459b3352965a5f4f663fbfc2
+  metadata.gz: 5041b9210824ce7add63097c50f67745af42c212197c33816ddef4a11e187e09c75d3e342b4d9e56d4a5e6132717cc5cef078518540e74494d98f7deb819c28b
+  data.tar.gz: 695290651c1db53bf9bfcd93f1567126f4eaba1c494d46b0cc6c4bbb221c4fcdd924a5c29d0ca25d8173b2b67fdbac46cfc344d59e9d28f58f8d30f19b71ea16

data/.vimproject

@@ -6,20 +6,34 @@ scout-essentials=/$PWD filter="*.rb *.txt *.md *.conf *.yaml" {
 
   review
   document.rb
+  documentation
+
+  cmd
+
+  timeout
  }
  doc=doc{
-  Annotation.md
-  CMD.md
-  ConcurrentStream.md
-  IndiferentHash.md
-  Log.md
-  NamedArray.md
-  Open.md
-  Path.md
-  Persist.md
-  Resource.md
-  SimpleOPT.md
-  TmpFile.md
+  Improvements.md
+  StartHere.md
+  user=user{
+   AnnotatingData.md
+   CachingResults.md
+   CommandLineOptions.md
+   Cookbook.md
+   HandlingStreams.md
+   LoggingAndProgress.md
+   ProducingResources.md
+   RunningCommands.md
+   WorkingWithFiles.md
+  }
+  developer=developer{
+   AnnotationSystem.md
+   Architecture.md
+   DesignPrinciples.md
+   PathResolution.md
+   PersistenceAndResources.md
+   StreamingModel.md
+  }
  }
  lib=lib {
   scout-essentials.rb

data/README.md

@@ -1,136 +1,107 @@
 # scout-essentials
 
-scout-essentials is the core library of the Scout framework. It provides a small, focused set of primitives used across the rest of the Scout ecosystem: process/stream management, file and path utilities, persistence and caching, logging and progress reporting, lightweight annotations for objects, and simple option parsing. The additional, domain-level functionality lives in companion packages such as `scout-gear`, `scout-ai`, etc. — see the mikisvaz GitHub account for those repositories.
+scout-essentials is a small Ruby library (gem `scout-essentials`, [VERSION](VERSION)) that provides the shared substrate the other Scout repositories are built on: path and resource resolution, persistence/caching, file and stream I/O, command execution, streaming, annotations, logging and progress reporting, configuration, CLI-option parsing, remote data access and locking.
 
-This README points you to the key modules, shows quick usage patterns and explains where to find more detailed documentation in the `doc/` directory.
+It deliberately contains no CLI executable, no TSV handling, no workflow engine and no job scheduler. Runtime dependencies are only `term-ansicolor`, `yaml`, `rake` and `listen` ([scout-essentials.gemspec](scout-essentials.gemspec)); it depends on no other scout-family or rbbt-family gem at runtime.
 
-## Overview
+## Central abstractions
 
-Core capabilities included in scout-essentials:
+| Abstraction | What it is |
+|---|---|
+| `Path` / `Resource` | `Path` is a `String` carrying annotations that resolves logical names (`data/config.yaml`) to real locations through configurable *path maps*; `Resource` modules declare claims over those paths and `produce` them on demand |
+| `Persist` | typed on-disk caching: `Persist.persist(key, type) { ... }` with per-type serialization drivers, staleness invalidation (`:update`/`:check`), an in-memory cache and lock-protected recompute |
+| `Open` | one interface for local, compressed (`.gz`/`.bgz`/`.zip`) and remote (`http(s)://`, `ssh:`) file and stream I/O, including atomic `sensible_write` and the `wget`/`rsync` remote cache |
+| `CMD` | the subprocess layer: `CMD.cmd` for shell and no-shell commands, stdin piping, timeouts, stderr capture and external-tool bootstrap |
+| `ConcurrentStream` | an `IO` extended with the producer threads/pids that feed it, plus join/abort callbacks — the lifecycle layer behind every pipe |
+| `Annotation` / `AnnotatedObject` / `AnnotatedArray` | a mixin that attaches named metadata to ordinary Ruby objects (Strings, Arrays, Hashes) without changing their class; the mechanism `Path`, `Resource` and `NamedArray` are built on |
+| `Log` / `Log::ProgressBar` | a severity-gated logger writing to STDERR (constants `Log::DEBUG … Log::NONE`) plus stacked progress bars rendered in the same place; colors via `term-ansicolor` |
+| `Scout::Config` | a flat `key -> [tokens, value]` registry where, among the entries whose token matches the caller's context, the one with the lowest priority number wins |
+| `SOPT` | a minimal option parser: options declared as a single string, a **destructive** consumer that edits `ARGV` in place, no subcommands and no coercion beyond booleans |
+| `TmpFile` | scratch files and directories under `$HOME/tmp/scout`, with `·`-flattened names and a 150-character cap |
+| `IndiferentHash` | a `Hash` extended so String and Symbol keys are interchangeable, plus the options-processing helpers built on it |
+| `NamedArray` | Arrays with named positions (`NamedArray.setup(arr, [:a, :b])`), giving both positional and field-name access |
+| `Misc` | small load-bearing helpers: the `Misc.format` family, `timespan`, digests, `insist`, filesystem and process utilities |
+| `Lockfile` / `Open.lock` | `Open.lock(file, &block)` is the locking primitive, built on the vendored `Lockfile` at `lib/scout/open/lock/lockfile.rb`; there are three lock namespaces under `$HOME/.scout/tmp` |
 
-- Process and stream management with safe concurrency: `ConcurrentStream`, `CMD`
-- Robust file/stream I/O and remote access: `Open`
-- File/path abstraction and package-oriented lookup: `Path`, `Resource`
-- Atomic persistence and caching: `Persist`, `TmpFile`
-- Logging, color output and progress reporting: `Log`
-- Lightweight typed annotations on arbitrary objects: `Annotation`, `NamedArray`
-- Flexible indifferent Hash and option helpers: `IndiferentHash`, `SimpleOPT` (SOPT)
+`require 'scout-essentials'` is the only entry point (`lib/scout-essentials.rb`); it loads the modules above plus `Open::NamedStream` and the exceptions (`Aborted`, `DontClose`, `ParameterException`, …). `NamedArray` and `Hook` are not auto-loaded and must be required explicitly.
 
-Each module is documented in the repository `doc/` directory; see the "Documentation" section below for direct links.
+## Getting started
 
----
-
-## Documentation
-
-Full module-level documentation is shipped in `doc/`. The most important documents are:
-
-- doc/Annotation.md — add typed annotations to objects and arrays; AnnotatedArray
-- doc/CMD.md — process execution, streaming, tool discovery and helper wrappers
-- doc/ConcurrentStream.md — concurrent stream lifecycle, joining, aborting and callbacks
-- doc/IndiferentHash.md — string/symbol indifferent Hash helpers and options utilities
-- doc/Log.md — logging, colors, fingerprinting and ProgressBar
-- doc/NamedArray.md — small record-like arrays with named fields and fuzzy matching
-- doc/Open.md — unified file/stream I/O, remote fetch (wget/ssh), atomic writes, sync
-- doc/Path.md — Path helpers, mapping, finding and extension utilities
-- doc/Persist.md — typed serialization, persistence/caching and `Persist.persist`
-- doc/Resource.md — resource production, claim/produce and rake-based producers
-- doc/SimpleOPT.md — small option parsing and usage generation (SOPT)
-- doc/TmpFile.md — temporary file/dir helpers and stable cache path generator
-
-Open those files for detailed API descriptions, examples and notes.
-
----
-
-## Quick start (examples)
-
-These short snippets show typical usage patterns — the docs in `doc/` contain more detail and examples.
-
-Annotation:
 ```ruby
-module Tag
-  extend Annotation
-  annotation :code, :note
-end
-
-s = "hello"
-Tag.setup(s, :code)    # s.code -> :code
-s2 = "other"
-s.annotate(s2)         # copies annotations
-```
+require 'scout-essentials'
 
-Open (reading a file, auto-decompress):
-```ruby
-content = Open.read("data.tsv.gz")
+Open.write('tmp/data.txt', "1\n2\n")   # creates parent directories
+Log.info "wrote 2 lines"
+Persist.persist('count', :integer) { Open.read('tmp/data.txt').lines.count }
 ```
 
-Persist (cache a computed value):
-```ruby
-value = Persist.persist("my-result", :json, dir: Path.setup("var/cache")) do
-  expensive_computation()
-end
-```
+Documentation lives in the [`doc/`](doc/StartHere.md) directory — start with [doc/StartHere.md](doc/StartHere.md).
 
-CMD + ConcurrentStream (run a pipeline):
-```ruby
-io = CMD.cmd("tail -n 100", :in => some_file_io, :pipe => true)
-io2 = CMD.cmd("grep foo", :in => io, :pipe => true)
-puts io2.read
-io2.join
-```
+## Relationship to the other Scout repositories
 
-Log + ProgressBar:
-```ruby
-Log::ProgressBar.with_bar(100, desc: "Working") do |bar|
-  100.times { bar.tick; work_item }
-end
-```
+scout-essentials is the lowest layer of the Scout stack; everything above it is implemented in separate repositories:
 
-Path + Resource:
-```ruby
-# Resource modules typically claim resources and produce them on demand.
-# Accessing a Path calls produce, so opening a resource Path triggers creation.
-p = Path.setup("share/data/myfile", 'mypkg')
-p.produce
-Open.read(p)
-```
+- **scout-gear** — adds TSV, `Workflow`/`Task`/`Step`, the `scout` CLI and `scout_commands/` dispatch, and the scheduler/HPC layer (SLURM/PBS/LSF, orchestrators) on top of essentials. It declares `scout-essentials` as a runtime dependency.
+- **scout-camp** — deploys workflows to the cloud: offsite/terraform/AWS provisioning around the CLI. Its bin loads scout-gear's `bin/scout`; the scheduler logic itself lives in gear.
+- **scout-rig** — consumes essentials' `Path`/`Resource` machinery at the code level (`require 'scout'`, `Path.add_path`); its gemspec does not declare the essentials edge.
+- **scout-ai** — reaches essentials transitively, via scout-rig and its own `require 'scout'`; it declares no direct dependency on essentials.
 
----
+Attribution facts worth keeping straight (verified against installed gem sources; see the attribution table in [Architecture](doc/developer/Architecture.md)):
 
-## Running tests
+| Concept | Where it actually lives |
+|---|---|
+| `TSV`, `TSV::Dumper`, `Annotation.tsv`, `Workflow`/`Task`/`Step`, `.info` files, the `scout` CLI and `scout_commands/` | scout-gear |
+| scheduler / HPC (SLURM, PBS, LSF, orchestrators) | scout-gear (scout-camp adds the cloud/offside deploy layer) |
+| `Misc.notify` / `Misc.send_email` and the `Bgzf` block API | legacy rbbt-util **only** — scout-essentials references them without a require or gemspec edge, so they are `NoMethodError`/`NameError` in an essentials-only install |
+| `deep_indifferent` | does not exist anywhere in the audited ecosystem |
+| rbbt-util `require_instead` shims | rbbt-util 6.0.5 redirects ~30 requires onto scout files; the *dependency direction* at gemspec level is UNVERIFIED |
 
-The test suite exercises the modules (unit tests use Test::Unit). To run the tests in this repository, use your normal Ruby test runner; the test files are under `test/` — examples:
+## Documentation
 
-```bash
-# from repository root
-ruby -Ilib test/scout/test_tmpfile.rb
-# or run the whole suite with your preferred runner
+22 Markdown pages under [`doc/`](doc/StartHere.md).
+
+Entry points:
+
+- [Start Here](doc/StartHere.md) — installation, the single require, what loads and what does not, what is *not* in this repo
+- [Improvements](doc/Improvements.md) — running log of known misbehaviours (bug log, not a roadmap)
+
+User guides:
+
+- [Annotating Data](doc/user/AnnotatingData.md) — attaching metadata to objects
+- [Working with Files](doc/user/WorkingWithFiles.md) — `Open` I/O, compression, grepping, atomic writes, locking, `TmpFile`
+- [Remote Data](doc/user/RemoteData.md) — HTTP(S)/FTP/SSH fetch, the URL cache, rsync helpers
+- [Running Commands](doc/user/RunningCommands.md) — `CMD.cmd`, piping, timeouts, stderr, tool bootstrap
+- [Handling Streams](doc/user/HandlingStreams.md) — using `ConcurrentStream` objects: join, abort, callbacks
+- [Caching Results](doc/user/CachingResults.md) — `Persist.persist`, serialization types, invalidation, in-memory cache
+- [Producing Resources](doc/user/ProducingResources.md) — the `claim` syntax and `produce`
+- [Logging and Progress](doc/user/LoggingAndProgress.md) — `Log`, severity, colors, fingerprints, progress bars
+- [Command-Line Options](doc/user/CommandLineOptions.md) — `SOPT`
+- [Cookbook](doc/user/Cookbook.md) — short executed recipes combining the modules
+
+Developer guides:
+
+- [Architecture](doc/developer/Architecture.md) — module dependency graph, load order, ecosystem boundaries
+- [Design Principles](doc/developer/DesignPrinciples.md) — composition by annotation, and the other repo-wide conventions
+- [Annotation System](doc/developer/AnnotationSystem.md) — the internals of annotations
+- [Path Resolution](doc/developer/PathResolution.md) — path maps, map order, `find`/`follow`, `Resource#method_missing`
+- [Persistence and Resources](doc/developer/PersistenceAndResources.md) — the `Persist` and `Resource#produce` write contracts
+- [Streaming Model](doc/developer/StreamingModel.md) — how `ConcurrentStream` is built and how errors travel
+- [Configuration](doc/developer/Configuration.md) — `Scout::Config` token priorities
+- [Error Handling](doc/developer/ErrorHandling.md) — the exception taxonomy and the control-flow signals
+- [Locking and Concurrency](doc/developer/LockingAndConcurrency.md) — the vendored `Lockfile` and the three lock namespaces
+- [Core Utilities](doc/developer/CoreUtilities.md) — `IndiferentHash`, `NamedArray`, `Misc`, `TmpFile` naming, `Hook`
+
+## Development and testing
+
+```sh
+bundle install
+rake test                                 # whole suite
+rake test TEST=test/scout/test_open.rb    # one file
+ruby -Ilib -Itest test/scout/test_misc.rb # run a file directly
 ```
 
-Tests in the suite show practical usages and edge cases for the provided utilities.
-
----
-
-## Related projects
-
-scout-essentials is intentionally focused on low-level primitives. Higher-level, domain-specific functionality is implemented in companion projects maintained on the mikisvaz GitHub account (look for repositories named `scout-gear`, `scout-ai`, etc.). Those packages build on what you find here to provide workflows, tools and integrations.
-
-GitHub: https://github.com/mikisvaz  
-Look for repositories that begin with `scout-` (e.g. `scout-gear`, `scout-ai`).
-
----
-
-## Contributing
-
-Contributions and improvements are welcome. Please follow the repository contribution guidelines (if present) or submit issues / pull requests on the project repository.
-
-If you extend or reuse code from this package in companion packages, prefer to keep core primitives here and implement domain logic in separate modules/packages (as done by the Scout ecosystem).
-
----
+The `test` task is defined in the [Rakefile](Rakefile) (`Rake::TestTask`, pattern `test/**/test_*.rb`, which matches 49 files on disk: 47 test files (one of which, `test/scout/log/test_color.rb`, is empty) plus the two `test_helper.rb` files, one of which itself defines `TestMiscHelper`). Tests are plain `Test::Unit` and mirror the `lib/scout/` layout. The gemspec is generated by juwelier from the Rakefile — regenerate it with `rake gemspec` rather than editing it.
 
 ## License
 
-See the repository LICENSE (if present) for licensing information.
-
----
-
-If you need help finding a specific API, open the corresponding file in `doc/` (listed above) or search the `lib/` tree for concrete implementations and tests in `test/` for usage examples.
+MIT-style license, Copyright (c) 2023 Miguel Vazquez — see [LICENSE.txt](LICENSE.txt).

data/VERSION

@@ -1 +1 @@
-1.8.8
+1.9.0

data/doc/Improvements.md

@@ -0,0 +1,226 @@
+# Improvements
+
+Actionable recommendations for improving scout-essentials code, discovered
+during the documentation effort. Each item includes a description, impact,
+and suggested action.
+
+## Status legend
+
+- 🔴 **Bug** — Incorrect behavior that should be fixed.
+- 🟡 **Improvement** — Works, but could be better.
+- 🟢 **Done** — Already resolved.
+
+---
+
+## High priority
+
+### I1. 🟡 ConcurrentStream join can deadlock with multi-stage pipelines
+
+When joining a multi-stage pipeline (producer → filter → consumer), if the
+producer is joined before the consumer has drained the pipe buffer, the
+join can deadlock because the pipe buffer fills and the producer blocks.
+
+**Impact:** Pipeline hangs in production.
+
+**Suggested action:** Document the correct join order (consumer first,
+producer last) and consider adding automatic consumer-joins-producer
+chaining via `autojoin`.
+
+---
+
+### I2. 🟡 Persist persistence_path can collide for short keys
+
+For short, filesystem-safe keys, Persist uses the key directly as the
+filename. If two modules use the same short key, the cache files collide.
+
+**Impact:** Subtle data corruption — one computation overwrites another's
+cache.
+
+**Suggested action:** Namespace cache paths by the calling module or
+library. Consider prefixing with a digest of the caller's file path.
+
+---
+
+### I3. 🟡 TmpFile cleanup not guaranteed on signal (SIGTERM/SIGKILL)
+
+TmpFile's `with_file` and `with_dir` use `ensure` blocks for cleanup.
+However, SIGTERM and SIGKILL bypass `ensure`, leaving temporary files on
+disk.
+
+**Impact:** Temporary file accumulation over time in long-running services.
+
+**Suggested action:** Register a PID-based temp directory and add a
+periodic cleanup task. Consider using `at_exit` for graceful shutdown.
+
+---
+
+### I4. 🟡 Open remote read via SSH not documented or tested
+
+Open supports reading from SSH paths (`user@host:path`), but this path is
+not well documented, and test coverage is limited.
+
+**Impact:** Users may not know SSH support exists; regressions may go
+undetected.
+
+**Suggested action:** Document SSH support in [Working with Files](user/WorkingWithFiles.md).
+Add tests. Add retry logic for network failures.
+
+---
+
+## Medium priority
+
+### I5. 🟡 No central registry for path maps
+
+Path maps are configured globally via `Path.path_maps` and
+`Path.map_order`. There is no central registry where applications can
+register their maps in a namespace-safe way.
+
+**Impact:** Map name collisions between applications.
+
+**Suggested action:** Introduce namespaced map registration: e.g.,
+`Path.add_path("myapp:data", "/opt/myapp/{PATH}")`.
+
+---
+
+### I6. 🟡 IndiferentHash deep_indifferent not recursive by default
+
+`IndiferentHash.setup` makes the top-level hash key-indifferent, but nested
+hashes are not automatically converted. `deep_indifferent` must be called
+explicitly.
+
+**Impact:** Surprising behavior when accessing nested keys with mixed
+string/symbol access.
+
+**Suggested action:** Document this clearly. Consider making
+`deep_indifferent` the default behavior or providing a clearer alternative.
+
+---
+
+### I7. 🟡 Path#find returns self even when file doesn't exist (for located paths)
+
+When a path is "located" (starts with `/`, `~`, or `./`), `find` returns
+`self` even if the file doesn't exist. This differs from non-located paths
+where `find` may return nil when no map resolves.
+
+**Impact:** Callers may assume `find` returns nil for nonexistent files.
+
+**Suggested action:** Document this behavior clearly. Consider adding a
+strict mode where `find` returns nil for nonexistent files regardless of
+location status.
+
+---
+
+### I8. 🟡 Resource production failures leave no trace
+
+When a Resource claim's production fails (e.g., download fails), the
+production is abandoned silently. No log message is emitted beyond the
+exception itself.
+
+**Impact:** Hard to debug resource production issues in production.
+
+**Suggested action:** Add `Log.warn` or `Log.error` calls in Resource
+production error paths.
+
+---
+
+### I9. 🟡 CMD tool registry not namespaced
+
+CMD tool registration (`CMD.add_tool`) uses a global registry. Multiple
+applications using scout-essentials share the same tool registry.
+
+**Impact:** Tool name collisions.
+
+**Suggested action:** Namespace tool registration by application or provide
+a scoped registry.
+
+---
+
+### I10. 🟡 No user-facing guide on custom Annotation modules
+
+While the annotation system is powerful, there was no user-facing guide on
+how to define and use custom annotation modules for application-specific
+metadata.
+
+**Impact:** Users may subclass instead of annotate, missing the idiom.
+
+**Suggested action:** The new [Annotating Data](user/AnnotatingData.md)
+guide covers this. It should be promoted more prominently.
+
+---
+
+## Low priority
+
+### I11. 🟢 Done — Documentation reorganized into three layers
+
+The documentation has been reorganized from a flat class-by-class structure
+into a three-layer structure (user/developer/research).
+
+---
+
+### I12. 🟡 Misc module is a catch-all
+
+The `Misc` module (`lib/scout/misc/`) contains many unrelated utilities:
+format, digest, math, system, hooks, monitor. This is a code smell.
+
+**Impact:** Hard to find utilities; import surface too broad.
+
+**Suggested action:** Consider splitting into separately named modules or
+documenting the sub-namespace usage more clearly.
+
+---
+
+### I13. 🟡 No CI configuration visible
+
+No CI configuration (`.travis.yml`, `.github/workflows/`, etc.) is visible
+in the repository.
+
+**Impact:** Tests may not run automatically on commit.
+
+**Suggested action:** Add a CI configuration to run tests on commit.
+
+---
+
+### I14. 🟡 Benchmarking suite absent
+
+There is no benchmarking suite to track performance across releases.
+
+**Impact:** Performance regressions may go unnoticed.
+
+**Suggested action:** Add a simple benchmark suite using `benchmark/ips`
+for key operations (Open.read, Persist.persist, Path.find).
+
+---
+
+### I15. 🟡 Annotation `setup` with frozen objects creates a duplicate
+
+When `setup` is called on a frozen object, it duplicates the object before
+extending it. The original frozen object is not modified.
+
+**Impact:** This is correct behavior but can surprise callers who expect
+the same object to be returned.
+
+**Suggested action:** Document this clearly in the Annotation documentation
+and user guides.
+
+---
+
+### I16. 🟡 Persist.persist can raise if cache_dir not writable
+
+If `Persist.cache_dir` is not writable, `persist` will fail with a
+permissions error.
+
+**Impact:** Can break production when cache_dir is on a read-only
+filesystem.
+
+**Suggested action:** Document this requirement and consider falling back
+to a temporary cache dir or raising a clearer error message.
+
+---
+
+## Summary
+
+| Category | Count |
+|----------|-------|
+| Bugs (🔴) | 0 |
+| Improvements (🟡) | 15 |
+| Done (🟢) | 1 |

data/doc/StartHere.md

@@ -0,0 +1,122 @@
+# Start Here
+
+`scout-essentials` is the lowest layer of the Scout stack: a Ruby gem of
+small, composable modules for paths, file I/O, command execution, caching,
+annotations, logging, progress bars and option parsing. It deliberately
+contains **no** CLI, no TSV handling, no workflow engine and no job
+scheduler — those live in other repos (see
+[Architecture](developer/Architecture.md), attribution table).
+
+## Installation and first require
+
+```ruby
+gem 'scout-essentials'      # Gemfile
+```
+
+```ruby
+require 'scout-essentials'
+Path.setup('tmp/data.txt')          # => annotated String
+Open.write('tmp/data.txt', "1\n2\n")
+Log.info "wrote 2 lines"
+```
+
+### What that single require actually loads
+
+`lib/scout-essentials.rb` is the whole surface — in this order:
+
+| require | gives you |
+| --- | --- |
+| `scout/exceptions` | `Aborted`, `DontClose`, `DontPersist`, `ParameterException`, … |
+| `scout/indiferent_hash` | `IndiferentHash` |
+| `scout/tmpfile` | `TmpFile` (also pulls `scout/open`) |
+| `scout/log` | `Log`, `Log::ProgressBar` |
+| `scout/path` | `Path`, `Resource`-adjacent path helpers |
+| `scout/simple_opt` | `SOPT` |
+| `scout/resource` | `Resource`, `Scout::Resource` |
+| `scout/resource/scout` | scout resource defaults (`pkgdir`, `subdir`) |
+| `scout/persist` | `Persist` |
+| `scout/config` | `Scout::Config` |
+
+Transitive pulls: `scout/open` (hence `Open`, `CMD`, `NamedStream`),
+`scout/annotation` (hence `Annotation`), `scout/misc` (via
+`scout/tmpfile`/`scout/open`). Notably **not** loaded by the umbrella
+require (verified by probe `tmp/rewrite_C/probe_01_requires.rb`):
+
+- `NamedArray` — `require 'scout/named_array'`
+- `Hook` (a **top-level** `module Hook`, *not* `Misc::Hook`) —
+  `require 'scout/misc/hook'`
+
+`Scout::Config` is loaded but you must use the namespaced name.
+
+## What is *not* in this repository
+
+| Not here | Where it lives |
+| --- | --- |
+| `bin/` CLI, `scout` executable, `scout_commands/` | scout-gear |
+| TSV, `TSV` parsing, `Annotation.tsv` | scout-gear |
+| `Workflow`, `Step`, job dependencies | scout-gear / scout-rig |
+| scheduler, HPC/DRMAA/LSF/SLURM orchestration | scout-workflows |
+| `notify`, `send_email` | rbbt-util |
+| Bgzf (`Open.bgunzip` is here, the Bgzf block API is not) | scout-gear |
+| `deep_indifferent` | nowhere — it does not exist |
+
+See the [attribution table in Architecture](developer/Architecture.md) for
+the full map.
+
+## Bug log
+
+[Improvements.md](Improvements.md) is the running log of known
+misbehaviours that were documented during the audit
+(`TmpFile.with_file` leaking on exception, `Persist` lock races, …). It is
+**not** a roadmap; it records what the code actually does versus what you
+might expect.
+
+## Development and testing
+
+```sh
+bundle install
+rake test          # runs the whole suite (49 test files)
+rake test TEST=test/scout/test_open.rb
+```
+
+The `test` task is defined at `Rakefile:26-28`; the suite lives under
+`test/scout/**` and mirrors the module layout (`test_path.rb`,
+`test_open.rb`, `test_persist.rb`, `test_annotation.rb`, …). Tests are
+plain `Test::Unit` and can be run directly:
+
+```sh
+ruby -Ilib -Itest test/scout/test_misc.rb
+```
+
+## Where to go next
+
+**User guides** — how to *use* the library:
+
+- [Annotating Data](user/AnnotatingData.md)
+- [Working with Files](user/WorkingWithFiles.md)
+- [Remote Data](user/RemoteData.md)
+- [Running Commands](user/RunningCommands.md)
+- [Handling Streams](user/HandlingStreams.md)
+- [Caching Results](user/CachingResults.md)
+- [Producing Resources](user/ProducingResources.md)
+- [Logging and Progress](user/LoggingAndProgress.md)
+- [Command-Line Options](user/CommandLineOptions.md)
+- [Cookbook](user/Cookbook.md)
+
+**Developer guides** — how the library *works*:
+
+- [Architecture](developer/Architecture.md)
+- [Design Principles](developer/DesignPrinciples.md)
+- [Annotation System](developer/AnnotationSystem.md)
+- [Path Resolution](developer/PathResolution.md)
+- [Persistence and Resources](developer/PersistenceAndResources.md)
+- [Streaming Model](developer/StreamingModel.md)
+- [Configuration](developer/Configuration.md)
+- [Error Handling](developer/ErrorHandling.md)
+- [Locking and Concurrency](developer/LockingAndConcurrency.md)
+- [Core Utilities](developer/CoreUtilities.md)
+
+**Research artifacts** (`../research/`) are non-normative supporting
+material: `research/doc_audit/*` holds the per-claim audit ledgers,
+`research/behavior-probes.md` the probe transcripts, and
+`research/implementation-inventory-*.md` the file:line inventories.