scout-essentials 1.8.7 → 1.9.0

data/research/io-paths-analysis.md

@@ -0,0 +1,228 @@
+# Investigation: File I/O, Paths, and Temporary Files
+
+**Status:** Non-normative investigation artifact. May be outdated.
+
+## Scope
+Open, Path, TmpFile, and their interactions.
+
+---
+
+## TmpFile
+
+### What it is
+A minimal module providing temporary file/directory paths and scoped helpers.
+
+### Key methods
+- `TmpFile.tmpdir` — base directory (default `~/tmp/scout/tmpfiles`).
+- `TmpFile.user_tmp(subdir)` — user-scoped tmp directory.
+- `TmpFile.random_name(prefix, max)` — prefix + random integer.
+- `TmpFile.tmp_file(prefix, max, dir)` — path inside dir (defaults to tmpdir).
+
+### Scoped helpers (create-then-cleanup)
+- `with_file(content=nil, erase=true, options={})` — create a temporary file,
+  optionally pre-populate it with content (String, IO, or StringIO), yield the
+  path, then delete it.
+  - IO content is read in chunks via `readpartial`.
+  - Options: `:prefix`, `:max`, `:tmpdir`, `:extension`.
+- `with_dir(erase=true, options={})` — create a temp directory, yield, cleanup.
+- `in_dir(*args)` — create temp dir, `chdir` into it, yield, cleanup.
+
+### Persistence path helper
+- `tmp_for_file(file, tmp_options={}, other_options={})` — generates a stable
+  cache filename from a logical filename + options.
+  - Replaces `/` with `·` (SLASH_REPLACE) to make a single filename.
+  - Truncates names over 150 chars, appends a digest.
+  - Appends a digest of `other_options` for parameter-variant uniqueness.
+  - Used pervasively by Persist to generate cache paths.
+
+### Design insight
+TmpFile is the **fundamental cache-path primitive** — it turns logical names
+into stable filesystem paths. It does not itself manage caching or locking;
+it just builds deterministic paths.
+
+---
+
+## Open
+
+### What it is
+A unified, high-level file/stream/remote I/O module. It wraps plain File I/O,
+streaming helpers, remote fetching (wget/ssh), atomic writes, locking,
+compression auto-handling, and filesystem utilities (grep, sort, collapse,
+wc, head, tail, mv, cp, rm).
+
+### Sub-modules
+- `Open/stream.rb` — streaming, tee streams, `sensible_write`, monitor streams.
+- `Open/util.rb` — existence checks, remote detection, compression detection,
+  realpath, gzip/bgzip/zip helpers, grep, sort, wc, head, tail, mv, cp, rm,
+  is_gzip?/is_bgzip?/is_zip? detection.
+- `Open/remote.rb` — remote detection, `wget` download, `ssh` file operations
+  (exists, open, read, write, mv, cp, upload, download), scp.
+- `Open/lock/lockfile.rb` — cross-process locking via lock files (a port of
+  the Lockfile gem). Supports `Open.lock(file) { ... }`.
+- `Open/final.rb` — streaming wrappers with guaranteed resource cleanup
+  (`stream_without_close`, etc.).
+
+### Key patterns
+
+#### Auto-decompression
+`Open.read("data.tsv.gz")` automatically detects `.gz`, `.bgz`, `.zip` and
+decompresses transparently. `Open.open` also auto-detects compression on the
+fly via magic byte detection.
+
+#### Remote access
+- `Open.remote?(path)` — detects `http://`, `https://`, `ssh:` prefixes.
+- `Open.open(remote_url)` — downloads via `wget` to a cache, then opens.
+- SSH-style: `Open.exists?("server:file")`, `Open.read("server:file")` etc.
+
+#### Atomic writes (sensible_write)
+`Open.sensible_write(file, content)` writes to a temporary file then renames
+atomically. Uses a lock to prevent concurrent writes to the same target.
+This is the safe way to write files that might be read concurrently.
+
+#### Cross-process locking
+`Open.lock(lockfile) { ... }` — file-based locks using `flock` + lock files.
+Used by Persist and Resource for concurrent-safe resource production.
+
+#### Stream handling
+- `Open.open(path) { |io| ... }` — streaming file open with auto-decompression.
+- `Open.stream(io)` — wraps an IO into a ConcurrentStream for lifecycle safety.
+- `Open.tee_stream(io)` — splits a stream into multiple consumers.
+- `Open.consume_stream(io, close=true)` — reads to end and closes.
+
+#### Grep/sort/wc utilities
+`Open.grep(stream, pattern)`, `Open.sort(stream)`, `Open.wc(stream)`,
+`Open.head(stream, n)`, `Open.collapse_stream(stream)` — stream-processing
+utilities that work on file paths or IO objects.
+
+### Compression detection
+- `Open.is_gzip?(file)` — magic bytes `\x1f\x8b`.
+- `Open.is_bgzip?(file)` — magic bytes `\x1f\x8b` + extra field in header.
+- `Open.is_zip?(file)` — magic bytes `PK\x03\x04`.
+- `Open.gzip?(file)` — file extension `.gz`.
+- `Open.bgzip?(file)` — file extension `.bgz`.
+- `Open.zip?(file)` — file extension `.zip`.
+
+---
+
+## Path
+
+### What it is
+A path abstraction layered on top of Annotation. Paths are annotated Strings
+that know how to **find** themselves across a configurable set of locations
+(path maps).
+
+### Annotations
+- `pkgdir` — logical package name (default 'scout').
+- `libdir` — library directory for resolving `lib` maps.
+- `path_maps` — hash of named location templates.
+- `map_order` — ordered list of map names to try.
+- `where` — which map resolved successfully during `find`.
+- `original` — the path before resolution.
+
+### Setup
+- `Path.setup(string, pkgdir, libdir, path_maps, map_order)` — extends a plain
+  String with Path annotations.
+- A Path **is** a String; it responds to all String methods plus path-specific
+  ones.
+
+### Path maps system
+Path maps are templates with placeholders:
+- `{PKGDIR}`, `{HOME}`, `{PWD}`, `{TOPLEVEL}`, `{SUBPATH}`, `{PATH}`,
+  `{BASENAME}`, `{LIBDIR}`, `{RESOURCE}`, `{MAPNAME}`.
+
+Default maps:
+```ruby
+:current => "{PWD}/{TOPLEVEL}/{SUBPATH}"
+:home    => "{HOME}/{TOPLEVEL}/{PKGDIR}/{SUBPATH}"
+:user    => "{HOME}/.{PKGDIR}/{TOPLEVEL}/{SUBPATH}"
+:global  => '/{TOPLEVEL}/{PKGDIR}/{SUBPATH}'
+:lib     => '{LIBDIR}/{TOPLEVEL}/{SUBPATH}'
+# etc.
+```
+
+### Resolution: `find`
+```ruby
+path = Path.setup("data/file.tsv", "mypkg")
+path.find  # tries each map in map_order, returns first existing location
+```
+
+- If the path is already located (absolute, ~/, ./), `find` returns immediately.
+- Otherwise, it iterates `map_order`, applies each map template, and checks
+  existence (including `.gz`/`.bgz` alternatives).
+- The first match wins; `where` annotation records which map resolved.
+
+### Method-missing as path builder
+```ruby
+path = Path.setup("data", "mypkg")
+path.file   # → "data/file"  (as annotated Path)
+path.data.tsv  # → "data/data/tsv"
+```
+This works because `method_missing(name, prev)` calls `join(name, prev)`,
+which returns an annotated Path. This is the idiomatic Scout way to build
+nested paths.
+
+### Operators
+- `path / "subpath"` — alias for `join`.
+- `path["subpath"]` — alias for `join`.
+- `path.join("subpath")` — returns a new annotated Path.
+
+### located? check
+`Path.located?(path)` — returns true if path starts with `/`, `~/`, or `./`.
+These paths are treated as already resolved and are not subject to map
+resolution.
+
+### Path manipulation
+- `_parts` — split by `/`.
+- `_toplevel` — first path component.
+- `_subpath` — everything after the first `/`.
+- `set_extension(ext)` — swap file extension.
+- `set_extension` — used by `find_with_extension`.
+
+### find_with_extension
+```ruby
+path.find_with_extension(['tsv', 'csv'], produce: true)
+```
+Tries the path as-is, then tries each extension. Useful when the exact
+extension is not known.
+
+---
+
+## Cross-module interactions
+
+- **Path depends on Annotation** — extends Annotation; paths are annotated strings.
+- **Path depends on IndiferentHash** — path_maps is an IndiferentHash.
+- **Open depends on Path** — for path resolution in read/write operations.
+- **Open depends on ConcurrentStream** — for streaming safety.
+- **TmpFile depends on Open** — uses `Open.mkdir` to ensure directories exist.
+- **Resource depends on Path and Open** — resources are annotated paths that
+  can be produced on demand.
+- **Persist depends on TmpFile** — uses `tmp_for_file` to generate cache paths.
+
+---
+
+## Gotchas and warnings
+
+1. **Path.map_order is a class variable (@@map_order)** — modifying it globally
+   affects all Path instances. Use instance-level `@map_order` for per-resource
+   customization.
+2. **Path.method_missing sends `to_*` methods to super** — `path.to_s` works
+   normally, but any custom method starting with `to_` is passed through.
+3. **Open.remote? requires the full URL prefix** — bare hostnames without
+   `http://` or `ssh:` are treated as local paths.
+3. **Open.sensible_write lock files are not always cleaned up** — if the
+   process crashes during write, lock files may persist in
+   `tmp/sensible_write_locks`.
+4. **Path.find may return a path that doesn't exist** — if no map resolves to
+   an existing file, `find` returns the `:default` map result (which may not
+   exist). Always check `exist?` or `File.exist?` after `find`.
+5. **TmpFile.tmp_for_file generates Path-annotated strings** — the return value
+   is a Path, not a plain String, so it responds to `find`, `produce`, etc.
+6. **Open.open auto-decompresses based on file content, not just extension** —
+   if a file named `.tsv` actually contains gzipped data, Open.open will
+   still decompress it.
+7. **ConcurrentStream.setup on streams returned by Open** — when you use
+   `Open.open` with a block, the stream is automatically set up with
+   ConcurrentStream. When using `Open.read`, the stream is consumed and
+   closed.
+8. **Open.read on a directory** — returns an empty string or raises, depending
+   on the Ruby version.

data/research/persistence-resources-analysis.md

@@ -0,0 +1,244 @@
+# Investigation: Persistence, Resources, and Concurrency
+
+**Status:** Non-normative investigation artifact. May be outdated.
+
+## Scope
+Persist, Resource, ConcurrentStream, and their interactions.
+
+---
+
+## Persist
+
+### What it is
+A caching module that saves the result of expensive computations to disk (or
+memory), keyed by a deterministic cache path. It provides type-aware
+serialization and deserialization, multiple serialization drivers, and
+concurrent-safe production via file locking.
+
+### Core pattern: `persist` / `persist`-with-block
+```ruby
+result = Persist.persist("my_data", :string, :path => "/cache/my_data") do
+  expensive_computation()
+end
+```
+- If `/cache/my_data` exists, the stored value is loaded.
+- If not, the block is executed, and the result is saved.
+- The `:path` determines where on disk.
+- The type (`:string`, `:marshal`, `:array`, etc.) determines serialization.
+
+### Serialization types
+Supported by `serialize`/`deserialize`:
+- `:string`, `:text`, `:integer`, `:float`, `:boolean` — basic type coercion.
+- `:array` — newline-joined.
+- `:yaml`, `:json`, `:marshal` — standard serializers.
+- `:binary` — raw bytes.
+- `:file` — stores a path to another file (pointer).
+- `:annotation`, `:annotations` — via `Annotation.tsv`.
+- `<type>_array` — array of the inner type (e.g., `:integer_array`).
+- `:memory` — uses the in-memory `Persist::MEMORY` hash.
+- Custom drivers — registered via `save_drivers`/`load_drivers` hashes.
+
+### `persist` lifecycle
+1. Generate a cache path (via `persist_path` if not given).
+2. Check if the cache file exists.
+3. If it exists: load and return.
+4. If not: acquire a lock (`Open.lock`), check again (double-check locking),
+   execute the block, save the result, release the lock, return.
+
+### `persist_path` generation
+When `:path` is not explicitly provided, Persist builds one from the name and
+options using `TmpFile.tmp_for_file`:
+```ruby
+Persist.persist_path("my_data", :some_option => "value")
+# => ~/tmp/scout/tmpfiles/my_data·SOME_OPTION=VALUE
+```
+This means the cache path is deterministic for the same name + options,
+enabling automatic cache reuse.
+
+### Concurrency model
+- `Open.lock(cache_path + '.lock')` — uses lockfiles for cross-process safety.
+- Double-check: checks existence again inside the lock.
+- Multiple processes attempting to produce the same cache will block on the
+  lock, and the second will find the file already present.
+
+### `persist_tsv` (legacy)
+`Persist.persist_tsv(database, file, data, type)` is a convenience for caching
+TSV-style data. It wraps the persist pattern with TSV-specific serialization.
+In scout-essentials, this exists but the main TSV class is not included —
+this method is for integration with downstream TSV-using code.
+
+---
+
+## Resource
+
+### What it is
+A module for declaring **named resources** that can be produced on demand.
+A Resource module acts as a namespace: it defines a root path (where files
+live), claims (how to produce each file), and a path resolution scheme.
+
+### Anatomy of a Resource module
+```ruby
+module MyResource
+  extend Resource
+  self.claim Path.setup("data/file.tsv"), :proc do |path|
+    # produce the file at `path`
+  end
+end
+```
+- `extend Resource` — becomes a resource namespace.
+- `claim(path, type, content = nil, &block)` — declare how to produce a path.
+- `self.root` — the filesystem root for this resource.
+- `self.pkgdir` — package name for path resolution.
+- `self.subdir` — subdirectory under the root.
+
+### Claim types
+1. `:string` — write the string content to the path.
+2. `:url` — download the URL to the path.
+3. `:proc` — call the block; it returns String/IO/Array/TSV to write.
+   - `arity == 0` → block takes no args, returns data.
+   - `arity == 1` → block receives the output path.
+4. `:rake` — run a Rake task (with `ScoutRake.run`).
+5. `:install` — install software (uses `Resource.install`).
+6. `:csv` — (declared but not implemented — raises).
+
+### Production lifecycle (`produce`)
+1. Resolve the path to find its current location (`path.find`).
+2. If the file exists at the resolved location, return.
+3. If a claim exists for this path:
+   - Acquire a lock (`Open.lock`).
+   - Re-check existence (double-check).
+   - Execute the claim (write string, download URL, run proc, run rake, etc.).
+   - If the claim type is missing, try `.gz` / `.bgz` variants.
+4. On error: remove the partial file and re-raise.
+5. After production, reset the path's location cache.
+
+### Resource → Path integration
+The `Resource/path.rb` file adds `produce`, `produce_and_find`, `produce_with_extension`,
+`relocate`, `identify`, `open`, `read`, `write`, `list`, `exists?`, and
+`find_with_extension` methods to all Path objects.
+
+This means **any Path can be produced on-demand** if its `pkgdir` is a
+Resource module. This is the key integration: `path.produce` checks existence,
+and if the file is missing, triggers production.
+
+### Software installation (`Resource/software.rb`)
+- `Resource.install(url_or_spec, name, software_dir)` — downloads and extracts
+  software packages.
+- `set_software_env(software_dir)` — adds installed software to PATH and
+  LD_LIBRARY_PATH.
+- Used by the `:install` claim type.
+
+### Resource relocation and identification
+- `Resource.relocate(path)` — moves a resource to a better location based on
+  resource identification.
+- `Resource.identify(path)` — determines the canonical identity of a path.
+
+---
+
+## ConcurrentStream
+
+### What it is
+A mixin that augments IO-like objects (pipes from `CMD`, file streams from
+`Open`, in-memory streams) with concurrency-aware lifecycle management:
+thread/pid tracking, join/abort semantics, callback chains, and error
+propagation.
+
+### Core attributes
+- `threads` — threads that produce the stream's data.
+- `pids` — subprocess PIDs that produce the stream's data.
+- `callback` — Proc chain called on successful `join`.
+- `abort_callback` — Proc chain called on `abort`.
+- `autojoin` — if true, the stream auto-joins on close/eof.
+- `no_fail` — if true, suppress process failures (log instead of raise).
+- `pair` — a paired stream (e.g., stdout/stderr pair) that is aborted together.
+- `lock` — an associated lock released on join/abort.
+- `stream_exception` — exception captured during streaming; raised on join.
+- `joined`, `aborted` — state flags.
+
+### Lifecycle: join
+1. `join_threads` — wait for all producer threads; check `Process::Status` for
+   failures; raise `ConcurrentStreamProcessFailed` on failure (unless `no_fail`).
+2. `join_pids` — `Process.waitpid` for each PID; check exit status.
+3. Check `stream_exception` — raise if set.
+4. `join_callback` — call the callback chain.
+5. `close` — close the underlying IO.
+6. Release the lock.
+
+### Lifecycle: abort
+1. Set `stream_exception` if not already set.
+2. Call `abort_callback` chain.
+3. `abort_threads` — raise `Aborted` exception in each producer thread.
+4. `abort_pids` — send `SIGINT` to each PID.
+5. Abort the pair stream if present.
+6. Close the IO.
+7. Release the lock.
+
+### Callback chaining
+`ConcurrentStream.setup(stream, :callback => proc1)` then
+`ConcurrentStream.setup(stream, :callback => proc2)` results in a chained
+callback that calls `proc1` then `proc2`. This allows multiple subsystems to
+attach cleanup logic to the same stream without coordination.
+
+### Annotate (propagation)
+`ConcurrentStream#annotate(stream)` copies threads, pids, callback, etc. to
+another stream. Used by CMD when transforming streams (e.g., decompressing).
+
+### `process_stream` class method
+`ConcurrentStream.process_stream(stream, **kwargs) { ... }` — a convenience
+wrapper that sets up the stream, runs the block, and ensures proper
+join/abort in case of exception. Used by stream-consuming code for
+guaranteed cleanup.
+
+### AbortedStream
+`AbortedStream.setup(obj, exception)` — marks a stream as aborted, attaching
+the exception. Consumers can check `AbortedStream === stream` and access
+`stream.exception`.
+
+---
+
+## Cross-module interactions
+
+- **Persist depends on Open** — for file I/O, locking, and atomic writes.
+- **Persist depends on TmpFile** — for cache path generation.
+- **Resource depends on Open, Path, TmpFile** — for production, path
+  resolution, and locking.
+- **CMD depends on ConcurrentStream** — CMD.cmd returns ConcurrentStream-enhanced
+  IO objects that track the subprocess PID and threads.
+- **Open depends on ConcurrentStream** — Open.stream wraps IO objects with
+  ConcurrentStream for lifecycle safety.
+- **ConcurrentStream depends on IndiferentHash** — for `process_options`.
+- **Log::ProgressBar depends on CMD** — `guess_obj_max` uses `CMD.cmd("wc -l")`
+  to estimate total lines in a file.
+
+---
+
+## Gotchas and warnings
+
+1. **Persist.persist silently returns nil on failure** — if the block returns
+   `nil`, `Persist.save` returns immediately without writing. The cache file
+   is not created, so the next call will re-execute the block.
+2. **Persist MEMORY type is process-local** — `:memory` persistence uses a
+   global `Persist::MEMORY` hash, which is lost when the process exits. It is
+   suitable only for within-process caching.
+3. **Resource.produce fallback to .gz/.bgz** — if no claim matches, produce
+   tries `path + '.gz'` then `path + '.bgz'`. This is usually desired but can
+   cause confusing errors if you expected a specific extension.
+4. **ConcurrentStream.join is idempotent** — calling `join` on an already-joined
+   stream is safe (returns without re-joining), but the `@joined` flag is set
+   even if the join raised an exception.
+5. **ConcurrentStream callback chains can grow** — each `setup` call with a
+   `:callback` prepends to the chain. Long chains from repeated `setup` calls
+   may cause unexpected ordering.
+6. **CMD no_fail streams** — a stream with `no_fail: true` will log errors but
+   not raise. Reading from it may produce truncated output. Check
+   `stream.exit_status` if you need to verify success.
+7. **Open.lock requires the lock directory to exist** — if the directory does
+   not exist, the lock file cannot be created and the lock fails silently.
+8. **Resource :csv claim type raises NotImplementedError** — declared but
+   not implemented. If you encounter this, use `:proc` with manual CSV parsing.
+9. **Persist caching of nil results** — if the computation block returns `nil`,
+   no cache file is written, leading to re-computation on every call. Use a
+   sentinel value if nil is a valid result.
+10. **ConcurrentStream.pair abort propagation** — aborting a stream also aborts
+    its pair. If you only want to abort one side of a stdout/stderr pair, you
+    must detach the pair first (set `pair = nil`).

data/research/synthesis-report.md

@@ -0,0 +1,80 @@
+# Synthesis Report
+
+**Status:** Non-normative investigation artifact. May be outdated.
+
+## Purpose
+
+Cross-check all investigation artifacts for consistency, overlaps, and gaps.
+Produce a concrete file mapping from investigation content to target
+documentation files.
+
+---
+
+## Module → Concept → Documentation mapping
+
+| Module | Concept | User doc | Developer doc |
+|--------|---------|----------|---------------|
+| Annotation | Annotating objects | AnnotatingData.md | AnnotationSystem.md |
+| NamedArray | Named arrays | AnnotatingData.md | AnnotationSystem.md |
+| IndiferentHash | Key-indifferent hashes | AnnotatingData.md | AnnotationSystem.md |
+| Open | File I/O | WorkingWithFiles.md | PathResolution.md |
+| Path | Path resolution | WorkingWithFiles.md | PathResolution.md |
+| TmpFile | Temp files | WorkingWithFiles.md | PathResolution.md |
+| CMD | Running commands | RunningCommands.md | (covered in user doc) |
+| Log | Logging | LoggingAndProgress.md | (covered in user doc) |
+| ConcurrentStream | Stream lifecycle | HandlingStreams.md | StreamingModel.md |
+| Persist | Caching | CachingResults.md | PersistenceAndResources.md |
+| Resource | On-demand resources | ProducingResources.md | PersistenceAndResources.md |
+| SOPT | CLI options | CommandLineOptions.md | (covered in user doc) |
+| Misc | Utility functions | (cross-cutting) | DesignPrinciples.md |
+
+---
+
+## Target documentation structure
+
+```
+doc/
+    StartHere.md
+    Improvements.md
+    user/
+        AnnotatingData.md
+        WorkingWithFiles.md
+        RunningCommands.md
+        LoggingAndProgress.md
+        HandlingStreams.md
+        CachingResults.md
+        ProducingResources.md
+        CommandLineOptions.md
+        Cookbook.md
+    developer/
+        Architecture.md
+        DesignPrinciples.md
+        AnnotationSystem.md
+        PathResolution.md
+        PersistenceAndResources.md
+        StreamingModel.md
+```
+
+---
+
+## Writing order (by priority)
+
+1. **StartHere.md** — entry point
+2. **developer/Architecture.md** — mental map
+3. **developer/DesignPrinciples.md** — coding style
+4. **user/AnnotatingData.md** — Annotation foundation
+5. **user/WorkingWithFiles.md** — most used
+6. **user/RunningCommands.md** — CMD
+7. **user/LoggingAndProgress.md** — Log
+8. **user/HandlingStreams.md** — ConcurrentStream
+9. **user/CachingResults.md** — Persist
+10. **user/ProducingResources.md** — Resource
+11. **user/CommandLineOptions.md** — SOPT
+12. **user/Cookbook.md** — recipes
+13. **developer/AnnotationSystem.md** — internals
+14. **developer/PathResolution.md** — path maps
+15. **developer/PersistenceAndResources.md** — caching internals
+16. **developer/StreamingModel.md** — stream internals
+17. **Improvements.md** — recommendations
+18. **research/** — curate SHARED artifacts
+19. **Validation** — check links, coverage

data/scout-essentials.gemspec

@@ -2,11 +2,11 @@
 # DO NOT EDIT THIS FILE DIRECTLY
 # Instead, edit Juwelier::Tasks in Rakefile, and run 'rake gemspec'
 # -*- encoding: utf-8 -*-
-# stub: scout-essentials 1.8.7 ruby lib
+# stub: scout-essentials 1.9.0 ruby lib
 
 Gem::Specification.new do |s|
   s.name = "scout-essentials".freeze
-  s.version = "1.8.7".freeze
+  s.version = "1.9.0".freeze
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0".freeze) if s.respond_to? :required_rubygems_version=
   s.require_paths = ["lib".freeze]
@@ -26,18 +26,28 @@ Gem::Specification.new do |s|
     "README.md",
     "Rakefile",
     "VERSION",
-    "doc/Annotation.md",
-    "doc/CMD.md",
-    "doc/ConcurrentStream.md",
-    "doc/IndiferentHash.md",
-    "doc/Log.md",
-    "doc/NamedArray.md",
-    "doc/Open.md",
-    "doc/Path.md",
-    "doc/Persist.md",
-    "doc/Resource.md",
-    "doc/SimpleOPT.md",
-    "doc/TmpFile.md",
+    "doc/Improvements.md",
+    "doc/StartHere.md",
+    "doc/developer/AnnotationSystem.md",
+    "doc/developer/Architecture.md",
+    "doc/developer/Configuration.md",
+    "doc/developer/CoreUtilities.md",
+    "doc/developer/DesignPrinciples.md",
+    "doc/developer/ErrorHandling.md",
+    "doc/developer/LockingAndConcurrency.md",
+    "doc/developer/PathResolution.md",
+    "doc/developer/PersistenceAndResources.md",
+    "doc/developer/StreamingModel.md",
+    "doc/user/AnnotatingData.md",
+    "doc/user/CachingResults.md",
+    "doc/user/CommandLineOptions.md",
+    "doc/user/Cookbook.md",
+    "doc/user/HandlingStreams.md",
+    "doc/user/LoggingAndProgress.md",
+    "doc/user/ProducingResources.md",
+    "doc/user/RemoteData.md",
+    "doc/user/RunningCommands.md",
+    "doc/user/WorkingWithFiles.md",
     "lib/scout-essentials.rb",
     "lib/scout/annotation.rb",
     "lib/scout/annotation/annotated_object.rb",
@@ -104,6 +114,18 @@ Gem::Specification.new do |s|
     "lib/scout/simple_opt/parse.rb",
     "lib/scout/simple_opt/setup.rb",
     "lib/scout/tmpfile.rb",
+    "research/annotations-data-analysis.md",
+    "research/behavior-probes.md",
+    "research/commands-streaming-analysis.md",
+    "research/design-philosophy-analysis.md",
+    "research/doc-audit-findings.md",
+    "research/ecosystem-attribution.md",
+    "research/implementation-inventory-core.md",
+    "research/implementation-inventory-open.md",
+    "research/implementation-inventory-path-persist-resource.md",
+    "research/io-paths-analysis.md",
+    "research/persistence-resources-analysis.md",
+    "research/synthesis-report.md",
     "scout-essentials.gemspec",
     "share/color/color_names",
     "share/color/diverging_colors.hex",
@@ -159,7 +181,7 @@ Gem::Specification.new do |s|
   ]
   s.homepage = "http://github.com/mikisvaz/scout-essentials".freeze
   s.licenses = ["MIT".freeze]
-  s.rubygems_version = "3.7.0.dev".freeze
+  s.rubygems_version = "3.7.2".freeze
   s.summary = "Scout essential tools".freeze
 
   s.specification_version = 4

data/test/scout/open/test_remote.rb

@@ -26,10 +26,9 @@ class TestOpenRemote < Test::Unit::TestCase
     teardown
     sss 0
     assert_raises ConcurrentStreamProcessFailed do
-      stream = Open.open("ftp://ftp.ncbi.nlm.nih.gov/pub/geo/DATA/SOFT/GDS/GDS3148.soft.gz", nocache: true)
+      stream = Open.open("ftp://ftp.ncbi.nlm.nih.gov/pub/geo/DATA/SOFT/GDS/GDS3148.soft.gz", nocache: true, no_fail: false)
       stream.read
     end
   end
-
 end