scout-essentials 1.7.1 → 1.8.0

data/doc/Path.md

@@ -0,0 +1,217 @@
+# Path
+
+Path is a lightweight path utility layered on top of the framework's Annotation system. It makes it easy to build, transform and locate resources by logical name across a variety of search maps (current, user, global, lib, etc.). Path objects are plain string values extended with Path behavior (via Path.setup or by extending instances). Many Open/Path helpers accept Path objects and will call `.find` / `.find_all` / `.produce_and_find` as needed.
+
+Key features:
+- Build and compose path strings fluently (join, /, method_missing).
+- Map logical names to physical locations using configurable path maps.
+- Find the first existing file across map order or list all matches.
+- Helpers for file extension manipulation, globbing, dirname/basename, sanitizing filenames.
+- Integration with Open and TmpFile utilities; supports annotation (pkgdir, libdir, map configuration).
+- Helpers for digest/MD5 summary for files and directories.
+
+---
+
+## Creating / wrapping Path values
+
+- Path.setup(str, pkgdir = nil)
+  - Convert a string into a Path (i.e., extend it with Path methods). Many test examples call `Path.setup("...")`.
+  - A Path is just a String extended with Path behavior and optional annotations (`pkgdir`, `libdir`, `path_maps`, `map_order`).
+
+Shortcuts on Path instances:
+- join(subpath, prevpath = nil) — join subpath to the Path (returns annotated Path)
+  - Aliases: [] and /
+  - Example:
+    ```ruby
+    p = Path.setup('/tmp')
+    p.join(:foo)        # => "/tmp/foo"
+    p[:bar, :foo]       # => "/tmp/bar/foo"
+    p / :foo            # => "/tmp/foo"
+    ```
+
+- method_missing is used to make `path.component` behave like join:
+  - `path.foo` → join("foo")
+  - Be careful: methods starting `to_` or blocks will not be treated as path components.
+
+---
+
+## Package / library defaults
+
+- Path.default_pkgdir and Path.default_pkgdir= — global default package dir (default `'scout'`).
+- Instance attributes:
+  - pkgdir — per-path override of package directory (defaults to Path.default_pkgdir).
+  - libdir — library directory (attempts to infer caller lib dir).
+  - path_maps — instance copy of global path maps (modifiable per-path).
+  - map_order — instance map order (derived from global map_order unless overridden).
+
+---
+
+## Path maps and find behavior
+
+Path maps let you define templates where logical paths can be found on disk. The module ships a sensible set of maps (current, user, global, usr, local, fast, cache, bulk, lib, tmp, …) and a default map order.
+
+- Path.path_maps — global IndiferentHash of map templates (strings containing placeholders like {TOPLEVEL}, {PKGDIR}, {SUBPATH}, {PATH}, {LIBDIR}, etc.)
+- Path.map_order / Path.basic_map_order — global order to search maps.
+- You can add or change maps:
+  - Path.add_path(name, map)
+  - Path.prepend_path(name, map)
+  - Path.append_path(name, map)
+  - For a Path instance you can also call add_path / prepend_path / append_path (instance-level override).
+
+Templates can reference:
+- {PKGDIR}, {HOME}, {RESOURCE}, {PWD}, {TOPLEVEL}, {SUBPATH}, {BASENAME}, {PATH}, {LIBDIR}, {MAPNAME}, and custom substitutions.
+
+Finding:
+- Path#follow(map_name = :default) — substitute template tokens and return the resulting path string (annotated). Does not check existence. Annotates result with `.where` and `.original` when requested (see annotate_found_where).
+- Path#find(where = nil) — search for the first existing file for the Path:
+  - If path is absolute (located?), returns itself if exists or checks alternatives (.gz, .bgz, .zip).
+  - If where is given, uses that map name only.
+  - If where == :all returns an array of matching paths (see find_all).
+  - Otherwise iterates configured map_order and returns the first found path (annotated with `.where` and `.original`).
+- Path#find_all — returns all existing matches across map_order (useful to locate duplicates).
+- Path#find_with_extension(extension, *args) — try original then the given extension.
+- Path.exists_file_or_alternatives(file) — helper that checks for file or file.gz/.bgz/.zip alternatives.
+
+Helpers for locating:
+- Path.located?(path) / instance located? — returns true if path is absolute (~/, /, ./).
+- Path.caller_file / Path.caller_lib_dir — helper to find the caller's script dir / lib directory (used to set libdir defaults and map values).
+- Path.follow supports advanced `{PATH/old/new}` style substitutions (see tests showing `Path.follow(path, "/some_dir/{PATH/scout/scout_commands}")`).
+
+When a find succeeds, Path#set attributes:
+- found.where — the map name used to find the file
+- found.original — original logical path string before substitution
+
+---
+
+## Globbing and directory utilities
+
+- Path#glob(pattern = "*") — list children matching pattern if this Path points to a directory; returns annotated Path instances.
+- Path#glob_all(pattern = nil) — search across path maps and return all matching annotated paths.
+- Path#directory? — true if the found path is a directory.
+- Path.dirname / Path.basename — string helpers returning annotated strings.
+
+---
+
+## Filename / extension utilities
+
+- Path.is_filename?(string, need_to_exists = true) — static predicate to test if a value looks like a filename.
+- Path.sanitize_filename(filename, length = 254) — shorten a filename safely preserving an extension and adding a digest postfix when needed.
+- Extension helpers:
+  - get_extension(multiple = false) — last extension or multiple.
+  - set_extension(extension) — return path with extension appended.
+  - unset_extension — remove last extension.
+  - remove_extension(extension = nil) — remove a specific extension or unset last.
+  - replace_extension(new_extension, multiple = false) — replace extension(s).
+- Path.relative_to(dir) — returns relative path from dir to this path (uses Misc.path_relative_to).
+
+---
+
+## Misc utilities
+
+- Path.digest_str — extension in path/digest.rb:
+  - If path is a file and exists → "File MD5: <md5>".
+  - If path is a directory → "Directory MD5: <digest of glob>".
+  - Otherwise returns quoted path string.
+
+- Path.no_method_missing — remove the module-level method_missing implementation (rarely used).
+
+- TmpFile.with_path — helper that yields a path string and ensures it is a Path (via Path.setup).
+
+- Path.newer?(path, file, by_link = false) — compare mtimes; returns truthy if `path` is newer than `file`. Handles non-existing files and optionally compares lstat (links).
+
+---
+
+## Integration & annotations
+
+Path extends the Annotation module; each Path can carry annotations:
+- `pkgdir`, `libdir`, `path_maps`, `map_order`. These let different packages or code contexts override search behavior for a path instance.
+
+`Path.setup` creates a Path with default annotations:
+- `pkgdir` defaults to Path.default_pkgdir (usually 'scout').
+- `libdir` defaults to caller library directory (Path.caller_lib_dir).
+
+Examples from tests and common use:
+
+- Compose paths:
+  ```ruby
+  p = Path.setup('/tmp')
+  p.join(:foo)          # => "/tmp/foo"
+  p[:bar, :foo]         # => "/tmp/bar/foo"
+  p.foo[:bar]           # => "/tmp/foo/bar"
+  ```
+
+- Find a file across path maps:
+  ```ruby
+  p = Path.setup("share/data/some_file", 'scout')
+  p.find(:usr)               # resolves using :usr map -> "/usr/share/scout/data/some_file"
+  p.find                     # searches map_order and returns first existing match
+  p.find.where               # map name where it was found (e.g. :current)
+  p.find.original            # original logical path prior to substitution
+  ```
+
+- Search all matches:
+  ```ruby
+  Path.setup("share/data/some_file", 'scout').find_all
+  ```
+
+- Add custom search paths:
+  ```ruby
+  file = Path.setup("somefile")
+  file.append_path('dir1', '/tmp/dir1')
+  file.prepend_path('dir2', '/opt/dir2')
+  ```
+
+- Work with extensions:
+  ```ruby
+  p = Path.setup("/home/.scout/dir/file.txt")
+  p.unset_extension   # => "/home/.scout/dir/file"
+  p.replace_extension('tsv')
+  ```
+
+- Glob directories:
+  ```ruby
+  dir = Path.setup(tmpdir)
+  dir.glob          # returns annotated Path children
+  ```
+
+---
+
+## Notes & edge cases
+
+- Path uses `Path.located?` to decide if a path is already absolute / explicitly located (leading `/`, `~/` or `./`).
+- If `find` is called on a non-located path, it searches maps in `map_order`. Map templates are string patterns — if a map is missing `Path.find` will fallback to the default map.
+- `find` will also check for compressed alternatives (filename.gz, .bgz, .zip) via helper `exists_file_or_alternatives`.
+- The system allows per-path overrides of `pkgdir` and `path_maps` — useful for testing and package-specific layouts.
+- `Path.follow` performs token substitution and supports nested `{PATH/.../...}` style replacements.
+- `caller_lib_dir` and `caller_file` try to infer the caller's library directory; used to set sensible defaults for `libdir`.
+- Many helpers in the framework accept Path objects and will call `.find` (or `.produce_and_find` when supported). Use Path.setup to annotate and pass Path objects to other APIs.
+
+---
+
+## API quick reference
+
+- Creation / basics:
+  - Path.setup(string, pkgdir=nil)
+  - path.join(subpath, prevpath=nil)  — aliases: path[:x], path / :x
+  - path.method_missing to allow `path.foo` as join
+
+- Mapping & locating:
+  - Path.path_maps, Path.map_order, Path.add_path / prepend_path / append_path
+  - path.follow(map_name=:default) — expand template without checking existence
+  - path.find(where = nil) — locate first existing match (annotates `.where` and `.original`)
+  - path.find_all — return all existing matches across maps
+  - path.find_with_extension(extension, *args)
+
+- Filesystem / filename helpers:
+  - path.glob(pattern="*"), path.glob_all(pattern=nil)
+  - path.dirname, path.basename
+  - path.get_extension, set_extension, unset_extension, remove_extension, replace_extension
+  - Path.sanitize_filename, Path.is_filename?
+
+- Utilities:
+  - Path.located?(s), path.located?
+  - Path.caller_file, Path.caller_lib_dir
+  - Path.digest_str (file/directory MD5 summary)
+  - Path.newer?(path, file, by_link = false)
+
+This document summarizes the Path module's purpose and primary surface area. Use Path.setup to get annotated strings that interoperate with Open and other framework utilities to find and manipulate package-oriented filesystem resources.

data/doc/Persist.md

@@ -0,0 +1,214 @@
+# Persist
+
+Persist provides a unified persistence layer for serializing, saving and loading Ruby objects to/from files, with helpers for common formats (JSON, YAML, Marshal), typed serialization, memory-backed persistence, atomic writes, caching, and safe concurrent access (locking). It integrates with Open, Path and the Lockfile-based locking in Open.lock.
+
+Key capabilities:
+- Typed serialization/deserialization for many common types (string, integer, float, boolean, array, json, yaml, marshal, path, file, binary, and *_array variants).
+- Save/load drivers can be customized (Persist.save_drivers / Persist.load_drivers).
+- Safe, atomic writes using Open.sensible_write.
+- `Persist.persist` — higher-level pattern: compute value if not present or stale, save it, and return result; supports streaming writes and keeping locks while streaming.
+- In-process memory persistence (`:memory`).
+- Helpers to load JSON/YAML/Marshal with Open (Open.json/Open.yaml/Open.marshal).
+- Configurable cache_dir and lock_dir for persisted artifacts and lockfiles.
+
+---
+
+## Configuration & defaults
+
+- Persist.cache_dir — default cache directory (Path) used when creating persistence paths. Defaults to Path.setup("var/cache/persistence").
+- Persist.lock_dir — directory used to create lockfiles for persist operations (default tmp/persist_locks).
+- Persist.SERIALIZER — default serializer symbol used when type is `:serializer` (default :json).
+- Persist.TRUE_STRINGS — strings that are treated as true when deserializing booleans.
+
+You can set:
+```ruby
+Persist.cache_dir = Path.setup("var/cache/persistence")   # or string
+Persist.lock_dir = Path.setup("tmp/persist_locks").find
+```
+
+---
+
+## Serialization helpers
+
+- Persist.serialize(content, type)
+  - Convert an object into a string/bytes suitable to save for the given `type`.
+  - Supported types include:
+    - nil, :string, :text, :integer, :float, :boolean, :file, :path, :select, :folder, :binary — converted to string representation (IO/StringIO contents read).
+    - :array — joined by newline
+    - :yaml — YAML.dump
+    - :json — JSON.generate
+    - :marshal — Marshal.dump
+    - :annotation / :annotations — annotation TSV (framework-specific)
+    - `<type>_array` variants — will serialize each entry with given `type` and join with newlines
+  - Raises if unknown type.
+
+- Persist.deserialize(serialized, type)
+  - Converts serialized string back into Ruby object for `type`.
+  - Supported types include :string, :integer, :float, :boolean, :array, :yaml, :json, :marshal, :path (returns Path.setup(...)), :file, :file_array, :annotation, and `<type>_array` variants.
+
+---
+
+## Low-level save/load
+
+- Persist.save(content, file, type = :serializer)
+  - Save `content` to `file` according to `type`.
+  - Uses `Persist.save_drivers[type]` when registered (callable).
+    - If the driver arity is 1, it should return serialized string (Persist will sensible_write it).
+    - If arity > 1, the driver is called with (file, content) and should perform the save itself.
+  - For binary, writes in 'wb' mode.
+  - Otherwise calls `serialize` and writes via `Open.sensible_write(file, serialized, :force => true)`.
+  - Special `type == :memory` stores content in in-memory hash.
+  - Returns nil (or driver-specific return).
+
+- Persist.load(file, type = :serializer)
+  - Load and return object saved at `file` according to `type`.
+  - Accepts `type` values described in deserialize; special handlers:
+    - :binary -> Open.read(file, mode: 'rb')
+    - :yaml / :json / :marshal -> uses Open.yaml/open.json/Open.marshal
+    - :stream -> returns Open.open(file) (an IO-like stream)
+    - :path -> Path.setup(file)
+    - :file -> reads content, handles leading `./` as relative path replacement; returns a filename or file path depending on contents
+    - :file_array -> reads lines and expands relative paths
+    - :memory -> if type is a Hash, returns type[file]
+  - Can use custom `Persist.load_drivers[type]` if registered.
+
+- Custom drivers:
+  - `Persist.save_drivers[type] = ->(file, content) { ... }` or ->(content) { ... }
+  - `Persist.load_drivers[type] = ->(file) { ... }`
+
+---
+
+## The high-level pattern: Persist.persist
+
+The most important API: `Persist.persist(name, type = :serializer, options = {}) { ... }`
+
+Purpose: compute or produce a value and cache it persistently; subsequent calls return cached value unless stale or update requested.
+
+Behavior summary:
+- `name` — logical name (used to build persistent path if path not provided).
+- `type` — serializer type (see above). `:serializer` resolves to Persist.SERIALIZER.
+- `options` — various options (see list below).
+- The block computes and returns the value to be persisted (or may write to an IO stream and return an IO to be consumed).
+- If a persisted file exists and not stale, `Persist.persist` returns the loaded value (or the file path if `:no_load` true).
+- If file absent or stale, it runs the block, saves result via `Persist.save`, and returns result (or file or stream depending on options/return).
+
+Common options (via IndiferentHash/persist-specific keys):
+- :dir or :path or :persist[:path] — explicit file path to use (otherwise generated by Persist.persistence_path).
+- :data — optional object passed to block when block accepts arity 1.
+- :no_load — if true, do not load existing file, return file path instead (or file if empty result).
+- :update — boolean or Path; if provided triggers recompute. If a Path is passed it is compared by mtime; if that mtime is newer than the cached file, block executes and caches new result.
+- :lockfile — specify lockfile path (or use defaults created under Persist.lock_dir)
+- :tee_copies — when block returns an IO stream, tee the stream into multiple copies (useful to both write to file and also return streams)
+- :prefix — when persistence_path is generated, can influence tmp filename prefix (used in tests)
+- :canfail — if true, swallow save errors (used by some callers)
+- :update can be used as a path or Time to force recalculation based on modification time
+
+Concurrency & locking:
+- Persist.persist uses `Open.lock` with a lockfile derived from persistence path (unless disabled) to avoid concurrent processes duplicating work.
+- When the block returns an IO / StringIO (stream), Persist persists via streaming: it creates tee copies, writes one copy to file (using Open.sensible_write) in a background thread and returns another readable stream to the caller.
+  - In that streaming case Persist raises a `KeepLocked` (internal sentinel) so the lock is kept while the background writer thread runs; the calling code obtains an IO to read the stream while the persisting thread writes to file under the same lock.
+  - Test usage demonstrates two processes racing to produce the same persisted stream; locking ensures only one writes and others read the saved result or receive a streamed IO.
+
+Return semantics:
+- If `no_load` true, returns path (string/Path) instead of loaded object.
+- If the block returns `nil`, behavior:
+  - If `no_load` true -> return file path (or file)
+  - If type nil -> returns nil (no persist)
+  - Else attempt to load from file and return result
+
+Memory variant:
+- `Persist.memory(name, options = {}, &block)` — convenience calling `Persist.persist` with `:memory` type (keeps persist in RAM).
+- `Persist.MEMORY_CACHE` holds memory entries.
+
+Persistence path helper:
+- `Persist.persistence_path(name, options = {})` — returns a path in cache_dir for the named persistence file (uses TmpFile.tmp_for_file under the hood). Caller can pass `:dir` override.
+
+---
+
+## Helpers & Open/Path integration
+
+- Open.json(file) / Open.yaml(file) / Open.marshal(file) — convenience wrappers reading & parsing using Open.open.
+- `Path#yaml` / `Path#json` / `Path#marshal` are added to Path via persist/path.rb as convenience to call Open.* on a Path.
+
+---
+
+## Examples (from tests)
+
+Save/load primitive types:
+```ruby
+Persist.save("ABC", "/tmp/x", :string)
+Persist.load("/tmp/x", :string)  # => "ABC"
+
+Persist.save([1,2], "/tmp/x", :integer_array)
+Persist.load("/tmp/x", :integer_array)  # => [1,2]
+```
+
+Using `persist` to cache a computed value:
+```ruby
+res = Persist.persist("myname", :json, dir: some_dir) do
+  expensive_compute()
+end
+# On subsequent calls returns cached value unless update requested.
+```
+
+Streaming content into persistence and returning an IO stream:
+```ruby
+io = Persist.persist("stream_name", :string, path: file) do
+  Open.open_pipe do |sin|
+    1000.times { |i| sin.puts "line #{i}" }
+  end
+end
+# io is a readable IO (stream), background thread writes file atomically while caller can read one copy.
+```
+
+Memory persistence:
+```ruby
+value = Persist.memory("key"){ expensive_compute }
+```
+
+Force update when a source file changed:
+```ruby
+Persist.persist("cache_key", :string, update: source_file) { produce_new_value() }
+```
+
+---
+
+## Custom drivers
+
+Register custom save/load behavior for a type:
+```ruby
+Persist.save_drivers[:mytype] = ->(file, content) { ... }   # or ->(content) { ... }
+Persist.load_drivers[:mytype] = ->(file) { ... }
+```
+If driver takes 1 argument it should return serialized string; Persist.save will sensible_write it. If driver expects file as first argument it should perform write itself.
+
+---
+
+## Error handling & notes
+
+- Persist.persist wraps work in an Open.lock to avoid duplicate computations; it will rethrow exceptions by default (unless `:canfail` true).
+- If a block raises, persist will normally propagate the exception; tests demonstrate that when the persisted file already exists the caller will get cached value even if block raises (depending on options).
+- When streaming, Persist keeps locks while the background writer thread runs (ensures atomicity and consistent readers).
+- Persist.save uses `Open.sensible_write` → atomic temp->mv behavior and uses locks when requested.
+- The `:path` and `:dir` options allow customizing where persisted files go (useful in tests).
+- Use `Persist.save_drivers` and `Persist.load_drivers` to support external formats or custom storage backends.
+
+---
+
+## API quick reference
+
+- Persist.serialize(content, type)
+- Persist.deserialize(serialized, type)
+- Persist.save(content, file, type = :serializer)
+- Persist.load(file, type = :serializer)
+- Persist.persist(name, type = :serializer, options = {}) { ... } — high-level caching pattern (with locking)
+- Persist.memory(name, options = {}, &block) — in-RAM persist wrapper
+- Persist.persistence_path(name, options = {}) — generate a path under Persist.cache_dir
+- Persist.cache_dir / Persist.lock_dir — configurable directories
+- Persist.save_drivers / Persist.load_drivers — custom driver registries
+- Open.json(file), Open.yaml(file), Open.marshal(file) — helpers to read parsed content
+- Path#json / Path#yaml / Path#marshal — Path helpers to read persisted data
+
+---
+
+Persist is intended for reproducible caching of computed results in scripts and workflows, where atomic writes and cross-process locking are required. Use `Persist.persist` for the common compute-and-cache pattern; register custom drivers for special file formats or storage backends.

data/doc/Resource.md

@@ -0,0 +1,229 @@
+# Resource
+
+The Resource module provides a filesystem “resource” abstraction and a production system for on-demand creation of files and directories. It integrates tightly with Path and Open: Path objects can be tied to a Resource, and when you attempt to open/read a Path the Resource system may produce that file (download, generate, run a rake task, install software, etc.). Resource also supports simple synchronization, installation helpers and per-package mapping of search paths.
+
+Main responsibilities:
+- Declare (claim) resources and how to produce them (string content, proc, URL, rake tasks, installers, etc.).
+- Produce (create) resources on demand, with atomic writes and locking to avoid races.
+- Map filesystem paths back to logical resource identifiers (identify / relocate).
+- Install software helpers and set up environment variables from installed packages.
+- Synchronize resources (rsync) into target locations.
+
+Resource is intended to be extended by modules representing package/resource collections. Example in the framework: `Scout` extends Resource and becomes the default resource provider.
+
+---
+
+## Key concepts
+
+- Resource module is extended into a module representing a package or resource collection.
+  - Example:
+    ```ruby
+    module MyPkg
+      extend Resource
+      self.pkgdir = 'mypkg'
+      self.subdir = Path.setup('share/mypkg')   # optional
+    end
+    ```
+- A Resource holds claims: mappings between logical paths and how to produce them.
+- Path objects can carry a `pkgdir` referring to a Resource; Path#produce will invoke the corresponding Resource produce logic.
+
+---
+
+## Claiming resources
+
+Use `claim` on a Resource to register how to create a particular logical path:
+
+- Resource.claim(path, type, content = nil, &block)
+  - `path` — a Path (or string that will be converted to Path).
+  - `type` — a symbol describing how to produce (see below).
+  - `content` or block — content, URL, proc, rakefile, installer script, etc.
+
+Common `type` values (used by the built-in produce logic):
+- `:string` — static string written to file.
+- `:proc` — a Proc that returns content (String, IO, Array, TSV/dumper, etc.). If the proc accepts an arity of 1 it may receive the final filename.
+- `:url` — remote URL; the resource is produced by downloading the URL (Open.wget/Open.open).
+- `:rake` — a Rakefile or proc to be executed to generate targets via Rake rules (see rake integration).
+- `:install` — calls Resource.install to run system-level install helpers for software.
+- custom types may be supported by extending Resource.produce or providing other logic.
+
+Examples:
+```ruby
+module TestResource
+  extend Resource
+  claim self.tmp.test.string, :string, "TEST"
+  claim self.tmp.test.proc, :proc do
+    "PROC TEST"
+  end
+  claim self.tmp.test.google, :url, "http://google.com"
+  claim self.tmp.test.rakefiles.Rakefile , :string, "file('foo'){ |t| Open.write(t.name,'FOO') }"
+  claim self.tmp.test.work.footest, :rake, TestResource.tmp.test.rakefiles.Rakefile
+end
+```
+
+---
+
+## Producing resources
+
+- Resource.produce(path, force = false)
+  - Produces (creates) the file for `path` according to the claim registered on the Resource.
+  - If `force` is true it will overwrite existing file.
+  - Uses a lock per target (TmpFile tmp_for_file) and performs atomic write using Open.sensible_write.
+  - Supports producing compressed alternatives (tries .gz and .bgz when appropriate).
+  - Handles `:string`, `:proc`, `:url`, `:rake`, `:install` and other types as implemented.
+
+Behavior notes:
+- For `:proc` type: the proc may return String, IO, Array (written as newlines), TSV dumper, etc. Open.sensible_write is used.
+- For streaming scenarios block may produce an IO stream; produce writes the stream atomically using tees and background thread under lock.
+- On failure the partial output is removed and exception re-raised.
+
+Convenience wrappers on Path trigger production:
+- `Path#produce(force = false)` — call resource-produce for the path.
+- `Path#produce_with_extension(ext, *args)` — try producing path or path.ext.
+- `Path#produce_and_find(extension = nil, *args)` — produce then return found path.
+- `Path#open`/`Path#read` will call `produce` before delegating to Open.
+
+---
+
+## Rake integration
+
+Resource supports producing files by invoking Rake tasks:
+
+- `Resource.claim(..., :rake, rakefile)` — cooker references a Rakefile (Path or proc or string). The Rake file should define file rules/tasks for targets.
+- Under the hood:
+  - `ScoutRake.run(rakefile, dir, task)` forks and runs Rake in a child, invoking the requested task (task is derived from relative path).
+  - `Rake::FileTask.define_task` is hooked to track file tasks.
+  - `Resource.run_rake(path, rakefile, rake_dir)` handles invoking the appropriate rake task for a path and supports retries moving up directories to find tasks.
+
+This makes it possible to register a Rakefile that generates many resources via Rake rules.
+
+---
+
+## Identification & relocation
+
+- Resource.identify(path)
+  - Given an absolute `path`, try to identify its logical package/path by matching the configured `path_maps` for the resource.
+  - Returns an unlocated Path (logical path within the package) or a Path annotated with the resource context.
+
+- Resource.relocate(path)
+  - If `path` does not currently exist locally, attempt to identify it and find it in configured maps (e.g., different locations, caches) returning an available path.
+
+- Path#identify (delegates to Resource.identify for Path objects with pkgdir set).
+- Use `Resource.identify` to map filesystem locations back to `TOPLEVEL/{PKGDIR}/{SUBPATH}` style logical identifiers.
+
+---
+
+## Sync / rsync support
+
+- `Resource.sync(path, map = nil, options = {})`
+  - Copy a resource (file or directory) into a target location determined by resource maps and `map` name (defaults :user).
+  - Uses `Open.sync` (rsync wrapper) for actual transfer.
+  - Can accept `:resource` option to choose a resource module other than the path's pkgdir.
+
+---
+
+## Software install helpers
+
+Resource includes helpers to install software in a per-resource `software` directory and to set environment variables:
+
+- `Resource.install(content, name, software_dir = Path.setup('software'))`
+  - `content` may be a script, Path, String, Hash describing git/src/jar/commands, or a block that returns script text.
+  - Builds a wrapper script with a common install helper preamble and executes installation commands (using CMD).
+  - After install, calls `Resource.set_software_env(software_dir)`.
+
+- `Resource.set_software_env(software_dir)`
+  - Scans `software/opt` configuration files (`.ld-paths`, `.c-paths`, `.pkgconfig-paths`, `.aclocal-paths`, `.java-classpaths`) and adds entries to environment variables (PATH, CLASSPATH, PKG_CONFIG_PATH, etc.).
+  - Adds `opt/bin` to PATH and loads `.post_install` exports.
+
+This supports installing package-local tools and updating runtime environment.
+
+---
+
+## Helpers / utilities
+
+- `rake_for(path)`, `has_rake?(path)` — find registered rake dirs that can produce a given path.
+- `run_rake(path, rakefile, rake_dir)` — run rake task to produce target.
+- `relocate(path)` — attempt to relocate a missing path via resource identification.
+- Method delegation: Resource defines `root` and `method_missing` so resource instances behave like a Path root; e.g. `MyResource.foo.bar` delegates to `MyResource.root.foo.bar`.
+
+---
+
+## Concurrency & atomicity
+
+- Resource.produce uses Open.lock to acquire a lockfile for the target before producing, avoiding concurrent producers colliding.
+- Writes are performed via `Open.sensible_write` (temp file then atomic mv) to avoid partial files being visible.
+- Rake-run and other production steps execute in subprocesses or controlled threads so they don't corrupt the workspace.
+
+---
+
+## Integration details
+
+- Path.open has been overridden to call `produce` for Path objects before delegating to Open.open, so many consumers transparently trigger production when opening resources.
+- `Resource.default_resource` can be set to a default package (the framework sets `Resource.default_resource = Scout`).
+- Resources use Path.path_maps to find logical locations; packages typically set `pkgdir` and `subdir` and may provide per-resource `path_maps`.
+
+---
+
+## Examples (from tests)
+
+Registering and producing resources:
+```ruby
+module TestResource
+  extend Resource
+  self.subdir = Path.setup('tmp/test-resource')
+
+  claim self.tmp.test.string, :string, "TEST"
+  claim self.tmp.test.proc, :proc do
+    "PROC TEST"
+  end
+  claim self.tmp.test.google, :url, "http://google.com"
+end
+
+TestResource.produce TestResource.tmp.test.string
+puts TestResource.tmp.test.string.read   # "TEST"
+TestResource.tmp.test.google.produce     # downloads google HTML
+```
+
+Rake-based production:
+```ruby
+# Rakefile claims and tasks:
+claim self.tmp.test.rakefiles.Rakefile , :string , <<-EOF
+file('foo') { |t| Open.write(t.name, "FOO") }
+rule(/.*/) do |t|
+  Open.write(t.name, "OTHER")
+end
+EOF
+
+claim self.tmp.test.work.footest, :rake, TestResource.tmp.test.rakefiles.Rakefile
+# produce targets:
+TestResource.produce TestResource.tmp.test.work.footest.foo  # runs rake task to create foo
+```
+
+Software installation:
+```ruby
+Resource.install nil, "scout_install_example", tmpdir.software do
+  <<-EOF
+echo "#!/bin/bash\necho WORKING" > $OPT_BIN_DIR/scout_install_example
+chmod +x $OPT_BIN_DIR/scout_install_example
+  EOF
+end
+# then the installed helper is available in PATH via set_software_env
+```
+
+Syncing:
+```ruby
+Resource.sync(source_path, :current)   # rsync source into resource's current map target
+```
+
+---
+
+## Notes & caveats
+
+- Resource assumes the availability of external commands when needed (wget, git, bash, rake, etc.).
+- Rake tasks are executed in forked subprocesses; Rake definitions must be loadable in that context.
+- Production steps must be idempotent or guarded by the lock/atomic write semantics.
+- When claiming URL resources, network failures or remote changes can affect production; consider caching strategies or update checks in callers.
+- `Resource.identify` relies on path templates and caller libdir heuristics — mapping may require customizing `path_maps` per resource.
+
+---
+
+Resource is intended to centralize how resources for a package are produced and located, to let clients simply ask for a Path and have the resource created on demand with safe concurrency and atomic writes. Use `claim` to declare resources and `produce`/Path.open to trigger creation.