scout-essentials 1.8.8 → 1.9.1

data/doc/user/LoggingAndProgress.md

@@ -0,0 +1,158 @@
+# Logging and Progress
+
+`Log` is the single reporting surface: a severity-gated logger that writes to
+STDERR, plus a stack of progress bars that render in the same place.
+`lib/scout/log.rb`, `log/color.rb`, `log/fingerprint.rb`, `log/progress.rb`,
+`log/progress/util.rb` and `log/progress/report.rb` are the whole
+implementation.
+
+## Severity ladder
+
+`Log::SEVERITY_NAMES` is `%w(DEBUG LOW MEDIUM HIGH INFO WARN ERROR NONE)` with
+integer constants `DEBUG=0 .. NONE=7`. `INFO` sits in the middle (4), not at
+the top: `LOW`/`MEDIUM`/`HIGH` are *more verbose than info*.
+
+The severity is set at require time from `ENV['SCOUT_LOG']` first
+(`lib/scout/log.rb:36-54`): the known names (`DEBUG`, `LOW`, `MEDIUM`, `HIGH`,
+`WARN`, `ERROR`, `NONE`) are honoured, and anything else — including an
+unrecognised value — falls back to `Log.default_severity`, which reads
+`~/.scout/etc/log_severity` if that file exists and is `INFO` otherwise
+(log.rb:23-34).
+
+Verified by `tmp/rewrite_C/probe_04_log_severity.rb` (out04.txt): ladder
+order, `Log.severity == 4` with no env/file, `Log.with_severity(level){}`
+restoring the original value after the block.
+
+Change it at runtime with `Log.severity = Log::DEBUG`, or scope a block with
+`Log.with_severity(level){ ... }`, or use the top-level `sss(level)` /
+`sss(level){ ... }` helper (log.rb:430).
+
+## Where output goes, and how it is protected
+
+- `Log.log_write` / `Log.log_puts` wrap every write in `Log::MUTEX`
+  (log.rb:141-163). If a logfile was installed, output goes there; otherwise
+  STDERR (IOError swallowed).
+- **`Log.logfile(path)` sets the logfile; `Log.logfile` with no argument does
+  NOT read it back — it resets it to `nil`** (log.rb:110-113). There is no
+  reader; the accessor is only `Log.logfile=` (`attr_writer`).
+- `Log::LAST` is one shared mutable `String` (starts as `"log"`, never frozen)
+  used as a protocol between `logn` and progress-bar printing so bars know how
+  many lines to move up and which kind of output came last
+  (`progress/report.rb` `print`/`report`). Probe out04.txt confirms
+  `class == String`, `frozen? == false`.
+
+## Message forms
+
+- `Log.debug "msg"` .. `Log.error "msg"` print when severity allows; the top
+  of the ladder is the `Log::NONE` constant, not a method.
+- `Log.debug { "expensive " + build }` — **the block is lazy message
+  evaluation, not a timer**: it is only called if the level passes. Probe
+  (out05.txt): a counter incremented inside a block passed to `Log.debug`
+  stays at 0 under ERROR severity and reaches 1 under DEBUG severity.
+- `Log.exception(e)` (log.rb:254) prints `BACKTRACE` lines derived from
+  `e.backtrace` after fingerprinting messages longer than 1000 chars; it
+  returns early when the message contains `NOLOG` and drops the backtrace when
+  it contains `NOSTACK`.
+- `Log.stack(stack)` (log.rb:318) prints a magenta header then the stack
+  **reversed** (innermost frame first) unless `SCOUT_ORIGINAL_STACK=true`.
+  Probe (out05.txt): a caller list `a.rb:1, b.rb:2, c.rb:3` prints as
+  `c.rb:3`, `b.rb:2`, `a.rb:1`.
+
+## Top-level debug helpers (defined on Object)
+
+`ppp(msg)` (log.rb:358) prints a cyan `PRINT:` banner plus the caller line and
+pretty-prints the message. `fff(obj)` (log.rb:374) logs the fingerprint at
+DEBUG. `ddd/lll/mmm/iii/wwww/eee` inspect an object at the matching severity,
+and the `f`-suffixed variants (`ddf`, `llf`, `mmf`, `iif`, `wwwf`, `eef`)
+fingerprint it first. `sss(level[, &blk])` sets or scopes severity. `ccc`
+(log.rb:439) enables `$scout_debug_log` around a block so nested `ccc` calls
+print. Probe out05.txt confirms all five are top-level methods defined inside
+`lib/scout/log.rb`.
+
+## Colors
+
+- `Log.color(:red, "text")` — symbol color name. Produces `"\e[31mtext\e[0m"`.
+- Passing a *String* name does not resolve the color table; the string is used
+  as the literal prefix and an escape is appended (`"redred\e[0m"`). Use
+  symbols.
+- `Log.uncolor(str)` strips ANSI codes. `Log.nocolor` is true when
+  `ENV['SCOUT_NOCOLOR'] == 'true'` (exact string), which also blanks the cursor helpers
+  `up_lines`/`down_lines`/`return_line`/`clear_line`.
+
+## Fingerprints
+
+`Log.fingerprint(obj)` (log/fingerprint.rb:18) builds a stable, truncated text
+representation for logging big values. It dispatches: objects responding to
+`fingerprint` are delegated to; `nil/true/false/Symbol` are rendered
+literally; strings longer than `FP_MAX_STRING` (150) are truncated to a
+digest-marked middle (`"xxx<...400 - 0d723...>xxx"`); arrays longer than 20
+show first/middle/last elements; hashes longer than 10 fall back to
+keys+values fingerprints; floats get 1/3/6 decimals depending on magnitude.
+
+The signature is `fingerprint(obj)` — **one positional argument**; there are no
+`max_length`/`sep` options (probe_05). Hash rendering separates pairs with a
+space, not a comma: `Log.fingerprint({a: 1, b: 2, c: 3})` gives
+`"{:a=>1 :b=>2 :c=>3}"`.
+
+## Progress bars
+
+`Log::ProgressBar` instances are created with the class helpers in
+`progress/util.rb`:
+
+- `Log::ProgressBar.new_bar(max, options = {})` — also accepts a plain Hash
+  alone (`new_bar(:max => 50)`); `cleanup_bars` runs first and `:depth`
+  defaults to the current bar-stack depth plus offset. `new_bar(true)` treats
+  `true` as "no max" (`max = nil if TrueClass === max`, progress.rb:38).
+- `Log::ProgressBar.with_bar(max = nil, options = {})` (util.rb:85) — wraps a
+  block; `KeepBar` keeps the bar alive, any other exception marks it errored
+  and re-raises.
+- `Log::ProgressBar.with_obj_bar(obj, bar = true, &block)` (util.rb:167) —
+  the *second* argument (not `obj`) picks the bar: a String is the
+  description, `true` calls `guess_obj_max(obj)` (which — see the caveat
+  below — returns `nil` in a bare scout-essentials process), a Numeric is the
+  explicit max, a Hash carries `:max` and other options, and an existing
+  `Log::ProgressBar` is reused
+  (util.rb:139-162). **The block receives only the bar**:
+  `with_obj_bar(list, 'Processing'){ |bar| ... }`.
+- `Log.no_bar` / `Log.no_bar=` / `SCOUT_NO_PROGRESS=true` disable ticking
+  entirely (progress.rb:5-11). There is no `Log.bar` module method.
+
+Instance API (`progress.rb`): `init`, `tick(step = 1)`, `pos(pos)`,
+`process(elem)`, `percent`, and the `max/ticks/frequency/depth/desc/file/
+bytes/process/callback/severity` accessors.
+
+- `percent` returns 0 with no ticks, **100 when `max == 0`**, and integer
+  division otherwise.
+- `tick` is a no-op when `Log.no_bar`; it reports when `diff >= frequency`
+  (default 2 s) or when the percent advanced and `diff > 0.3` — that ~0.3 s
+  throttle plus the frequency gate is the render throttling.
+- `report(io = STDERR)` redraws the whole active-bar stack with cursor
+  movement. `report_msg` shows **elapsed time and rate** and, when `max` is
+  set, an ETA: `· <rate> per sec. -- <dots> <pct>% <eta> => <elapsed> -
+  <ticks> of <max> items · <desc>`; with no `max` the ETA part is replaced
+  by `<ticks> items` (probe_12 shows both forms).
+- `add_offset` / `remove_offset` / `offset` (util.rb:9-28) indent bars created
+  in nested threads.
+- `file:` option: `save` writes the bar state to YAML (`:desc, :last_count,
+  :last_percent, :last_time, :max, :start, :ticks`) and `done`/`error` remove
+  the file, so a bar given a `file:` **resumes across runs** — the persisted
+  YAML is reloaded into `ticks` (probe_06 round-trips 3 ticks).
+
+All bar bookkeeping is guarded by `BAR_MUTEX`; per-bar `tick` state is not
+synchronized and races by design.
+
+### A scout-essentials-only caveat
+
+`guess_obj_max` (util.rb:101-136) matches `TSV` and `Step` *before* `Array` and
+`Hash`. In a plain `require 'scout-essentials'` process neither `TSV` nor
+`Step` exists, so the `when TSV` arm raises `NameError`, the rescue turns it
+into `nil`, and **`guess_obj_max` always returns `nil`** — i.e.
+`get_obj_bar(list, true).max == nil` (probe_06). Pass an explicit Numeric or a
+`:max` hash, or define those constants in the embedding repo, to get a real
+max.
+
+## Related
+
+- [Cookbook](Cookbook.md) — progress bars inside real recipes.
+- [Architecture](../developer/Architecture.md) — where `Log` sits and
+  cross-repo attribution.

data/doc/user/ProducingResources.md

@@ -0,0 +1,177 @@
+# Producing Resources
+
+A **resource** is a logical file — a name like `data/config.yaml` — that a
+software package can materialize on demand. This page explains the `claim`
+syntax, how `produce` uses claims to write files, and the helpers around
+them.
+
+## Declaring a resource
+
+Any module that `extend`s `Resource` becomes a resource package:
+
+```ruby
+require 'scout-essentials'
+
+module MyApp
+  extend Resource
+  annotation :pkgdir
+  self.pkgdir = 'myapp'
+end
+
+MyApp.claim MyApp.data.config, :string, "key=value\n"
+```
+
+Inside such a module the idiomatic form (note: no `self.` prefix on `claim`)
+is:
+
+```ruby
+module MyApp
+  extend Resource
+  annotation :pkgdir
+  self.pkgdir = 'myapp'
+
+  claim data.file, :string, "content\n"
+end
+```
+
+
+- `claim(path, type, content = nil, &block)` — `type` is a **mandatory
+  positional** argument (`claim path { }` raises `ArgumentError`). Passing
+  only the path plus a block is not a valid call.
+- `type` is one of `:string`, `:url`, `:proc`, `:rake`, `:install`.
+  `:csv` is an unimplemented stub that raises `RuntimeError
+  "TSV/CSV Not implemented yet"` when produced. There is **no `:annotation`
+  claim type**.
+- `path` is normally a `Path` built with `method_missing` segments
+  (`MyApp.data.config`); a plain `String` is also accepted.
+
+## Claim types
+
+| Type | `content` | Effect |
+|------|-----------|--------|
+| `:string` | String | written verbatim |
+| `:url` | source path/URL | the source is opened (with `:noz` when it looks compressed) and copied to the final path |
+| `:proc` | `Proc` (arity 0 or 1) | called with no args (arity 0) or the final path (arity 1); the result is dispatched (below) |
+| `:rake` | Rakefile directory/task | `run_rake` builds the file (below) |
+| `:install` | software spec hash | `Resource.install` installs software into `share/software` (below) |
+
+`:proc` result dispatch — the returned value drives how the file is written:
+
+- `String`, `IO`, `StringIO` → `Open.sensible_write(final_path, data)`
+- `Array` → elements joined with `"\n"` then written
+- `TSV` / `TSV::Dumper` → the dumper stream is written (requires
+  `rbbt-util`/scout-gear; **a `nil` return trips the `when TSV` branch
+  first** and raises `NameError: uninitialized constant Resource::TSV`, so
+  procs that may return nil must guard themselves)
+- `nil` → nothing is written (file left missing) — but see the caveat above
+- anything else → `RuntimeError "Unkown object produced: ..."`
+
+## Producing
+
+`Path#produce` (in `lib/scout/resource/path.rb`) is the entry point:
+
+```ruby
+MyApp.data.config.produce   # returns the logical path (a Path)
+MyApp.data.config.find      # => "$HOME/.myapp/data/config"  (now existing)
+Open.read(MyApp.data.config.find) # => "key=value\n"
+```
+
+- It never returns nil: on `ResourceNotFound` (nothing claims the path) it
+  stores `false` and returns it, and that decision is **latched** in
+  `@produced` — a later call does not retry unless you pass `force`.
+- A raised `Exception` stored in `@produced` is re-raised on the next call.
+- On a successful produce the target is written under a **produce lock**
+  (`TmpFile.tmp_for_file(final_path, dir: lock_dir)` + `Open.lock`).
+- If nothing claims the plain path but a `.gz` or `.bgz` variant is claimed,
+  production falls through to those extensions. The fall-through is
+  asymmetric: producing `x.txt` may materialize `x.txt.gz`, but asking for
+  `x.gz` when only `x.txt` is claimed fails.
+- `force: true` (or `produce(true)`) removes the existing file and reruns.
+
+```ruby
+MyApp.data.config.produce(true) # force re-production
+```
+
+### Extension-aware helpers
+
+- `produce_and_find(extension = nil, *args)` — produce (falling back to the
+  given extension with `produce_with_extension`) and return the located
+  path; raises `RuntimeError "Not found"` if the result is still missing.
+- `produce_with_extension(extension, *args)` — try the plain path first, then
+  the variant with `extension` appended; raise the *first* exception if both
+  fail.
+- `find_with_extension(ext, *args, produce: true)` — `find` first; if the
+  plain result does not exist, look for `self.set_extension(ext)`.
+  `produce:` defaults to **true**, so it may trigger a produce.
+- `exists?(produce: true)` — default **produces** before answering; pass
+  `false` for a pure existence check.
+
+### Produce-aware `Path` I/O
+
+`Path#read` and `Path#open` call `produce` first; `Path#list` is
+`produce_and_find('list')` then `Open.list`. `Path#write` does **not**
+produce — it writes straight to `self.find` (use it to place a manual file
+under the resource tree):
+
+```ruby
+MyApp.data.manual.write("manual\n")
+# => writes to the found location, no claim consulted
+```
+
+## Rake claims
+
+A rake claim points at a directory containing a `Rakefile`:
+
+- `rake_dirs` lists candidate directories; `has_rake?(path)` is true when a
+  rake prefix matches, and `rake_for(path)` returns the longest matching
+  prefix (an empty-string prefix matches everything).
+- Production runs the task through `ScoutRake` (forked execution). If rake
+  reports `Don't know how to build task`, the walk-up retry joins the task
+  name with the parent directory and retries; if that also fails, the error
+  is converted to `ResourceNotFound`.
+- A `Path` whose pkgdir is the resource module is produced with
+  `pkgdir.produce(self)`; `Path#produce` on a String path returns `false`.
+
+```ruby
+# Rakefile in ./ with: file "data/built.txt" do |t| Open.write(t.name, "x") end
+MyApp.data.built.txt.produce
+Open.read(MyApp.data.built.txt.find) # => "x"
+```
+
+## Software installs
+
+`:install` claims (and `Resource.install(name, software_dir, options)`)
+install software binaries under `share/software/<name>` by sourcing
+`share/software/install_helpers` and running the recipe from the spec hash
+(`:git`, `:src`, `:url`, `:jar`, `:extra`, `:configure`, `:make`, ...).
+`Resource.set_software_env(software_dir)` runs at load time on the default
+`software` dir to expose installed binaries on `PATH`/`JAVA_CLASSPATH`-style
+variables.
+
+## `Resource.sync`
+
+```ruby
+Resource.sync(path, map = nil, options = {})
+```
+
+A module-level method (there is **no** `Path#sync`) that walks the map
+order copying/symlinking resource files found in other locations into the
+map's directory, so that unlocated paths resolve locally. It is unrelated to
+`Open.rsync`.
+
+## Scout itself is a Resource
+
+`Scout` extends `Resource` with `pkgdir 'scout'` and is
+`Resource.default_resource`. `Scout.etc`, `Scout.tmp`, ... are ordinary
+`Path` segment builders over the `Scout` package:
+
+```ruby
+Resource.default_resource # => Scout
+Scout.etc.find            # => "$HOME/.scout/etc"
+```
+
+## Related
+
+- [Path Resolution](../developer/PathResolution.md) for `find`/`follow`.
+- [Persistence and Resources](../developer/PersistenceAndResources.md) for the
+  produce lock namespace and claim semantics from a developer angle.

data/doc/user/RemoteData.md

@@ -0,0 +1,157 @@
+# Remote Data
+
+Fetching remote files over HTTP(S)/FTP/SSH, the URL cache that backs
+`Open.wget`, and the rsync helpers used to move directories between hosts.
+Source: `lib/scout/open/remote.rb`, `lib/scout/open/sync.rb`,
+`lib/scout/resource/sync.rb`.
+
+Executed examples come from `tmp/rewrite_D/probe_07_remote.rb`, which
+starts a throwaway local `python3 -m http.server` to serve real URLs;
+transcripts live in `tmp/rewrite_D/BATCH_D_PROBE_TRANSCRIPTS.txt`.
+
+For the local side of `Open` see [WorkingWithFiles.md](WorkingWithFiles.md);
+for `Resource` itself see [ProducingResources.md](ProducingResources.md).
+
+## Recognising remote paths
+
+```ruby
+Open.remote?('http://a')   # => true
+Open.remote?('https://a')  # => true
+Open.remote?('ssh://a:x')  # => true
+Open.remote?('ftp://a')    # => true
+Open.remote?('/local')     # => false
+Open.ssh?('ssh://a:x')     # => true
+Open.ssh?('http://a')      # => false
+```
+
+Both are pure regexes (`/^(?:https?|ftp|ssh):\/\//` and `/^ssh:\/\//`). A
+path without a scheme is never remote. Verified by
+`tmp/rewrite_D/probe_07_remote.rb`.
+
+## `Open.ssh` — reading over SSH
+
+`Open.ssh('ssh://server:path')` parses server and path from the URI. When
+the server is exactly `localhost` it bypasses SSH entirely and calls
+`Open.open(file)` on the bare path — handy for tests. Otherwise it returns a
+pipe from `CMD.cmd("ssh '<server>' cat '<file>'", :pipe => true, :autojoin
+=> true)`, i.e. a `ConcurrentStream`, not a String.
+
+## `Open.wget` and the URL cache
+
+`Open.wget(url, options)` is the main entry point for HTTP/FTP. It has a
+**permanent file cache** keyed by a digest of the request, and the default
+behaviour is to serve from that cache without touching the network.
+
+### Cache location and key
+
+```ruby
+Open.remote_cache_dir            # $HOME/.scout/var/cache/open-remote
+Open.remote_cache_dir = dir      # override (probe sets it to a tmpdir)
+Open.cache_file(url, options)    # <cache_dir>/<digest>
+```
+
+The digest (`Open.digest_url`) covers the URL, the `--post-data` value and
+the **sorted lines** of `--post-file`. Different post-data therefore gets a
+different cache entry (probe: three URLs/options => three distinct cache
+paths).
+
+### No TTL, no expiry
+
+There is no timestamp check and no maximum age. Once a URL is cached, a
+later `Open.wget` of the same URL/options returns the cached bytes even if
+the server has changed. The probe demonstrates it: the file was edited on
+the server three times (`HELLO` → `FIRST` → `SECOND` → `THIRD`) while the
+second `Open.wget` still reported the old cached content.
+
+### Forcing a refresh
+
+```ruby
+Open.wget(url, :nocache => 'update')  # re-fetch, write cache, read cache
+Open.wget(url, :force  => true)       # skip the cache check entirely
+Open.remove_from_cache(url, options)  # delete the cache entry
+```
+
+**Exactly `'update'`** re-fetches and re-writes the cache, then *reads it
+back* (probe: after `Open.wget(url, :nocache => 'update')` the returned file
+is the cache file again, not the raw pipe). Any other truthy `nocache`
+value takes the other branch and returns the raw wget pipe **without**
+touching the cache. `:force` only bypasses the initial `in_cache?` lookup,
+so a `:force` fetch without `nocache` still ends by writing and re-opening
+the cache.
+
+Other `wget` passthroughs: `--user-agent=` (defaults to `rbbt`),
+`:post` → `--post-data=`, `:cookies => file` → `--save-cookies`,
+`--load-cookies`, `--keep-session-cookies`, plus any literal wget flag.
+`:nice` / `:nice_key` route through `Open.wait` (below). Network errors are
+wrapped in `OpenURLError` with the message
+`Error reading remote url: <url>. <cause>`.
+
+## `Open.download` and `Open.scp`
+
+`Open.download(url, file)` runs `wget '<url>' -O '<file>'` via
+`CMD.cmd_log`; on failure it removes the partial output file and re-raises
+the original error. Verified: downloading a live URL produced the current
+server bytes (no cache involved).
+
+`Open.scp(source_file, target_file, target:, source:)` first creates the
+remote parent directory with `ssh <target> mkdir -p ...`, then runs
+`scp -r`. Both keyword arguments are optional and only prefix the paths with
+`host:` when they are not already prefixed.
+
+## `Open.rsync` and `Open.sync`
+
+`Open.rsync(source, target, options)` shells out to
+`rsync -avztHP --copy-unsafe-links --omit-dir-times` with:
+
+- `:excludes` — defaults to `%w(.save .crap .source tmp filecache
+  open-remote)`; a String is split on commas unless it already contains
+  `--exclude`.
+- `:files` — a list of relative filenames written to a temp file and passed
+  as `--files-from=`.
+- `:hard_link` — adds `--link-dest '<source>'` (local sources only).
+- `:test` — adds `-nv` (dry run).
+- `:delete` — appends `&& rm -Rf <source>` when no `:files`, or removes the
+  listed files and empties their directories afterwards.
+- `:source` / `:target` — turn one side into `server:'path'` URIs, creating
+  the remote directory with `ssh <server> mkdir -p` when needed.
+- `:print => true` — return the command string instead of running it.
+- `:other` — a String or Array appended verbatim.
+
+Trailing-slash semantics follow rsync: if either side is a directory (or ends
+in `/`), both get a trailing `/`, which copies *contents* rather than the
+directory itself. A local-to-local sync of the identical path logs a warning
+and returns.
+
+`Open.sync(...)` is a literal alias: `Open.sync` and `Open.rsync` are the
+same implementation forwarded with Ruby's `...` (probe: same owner). Use
+either name — they are interchangeable.
+
+## `Resource.sync(path, map, options)`
+
+`Resource.sync` copies a `Resource` path into a mapped location:
+`resource.identify(path).find(map)` is the target, and each existing file or
+globbed match is synced with `Open.sync(source, target, options)`. The
+resource is resolved from `:resource`, from the path's `pkgdir` when it is
+already a `Path` over a `Resource`, or from `Resource.default_resource`.
+`map` defaults to `'user'`. All `Open.rsync` options pass through.
+
+## `Open.wait` — client-side rate limiting
+
+```ruby
+Open.wait(lag, key = nil)
+```
+
+A tiny rate limiter backed by the module-level `LAST_TIME` hash. If the
+last call *for that key* was less than `lag` seconds ago, it sleeps the
+remainder. `Open.wget` uses it via `:nice` / `:nice_key`. Probe: two
+`Open.wait(0.3, :k)` calls in a row took 0.3 s wall clock.
+
+Note the process-local `LAST_TIME` map is shared mutable state with no
+mutex; see [LockingAndConcurrency.md](../developer/LockingAndConcurrency.md).
+
+## Where this is used
+
+- [CachingResults.md](CachingResults.md) — the other caching layers
+  (`Persist`) and how they differ from this URL cache.
+- [WorkingWithFiles.md](WorkingWithFiles.md) — `Open.open` dispatching to
+  `wget`/`ssh` when a path looks remote.