scout-essentials 1.8.7 → 1.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.vimproject +26 -12
- data/README.md +83 -112
- data/VERSION +1 -1
- data/doc/Improvements.md +226 -0
- data/doc/StartHere.md +122 -0
- data/doc/developer/AnnotationSystem.md +184 -0
- data/doc/developer/Architecture.md +147 -0
- data/doc/developer/Configuration.md +238 -0
- data/doc/developer/CoreUtilities.md +265 -0
- data/doc/developer/DesignPrinciples.md +129 -0
- data/doc/developer/ErrorHandling.md +203 -0
- data/doc/developer/LockingAndConcurrency.md +157 -0
- data/doc/developer/PathResolution.md +200 -0
- data/doc/developer/PersistenceAndResources.md +119 -0
- data/doc/developer/StreamingModel.md +236 -0
- data/doc/user/AnnotatingData.md +202 -0
- data/doc/user/CachingResults.md +183 -0
- data/doc/user/CommandLineOptions.md +189 -0
- data/doc/user/Cookbook.md +211 -0
- data/doc/user/HandlingStreams.md +236 -0
- data/doc/user/LoggingAndProgress.md +158 -0
- data/doc/user/ProducingResources.md +177 -0
- data/doc/user/RemoteData.md +157 -0
- data/doc/user/RunningCommands.md +218 -0
- data/doc/user/WorkingWithFiles.md +217 -0
- data/lib/scout/cmd.rb +343 -40
- data/lib/scout/concurrent_stream.rb +14 -1
- data/lib/scout/indiferent_hash.rb +1 -1
- data/lib/scout/log/fingerprint.rb +13 -8
- data/lib/scout/log/progress/report.rb +1 -1
- data/lib/scout/log.rb +4 -1
- data/lib/scout/misc/digest.rb +6 -5
- data/lib/scout/misc/format.rb +24 -0
- data/lib/scout/named_array.rb +1 -1
- data/lib/scout/open/stream.rb +2 -2
- data/lib/scout/open/util.rb +8 -4
- data/lib/scout/open.rb +3 -3
- data/lib/scout/path/find.rb +3 -2
- data/lib/scout/persist.rb +14 -10
- data/lib/scout/resource/produce.rb +9 -1
- data/research/annotations-data-analysis.md +206 -0
- data/research/behavior-probes.md +1925 -0
- data/research/commands-streaming-analysis.md +272 -0
- data/research/design-philosophy-analysis.md +383 -0
- data/research/doc-audit-findings.md +294 -0
- data/research/ecosystem-attribution.md +118 -0
- data/research/implementation-inventory-core.md +1029 -0
- data/research/implementation-inventory-open.md +417 -0
- data/research/implementation-inventory-path-persist-resource.md +774 -0
- data/research/io-paths-analysis.md +228 -0
- data/research/persistence-resources-analysis.md +244 -0
- data/research/synthesis-report.md +80 -0
- data/scout-essentials.gemspec +37 -15
- data/test/scout/open/test_remote.rb +1 -2
- data/test/scout/test_cmd.rb +411 -0
- metadata +36 -14
- data/doc/Annotation.md +0 -352
- data/doc/CMD.md +0 -363
- data/doc/ConcurrentStream.md +0 -163
- data/doc/IndiferentHash.md +0 -240
- data/doc/Log.md +0 -235
- data/doc/NamedArray.md +0 -174
- data/doc/Open.md +0 -331
- data/doc/Path.md +0 -217
- data/doc/Persist.md +0 -214
- data/doc/Resource.md +0 -229
- data/doc/SimpleOPT.md +0 -236
- data/doc/TmpFile.md +0 -154
data/doc/Open.md
DELETED
|
@@ -1,331 +0,0 @@
|
|
|
1
|
-
# Open
|
|
2
|
-
|
|
3
|
-
The Open module provides unified, high-level file/stream/remote I/O and filesystem utilities. It wraps plain File I/O, streaming helpers, remote fetching (wget/ssh), atomic/sensible writes, pipe/fifo helpers, gzip/bgzip/zip helpers, file-system operations (mkdir, mv, ln, cp, rm, etc.), and a lock wrapper (Lockfile). Use Open when you need robust file access, streaming, temporary/atomic writes, remote access and process-safe locking.
|
|
4
|
-
|
|
5
|
-
Sections:
|
|
6
|
-
- Opening / reading / writing files
|
|
7
|
-
- Streams, pipes and tees
|
|
8
|
-
- Sensible / atomic writes
|
|
9
|
-
- Remote fetching, caching and downloads
|
|
10
|
-
- File / filesystem helpers
|
|
11
|
-
- Locking
|
|
12
|
-
- Sync (rsync)
|
|
13
|
-
- Utilities (gzip/bgzip/grep/sort/collapse)
|
|
14
|
-
- Examples
|
|
15
|
-
- Notes and edge cases
|
|
16
|
-
|
|
17
|
-
---
|
|
18
|
-
|
|
19
|
-
## Opening / reading / writing files
|
|
20
|
-
|
|
21
|
-
Open unifies access and transparently handles compressed and remote files.
|
|
22
|
-
|
|
23
|
-
- Open.open(file, options = {}) { |io| ... } or returns an IO-like stream
|
|
24
|
-
- Accepts File paths, Path objects, IO, StringIO.
|
|
25
|
-
- Options (via IndiferentHash):
|
|
26
|
-
- :mode (default 'r') — file open mode (e.g., 'r', 'w', 'rb', etc.)
|
|
27
|
-
- :grep, :invert_grep, :fixed_grep — pipe the stream through grep
|
|
28
|
-
- :noz — if true, do not auto-decompress zip/gzip/bgz
|
|
29
|
-
- :gzip / :bgzip / :zip — force decompression
|
|
30
|
-
- For compressed files: Open detects .gz, .bgz, .zip and pipes through gzip/bgzip/unzip unless mode includes "w" (noz true).
|
|
31
|
-
- The returned stream is extended with NamedStream (has .filename and digest_str helper).
|
|
32
|
-
|
|
33
|
-
- Open.file_open(file, grep = false, mode = 'r', invert_grep = false, fixed_grep = true, options = {})
|
|
34
|
-
- Returns the basic stream (no auto-decompress). Uses get_stream to handle remote/ssh/wget or open file.
|
|
35
|
-
|
|
36
|
-
- Open.get_stream(file, mode = 'r', options = {})
|
|
37
|
-
- Low-level stream getter:
|
|
38
|
-
- If file is already a stream, returns it.
|
|
39
|
-
- If file responds to .stream, returns file.stream.
|
|
40
|
-
- If Path, resolves .find.
|
|
41
|
-
- If remote URL, delegates to Open.ssh or Open.wget.
|
|
42
|
-
- Finally falls back to File.open(File.expand_path(file), mode).
|
|
43
|
-
|
|
44
|
-
- Open.read(file, options = {}) { |line| ... } or returns String
|
|
45
|
-
- If block given: yields lines from file (fixes UTF-8 by default; suppressed via :nofix).
|
|
46
|
-
- If no block: returns full file contents (UTF-8 fixed by default).
|
|
47
|
-
- Supports :grep and :invert_grep (see tests).
|
|
48
|
-
|
|
49
|
-
- Open.write(file, content = nil, options = {})
|
|
50
|
-
- Atomic/robust writing wrapper:
|
|
51
|
-
- options default includes :mode => 'w'
|
|
52
|
-
- If mode includes 'w' ensures parent directory exists.
|
|
53
|
-
- If a block is given, yields a File object opened in mode and ensures close; on exception removes target file.
|
|
54
|
-
- If content is nil => writes empty file.
|
|
55
|
-
- If content is String => writes content.
|
|
56
|
-
- If content is an IO/StringIO => streams content into file with locking.
|
|
57
|
-
- On success calls Open.notify_write(file) and returns nil.
|
|
58
|
-
- Use for simple writes; for safer concurrency-sensitive writes prefer Open.sensible_write (below).
|
|
59
|
-
|
|
60
|
-
---
|
|
61
|
-
|
|
62
|
-
## Streams, pipes and tees
|
|
63
|
-
|
|
64
|
-
Open contains multiple helpers to produce and manage streams:
|
|
65
|
-
|
|
66
|
-
- Open.consume_stream(io, in_thread = false, into = nil, into_close = true)
|
|
67
|
-
- Consumes `io` reading blocks of size BLOCK_SIZE and writes into `into` (file or IO) or discards.
|
|
68
|
-
- If in_thread true, spawns a consumer thread and returns it (thread named and stored).
|
|
69
|
-
- Handles exceptions: closes and deletes partial files on failure; forwards abort to stream if supported.
|
|
70
|
-
|
|
71
|
-
- Open.pipe
|
|
72
|
-
- Creates an IO.pipe pair [reader, writer] and records writer for management. Returns [sout, sin] (sout reader, sin writer).
|
|
73
|
-
- Caller typically uses sout as stream to read and sin to write.
|
|
74
|
-
|
|
75
|
-
- Open.open_pipe { |sin| ... } -> returns sout
|
|
76
|
-
- Creates a pipe and runs the block with `sin` (writer) in a new thread (or fork if requested). The returned `sout` is a ConcurrentStream configured to join/handle threads/pids.
|
|
77
|
-
- Example: use for producing a stream on-the-fly for other consumers.
|
|
78
|
-
|
|
79
|
-
- Open.open_pipe(do_fork = false, close = true) yields sin in child/forked thread and returns sout for parent.
|
|
80
|
-
|
|
81
|
-
- Open.tee_stream_thread(stream) / Open.tee_stream_thread_multiple(stream, num)
|
|
82
|
-
- Duplicate input stream into multiple output pipes; returns out pipes.
|
|
83
|
-
- Uses a splitter thread that reads from the source and writes to each pipe; sets up abort callbacks and cleanup.
|
|
84
|
-
- Useful to fan-out a single stream to multiple consumers without re-reading source.
|
|
85
|
-
|
|
86
|
-
- Open.tee_stream(stream) — convenience returns two outputs.
|
|
87
|
-
|
|
88
|
-
- Open.read_stream / Open.read_stream(stream, size)
|
|
89
|
-
- Blocking reads helper to ensure reading exactly `size` bytes; raises ClosedStream if stream EOF.
|
|
90
|
-
|
|
91
|
-
- Open.with_fifo(path = nil, clean = true) { |path| ... }
|
|
92
|
-
- Create FIFO in temp path and yield it; removes it after block.
|
|
93
|
-
|
|
94
|
-
---
|
|
95
|
-
|
|
96
|
-
## Sensible / atomic writes
|
|
97
|
-
|
|
98
|
-
Use `Open.sensible_write(path, content, options = {})` for safe writes that avoid overwriting existing targets and use temporary files + atomic rename.
|
|
99
|
-
|
|
100
|
-
- Behavior:
|
|
101
|
-
- If path exists and :force not true, will consume source and skip update.
|
|
102
|
-
- Writes to a temporary file in `Open.sensible_write_dir` then moves (Open.mv) into place.
|
|
103
|
-
- Supports lock options via `:lock` key (uses Open.lock). Accepts hash of lock settings or Lockfile instance.
|
|
104
|
-
- Ensures cleanup of temp files on exception; preserves existing target if write fails.
|
|
105
|
-
- On successful move, calls Open.notify_write(path).
|
|
106
|
-
|
|
107
|
-
- Open.sensible_write_lock_dir / Open.sensible_write_dir are configurable directories (Paths) used for temporary files and lock state.
|
|
108
|
-
|
|
109
|
-
- Open.sensible_write uses Open.lock to protect move operations.
|
|
110
|
-
|
|
111
|
-
- For basic atomic writes, Open.write does attempt file lock during write (f.flock File::LOCK_EX) but sensible_write also uses safer tmp->mv semantics and optionally locking for concurrent processes.
|
|
112
|
-
|
|
113
|
-
---
|
|
114
|
-
|
|
115
|
-
## Remote fetching, caching and downloads
|
|
116
|
-
|
|
117
|
-
Open supports remote URLs and SSH-style access:
|
|
118
|
-
|
|
119
|
-
- Open.remote?(file) -> Boolean if URL-like (http|https|ftp|ssh)
|
|
120
|
-
- Open.ssh?(file) -> ssh:// scheme detection
|
|
121
|
-
- Open.ssh(file, options = {})
|
|
122
|
-
- Parses ssh://server:path and streams via `ssh server cat 'path'` (if server != 'localhost').
|
|
123
|
-
- For localhost returns Open.open(file) (local path handling).
|
|
124
|
-
|
|
125
|
-
- Open.wget(url, options = {})
|
|
126
|
-
- Download via `wget` (through CMD.cmd), returns an IO-like stream (ConcurrentStream).
|
|
127
|
-
- Options:
|
|
128
|
-
- :pipe => true (default), :autojoin => true
|
|
129
|
-
- supports `--post-data=`, cookies, quiet mode, :force, :nocache
|
|
130
|
-
- caching: unless :nocache true, saves to remote_cache_dir under a digest filename (Open.add_cache) and returns Open.open on cache file.
|
|
131
|
-
- :nice / :nice_key for throttling repeated requests with a wait
|
|
132
|
-
- Errors raise OpenURLError on failure
|
|
133
|
-
- Example: Open.wget('http://example.com', quiet: true, nocache: true).read
|
|
134
|
-
|
|
135
|
-
- Open.cache_file(url, options), Open.in_cache(url, options), Open.add_cache(url, data, options), Open.open_cache(url)
|
|
136
|
-
- Support caching remote requests to `Open.remote_cache_dir`.
|
|
137
|
-
|
|
138
|
-
- Open.download(url, file) — wrapper to run wget into local file with logging.
|
|
139
|
-
|
|
140
|
-
- Open.digest_url(url, options) — compute cache key based on url and post data/file.
|
|
141
|
-
|
|
142
|
-
- Open.scp(source_file, target_file, target:, source:) — convenience wrapper for scp and remote mkdir.
|
|
143
|
-
|
|
144
|
-
---
|
|
145
|
-
|
|
146
|
-
## File / filesystem helpers
|
|
147
|
-
|
|
148
|
-
Common filesystem operations with Path support:
|
|
149
|
-
|
|
150
|
-
- Open.mkdir(path) — ensure directory exists (mkdir_p). Accepts Path.
|
|
151
|
-
- Open.mkfiledir(target) — ensure parent dir exists for file target.
|
|
152
|
-
- Open.mv(source, target) — move with tmp intermediate to reduce risk (move to .tmp_mv.* then rename).
|
|
153
|
-
- Open.rm(file) — remove file if exists or broken symlink.
|
|
154
|
-
- Open.rm_rf(file) — recursive remove
|
|
155
|
-
- Open.touch(file) — create or update mtime (ensures parent dir).
|
|
156
|
-
- Open.cp(source, target) — copy (uses cp_r, removes existing target).
|
|
157
|
-
- Open.directory?(file)
|
|
158
|
-
- Open.exists?(file) / Open.exist? alias — existence check (Path supported).
|
|
159
|
-
- Open.ctime(file), Open.mtime(file) — time helpers; mtime has logic to follow symlinks and handle special Step info file cases.
|
|
160
|
-
- Open.size(file)
|
|
161
|
-
- Open.ln_s(source, target) — create symbolic link (ensures parent dir and remove existing).
|
|
162
|
-
- Open.ln(source, target) — create hard link (removing target if present).
|
|
163
|
-
- Open.ln_h(source, target) — attempt hard link via `ln -L`, fallback to copy on failure.
|
|
164
|
-
- Open.link(source, target) — tries ln then ln_s as fallback.
|
|
165
|
-
- Open.link_dir(source, target) — cp with hard-links (cp_lr).
|
|
166
|
-
- Open.same_file(file1, file2) — File.identical?
|
|
167
|
-
- Open.writable?(path) — checks writability handling symlinks and non-existing files.
|
|
168
|
-
- Open.realpath(file) — returns canonical realpath (resolves symlinks).
|
|
169
|
-
- Open.list(file) — returns file contents split on newline (convenience).
|
|
170
|
-
|
|
171
|
-
---
|
|
172
|
-
|
|
173
|
-
## Locking
|
|
174
|
-
|
|
175
|
-
Open wraps Lockfile to provide safe locking primitives and a simpler interface.
|
|
176
|
-
|
|
177
|
-
- Open.lock(file, unlock = true, options = {}) { |lockfile| ... }
|
|
178
|
-
- Acquire a lock (Lockfile) for a given path.
|
|
179
|
-
- `file` may be:
|
|
180
|
-
- a Lockfile instance (used directly),
|
|
181
|
-
- Path/String (lockfile path defaulting to `file + '.lock'`),
|
|
182
|
-
- nil with options[:lock] being a Lockfile instance or false.
|
|
183
|
-
- `unlock` default true; set false to keep lock after block (or raise KeepLocked inside block to keep lock and return payload).
|
|
184
|
-
- Options passed to Lockfile constructor (min_sleep, max_sleep, sleep_inc, max_age, refresh, timeout, etc.).
|
|
185
|
-
- Handles exceptions and unlocks safely in ensure.
|
|
186
|
-
- Example (from tests):
|
|
187
|
-
```ruby
|
|
188
|
-
Open.lock lockfile_path, min_sleep: 0.01, max_sleep: 0.05 do
|
|
189
|
-
# critical section
|
|
190
|
-
end
|
|
191
|
-
```
|
|
192
|
-
|
|
193
|
-
- Lockfile class is included in `open/lock/lockfile.rb` — classic NFS-safe lockfile implementation (supports refreshing, stealing detection, sweeps, retries, timeouts, etc.). Use its options via Open.lock(..., options).
|
|
194
|
-
|
|
195
|
-
---
|
|
196
|
-
|
|
197
|
-
## Sync (rsync)
|
|
198
|
-
|
|
199
|
-
- Open.rsync(source, target, options = {})
|
|
200
|
-
- Wrapper to build and execute an `rsync` command with common options.
|
|
201
|
-
- Options processed via IndiferentHash:
|
|
202
|
-
- :excludes, :files (list of files to transfer), :hard_link (use --link-dest), :test (dry-run), :print (return command), :delete, :source, :target (server strings), :other (extra args)
|
|
203
|
-
- Handles directory trailing slashes, remote server prefixes, ensures target dirs exist (remote mkdir via ssh when needed).
|
|
204
|
-
- Uses TMP files for --files-from when passing a list.
|
|
205
|
-
- Example:
|
|
206
|
-
```ruby
|
|
207
|
-
Open.rsync(source_dir, target_dir, excludes: 'tmp_dir', delete: true)
|
|
208
|
-
```
|
|
209
|
-
|
|
210
|
-
- Open.sync is alias for rsync.
|
|
211
|
-
|
|
212
|
-
---
|
|
213
|
-
|
|
214
|
-
## Utilities
|
|
215
|
-
|
|
216
|
-
- Compression helpers:
|
|
217
|
-
- Open.gzip?(file) / Open.bgzip?(file) / Open.zip?(file) — simple extension checks.
|
|
218
|
-
- Open.gunzip(stream), Open.gzip(stream), Open.bgzip(stream) — spawn subprocesses (zcat/gzip/bgzip) returning a piped IO.
|
|
219
|
-
- Open.gzip_pipe(file) — returns shell-friendly expression for gzip handling.
|
|
220
|
-
|
|
221
|
-
- Open.grep(stream, grep, invert = false, fixed = nil, options = {})
|
|
222
|
-
- Uses system grep (GREP_CMD) to filter stream. Accepts Array of patterns (written to temporary file and used with -f) or single pattern.
|
|
223
|
-
|
|
224
|
-
- Open.sort_stream(stream, header_hash: "#", cmd_args: nil, memory: false)
|
|
225
|
-
- Sort stream while preserving header lines (lines starting with header_hash).
|
|
226
|
-
- For memory=false runs external sort (env LC_ALL=C sort).
|
|
227
|
-
- Splits into substreams to avoid loading entire stream into memory for large inputs.
|
|
228
|
-
|
|
229
|
-
- Open.collapse_stream(s, line: nil, sep: "\t", header: nil, compact: false, &block)
|
|
230
|
-
- Collapses consecutive lines with same key (first field) merging rest columns with `|` separators or processed by provided block.
|
|
231
|
-
- Useful for aggregating grouped data in streaming fashion.
|
|
232
|
-
|
|
233
|
-
- Open.consume_stream described above.
|
|
234
|
-
|
|
235
|
-
- Open.notify_write(file)
|
|
236
|
-
- If `<file>.notify` exists, reads its contents and sends notification (email or system notify) and removes .notify file.
|
|
237
|
-
|
|
238
|
-
- Open.broken_link?(path) — true if symlink target missing
|
|
239
|
-
- Open.exist_or_link?(file) — exists or symlink
|
|
240
|
-
- Open.list(file) — read as lines
|
|
241
|
-
|
|
242
|
-
- Lockfile utility: Lockfile.create(path) creates lock and opens file (used internally).
|
|
243
|
-
|
|
244
|
-
---
|
|
245
|
-
|
|
246
|
-
## Examples (from tests)
|
|
247
|
-
|
|
248
|
-
Reading and line-wise processing:
|
|
249
|
-
```ruby
|
|
250
|
-
sum = 0
|
|
251
|
-
Open.read(file) { |line| sum += line.to_i }
|
|
252
|
-
```
|
|
253
|
-
|
|
254
|
-
Open compressed file:
|
|
255
|
-
```ruby
|
|
256
|
-
Open.read("file.txt.gz") # decompresses and returns content
|
|
257
|
-
```
|
|
258
|
-
|
|
259
|
-
Sensible write:
|
|
260
|
-
```ruby
|
|
261
|
-
Open.sensible_write(target_path, File.open(source)) # safe atomic write from stream
|
|
262
|
-
```
|
|
263
|
-
|
|
264
|
-
Pipe and open_pipe:
|
|
265
|
-
```ruby
|
|
266
|
-
sout = Open.open_pipe do |sin|
|
|
267
|
-
10.times { |i| sin.puts "line #{i}" }
|
|
268
|
-
end
|
|
269
|
-
# sout is a readable stream; consume:
|
|
270
|
-
Open.consume_stream(sout, false, target_file)
|
|
271
|
-
```
|
|
272
|
-
|
|
273
|
-
Tee stream to two consumers:
|
|
274
|
-
```ruby
|
|
275
|
-
sout = Open.open_pipe do |sin|
|
|
276
|
-
2000.times { |i| sin.puts "line #{i}" }
|
|
277
|
-
end
|
|
278
|
-
s1, s2 = Open.tee_stream_thread(sout)
|
|
279
|
-
t1 = Open.consume_stream(s1, true, tmp.file1)
|
|
280
|
-
t2 = Open.consume_stream(s2, true, tmp.file2)
|
|
281
|
-
t1.join; t2.join
|
|
282
|
-
```
|
|
283
|
-
|
|
284
|
-
Locking (concurrency safe):
|
|
285
|
-
```ruby
|
|
286
|
-
Open.lock(lockfile_path, min_sleep: 0.01, max_sleep: 0.05) do
|
|
287
|
-
# critical section
|
|
288
|
-
end
|
|
289
|
-
```
|
|
290
|
-
|
|
291
|
-
Rsync:
|
|
292
|
-
```ruby
|
|
293
|
-
Open.rsync(source, target)
|
|
294
|
-
Open.sync(source, target) # alias
|
|
295
|
-
```
|
|
296
|
-
|
|
297
|
-
Sorting a stream while preserving headers:
|
|
298
|
-
```ruby
|
|
299
|
-
sorted = Open.sort_stream(string_io)
|
|
300
|
-
puts sorted.read
|
|
301
|
-
```
|
|
302
|
-
|
|
303
|
-
Collapse grouped rows:
|
|
304
|
-
```ruby
|
|
305
|
-
stream = Open.collapse_stream(s, sep: " ") do |parts|
|
|
306
|
-
parts.map(&:upcase) # or aggregate
|
|
307
|
-
end
|
|
308
|
-
```
|
|
309
|
-
|
|
310
|
-
Remote fetch:
|
|
311
|
-
```ruby
|
|
312
|
-
io = Open.wget('http://example.com', quiet: true)
|
|
313
|
-
puts io.read
|
|
314
|
-
```
|
|
315
|
-
|
|
316
|
-
---
|
|
317
|
-
|
|
318
|
-
## Notes & edge cases
|
|
319
|
-
|
|
320
|
-
- Many functions accept Path objects and will call `.find` or `.produce_and_find` where appropriate.
|
|
321
|
-
- Remote functions rely on external commands (wget, ssh). Errors from those commands are wrapped/propagated (OpenURLError, ConcurrentStreamProcessFailed, etc.).
|
|
322
|
-
- Open.sensible_write and Open.write try to avoid inconsistent partial files; sensible_write uses tmp-file + mv and optional Lockfile to avoid races.
|
|
323
|
-
- Stream utilities use a ConcurrentStream abstraction (not documented here) to manage thread/pid/join semantics.
|
|
324
|
-
- Tee/splitter threads forward aborts and exceptions to downstream consumers; callers must handle cleanup and join threads.
|
|
325
|
-
- Open.lock relies on the included Lockfile implementation which supports NFS-safe locking, lock refreshing, stealing detection and sweeping stale locks.
|
|
326
|
-
- gzip/bgzip/unzip operations spawn external processes and return piped IOs — ensure you consume/join and close these streams to avoid zombies.
|
|
327
|
-
- Open.grep handles Array of patterns by writing them to a tmp file and using `-f` grep; fixed matching uses -F and -w by default.
|
|
328
|
-
|
|
329
|
-
---
|
|
330
|
-
|
|
331
|
-
This document covers the main public behaviors of the Open module: unified file/stream opening, robust writing, streaming utilities, remote fetching and caching, filesystem helpers, locking and synchronization, and convenience utilities for sorting, collapsing and grepping streams. Use Open for safe, composable I/O operations in scripts and concurrent code.
|
data/doc/Path.md
DELETED
|
@@ -1,217 +0,0 @@
|
|
|
1
|
-
# Path
|
|
2
|
-
|
|
3
|
-
Path is a lightweight path utility layered on top of the framework's Annotation system. It makes it easy to build, transform and locate resources by logical name across a variety of search maps (current, user, global, lib, etc.). Path objects are plain string values extended with Path behavior (via Path.setup or by extending instances). Many Open/Path helpers accept Path objects and will call `.find` / `.find_all` / `.produce_and_find` as needed.
|
|
4
|
-
|
|
5
|
-
Key features:
|
|
6
|
-
- Build and compose path strings fluently (join, /, method_missing).
|
|
7
|
-
- Map logical names to physical locations using configurable path maps.
|
|
8
|
-
- Find the first existing file across map order or list all matches.
|
|
9
|
-
- Helpers for file extension manipulation, globbing, dirname/basename, sanitizing filenames.
|
|
10
|
-
- Integration with Open and TmpFile utilities; supports annotation (pkgdir, libdir, map configuration).
|
|
11
|
-
- Helpers for digest/MD5 summary for files and directories.
|
|
12
|
-
|
|
13
|
-
---
|
|
14
|
-
|
|
15
|
-
## Creating / wrapping Path values
|
|
16
|
-
|
|
17
|
-
- Path.setup(str, pkgdir = nil)
|
|
18
|
-
- Convert a string into a Path (i.e., extend it with Path methods). Many test examples call `Path.setup("...")`.
|
|
19
|
-
- A Path is just a String extended with Path behavior and optional annotations (`pkgdir`, `libdir`, `path_maps`, `map_order`).
|
|
20
|
-
|
|
21
|
-
Shortcuts on Path instances:
|
|
22
|
-
- join(subpath, prevpath = nil) — join subpath to the Path (returns annotated Path)
|
|
23
|
-
- Aliases: [] and /
|
|
24
|
-
- Example:
|
|
25
|
-
```ruby
|
|
26
|
-
p = Path.setup('/tmp')
|
|
27
|
-
p.join(:foo) # => "/tmp/foo"
|
|
28
|
-
p[:bar, :foo] # => "/tmp/bar/foo"
|
|
29
|
-
p / :foo # => "/tmp/foo"
|
|
30
|
-
```
|
|
31
|
-
|
|
32
|
-
- method_missing is used to make `path.component` behave like join:
|
|
33
|
-
- `path.foo` → join("foo")
|
|
34
|
-
- Be careful: methods starting `to_` or blocks will not be treated as path components.
|
|
35
|
-
|
|
36
|
-
---
|
|
37
|
-
|
|
38
|
-
## Package / library defaults
|
|
39
|
-
|
|
40
|
-
- Path.default_pkgdir and Path.default_pkgdir= — global default package dir (default `'scout'`).
|
|
41
|
-
- Instance attributes:
|
|
42
|
-
- pkgdir — per-path override of package directory (defaults to Path.default_pkgdir).
|
|
43
|
-
- libdir — library directory (attempts to infer caller lib dir).
|
|
44
|
-
- path_maps — instance copy of global path maps (modifiable per-path).
|
|
45
|
-
- map_order — instance map order (derived from global map_order unless overridden).
|
|
46
|
-
|
|
47
|
-
---
|
|
48
|
-
|
|
49
|
-
## Path maps and find behavior
|
|
50
|
-
|
|
51
|
-
Path maps let you define templates where logical paths can be found on disk. The module ships a sensible set of maps (current, user, global, usr, local, fast, cache, bulk, lib, tmp, …) and a default map order.
|
|
52
|
-
|
|
53
|
-
- Path.path_maps — global IndiferentHash of map templates (strings containing placeholders like {TOPLEVEL}, {PKGDIR}, {SUBPATH}, {PATH}, {LIBDIR}, etc.)
|
|
54
|
-
- Path.map_order / Path.basic_map_order — global order to search maps.
|
|
55
|
-
- You can add or change maps:
|
|
56
|
-
- Path.add_path(name, map)
|
|
57
|
-
- Path.prepend_path(name, map)
|
|
58
|
-
- Path.append_path(name, map)
|
|
59
|
-
- For a Path instance you can also call add_path / prepend_path / append_path (instance-level override).
|
|
60
|
-
|
|
61
|
-
Templates can reference:
|
|
62
|
-
- {PKGDIR}, {HOME}, {RESOURCE}, {PWD}, {TOPLEVEL}, {SUBPATH}, {BASENAME}, {PATH}, {LIBDIR}, {MAPNAME}, and custom substitutions.
|
|
63
|
-
|
|
64
|
-
Finding:
|
|
65
|
-
- Path#follow(map_name = :default) — substitute template tokens and return the resulting path string (annotated). Does not check existence. Annotates result with `.where` and `.original` when requested (see annotate_found_where).
|
|
66
|
-
- Path#find(where = nil) — search for the first existing file for the Path:
|
|
67
|
-
- If path is absolute (located?), returns itself if exists or checks alternatives (.gz, .bgz, .zip).
|
|
68
|
-
- If where is given, uses that map name only.
|
|
69
|
-
- If where == :all returns an array of matching paths (see find_all).
|
|
70
|
-
- Otherwise iterates configured map_order and returns the first found path (annotated with `.where` and `.original`).
|
|
71
|
-
- Path#find_all — returns all existing matches across map_order (useful to locate duplicates).
|
|
72
|
-
- Path#find_with_extension(extension, *args) — try original then the given extension.
|
|
73
|
-
- Path.exists_file_or_alternatives(file) — helper that checks for file or file.gz/.bgz/.zip alternatives.
|
|
74
|
-
|
|
75
|
-
Helpers for locating:
|
|
76
|
-
- Path.located?(path) / instance located? — returns true if path is absolute (~/, /, ./).
|
|
77
|
-
- Path.caller_file / Path.caller_lib_dir — helper to find the caller's script dir / lib directory (used to set libdir defaults and map values).
|
|
78
|
-
- Path.follow supports advanced `{PATH/old/new}` style substitutions (see tests showing `Path.follow(path, "/some_dir/{PATH/scout/scout_commands}")`).
|
|
79
|
-
|
|
80
|
-
When a find succeeds, Path#set attributes:
|
|
81
|
-
- found.where — the map name used to find the file
|
|
82
|
-
- found.original — original logical path string before substitution
|
|
83
|
-
|
|
84
|
-
---
|
|
85
|
-
|
|
86
|
-
## Globbing and directory utilities
|
|
87
|
-
|
|
88
|
-
- Path#glob(pattern = "*") — list children matching pattern if this Path points to a directory; returns annotated Path instances.
|
|
89
|
-
- Path#glob_all(pattern = nil) — search across path maps and return all matching annotated paths.
|
|
90
|
-
- Path#directory? — true if the found path is a directory.
|
|
91
|
-
- Path.dirname / Path.basename — string helpers returning annotated strings.
|
|
92
|
-
|
|
93
|
-
---
|
|
94
|
-
|
|
95
|
-
## Filename / extension utilities
|
|
96
|
-
|
|
97
|
-
- Path.is_filename?(string, need_to_exists = true) — static predicate to test if a value looks like a filename.
|
|
98
|
-
- Path.sanitize_filename(filename, length = 254) — shorten a filename safely preserving an extension and adding a digest postfix when needed.
|
|
99
|
-
- Extension helpers:
|
|
100
|
-
- get_extension(multiple = false) — last extension or multiple.
|
|
101
|
-
- set_extension(extension) — return path with extension appended.
|
|
102
|
-
- unset_extension — remove last extension.
|
|
103
|
-
- remove_extension(extension = nil) — remove a specific extension or unset last.
|
|
104
|
-
- replace_extension(new_extension, multiple = false) — replace extension(s).
|
|
105
|
-
- Path.relative_to(dir) — returns relative path from dir to this path (uses Misc.path_relative_to).
|
|
106
|
-
|
|
107
|
-
---
|
|
108
|
-
|
|
109
|
-
## Misc utilities
|
|
110
|
-
|
|
111
|
-
- Path.digest_str — extension in path/digest.rb:
|
|
112
|
-
- If path is a file and exists → "File MD5: <md5>".
|
|
113
|
-
- If path is a directory → "Directory MD5: <digest of glob>".
|
|
114
|
-
- Otherwise returns quoted path string.
|
|
115
|
-
|
|
116
|
-
- Path.no_method_missing — remove the module-level method_missing implementation (rarely used).
|
|
117
|
-
|
|
118
|
-
- TmpFile.with_path — helper that yields a path string and ensures it is a Path (via Path.setup).
|
|
119
|
-
|
|
120
|
-
- Path.newer?(path, file, by_link = false) — compare mtimes; returns truthy if `path` is newer than `file`. Handles non-existing files and optionally compares lstat (links).
|
|
121
|
-
|
|
122
|
-
---
|
|
123
|
-
|
|
124
|
-
## Integration & annotations
|
|
125
|
-
|
|
126
|
-
Path extends the Annotation module; each Path can carry annotations:
|
|
127
|
-
- `pkgdir`, `libdir`, `path_maps`, `map_order`. These let different packages or code contexts override search behavior for a path instance.
|
|
128
|
-
|
|
129
|
-
`Path.setup` creates a Path with default annotations:
|
|
130
|
-
- `pkgdir` defaults to Path.default_pkgdir (usually 'scout').
|
|
131
|
-
- `libdir` defaults to caller library directory (Path.caller_lib_dir).
|
|
132
|
-
|
|
133
|
-
Examples from tests and common use:
|
|
134
|
-
|
|
135
|
-
- Compose paths:
|
|
136
|
-
```ruby
|
|
137
|
-
p = Path.setup('/tmp')
|
|
138
|
-
p.join(:foo) # => "/tmp/foo"
|
|
139
|
-
p[:bar, :foo] # => "/tmp/bar/foo"
|
|
140
|
-
p.foo[:bar] # => "/tmp/foo/bar"
|
|
141
|
-
```
|
|
142
|
-
|
|
143
|
-
- Find a file across path maps:
|
|
144
|
-
```ruby
|
|
145
|
-
p = Path.setup("share/data/some_file", 'scout')
|
|
146
|
-
p.find(:usr) # resolves using :usr map -> "/usr/share/scout/data/some_file"
|
|
147
|
-
p.find # searches map_order and returns first existing match
|
|
148
|
-
p.find.where # map name where it was found (e.g. :current)
|
|
149
|
-
p.find.original # original logical path prior to substitution
|
|
150
|
-
```
|
|
151
|
-
|
|
152
|
-
- Search all matches:
|
|
153
|
-
```ruby
|
|
154
|
-
Path.setup("share/data/some_file", 'scout').find_all
|
|
155
|
-
```
|
|
156
|
-
|
|
157
|
-
- Add custom search paths:
|
|
158
|
-
```ruby
|
|
159
|
-
file = Path.setup("somefile")
|
|
160
|
-
file.append_path('dir1', '/tmp/dir1')
|
|
161
|
-
file.prepend_path('dir2', '/opt/dir2')
|
|
162
|
-
```
|
|
163
|
-
|
|
164
|
-
- Work with extensions:
|
|
165
|
-
```ruby
|
|
166
|
-
p = Path.setup("/home/.scout/dir/file.txt")
|
|
167
|
-
p.unset_extension # => "/home/.scout/dir/file"
|
|
168
|
-
p.replace_extension('tsv')
|
|
169
|
-
```
|
|
170
|
-
|
|
171
|
-
- Glob directories:
|
|
172
|
-
```ruby
|
|
173
|
-
dir = Path.setup(tmpdir)
|
|
174
|
-
dir.glob # returns annotated Path children
|
|
175
|
-
```
|
|
176
|
-
|
|
177
|
-
---
|
|
178
|
-
|
|
179
|
-
## Notes & edge cases
|
|
180
|
-
|
|
181
|
-
- Path uses `Path.located?` to decide if a path is already absolute / explicitly located (leading `/`, `~/` or `./`).
|
|
182
|
-
- If `find` is called on a non-located path, it searches maps in `map_order`. Map templates are string patterns — if a map is missing `Path.find` will fallback to the default map.
|
|
183
|
-
- `find` will also check for compressed alternatives (filename.gz, .bgz, .zip) via helper `exists_file_or_alternatives`.
|
|
184
|
-
- The system allows per-path overrides of `pkgdir` and `path_maps` — useful for testing and package-specific layouts.
|
|
185
|
-
- `Path.follow` performs token substitution and supports nested `{PATH/.../...}` style replacements.
|
|
186
|
-
- `caller_lib_dir` and `caller_file` try to infer the caller's library directory; used to set sensible defaults for `libdir`.
|
|
187
|
-
- Many helpers in the framework accept Path objects and will call `.find` (or `.produce_and_find` when supported). Use Path.setup to annotate and pass Path objects to other APIs.
|
|
188
|
-
|
|
189
|
-
---
|
|
190
|
-
|
|
191
|
-
## API quick reference
|
|
192
|
-
|
|
193
|
-
- Creation / basics:
|
|
194
|
-
- Path.setup(string, pkgdir=nil)
|
|
195
|
-
- path.join(subpath, prevpath=nil) — aliases: path[:x], path / :x
|
|
196
|
-
- path.method_missing to allow `path.foo` as join
|
|
197
|
-
|
|
198
|
-
- Mapping & locating:
|
|
199
|
-
- Path.path_maps, Path.map_order, Path.add_path / prepend_path / append_path
|
|
200
|
-
- path.follow(map_name=:default) — expand template without checking existence
|
|
201
|
-
- path.find(where = nil) — locate first existing match (annotates `.where` and `.original`)
|
|
202
|
-
- path.find_all — return all existing matches across maps
|
|
203
|
-
- path.find_with_extension(extension, *args)
|
|
204
|
-
|
|
205
|
-
- Filesystem / filename helpers:
|
|
206
|
-
- path.glob(pattern="*"), path.glob_all(pattern=nil)
|
|
207
|
-
- path.dirname, path.basename
|
|
208
|
-
- path.get_extension, set_extension, unset_extension, remove_extension, replace_extension
|
|
209
|
-
- Path.sanitize_filename, Path.is_filename?
|
|
210
|
-
|
|
211
|
-
- Utilities:
|
|
212
|
-
- Path.located?(s), path.located?
|
|
213
|
-
- Path.caller_file, Path.caller_lib_dir
|
|
214
|
-
- Path.digest_str (file/directory MD5 summary)
|
|
215
|
-
- Path.newer?(path, file, by_link = false)
|
|
216
|
-
|
|
217
|
-
This document summarizes the Path module's purpose and primary surface area. Use Path.setup to get annotated strings that interoperate with Open and other framework utilities to find and manipulate package-oriented filesystem resources.
|