scout-essentials 1.8.8 → 1.9.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (65) hide show
  1. checksums.yaml +4 -4
  2. data/.vimproject +26 -12
  3. data/README.md +83 -112
  4. data/VERSION +1 -1
  5. data/doc/Improvements.md +226 -0
  6. data/doc/StartHere.md +122 -0
  7. data/doc/developer/AnnotationSystem.md +184 -0
  8. data/doc/developer/Architecture.md +147 -0
  9. data/doc/developer/Configuration.md +238 -0
  10. data/doc/developer/CoreUtilities.md +265 -0
  11. data/doc/developer/DesignPrinciples.md +129 -0
  12. data/doc/developer/ErrorHandling.md +203 -0
  13. data/doc/developer/LockingAndConcurrency.md +157 -0
  14. data/doc/developer/PathResolution.md +200 -0
  15. data/doc/developer/PersistenceAndResources.md +119 -0
  16. data/doc/developer/StreamingModel.md +236 -0
  17. data/doc/user/AnnotatingData.md +202 -0
  18. data/doc/user/CachingResults.md +183 -0
  19. data/doc/user/CommandLineOptions.md +189 -0
  20. data/doc/user/Cookbook.md +211 -0
  21. data/doc/user/HandlingStreams.md +236 -0
  22. data/doc/user/LoggingAndProgress.md +158 -0
  23. data/doc/user/ProducingResources.md +177 -0
  24. data/doc/user/RemoteData.md +157 -0
  25. data/doc/user/RunningCommands.md +218 -0
  26. data/doc/user/WorkingWithFiles.md +217 -0
  27. data/lib/scout/cmd.rb +343 -40
  28. data/lib/scout/concurrent_stream.rb +14 -1
  29. data/lib/scout/indiferent_hash.rb +1 -1
  30. data/lib/scout/log/fingerprint.rb +13 -8
  31. data/lib/scout/log/progress/report.rb +1 -1
  32. data/lib/scout/log.rb +4 -1
  33. data/lib/scout/misc/format.rb +24 -0
  34. data/lib/scout/named_array.rb +1 -1
  35. data/lib/scout/open/stream.rb +2 -2
  36. data/lib/scout/open/util.rb +4 -0
  37. data/lib/scout/path/find.rb +3 -2
  38. data/lib/scout/persist.rb +14 -10
  39. data/research/annotations-data-analysis.md +206 -0
  40. data/research/behavior-probes.md +1925 -0
  41. data/research/commands-streaming-analysis.md +272 -0
  42. data/research/design-philosophy-analysis.md +383 -0
  43. data/research/doc-audit-findings.md +294 -0
  44. data/research/ecosystem-attribution.md +118 -0
  45. data/research/implementation-inventory-core.md +1029 -0
  46. data/research/implementation-inventory-open.md +417 -0
  47. data/research/implementation-inventory-path-persist-resource.md +774 -0
  48. data/research/io-paths-analysis.md +228 -0
  49. data/research/persistence-resources-analysis.md +244 -0
  50. data/research/synthesis-report.md +80 -0
  51. data/scout-essentials.gemspec +37 -15
  52. data/test/scout/test_cmd.rb +411 -0
  53. metadata +36 -14
  54. data/doc/Annotation.md +0 -352
  55. data/doc/CMD.md +0 -363
  56. data/doc/ConcurrentStream.md +0 -163
  57. data/doc/IndiferentHash.md +0 -240
  58. data/doc/Log.md +0 -235
  59. data/doc/NamedArray.md +0 -174
  60. data/doc/Open.md +0 -331
  61. data/doc/Path.md +0 -217
  62. data/doc/Persist.md +0 -214
  63. data/doc/Resource.md +0 -229
  64. data/doc/SimpleOPT.md +0 -236
  65. data/doc/TmpFile.md +0 -154
data/doc/TmpFile.md DELETED
@@ -1,154 +0,0 @@
1
- # TmpFile
2
-
3
- TmpFile provides small helpers to create and manage temporary files and directories used throughout the framework. It offers safe temporary-path generation, scoped helpers that create and remove temporary files/dirs automatically, and a persistence-path helper (tmp_for_file) used by caching/persistence code to build stable cache filenames.
4
-
5
- Files: lib/scout/tmpfile.rb
6
-
7
- ---
8
-
9
- ## Key constants & configuration
10
-
11
- - TmpFile.MAX_FILE_LENGTH = 150 — max length used by tmp_for_file before truncating and appending a digest.
12
- - TmpFile.tmpdir — base temporary directory used by tmp utilities (defaults to user tmp under $HOME: `~/tmp/scout/tmpfiles`).
13
- - You can set: `TmpFile.tmpdir = "/some/dir"`.
14
-
15
- Helpers:
16
- - TmpFile.user_tmp(subdir = nil) — returns user-scoped base tmp dir (under $HOME/tmp/scout). If `subdir` provided it is appended.
17
-
18
- ---
19
-
20
- ## Filename helpers
21
-
22
- - TmpFile.random_name(prefix = 'tmp-', max = 1_000_000_000)
23
- - Return a random name with the given prefix and a random integer (0..max).
24
-
25
- - TmpFile.tmp_file(prefix = 'tmp-', max = 1_000_000_000, dir = nil)
26
- - Returns a path inside `dir` (defaults to `TmpFile.tmpdir`) composed of prefix + random number.
27
- - If `dir` is a Path it will be `.find`ed.
28
-
29
- ---
30
-
31
- ## Scoped helpers
32
-
33
- These helpers create temporary files/directories, yield them to the caller, and delete them afterward (by default).
34
-
35
- - TmpFile.with_file(content = nil, erase = true, options = {}) { |tmpfile| ... }
36
- - Create a temporary file path and optionally pre-populate it with `content`.
37
- - Parameters:
38
- - `content`:
39
- - If String: write content into file.
40
- - If IO/StringIO: read its contents and write into tmp file.
41
- - If nil: tmp file is created empty.
42
- - If `content` is a Hash, it is treated as options (content=nil).
43
- - `erase` (default true): remove the tmp file after the block completes.
44
- - `options` (Hash):
45
- - `:prefix` — filename prefix (default `'tmp-'`).
46
- - `:max` — random suffix max integer.
47
- - `:tmpdir` — directory to write the tmp file into.
48
- - `:extension` — append `.extension` to tmp file name.
49
- - Behavior:
50
- - Ensures tmpdir exists (Open.mkdir).
51
- - Handles IO content safely by reading readpartial until EOF.
52
- - Yields the tmp file path (string) to the block.
53
- - After the block returns, removes the tmp file when `erase` is true and file exists.
54
- - Examples:
55
- ```ruby
56
- TmpFile.with_file("Hello") do |file|
57
- puts File.read(file) # => "Hello"
58
- end
59
- ```
60
-
61
- - TmpFile.with_dir(erase = true, options = {}) { |tmpdir| ... }
62
- - Create a temporary directory (using tmp_file for a unique name), yield its path, and remove it after block if `erase` true.
63
- - `options[:prefix]` may change directory name prefix.
64
- - Example:
65
- ```ruby
66
- TmpFile.with_dir do |dir|
67
- # dir is a path to a temporary directory
68
- end
69
- ```
70
-
71
- - TmpFile.in_dir(*args) { |dir| ... }
72
- - Convenience that creates a temporary directory and executes the block with the current working directory changed to that directory (uses `Misc.in_dir` internally).
73
-
74
- ---
75
-
76
- ## Persistence path helper
77
-
78
- - TmpFile.tmp_for_file(file, tmp_options = {}, other_options = {})
79
- - Generates a stable temporary/persistent filename for a logical file name plus options. Used by persistence/caching logic to build consistent cache files per logical input and options.
80
- - Returns a Path (string extended with Path) under the chosen persistence directory.
81
- - Parameters:
82
- - `file` — logical filename or Path used to build the identifier.
83
- - `tmp_options` may include:
84
- - `:file` — return value override (internal)
85
- - `:prefix` — prefix for the identifier (default: based on file)
86
- - `:key` — optional key appended in identifier (`[...]`).
87
- - `:dir` — base directory for persistence (defaults to `TmpFile.tmpdir`).
88
- - `other_options` — additional options whose digest will be appended to the filename (used to make identifier unique for variations like filters).
89
- - Special handling:
90
- - Replaces path separators with `SLASH_REPLACE` (character `·`) to make a single filename.
91
- - Truncates long filenames (over MAX_FILE_LENGTH) and appends a short digest to avoid filesystem limits.
92
- - Appends a digest of `other_options` (unless empty) to ensure uniqueness when options differ.
93
- - Use cases:
94
- - Build cache file path for a content produced from input + parameters.
95
- - Example (simplified):
96
- ```ruby
97
- p = TmpFile.tmp_for_file("data.tsv", dir: Path.setup("var/cache"))
98
- ```
99
-
100
- ---
101
-
102
- ## Behavior / edge cases
103
-
104
- - `with_file` and `with_dir` remove created resources after the block only if `erase` true and the file/dir exists.
105
- - `with_file` supports passing options as the first argument (if `content` is a Hash).
106
- - `with_file` writes IO content using `readpartial` in chunks (handles large IOs without loading entire contents into memory).
107
- - `tmp_for_file` uses a safe character `SLASH_REPLACE` (`'·'`) for replaced `/` characters — results in single-file names representing nested logical paths.
108
- - `tmp_for_file` truncates overly long identifiers and appends a digest of the remainder to keep the filename length reasonable.
109
- - The helper returns a Path-like value when `persistence_dir` is a Path.
110
-
111
- ---
112
-
113
- ## Examples (from tests)
114
-
115
- - Create a temporary file with content:
116
- ```ruby
117
- TmpFile.with_file("Hello World!") do |file|
118
- assert_equal "Hello World!", File.read(file)
119
- end
120
- ```
121
-
122
- - Create a temporary file from an IO and consume into another temporary:
123
- ```ruby
124
- TmpFile.with_file("Hello") do |file1|
125
- Open.open(file1) do |io|
126
- TmpFile.with_file(io) do |file2|
127
- assert_equal "Hello", File.read(file2)
128
- end
129
- end
130
- end
131
- ```
132
-
133
- - Temporary directory and change into it:
134
- ```ruby
135
- TmpFile.in_dir do |dir|
136
- # current working directory is dir inside the block
137
- end
138
- ```
139
-
140
- - Build a persistent cache path for a logical filename + options:
141
- ```ruby
142
- cache_path = TmpFile.tmp_for_file("input.tsv", dir: Path.setup("var/cache"))
143
- ```
144
-
145
- ---
146
-
147
- ## Implementation notes
148
-
149
- - TmpFile uses `Open` utilities to create directories and write files.
150
- - `tmp_file` returns plain string paths; callers often wrap them in Path.setup when needed.
151
- - Filenames are sanitized (spaces replaced with `_`) and slashes replaced with `·` for single-file representation of nested paths in `tmp_for_file`.
152
- - The module is intentionally minimal but used pervasively by other framework components like Persist and Resource to generate stable temporary / cache paths.
153
-
154
- Use TmpFile helpers when you need temporary files/dirs or stable cache filenames with predictable cleanup behavior.