scout-gear 10.12.2 → 10.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.vimproject +32 -8
- data/README.md +179 -329
- data/VERSION +1 -1
- data/doc/Improvements.md +239 -0
- data/doc/StartHere.md +71 -0
- data/doc/developer/Architecture.md +216 -0
- data/doc/developer/ConcurrencyModel.md +222 -0
- data/doc/developer/DesignPrinciples.md +246 -0
- data/doc/developer/EntitySystem.md +231 -0
- data/doc/developer/PersistenceEngines.md +163 -0
- data/doc/developer/TSVInternals.md +310 -0
- data/doc/developer/WorkflowEngine.md +296 -0
- data/doc/user/BuildingWorkflows.md +394 -0
- data/doc/user/CachingData.md +189 -0
- data/doc/user/Cookbook.md +255 -0
- data/doc/user/HPCBatchExecution.md +159 -0
- data/doc/user/ManagingRelationships.md +207 -0
- data/doc/user/ProcessingTabularData.md +315 -0
- data/doc/user/RunningParallelWork.md +196 -0
- data/doc/user/WorkingWithEntities.md +238 -0
- data/lib/scout/knowledge_base/query.rb +2 -2
- data/lib/scout/persist/tsv/adapter/base.rb +5 -1
- data/lib/scout/persist/tsv/serialize.rb +3 -2
- data/lib/scout/tsv/transformer.rb +11 -3
- data/lib/scout/workflow/definition.rb +14 -14
- data/lib/scout/workflow/deployment/scheduler/lfs.rb +1 -1
- data/lib/scout/workflow/entity.rb +1 -1
- data/lib/scout/workflow/step/file.rb +2 -2
- data/lib/scout/workflow/step/info.rb +17 -2
- data/lib/scout/workflow/step/load.rb +3 -1
- data/lib/scout/workflow/step/provenance.rb +1 -1
- data/lib/scout/workflow/task/dependencies.rb +1 -2
- data/lib/scout/workflow/task.rb +6 -1
- data/lib/scout/workflow/usage.rb +0 -1
- data/lib/scout/workflow.rb +11 -5
- data/lib/scout-gear.rb +2 -3
- data/research/00_scope_and_themes.md +150 -0
- data/research/design-philosophy-analysis.md +263 -0
- data/research/downstream-usage.md +130 -0
- data/research/entity-association-kb-analysis.md +265 -0
- data/research/identifiers-mechanism.md +170 -0
- data/research/persistence-concurrency-analysis.md +316 -0
- data/research/repo-map.md +124 -0
- data/research/synthesis-report.md +106 -0
- data/research/tsv-internals-analysis.md +221 -0
- data/research/validation-report.md +136 -0
- data/research/workflow-engine-analysis.md +213 -0
- data/scout-gear.gemspec +31 -11
- data/scout_commands/kb/query +1 -1
- data/scout_commands/kb/traverse +1 -1
- data/scout_commands/rbbt +25 -23
- data/scout_commands/system/status +1 -1
- data/scout_commands/workflow/prov +1 -1
- data/test/scout/workflow/test_step.rb +31 -0
- metadata +30 -10
- data/doc/Association.md +0 -288
- data/doc/Entity.md +0 -296
- data/doc/KnowledgeBase.md +0 -433
- data/doc/Persist.md +0 -356
- data/doc/Semaphore.md +0 -171
- data/doc/TSV.md +0 -449
- data/doc/WorkQueue.md +0 -359
- data/doc/Workflow.md +0 -744
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9a700bc307360c342dd671a87d0a5ddf69519506c52a0ac50d9d1580705a6f3e
|
|
4
|
+
data.tar.gz: 66694104c9005d0ba4e979f6a0120e029afe42c34070f013b0105646e0c6619f
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 664c8e59a165edfc9d8291dce01e15bd913c388f6de455c21f88108f5dbc23147b8d3312edf79b37516303d4ecb37bb0b2dc4473b32668d8dfddf91515d80710
|
|
7
|
+
data.tar.gz: 41417a3686d7d2bea6173296fbe21eb8862e3dc0a00de45a0d663340e87b25834df9ca6aefa5e7e5fdcd9dde65649f94df734249e06bbbe37601da3d3c55737a
|
data/.vimproject
CHANGED
|
@@ -2,6 +2,9 @@ scout-gear=/$PWD filter="*.rb *.yaml" {
|
|
|
2
2
|
Rakefile
|
|
3
3
|
README.md
|
|
4
4
|
chats=chats filter="*"{
|
|
5
|
+
test=test{
|
|
6
|
+
kb.rb
|
|
7
|
+
}
|
|
5
8
|
|
|
6
9
|
purge
|
|
7
10
|
|
|
@@ -25,6 +28,15 @@ scout-gear=/$PWD filter="*.rb *.yaml" {
|
|
|
25
28
|
doc=doc{
|
|
26
29
|
documenter.rb
|
|
27
30
|
}
|
|
31
|
+
|
|
32
|
+
documentation
|
|
33
|
+
|
|
34
|
+
README_doc
|
|
35
|
+
|
|
36
|
+
documentation_fix
|
|
37
|
+
doc_learning
|
|
38
|
+
|
|
39
|
+
doc_learning_chat_analyst
|
|
28
40
|
}
|
|
29
41
|
bin=bin filter="*"{
|
|
30
42
|
scout
|
|
@@ -316,13 +328,25 @@ scout-gear=/$PWD filter="*.rb *.yaml" {
|
|
|
316
328
|
}
|
|
317
329
|
}
|
|
318
330
|
doc=doc filter="*"{
|
|
319
|
-
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
331
|
+
Improvements.md
|
|
332
|
+
StartHere.md
|
|
333
|
+
user=user{
|
|
334
|
+
BuildingWorkflows.md
|
|
335
|
+
CachingData.md
|
|
336
|
+
Cookbook.md
|
|
337
|
+
ManagingRelationships.md
|
|
338
|
+
ProcessingTabularData.md
|
|
339
|
+
RunningParallelWork.md
|
|
340
|
+
WorkingWithEntities.md
|
|
341
|
+
}
|
|
342
|
+
developer=developer{
|
|
343
|
+
Architecture.md
|
|
344
|
+
ConcurrencyModel.md
|
|
345
|
+
DesignPrinciples.md
|
|
346
|
+
EntitySystem.md
|
|
347
|
+
PersistenceEngines.md
|
|
348
|
+
TSVInternals.md
|
|
349
|
+
WorkflowEngine.md
|
|
350
|
+
}
|
|
327
351
|
}
|
|
328
352
|
}
|
data/README.md
CHANGED
|
@@ -1,352 +1,202 @@
|
|
|
1
1
|
# Scout Gear
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
---
|
|
30
|
-
|
|
31
|
-
## How command-line interfaces work (scout …)
|
|
32
|
-
|
|
33
|
-
Scout provides a single “scout” command that discovers and runs nested subcommands from any installed Scout package. Scripts are discovered using the Path subsystem across PATH-like roots, enabling workflows or packages to inject their own commands.
|
|
34
|
-
|
|
35
|
-
Basics:
|
|
36
|
-
- The CLI resolves terms left-to-right until a file is found under a scout_commands tree.
|
|
37
|
-
- Example: scout workflow task runs scout_commands/workflow/task
|
|
38
|
-
- Example: all TSV-related scripts are under scout_commands/tsv and can be listed with scout tsv
|
|
39
|
-
- If the path resolves to a directory instead of a script, a list of available subcommands in that directory is shown.
|
|
40
|
-
- Remaining ARGV is parsed by the selected script using SimpleOPT (SOPT) or compatible parsers.
|
|
41
|
-
- Because discovery uses Path maps, commands contributed by other packages or installed workflows are automatically found.
|
|
42
|
-
|
|
43
|
-
See the per-module CLI sections below for TSV, Workflow, and KnowledgeBase.
|
|
44
|
-
|
|
45
|
-
---
|
|
46
|
-
|
|
47
|
-
## Scout Essentials: Core building blocks
|
|
48
|
-
|
|
49
|
-
Scout Gear depends on the following main modules from scout-essentials. You’ll use these directly for filesystem/resource orchestration, external command execution, caching, and options handling.
|
|
50
|
-
|
|
51
|
-
### Path
|
|
52
|
-
|
|
53
|
-
doc/Path.md
|
|
54
|
-
|
|
55
|
-
Path is a lightweight, annotation-enabled “smart string” for composing and locating project resources across multiple search maps (current/user/global/lib/tmp, etc.). It integrates with Open and Persist.
|
|
56
|
-
|
|
57
|
-
Highlights:
|
|
58
|
-
- Path.setup("str") turns a String into a Path with join via [], /, or method_missing (path.foo.bar)
|
|
59
|
-
- Map logical locations to physical roots with path maps; find the first match across map order with path.find (and path.find_all)
|
|
60
|
-
- Filename helpers: get/set/replace/unset extensions; sanitize filenames; relative paths
|
|
61
|
-
- Directory helpers: glob and glob_all over maps; dirname/basename; realpath; newer?
|
|
62
|
-
- Digest summaries: path.digest_str summarizes files/dirs for logging/debugging
|
|
63
|
-
|
|
64
|
-
Usage:
|
|
65
|
-
```ruby
|
|
66
|
-
p = Path.setup('share/data/myfile')
|
|
67
|
-
p.find # resolve across configured maps
|
|
68
|
-
p[:subdir, :file] # joins => share/data/subdir/file
|
|
3
|
+
**Workflows, data, entities and persistence for the Scout ecosystem.**
|
|
4
|
+
|
|
5
|
+
Scout turns computation into persistent, composable, inspectable work.
|
|
6
|
+
Scout-gear is the workflow and data-processing layer of the Scout
|
|
7
|
+
ecosystem: it provides a workflow engine for computational pipelines, a
|
|
8
|
+
tabular data system (TSV), an entity system for typed identifiers, a
|
|
9
|
+
knowledge base for relationship queries, and specialized persistence
|
|
10
|
+
engines. It builds on
|
|
11
|
+
[scout-essentials](https://github.com/mikisvaz/scout-essentials), which
|
|
12
|
+
provides the foundational utilities (file I/O, annotations, paths,
|
|
13
|
+
logging, concurrency streams).
|
|
14
|
+
|
|
15
|
+
New to Scout? Start with [StartHere](doc/StartHere.md) for an
|
|
16
|
+
audience-oriented route into the documentation.
|
|
17
|
+
|
|
18
|
+
## The Scout model
|
|
19
|
+
|
|
20
|
+
In scout-gear a task is not a transient function call. Running a job
|
|
21
|
+
creates a **Step**: a concrete unit of work with typed inputs, declared
|
|
22
|
+
dependencies, and a unique path derived from those inputs. Because the
|
|
23
|
+
path identifies the work, the result is written to disk, reused on
|
|
24
|
+
re-run, and only recomputed when the inputs change. Everything that
|
|
25
|
+
happened to produce it is recorded with it.
|
|
26
|
+
|
|
27
|
+
```text
|
|
28
|
+
task -> job (Step) -> persistent result -> provenance -> composition / orchestration
|
|
69
29
|
```
|
|
70
30
|
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
doc/Open.md
|
|
31
|
+
The canonical example:
|
|
74
32
|
|
|
75
|
-
Open unifies file/stream/remote I/O, atomic writes, pipes/tees/FIFOs, (bg)zip helpers, rsync/sync, and lock handling.
|
|
76
|
-
|
|
77
|
-
Highlights:
|
|
78
|
-
- Open.open/read/write with auto-(de)compression for .gz/.bgz/.zip and remote urls (wget/ssh)
|
|
79
|
-
- Streams: open_pipe, tee_stream, consume_stream, with_fifo
|
|
80
|
-
- Safe writes: sensible_write (tmp + atomic rename + optional locks)
|
|
81
|
-
- Remote: wget with caching, ssh/scp, digest_url, remote cache
|
|
82
|
-
- Filesystem: mkdir/mkfiledir, mv/cp/ln/link_dir, rm/rm_rf, same_file?, exists?, writable?
|
|
83
|
-
- Locking: Open.lock wraps a robust Lockfile (NFS-safe) with refresh/timeout/steal
|
|
84
|
-
|
|
85
|
-
Example:
|
|
86
|
-
```ruby
|
|
87
|
-
Open.sensible_write("out.txt", Open.open("http://example.com"))
|
|
88
|
-
Open.with_fifo { |fifo| ... }
|
|
89
|
-
Open.rsync("src/", "user@server:dst/", delete: true)
|
|
90
|
-
```
|
|
91
|
-
|
|
92
|
-
### CMD
|
|
93
|
-
|
|
94
|
-
doc/CMD.md
|
|
95
|
-
|
|
96
|
-
CMD wraps Open3.popen3 with robust patterns for streaming, stderr logging, stdin feeding, auto-join of producers, and tool discovery/installation.
|
|
97
|
-
|
|
98
|
-
Highlights:
|
|
99
|
-
- CMD.cmd("tool args", pipe: true, in: io_or_string, stderr: Log::HIGH, autojoin: true)
|
|
100
|
-
- ConcurrentStream-enabled stdout with join/error propagation
|
|
101
|
-
- Convenience: CMD.bash("bash -l -c '...'"), cmd_pid/cmd_log
|
|
102
|
-
- Tool registry: CMD.tool, CMD.get_tool (auto-install via conda or producers), version scanning
|
|
103
|
-
|
|
104
|
-
Example:
|
|
105
|
-
```ruby
|
|
106
|
-
io = CMD.cmd("cut", "-f" => 2, "-d" => " ", in: "a b", pipe: true)
|
|
107
|
-
io.read # => "b\n"; io.join
|
|
108
|
-
```
|
|
109
|
-
|
|
110
|
-
### IndiferentHash
|
|
111
|
-
|
|
112
|
-
doc/IndiferentHash.md
|
|
113
|
-
|
|
114
|
-
Hash mixin for indifferent access (string/symbol keys equal), deep-merge, options parsing, and string<->hash conversions.
|
|
115
|
-
|
|
116
|
-
Highlights:
|
|
117
|
-
- IndiferentHash.setup(hash) to extend a single hash instance
|
|
118
|
-
- Access with h[:a] == h["a"]; delete/include? are indifferent
|
|
119
|
-
- Helpers: deep_merge, values_at with indifferent keys, slice, except
|
|
120
|
-
- Options utilities: parse_options, process_options, positional2hash, hash2string/string2hash
|
|
121
|
-
|
|
122
|
-
Example:
|
|
123
|
-
```ruby
|
|
124
|
-
opts = IndiferentHash.parse_options('limit=10 title="A title"')
|
|
125
|
-
opts[:title] # => "A title"
|
|
126
|
-
```
|
|
127
|
-
|
|
128
|
-
### Persist (core serialization/caching)
|
|
129
|
-
|
|
130
|
-
doc/Persist.md (essentials)
|
|
131
|
-
|
|
132
|
-
Typed serialization (json/yaml/marshal/binary/arrays), atomic saves, and the high-level persist pattern with locking and streaming.
|
|
133
|
-
|
|
134
|
-
Highlights:
|
|
135
|
-
- Persist.save/load(obj, file, type)
|
|
136
|
-
- Persist.persist(name, type, dir: ...) { compute_or_stream }
|
|
137
|
-
- Locking and tmp-to-final atomic writes
|
|
138
|
-
- Streaming tee: one copy to file, one to caller
|
|
139
|
-
- Memory cache: Persist.memory(name) { ... }
|
|
140
|
-
- Helpers to parse YAML/JSON/Marshal via Open
|
|
141
|
-
|
|
142
|
-
Example:
|
|
143
|
-
```ruby
|
|
144
|
-
val = Persist.persist("expensive", :json) { compute_hash }
|
|
145
|
-
# subsequent calls load cached JSON unless :update or stale
|
|
146
|
-
```
|
|
147
|
-
|
|
148
|
-
### Resource
|
|
149
|
-
|
|
150
|
-
doc/Resource.md
|
|
151
|
-
|
|
152
|
-
Resource system to claim and produce files on demand (string/proc/url/rake/installers), integrated with Path/Open and locking.
|
|
153
|
-
|
|
154
|
-
Highlights:
|
|
155
|
-
- claim path => (:string, :proc, :url, :rake, :install)
|
|
156
|
-
- Produce on demand via path.produce and path.open/read
|
|
157
|
-
- Rake integration: drive file tasks/rules to generate outputs
|
|
158
|
-
- Install software into a per-resource “software” dir and update env
|
|
159
|
-
|
|
160
|
-
Example:
|
|
161
|
-
```ruby
|
|
162
|
-
module MyPkg
|
|
163
|
-
extend Resource
|
|
164
|
-
claim root.tmp.test.hello, :string, "Hello"
|
|
165
|
-
end
|
|
166
|
-
MyPkg.tmp.test.hello.read # produces if missing, then reads
|
|
167
|
-
```
|
|
168
|
-
|
|
169
|
-
Other essentials you’ll encounter:
|
|
170
|
-
- Annotation / AnnotatedArray / NamedArray: lightweight typed attributes on objects and arrays; named tuple-style rows
|
|
171
|
-
- ConcurrentStream: concurrency-aware streams with join/abort/callbacks
|
|
172
|
-
- SimpleOPT (SOPT): tiny CLI option DSL/parser; used by scout commands
|
|
173
|
-
- Log: leveled, colored logging; progress bars; fingerprint utilities
|
|
174
|
-
- TmpFile: temp files/dirs and stable tmp path generator for caches
|
|
175
|
-
|
|
176
|
-
---
|
|
177
|
-
|
|
178
|
-
## Scout Gear modules
|
|
179
|
-
|
|
180
|
-
Scout Gear builds on essentials to deliver domain abstractions and engines.
|
|
181
|
-
|
|
182
|
-
### TSV
|
|
183
|
-
|
|
184
|
-
doc/TSV.md
|
|
185
|
-
|
|
186
|
-
A flexible, typed table abstraction with robust parser, streaming dumper/transformer, parallel traversal, joins/attachments, identifier translation, on-disk persistence (TokyoCabinet/Tkrzw), and range/position indices.
|
|
187
|
-
|
|
188
|
-
Highlights:
|
|
189
|
-
- Shapes: :double, :list, :flat, :single; key_field + fields
|
|
190
|
-
- Parse TSV/CSV from files/streams/strings with rich header options (sep, type, cast, merge)
|
|
191
|
-
- Dumper/Transformer for streaming pipelines
|
|
192
|
-
- TSV.traverse(obj, cpus: N, into: …) for parallel iteration
|
|
193
|
-
- Attach, change_key, change_id, translate via identifier indices
|
|
194
|
-
- Persistence via TSVAdapter over HDB/BDB/Tkrzw/FWT/PKI/Sharder
|
|
195
|
-
- Streaming paste/concat/collapse utilities; filters with persisted sets
|
|
196
|
-
|
|
197
|
-
Example:
|
|
198
|
-
```ruby
|
|
199
|
-
tsv = TSV.open(path, persist: true, type: :double)
|
|
200
|
-
tsv.attach(other, complete: true)
|
|
201
|
-
index = TSV.index(tsv, target: "FieldA")
|
|
202
|
-
```
|
|
203
|
-
|
|
204
|
-
CLI (scout tsv):
|
|
205
|
-
- Scripts live under scout_commands/tsv; list with scout tsv
|
|
206
|
-
- Run a specific subcommand: scout tsv <subcommand> [options] [args...]
|
|
207
|
-
- If you hit a directory, available subcommands are listed
|
|
208
|
-
- Subcommands parse options with SOPT (see each script’s help)
|
|
209
|
-
|
|
210
|
-
### Workflow
|
|
211
|
-
|
|
212
|
-
doc/Workflow.md
|
|
213
|
-
|
|
214
|
-
A lightweight workflow engine. Define tasks with typed inputs and dependencies, create jobs (Steps), and run them with persistence, streaming, provenance, and orchestration under resource rules.
|
|
215
|
-
|
|
216
|
-
Highlights:
|
|
217
|
-
- input/dep/task DSL with helper methods; task_alias and overrides
|
|
218
|
-
- Jobs (Step): run/load/stream/join, info files, files_dir, provenance
|
|
219
|
-
- Orchestrator: schedule dependent jobs under cpus/IO constraints; retry recoverable errors; archive/erase deps per rules
|
|
220
|
-
- EntityWorkflow: entity-centric tasks and properties
|
|
221
|
-
- Queue helpers to enqueue and process jobs
|
|
222
|
-
|
|
223
|
-
Example:
|
|
224
33
|
```ruby
|
|
225
34
|
module Baking
|
|
226
35
|
extend Workflow
|
|
36
|
+
input :name, :string
|
|
227
37
|
task :say => :string do |name| "Hi #{name}" end
|
|
228
38
|
end
|
|
229
|
-
|
|
230
|
-
Baking.job(:say, "Miguel").run # => "Hi Miguel"
|
|
231
|
-
```
|
|
232
|
-
|
|
233
|
-
CLI (scout workflow):
|
|
234
|
-
- List workflows: scout workflow list
|
|
235
|
-
- Run a task: scout workflow task <workflow> <task> [--jobname NAME] [input options...]
|
|
236
|
-
- Options include --fork, --nostream, --update, --printpath, --provenance, --clean, --recursive_clean, --override_deps, --deploy (serial|local|queue|SLURM|server)
|
|
237
|
-
- Show job info: scout workflow info <step_path> [--inputs|--recursive_inputs]
|
|
238
|
-
- Provenance: scout workflow prov <step_path> [--plot file.png] […]
|
|
239
|
-
- Trace execution: scout workflow trace <job-result> [options]
|
|
240
|
-
- Process queue: scout workflow process [filters] [--continuous] [--produce_cpus N] […]
|
|
241
|
-
|
|
242
|
-
You can also dispatch workflow-specific custom commands via:
|
|
243
|
-
- scout workflow cmd <workflow> <subcommand> … (discovers scripts under <workflow>/share/scout_commands/workflow)
|
|
244
|
-
|
|
245
|
-
### KnowledgeBase
|
|
246
|
-
|
|
247
|
-
doc/KnowledgeBase.md
|
|
248
|
-
|
|
249
|
-
A thin orchestrator around Association, TSV, Entity, and Persist to register multiple association databases, normalize/index them, query/traverse across them, manage entity lists, and generate markdown descriptions.
|
|
250
|
-
|
|
251
|
-
Highlights:
|
|
252
|
-
- Register databases with source/target specs and identifier files
|
|
253
|
-
- get_database/get_index (BDB-backed) with undirected options
|
|
254
|
-
- Query: all, subset (children/parents/neighbours), identify/translate entities
|
|
255
|
-
- Lists: save/load/delete/enumerate typed lists
|
|
256
|
-
- Traversal DSL: multi-hop path finding with wildcards/conditions
|
|
257
|
-
- Markdown descriptions from registry/README files
|
|
258
|
-
|
|
259
|
-
Example:
|
|
260
|
-
```ruby
|
|
261
|
-
kb = KnowledgeBase.new(Path.setup("var/kb"), "Hsa")
|
|
262
|
-
kb.register :brothers, datafile_test(:person).brothers, undirected: true
|
|
263
|
-
kb.children(:brothers, "Miki") # => ["Miki~Isa", ...]
|
|
39
|
+
Baking.job(:say, name: "Miguel").run # => "Hi Miguel"
|
|
264
40
|
```
|
|
265
41
|
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
-
|
|
269
|
-
|
|
270
|
-
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
42
|
+
From this one idea follow the properties that make workflows tractable at scale:
|
|
43
|
+
|
|
44
|
+
- **Caching** — jobs have a unique path derived from their inputs; the
|
|
45
|
+
same job run twice returns the cached result without re-executing.
|
|
46
|
+
- **Dependencies** — the engine resolves the dependency graph, runs
|
|
47
|
+
tasks in the right order, and streams results between them instead of
|
|
48
|
+
materializing intermediates.
|
|
49
|
+
- **Provenance** — what ran, when, and with what inputs is tracked and
|
|
50
|
+
inspectable on every job.
|
|
51
|
+
- **Orchestration** — jobs run under resource rules (multiple
|
|
52
|
+
processes, bounded by semaphores) and can be deployed to HPC
|
|
53
|
+
schedulers (SLURM, PBS, LSF) or containers (Singularity).
|
|
54
|
+
|
|
55
|
+
See [Building Workflows](doc/user/BuildingWorkflows.md) and the
|
|
56
|
+
[Workflow Engine](doc/developer/WorkflowEngine.md) internals.
|
|
57
|
+
|
|
58
|
+
## Why these subsystems belong together
|
|
59
|
+
|
|
60
|
+
The subsystems in scout-gear are not a grab bag of modules; they are one
|
|
61
|
+
system for making large, structured, persistent computations composable.
|
|
62
|
+
The workflow engine needs data that can move through pipes (TSV), data
|
|
63
|
+
that survives between runs (Persist), data that carries meaning
|
|
64
|
+
(Entity/Association/KnowledgeBase), and execution bounded by real
|
|
65
|
+
resources (WorkQueue/Semaphore).
|
|
66
|
+
|
|
67
|
+
| Subsystem | What it contributes |
|
|
68
|
+
| --- | --- |
|
|
69
|
+
| TSV | typed tabular data, streaming, parallel traversal, persistence, indexing |
|
|
70
|
+
| Entity / Association / KnowledgeBase | typed identifiers, properties, relationship indices, traversal queries |
|
|
71
|
+
| Persist | transparent caching of expensive computation |
|
|
72
|
+
| WorkQueue / Semaphore | multi-process parallelism bounded by resources |
|
|
73
|
+
|
|
74
|
+
This coherence is anchored in three design principles shared across the
|
|
75
|
+
codebase:
|
|
76
|
+
|
|
77
|
+
1. **Annotate, don't subclass** — attach behavior to existing objects
|
|
78
|
+
(a TSV is a plain Hash, an Entity is a plain String) instead of
|
|
79
|
+
creating new classes.
|
|
80
|
+
2. **Stream, don't load** — process data row by row through pipes
|
|
81
|
+
instead of materializing datasets in memory.
|
|
82
|
+
3. **Persist as cache** — expensive computation is transparently cached
|
|
83
|
+
and recomputed only when inputs change.
|
|
84
|
+
|
|
85
|
+
The principles are explained with idiomatic and non-idiomatic examples in
|
|
86
|
+
[Design Principles](doc/developer/DesignPrinciples.md).
|
|
87
|
+
|
|
88
|
+
## The Scout stack
|
|
89
|
+
|
|
90
|
+
```text
|
|
91
|
+
+------------------------------------------------------------+
|
|
92
|
+
| scout-ai |
|
|
93
|
+
| reasoning, conversations, agents |
|
|
94
|
+
+------------------------------------------------------------+
|
|
95
|
+
| scout-rig / scout-camp (optional ecosystem) |
|
|
96
|
+
| bridging other languages / remote servers, cloud, web |
|
|
97
|
+
+------------------------------------------------------------+
|
|
98
|
+
| scout-gear << THIS REPO >> |
|
|
99
|
+
| workflows, data, entities, persistence, concurrency, |
|
|
100
|
+
| knowledge, provenance |
|
|
101
|
+
+------------------------------------------------------------+
|
|
102
|
+
| scout-essentials |
|
|
103
|
+
| paths, IO, resources, caching, streams |
|
|
104
|
+
+------------------------------------------------------------+
|
|
286
105
|
```
|
|
287
106
|
|
|
288
|
-
###
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
107
|
+
### Scout-AI
|
|
108
|
+
|
|
109
|
+
[Scout-AI](https://github.com/mikisvaz/scout-ai) is an agent and LLM
|
|
110
|
+
layer built on top of Scout. It provides a reproducible conversation
|
|
111
|
+
format (Chat), tool calling backed by real Scout workflows, knowledge
|
|
112
|
+
bases and MCP servers, and multi-agent orchestration encoded as typed,
|
|
113
|
+
inspectable workflow jobs. In Scout-AI a tool is therefore not just an
|
|
114
|
+
API wrapper: it can be a real Scout workflow with steps, dependencies,
|
|
115
|
+
persistence and provenance. The point of scout-gear is that it is the
|
|
116
|
+
computational substrate AI systems operate in.
|
|
117
|
+
|
|
118
|
+
## Where to start
|
|
119
|
+
|
|
120
|
+
The entry point for all documentation is
|
|
121
|
+
[StartHere](doc/StartHere.md), which routes readers by intent.
|
|
122
|
+
|
|
123
|
+
### User guides
|
|
124
|
+
|
|
125
|
+
- [Building Workflows](doc/user/BuildingWorkflows.md) — define tasks,
|
|
126
|
+
inputs, dependencies, and run jobs.
|
|
127
|
+
- [Processing Tabular Data](doc/user/ProcessingTabularData.md) — open,
|
|
128
|
+
create, transform, and filter TSV data.
|
|
129
|
+
- [Working with Entities](doc/user/WorkingWithEntities.md) — attach
|
|
130
|
+
types and properties to identifiers.
|
|
131
|
+
- [Managing Relationships](doc/user/ManagingRelationships.md) — build
|
|
132
|
+
knowledge bases and query relationships.
|
|
133
|
+
- [Running Parallel Work](doc/user/RunningParallelWork.md) — distribute
|
|
134
|
+
work across multiple processes.
|
|
135
|
+
- [HPC / Batch Execution](doc/user/HPCBatchExecution.md) — run workflows on
|
|
136
|
+
SLURM, PBS, or LSF clusters.
|
|
137
|
+
- [Caching Data](doc/user/CachingData.md) — persist results to avoid
|
|
138
|
+
redundant computation.
|
|
139
|
+
- [Cookbook](doc/user/Cookbook.md) — practical recipes combining
|
|
140
|
+
multiple subsystems.
|
|
141
|
+
|
|
142
|
+
### Developer docs
|
|
143
|
+
|
|
144
|
+
- [Architecture](doc/developer/Architecture.md) — subsystem map and
|
|
145
|
+
dependency graph.
|
|
146
|
+
- [Design Principles](doc/developer/DesignPrinciples.md) — coding
|
|
147
|
+
philosophy and idioms.
|
|
148
|
+
- [Workflow Engine](doc/developer/WorkflowEngine.md) — task lifecycle,
|
|
149
|
+
Step execution, dependency resolution.
|
|
150
|
+
- [TSV Internals](doc/developer/TSVInternals.md) — parser/dumper/
|
|
151
|
+
transformer pipeline, traverse, indexing.
|
|
152
|
+
- [Entity System](doc/developer/EntitySystem.md) — property dispatch,
|
|
153
|
+
format registry, KnowledgeBase traversal.
|
|
154
|
+
- [Persistence Engines](doc/developer/PersistenceEngines.md) — database
|
|
155
|
+
engines and the TSVAdapter pattern.
|
|
156
|
+
- [Concurrency Model](doc/developer/ConcurrencyModel.md) — WorkQueue
|
|
157
|
+
fork+IPC and Semaphore synchronization.
|
|
158
|
+
|
|
159
|
+
### Improvements and research
|
|
160
|
+
|
|
161
|
+
- [Improvements](doc/Improvements.md) — prioritized, actionable
|
|
162
|
+
recommendations for code and architecture.
|
|
163
|
+
- [Research artifacts](research/) — non-normative investigation
|
|
164
|
+
documents recording code-level detail and design reasoning; useful
|
|
165
|
+
background, but not the specification.
|
|
166
|
+
|
|
167
|
+
## Install
|
|
168
|
+
|
|
169
|
+
Scout-gear is published as the `scout-gear` gem and installs a single
|
|
170
|
+
`scout` executable:
|
|
171
|
+
|
|
172
|
+
```sh
|
|
173
|
+
gem install scout-gear
|
|
314
174
|
```
|
|
315
175
|
|
|
316
|
-
|
|
317
|
-
|
|
318
|
-
|
|
319
|
-
|
|
320
|
-
Concurrency helpers based on POSIX named semaphores (via RubyInline C bindings), plus higher-level helpers to bound concurrency with forks/threads.
|
|
176
|
+
The `scout` command discovers subcommands contributed by installed Scout
|
|
177
|
+
packages, so the set of commands grows with what you have installed. A
|
|
178
|
+
good pointer into the workflow tooling is:
|
|
321
179
|
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
ScoutSemaphore.with_semaphore(2) do |sem|
|
|
325
|
-
ScoutSemaphore.synchronize(sem){ critical_work }
|
|
326
|
-
end
|
|
180
|
+
```sh
|
|
181
|
+
scout workflow
|
|
327
182
|
```
|
|
328
183
|
|
|
329
|
-
|
|
330
|
-
|
|
331
|
-
## Examples and further reading
|
|
184
|
+
## Ecosystem and lineage
|
|
332
185
|
|
|
333
|
-
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
- For numerous end-to-end examples and real datasets, explore the Rbbt-Workflows organization:
|
|
342
|
-
- https://github.com/Rbbt-Workflows
|
|
343
|
-
- For foundational utilities (Path, Open, CMD, IndiferentHash, Persist, Resource, etc.), consult the scout-essentials documentation:
|
|
344
|
-
- Those modules are summarized above and used pervasively across Scout Gear.
|
|
186
|
+
- [scout-essentials](https://github.com/mikisvaz/scout-essentials) —
|
|
187
|
+
foundational utilities: paths, IO, resources, caching, streams.
|
|
188
|
+
- [scout-camp](https://github.com/mikisvaz/scout-camp) — remote
|
|
189
|
+
servers, cloud deployments, web interfaces, cross-site operations.
|
|
190
|
+
- [scout-rig](https://github.com/mikisvaz/scout-rig) — bridge to other
|
|
191
|
+
languages (for example Python).
|
|
192
|
+
- [scout-ai](https://github.com/mikisvaz/scout-ai) — agents, LLMs and
|
|
193
|
+
tool calling on top of Scout.
|
|
345
194
|
|
|
346
|
-
|
|
195
|
+
Many of Scout's ideas and utilities originated in
|
|
196
|
+
[Rbbt](https://github.com/Rbbt-Workflows), which still carries many
|
|
197
|
+
real-world workflow examples.
|
|
347
198
|
|
|
348
|
-
##
|
|
199
|
+
## License
|
|
349
200
|
|
|
350
|
-
|
|
351
|
-
|
|
352
|
-
- Discovery and composition: the Path subsystem and Resource claims make it easy to build portable projects with on-demand production of resources and discoverable commands.
|
|
201
|
+
MIT — see [LICENSE.txt](LICENSE.txt) (Copyright (c) 2023 Miguel
|
|
202
|
+
Vazquez).
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
10.
|
|
1
|
+
10.13.0
|