scout-gear 10.12.2 → 10.13.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (64) hide show
  1. checksums.yaml +4 -4
  2. data/.vimproject +32 -8
  3. data/README.md +179 -329
  4. data/VERSION +1 -1
  5. data/doc/Improvements.md +239 -0
  6. data/doc/StartHere.md +71 -0
  7. data/doc/developer/Architecture.md +216 -0
  8. data/doc/developer/ConcurrencyModel.md +222 -0
  9. data/doc/developer/DesignPrinciples.md +246 -0
  10. data/doc/developer/EntitySystem.md +231 -0
  11. data/doc/developer/PersistenceEngines.md +163 -0
  12. data/doc/developer/TSVInternals.md +310 -0
  13. data/doc/developer/WorkflowEngine.md +296 -0
  14. data/doc/user/BuildingWorkflows.md +394 -0
  15. data/doc/user/CachingData.md +189 -0
  16. data/doc/user/Cookbook.md +255 -0
  17. data/doc/user/HPCBatchExecution.md +159 -0
  18. data/doc/user/ManagingRelationships.md +207 -0
  19. data/doc/user/ProcessingTabularData.md +315 -0
  20. data/doc/user/RunningParallelWork.md +196 -0
  21. data/doc/user/WorkingWithEntities.md +238 -0
  22. data/lib/scout/knowledge_base/query.rb +2 -2
  23. data/lib/scout/persist/tsv/adapter/base.rb +5 -1
  24. data/lib/scout/persist/tsv/serialize.rb +3 -2
  25. data/lib/scout/tsv/transformer.rb +11 -3
  26. data/lib/scout/workflow/definition.rb +14 -14
  27. data/lib/scout/workflow/deployment/scheduler/lfs.rb +1 -1
  28. data/lib/scout/workflow/entity.rb +1 -1
  29. data/lib/scout/workflow/step/file.rb +2 -2
  30. data/lib/scout/workflow/step/info.rb +17 -2
  31. data/lib/scout/workflow/step/load.rb +3 -1
  32. data/lib/scout/workflow/step/provenance.rb +1 -1
  33. data/lib/scout/workflow/task/dependencies.rb +1 -2
  34. data/lib/scout/workflow/task.rb +6 -1
  35. data/lib/scout/workflow/usage.rb +0 -1
  36. data/lib/scout/workflow.rb +11 -5
  37. data/lib/scout-gear.rb +2 -3
  38. data/research/00_scope_and_themes.md +150 -0
  39. data/research/design-philosophy-analysis.md +263 -0
  40. data/research/downstream-usage.md +130 -0
  41. data/research/entity-association-kb-analysis.md +265 -0
  42. data/research/identifiers-mechanism.md +170 -0
  43. data/research/persistence-concurrency-analysis.md +316 -0
  44. data/research/repo-map.md +124 -0
  45. data/research/synthesis-report.md +106 -0
  46. data/research/tsv-internals-analysis.md +221 -0
  47. data/research/validation-report.md +136 -0
  48. data/research/workflow-engine-analysis.md +213 -0
  49. data/scout-gear.gemspec +31 -11
  50. data/scout_commands/kb/query +1 -1
  51. data/scout_commands/kb/traverse +1 -1
  52. data/scout_commands/rbbt +25 -23
  53. data/scout_commands/system/status +1 -1
  54. data/scout_commands/workflow/prov +1 -1
  55. data/test/scout/workflow/test_step.rb +31 -0
  56. metadata +30 -10
  57. data/doc/Association.md +0 -288
  58. data/doc/Entity.md +0 -296
  59. data/doc/KnowledgeBase.md +0 -433
  60. data/doc/Persist.md +0 -356
  61. data/doc/Semaphore.md +0 -171
  62. data/doc/TSV.md +0 -449
  63. data/doc/WorkQueue.md +0 -359
  64. data/doc/Workflow.md +0 -744
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 72599e4b76e2a08e984f458d75d268fc566c59a91132607ff1dba7ea7dbffe29
4
- data.tar.gz: 26f292e80a00565488becc8cbdc4ca7d7b1821571c8b6c8d4ae7dde5e46710e5
3
+ metadata.gz: 9a700bc307360c342dd671a87d0a5ddf69519506c52a0ac50d9d1580705a6f3e
4
+ data.tar.gz: 66694104c9005d0ba4e979f6a0120e029afe42c34070f013b0105646e0c6619f
5
5
  SHA512:
6
- metadata.gz: b28d223415b0dbb4e4f6d1de498f3e00811c3f79c4038ee3d44b7e436b5d815d2ddbe69892b9ca6072c44344a175a0332c5d3ee522b13decdf7f8ed21f9712b5
7
- data.tar.gz: 94c3d47de97f1aa38d1bc298066882d683e845d35d39e8ce8b9a601fda752f5fc9bb1a516c4bcd1f58a0a3ccece9cd1c5d335d55a844f52cc307fbaffd079c7c
6
+ metadata.gz: 664c8e59a165edfc9d8291dce01e15bd913c388f6de455c21f88108f5dbc23147b8d3312edf79b37516303d4ecb37bb0b2dc4473b32668d8dfddf91515d80710
7
+ data.tar.gz: 41417a3686d7d2bea6173296fbe21eb8862e3dc0a00de45a0d663340e87b25834df9ca6aefa5e7e5fdcd9dde65649f94df734249e06bbbe37601da3d3c55737a
data/.vimproject CHANGED
@@ -2,6 +2,9 @@ scout-gear=/$PWD filter="*.rb *.yaml" {
2
2
  Rakefile
3
3
  README.md
4
4
  chats=chats filter="*"{
5
+ test=test{
6
+ kb.rb
7
+ }
5
8
 
6
9
  purge
7
10
 
@@ -25,6 +28,15 @@ scout-gear=/$PWD filter="*.rb *.yaml" {
25
28
  doc=doc{
26
29
  documenter.rb
27
30
  }
31
+
32
+ documentation
33
+
34
+ README_doc
35
+
36
+ documentation_fix
37
+ doc_learning
38
+
39
+ doc_learning_chat_analyst
28
40
  }
29
41
  bin=bin filter="*"{
30
42
  scout
@@ -316,13 +328,25 @@ scout-gear=/$PWD filter="*.rb *.yaml" {
316
328
  }
317
329
  }
318
330
  doc=doc filter="*"{
319
- Association.md
320
- Entity.md
321
- KnowledgeBase.md
322
- Persist.md
323
- Semaphore.md
324
- TSV.md
325
- WorkQueue.md
326
- Workflow.md
331
+ Improvements.md
332
+ StartHere.md
333
+ user=user{
334
+ BuildingWorkflows.md
335
+ CachingData.md
336
+ Cookbook.md
337
+ ManagingRelationships.md
338
+ ProcessingTabularData.md
339
+ RunningParallelWork.md
340
+ WorkingWithEntities.md
341
+ }
342
+ developer=developer{
343
+ Architecture.md
344
+ ConcurrencyModel.md
345
+ DesignPrinciples.md
346
+ EntitySystem.md
347
+ PersistenceEngines.md
348
+ TSVInternals.md
349
+ WorkflowEngine.md
350
+ }
327
351
  }
328
352
  }
data/README.md CHANGED
@@ -1,352 +1,202 @@
1
1
  # Scout Gear
2
2
 
3
- Scout Gear is the core, higher-level module set of the Scout framework. It bundles rich, production-grade data and workflow tooling built on top of the lower-level primitives in scout-essentials, and adds domain abstractions such as TSV processing, workflows, knowledge bases, entity typing, parallel work queues, and more.
4
-
5
- Layering:
6
- - scout-essentials: foundational utilities used everywhere (Path, Open, CMD, IndiferentHash, Persist, Resource, etc.)
7
- - scout-gear (this repo): TSV, Workflow, KnowledgeBase, Entity/Association, WorkQueue, Semaphore, and glue code
8
- - Additional packages:
9
- - scout-camp: remote servers, cloud deployments, web interfaces, cross-site operations
10
- - scout-ai: model training and chat agents
11
- - scout-rig: connect with other languages (e.g., Python)
12
-
13
- Related ecosystem:
14
- - Rbbt (Ruby bioinformatics): Many of Scout’s ideas and utilities originated in Rbbt. It still provides a broad set of bioinformatics workflows and tools. See the Rbbt-Workflows organization for many real-world examples and usage patterns:
15
- - https://github.com/Rbbt-Workflows
16
-
17
- For module-specific guides, see doc/*.md in this repository (linked below).
18
-
19
- - TSV: doc/TSV.md
20
- - Workflow: doc/Workflow.md
21
- - KnowledgeBase: doc/KnowledgeBase.md
22
- - Association: doc/Association.md
23
- - Entity: doc/Entity.md
24
- - WorkQueue: doc/WorkQueue.md
25
- - Semaphore: doc/Semaphore.md
26
-
27
- Additionally, Scout Gear reuses and exposes core facilities from scout-essentials. Summaries of those core modules are included below for convenience.
28
-
29
- ---
30
-
31
- ## How command-line interfaces work (scout …)
32
-
33
- Scout provides a single “scout” command that discovers and runs nested subcommands from any installed Scout package. Scripts are discovered using the Path subsystem across PATH-like roots, enabling workflows or packages to inject their own commands.
34
-
35
- Basics:
36
- - The CLI resolves terms left-to-right until a file is found under a scout_commands tree.
37
- - Example: scout workflow task runs scout_commands/workflow/task
38
- - Example: all TSV-related scripts are under scout_commands/tsv and can be listed with scout tsv
39
- - If the path resolves to a directory instead of a script, a list of available subcommands in that directory is shown.
40
- - Remaining ARGV is parsed by the selected script using SimpleOPT (SOPT) or compatible parsers.
41
- - Because discovery uses Path maps, commands contributed by other packages or installed workflows are automatically found.
42
-
43
- See the per-module CLI sections below for TSV, Workflow, and KnowledgeBase.
44
-
45
- ---
46
-
47
- ## Scout Essentials: Core building blocks
48
-
49
- Scout Gear depends on the following main modules from scout-essentials. You’ll use these directly for filesystem/resource orchestration, external command execution, caching, and options handling.
50
-
51
- ### Path
52
-
53
- doc/Path.md
54
-
55
- Path is a lightweight, annotation-enabled “smart string” for composing and locating project resources across multiple search maps (current/user/global/lib/tmp, etc.). It integrates with Open and Persist.
56
-
57
- Highlights:
58
- - Path.setup("str") turns a String into a Path with join via [], /, or method_missing (path.foo.bar)
59
- - Map logical locations to physical roots with path maps; find the first match across map order with path.find (and path.find_all)
60
- - Filename helpers: get/set/replace/unset extensions; sanitize filenames; relative paths
61
- - Directory helpers: glob and glob_all over maps; dirname/basename; realpath; newer?
62
- - Digest summaries: path.digest_str summarizes files/dirs for logging/debugging
63
-
64
- Usage:
65
- ```ruby
66
- p = Path.setup('share/data/myfile')
67
- p.find # resolve across configured maps
68
- p[:subdir, :file] # joins => share/data/subdir/file
3
+ **Workflows, data, entities and persistence for the Scout ecosystem.**
4
+
5
+ Scout turns computation into persistent, composable, inspectable work.
6
+ Scout-gear is the workflow and data-processing layer of the Scout
7
+ ecosystem: it provides a workflow engine for computational pipelines, a
8
+ tabular data system (TSV), an entity system for typed identifiers, a
9
+ knowledge base for relationship queries, and specialized persistence
10
+ engines. It builds on
11
+ [scout-essentials](https://github.com/mikisvaz/scout-essentials), which
12
+ provides the foundational utilities (file I/O, annotations, paths,
13
+ logging, concurrency streams).
14
+
15
+ New to Scout? Start with [StartHere](doc/StartHere.md) for an
16
+ audience-oriented route into the documentation.
17
+
18
+ ## The Scout model
19
+
20
+ In scout-gear a task is not a transient function call. Running a job
21
+ creates a **Step**: a concrete unit of work with typed inputs, declared
22
+ dependencies, and a unique path derived from those inputs. Because the
23
+ path identifies the work, the result is written to disk, reused on
24
+ re-run, and only recomputed when the inputs change. Everything that
25
+ happened to produce it is recorded with it.
26
+
27
+ ```text
28
+ task -> job (Step) -> persistent result -> provenance -> composition / orchestration
69
29
  ```
70
30
 
71
- ### Open
72
-
73
- doc/Open.md
31
+ The canonical example:
74
32
 
75
- Open unifies file/stream/remote I/O, atomic writes, pipes/tees/FIFOs, (bg)zip helpers, rsync/sync, and lock handling.
76
-
77
- Highlights:
78
- - Open.open/read/write with auto-(de)compression for .gz/.bgz/.zip and remote urls (wget/ssh)
79
- - Streams: open_pipe, tee_stream, consume_stream, with_fifo
80
- - Safe writes: sensible_write (tmp + atomic rename + optional locks)
81
- - Remote: wget with caching, ssh/scp, digest_url, remote cache
82
- - Filesystem: mkdir/mkfiledir, mv/cp/ln/link_dir, rm/rm_rf, same_file?, exists?, writable?
83
- - Locking: Open.lock wraps a robust Lockfile (NFS-safe) with refresh/timeout/steal
84
-
85
- Example:
86
- ```ruby
87
- Open.sensible_write("out.txt", Open.open("http://example.com"))
88
- Open.with_fifo { |fifo| ... }
89
- Open.rsync("src/", "user@server:dst/", delete: true)
90
- ```
91
-
92
- ### CMD
93
-
94
- doc/CMD.md
95
-
96
- CMD wraps Open3.popen3 with robust patterns for streaming, stderr logging, stdin feeding, auto-join of producers, and tool discovery/installation.
97
-
98
- Highlights:
99
- - CMD.cmd("tool args", pipe: true, in: io_or_string, stderr: Log::HIGH, autojoin: true)
100
- - ConcurrentStream-enabled stdout with join/error propagation
101
- - Convenience: CMD.bash("bash -l -c '...'"), cmd_pid/cmd_log
102
- - Tool registry: CMD.tool, CMD.get_tool (auto-install via conda or producers), version scanning
103
-
104
- Example:
105
- ```ruby
106
- io = CMD.cmd("cut", "-f" => 2, "-d" => " ", in: "a b", pipe: true)
107
- io.read # => "b\n"; io.join
108
- ```
109
-
110
- ### IndiferentHash
111
-
112
- doc/IndiferentHash.md
113
-
114
- Hash mixin for indifferent access (string/symbol keys equal), deep-merge, options parsing, and string<->hash conversions.
115
-
116
- Highlights:
117
- - IndiferentHash.setup(hash) to extend a single hash instance
118
- - Access with h[:a] == h["a"]; delete/include? are indifferent
119
- - Helpers: deep_merge, values_at with indifferent keys, slice, except
120
- - Options utilities: parse_options, process_options, positional2hash, hash2string/string2hash
121
-
122
- Example:
123
- ```ruby
124
- opts = IndiferentHash.parse_options('limit=10 title="A title"')
125
- opts[:title] # => "A title"
126
- ```
127
-
128
- ### Persist (core serialization/caching)
129
-
130
- doc/Persist.md (essentials)
131
-
132
- Typed serialization (json/yaml/marshal/binary/arrays), atomic saves, and the high-level persist pattern with locking and streaming.
133
-
134
- Highlights:
135
- - Persist.save/load(obj, file, type)
136
- - Persist.persist(name, type, dir: ...) { compute_or_stream }
137
- - Locking and tmp-to-final atomic writes
138
- - Streaming tee: one copy to file, one to caller
139
- - Memory cache: Persist.memory(name) { ... }
140
- - Helpers to parse YAML/JSON/Marshal via Open
141
-
142
- Example:
143
- ```ruby
144
- val = Persist.persist("expensive", :json) { compute_hash }
145
- # subsequent calls load cached JSON unless :update or stale
146
- ```
147
-
148
- ### Resource
149
-
150
- doc/Resource.md
151
-
152
- Resource system to claim and produce files on demand (string/proc/url/rake/installers), integrated with Path/Open and locking.
153
-
154
- Highlights:
155
- - claim path => (:string, :proc, :url, :rake, :install)
156
- - Produce on demand via path.produce and path.open/read
157
- - Rake integration: drive file tasks/rules to generate outputs
158
- - Install software into a per-resource “software” dir and update env
159
-
160
- Example:
161
- ```ruby
162
- module MyPkg
163
- extend Resource
164
- claim root.tmp.test.hello, :string, "Hello"
165
- end
166
- MyPkg.tmp.test.hello.read # produces if missing, then reads
167
- ```
168
-
169
- Other essentials you’ll encounter:
170
- - Annotation / AnnotatedArray / NamedArray: lightweight typed attributes on objects and arrays; named tuple-style rows
171
- - ConcurrentStream: concurrency-aware streams with join/abort/callbacks
172
- - SimpleOPT (SOPT): tiny CLI option DSL/parser; used by scout commands
173
- - Log: leveled, colored logging; progress bars; fingerprint utilities
174
- - TmpFile: temp files/dirs and stable tmp path generator for caches
175
-
176
- ---
177
-
178
- ## Scout Gear modules
179
-
180
- Scout Gear builds on essentials to deliver domain abstractions and engines.
181
-
182
- ### TSV
183
-
184
- doc/TSV.md
185
-
186
- A flexible, typed table abstraction with robust parser, streaming dumper/transformer, parallel traversal, joins/attachments, identifier translation, on-disk persistence (TokyoCabinet/Tkrzw), and range/position indices.
187
-
188
- Highlights:
189
- - Shapes: :double, :list, :flat, :single; key_field + fields
190
- - Parse TSV/CSV from files/streams/strings with rich header options (sep, type, cast, merge)
191
- - Dumper/Transformer for streaming pipelines
192
- - TSV.traverse(obj, cpus: N, into: …) for parallel iteration
193
- - Attach, change_key, change_id, translate via identifier indices
194
- - Persistence via TSVAdapter over HDB/BDB/Tkrzw/FWT/PKI/Sharder
195
- - Streaming paste/concat/collapse utilities; filters with persisted sets
196
-
197
- Example:
198
- ```ruby
199
- tsv = TSV.open(path, persist: true, type: :double)
200
- tsv.attach(other, complete: true)
201
- index = TSV.index(tsv, target: "FieldA")
202
- ```
203
-
204
- CLI (scout tsv):
205
- - Scripts live under scout_commands/tsv; list with scout tsv
206
- - Run a specific subcommand: scout tsv <subcommand> [options] [args...]
207
- - If you hit a directory, available subcommands are listed
208
- - Subcommands parse options with SOPT (see each script’s help)
209
-
210
- ### Workflow
211
-
212
- doc/Workflow.md
213
-
214
- A lightweight workflow engine. Define tasks with typed inputs and dependencies, create jobs (Steps), and run them with persistence, streaming, provenance, and orchestration under resource rules.
215
-
216
- Highlights:
217
- - input/dep/task DSL with helper methods; task_alias and overrides
218
- - Jobs (Step): run/load/stream/join, info files, files_dir, provenance
219
- - Orchestrator: schedule dependent jobs under cpus/IO constraints; retry recoverable errors; archive/erase deps per rules
220
- - EntityWorkflow: entity-centric tasks and properties
221
- - Queue helpers to enqueue and process jobs
222
-
223
- Example:
224
33
  ```ruby
225
34
  module Baking
226
35
  extend Workflow
36
+ input :name, :string
227
37
  task :say => :string do |name| "Hi #{name}" end
228
38
  end
229
-
230
- Baking.job(:say, "Miguel").run # => "Hi Miguel"
231
- ```
232
-
233
- CLI (scout workflow):
234
- - List workflows: scout workflow list
235
- - Run a task: scout workflow task <workflow> <task> [--jobname NAME] [input options...]
236
- - Options include --fork, --nostream, --update, --printpath, --provenance, --clean, --recursive_clean, --override_deps, --deploy (serial|local|queue|SLURM|server)
237
- - Show job info: scout workflow info <step_path> [--inputs|--recursive_inputs]
238
- - Provenance: scout workflow prov <step_path> [--plot file.png] […]
239
- - Trace execution: scout workflow trace <job-result> [options]
240
- - Process queue: scout workflow process [filters] [--continuous] [--produce_cpus N] […]
241
-
242
- You can also dispatch workflow-specific custom commands via:
243
- - scout workflow cmd <workflow> <subcommand> … (discovers scripts under <workflow>/share/scout_commands/workflow)
244
-
245
- ### KnowledgeBase
246
-
247
- doc/KnowledgeBase.md
248
-
249
- A thin orchestrator around Association, TSV, Entity, and Persist to register multiple association databases, normalize/index them, query/traverse across them, manage entity lists, and generate markdown descriptions.
250
-
251
- Highlights:
252
- - Register databases with source/target specs and identifier files
253
- - get_database/get_index (BDB-backed) with undirected options
254
- - Query: all, subset (children/parents/neighbours), identify/translate entities
255
- - Lists: save/load/delete/enumerate typed lists
256
- - Traversal DSL: multi-hop path finding with wildcards/conditions
257
- - Markdown descriptions from registry/README files
258
-
259
- Example:
260
- ```ruby
261
- kb = KnowledgeBase.new(Path.setup("var/kb"), "Hsa")
262
- kb.register :brothers, datafile_test(:person).brothers, undirected: true
263
- kb.children(:brothers, "Miki") # => ["Miki~Isa", ...]
39
+ Baking.job(:say, name: "Miguel").run # => "Hi Miguel"
264
40
  ```
265
41
 
266
- CLI (scout kb):
267
- - Configure KB: scout kb config [options] <name>
268
- - Register DB: scout kb register [options] <name> <filename>
269
- - Declare entities: scout kb entities <entity> <identifier_files>
270
- - Show info: scout kb show [<name>]
271
- - Query: scout kb query <name> <entity_spec>
272
- - Lists: scout kb list [<list_name>]
273
- - Traverse: scout kb traverse [options] "<rules,comma,separated>"
274
-
275
- ### Association
276
-
277
- doc/Association.md
278
-
279
- Utilities to normalize source/target field specifications from TSVs, open normalized association databases with optional identifier translation, and build pairwise “source~target” indices (optionally undirected). Also includes AssociationItem for entity-like behavior over pair strings and utilities to build incidence/adjacency matrices.
280
-
281
- Example:
282
- ```ruby
283
- idx = Association.index(file, source: "=>Name", target: "Parent=>Name", undirected: true)
284
- idx.match("Clei") # => ["Clei~Guille"]
285
- idx.to_matrix # boolean incidence matrix
42
+ From this one idea follow the properties that make workflows tractable at scale:
43
+
44
+ - **Caching** jobs have a unique path derived from their inputs; the
45
+ same job run twice returns the cached result without re-executing.
46
+ - **Dependencies** the engine resolves the dependency graph, runs
47
+ tasks in the right order, and streams results between them instead of
48
+ materializing intermediates.
49
+ - **Provenance** what ran, when, and with what inputs is tracked and
50
+ inspectable on every job.
51
+ - **Orchestration** — jobs run under resource rules (multiple
52
+ processes, bounded by semaphores) and can be deployed to HPC
53
+ schedulers (SLURM, PBS, LSF) or containers (Singularity).
54
+
55
+ See [Building Workflows](doc/user/BuildingWorkflows.md) and the
56
+ [Workflow Engine](doc/developer/WorkflowEngine.md) internals.
57
+
58
+ ## Why these subsystems belong together
59
+
60
+ The subsystems in scout-gear are not a grab bag of modules; they are one
61
+ system for making large, structured, persistent computations composable.
62
+ The workflow engine needs data that can move through pipes (TSV), data
63
+ that survives between runs (Persist), data that carries meaning
64
+ (Entity/Association/KnowledgeBase), and execution bounded by real
65
+ resources (WorkQueue/Semaphore).
66
+
67
+ | Subsystem | What it contributes |
68
+ | --- | --- |
69
+ | TSV | typed tabular data, streaming, parallel traversal, persistence, indexing |
70
+ | Entity / Association / KnowledgeBase | typed identifiers, properties, relationship indices, traversal queries |
71
+ | Persist | transparent caching of expensive computation |
72
+ | WorkQueue / Semaphore | multi-process parallelism bounded by resources |
73
+
74
+ This coherence is anchored in three design principles shared across the
75
+ codebase:
76
+
77
+ 1. **Annotate, don't subclass** — attach behavior to existing objects
78
+ (a TSV is a plain Hash, an Entity is a plain String) instead of
79
+ creating new classes.
80
+ 2. **Stream, don't load** — process data row by row through pipes
81
+ instead of materializing datasets in memory.
82
+ 3. **Persist as cache** — expensive computation is transparently cached
83
+ and recomputed only when inputs change.
84
+
85
+ The principles are explained with idiomatic and non-idiomatic examples in
86
+ [Design Principles](doc/developer/DesignPrinciples.md).
87
+
88
+ ## The Scout stack
89
+
90
+ ```text
91
+ +------------------------------------------------------------+
92
+ | scout-ai |
93
+ | reasoning, conversations, agents |
94
+ +------------------------------------------------------------+
95
+ | scout-rig / scout-camp (optional ecosystem) |
96
+ | bridging other languages / remote servers, cloud, web |
97
+ +------------------------------------------------------------+
98
+ | scout-gear << THIS REPO >> |
99
+ | workflows, data, entities, persistence, concurrency, |
100
+ | knowledge, provenance |
101
+ +------------------------------------------------------------+
102
+ | scout-essentials |
103
+ | paths, IO, resources, caching, streams |
104
+ +------------------------------------------------------------+
286
105
  ```
287
106
 
288
- ### Entity
289
-
290
- doc/Entity.md
291
-
292
- Annotate plain values or arrays as entities with behavior-rich “properties”, automatic format mapping, identifier translation (Entity::Identified), array-aware property batching/caching, and persistence for property results via Persist.
293
-
294
- Example:
295
- ```ruby
296
- module Person
297
- extend Entity
298
- property :greet => :single do "Hi #{self}" end
299
- end
300
- Person.setup("Miki").greet
301
- ```
302
-
303
- ### WorkQueue
304
-
305
- doc/WorkQueue.md
306
-
307
- A multi-process work pipeline (forked workers + semaphore-guarded sockets) to parallelize processing over a stream of inputs, with robust error propagation.
308
-
309
- Example:
310
- ```ruby
311
- q = WorkQueue.new(4){|x| x * 2}
312
- out = []; q.process{|y| out << y}
313
- (1..100).each{|i| q.write i}; q.close; q.join
107
+ ### Scout-AI
108
+
109
+ [Scout-AI](https://github.com/mikisvaz/scout-ai) is an agent and LLM
110
+ layer built on top of Scout. It provides a reproducible conversation
111
+ format (Chat), tool calling backed by real Scout workflows, knowledge
112
+ bases and MCP servers, and multi-agent orchestration encoded as typed,
113
+ inspectable workflow jobs. In Scout-AI a tool is therefore not just an
114
+ API wrapper: it can be a real Scout workflow with steps, dependencies,
115
+ persistence and provenance. The point of scout-gear is that it is the
116
+ computational substrate AI systems operate in.
117
+
118
+ ## Where to start
119
+
120
+ The entry point for all documentation is
121
+ [StartHere](doc/StartHere.md), which routes readers by intent.
122
+
123
+ ### User guides
124
+
125
+ - [Building Workflows](doc/user/BuildingWorkflows.md) — define tasks,
126
+ inputs, dependencies, and run jobs.
127
+ - [Processing Tabular Data](doc/user/ProcessingTabularData.md) — open,
128
+ create, transform, and filter TSV data.
129
+ - [Working with Entities](doc/user/WorkingWithEntities.md) — attach
130
+ types and properties to identifiers.
131
+ - [Managing Relationships](doc/user/ManagingRelationships.md) build
132
+ knowledge bases and query relationships.
133
+ - [Running Parallel Work](doc/user/RunningParallelWork.md) — distribute
134
+ work across multiple processes.
135
+ - [HPC / Batch Execution](doc/user/HPCBatchExecution.md) — run workflows on
136
+ SLURM, PBS, or LSF clusters.
137
+ - [Caching Data](doc/user/CachingData.md) — persist results to avoid
138
+ redundant computation.
139
+ - [Cookbook](doc/user/Cookbook.md) — practical recipes combining
140
+ multiple subsystems.
141
+
142
+ ### Developer docs
143
+
144
+ - [Architecture](doc/developer/Architecture.md) — subsystem map and
145
+ dependency graph.
146
+ - [Design Principles](doc/developer/DesignPrinciples.md) — coding
147
+ philosophy and idioms.
148
+ - [Workflow Engine](doc/developer/WorkflowEngine.md) — task lifecycle,
149
+ Step execution, dependency resolution.
150
+ - [TSV Internals](doc/developer/TSVInternals.md) — parser/dumper/
151
+ transformer pipeline, traverse, indexing.
152
+ - [Entity System](doc/developer/EntitySystem.md) — property dispatch,
153
+ format registry, KnowledgeBase traversal.
154
+ - [Persistence Engines](doc/developer/PersistenceEngines.md) — database
155
+ engines and the TSVAdapter pattern.
156
+ - [Concurrency Model](doc/developer/ConcurrencyModel.md) — WorkQueue
157
+ fork+IPC and Semaphore synchronization.
158
+
159
+ ### Improvements and research
160
+
161
+ - [Improvements](doc/Improvements.md) — prioritized, actionable
162
+ recommendations for code and architecture.
163
+ - [Research artifacts](research/) — non-normative investigation
164
+ documents recording code-level detail and design reasoning; useful
165
+ background, but not the specification.
166
+
167
+ ## Install
168
+
169
+ Scout-gear is published as the `scout-gear` gem and installs a single
170
+ `scout` executable:
171
+
172
+ ```sh
173
+ gem install scout-gear
314
174
  ```
315
175
 
316
- ### Semaphore (ScoutSemaphore)
317
-
318
- doc/Semaphore.md
319
-
320
- Concurrency helpers based on POSIX named semaphores (via RubyInline C bindings), plus higher-level helpers to bound concurrency with forks/threads.
176
+ The `scout` command discovers subcommands contributed by installed Scout
177
+ packages, so the set of commands grows with what you have installed. A
178
+ good pointer into the workflow tooling is:
321
179
 
322
- Example:
323
- ```ruby
324
- ScoutSemaphore.with_semaphore(2) do |sem|
325
- ScoutSemaphore.synchronize(sem){ critical_work }
326
- end
180
+ ```sh
181
+ scout workflow
327
182
  ```
328
183
 
329
- ---
330
-
331
- ## Examples and further reading
184
+ ## Ecosystem and lineage
332
185
 
333
- - This repository’s docs directory provides in-depth guides for each module:
334
- - TSV: doc/TSV.md
335
- - Workflow: doc/Workflow.md
336
- - KnowledgeBase: doc/KnowledgeBase.md
337
- - Association: doc/Association.md
338
- - Entity: doc/Entity.md
339
- - WorkQueue: doc/WorkQueue.md
340
- - Semaphore: doc/Semaphore.md
341
- - For numerous end-to-end examples and real datasets, explore the Rbbt-Workflows organization:
342
- - https://github.com/Rbbt-Workflows
343
- - For foundational utilities (Path, Open, CMD, IndiferentHash, Persist, Resource, etc.), consult the scout-essentials documentation:
344
- - Those modules are summarized above and used pervasively across Scout Gear.
186
+ - [scout-essentials](https://github.com/mikisvaz/scout-essentials)
187
+ foundational utilities: paths, IO, resources, caching, streams.
188
+ - [scout-camp](https://github.com/mikisvaz/scout-camp) — remote
189
+ servers, cloud deployments, web interfaces, cross-site operations.
190
+ - [scout-rig](https://github.com/mikisvaz/scout-rig) — bridge to other
191
+ languages (for example Python).
192
+ - [scout-ai](https://github.com/mikisvaz/scout-ai) — agents, LLMs and
193
+ tool calling on top of Scout.
345
194
 
346
- ---
195
+ Many of Scout's ideas and utilities originated in
196
+ [Rbbt](https://github.com/Rbbt-Workflows), which still carries many
197
+ real-world workflow examples.
347
198
 
348
- ## Notes
199
+ ## License
349
200
 
350
- - Streaming everywhere: many APIs return ConcurrentStream-enabled IOs. Always read to EOF and join (or rely on autojoin) to ensure producers exit and errors are surfaced.
351
- - Atomicity and locking: Open.sensible_write and Persist.persist use tmp+mv and lockfiles to provide robust cross-process behavior.
352
- - Discovery and composition: the Path subsystem and Resource claims make it easy to build portable projects with on-demand production of resources and discoverable commands.
201
+ MIT see [LICENSE.txt](LICENSE.txt) (Copyright (c) 2023 Miguel
202
+ Vazquez).
data/VERSION CHANGED
@@ -1 +1 @@
1
- 10.12.2
1
+ 10.13.0