scout-gear 10.8.4 → 10.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 97c074aa2b85744eccbcba77c7be3eaa733d7280f6c5875211a3a1b771234e04
-  data.tar.gz: c284bf58c6954699fdc0b0742f6331ea1af7db7188b76b99325ad561e19de6cf
+  metadata.gz: 3900d5a0e546d494ae3224176db5feb92344dc1f6ba311461c74d02f5b137a9c
+  data.tar.gz: 35421139fea183944db794bad361bdfa9db78b086e13fab6367d6bbcfebc29e3
 SHA512:
-  metadata.gz: f381c4c3167580d337ec5ac43bec9e12e8db0b4b423603fe134b8751d3a9aa8b9930797c87daa9cd0b7cf7f9c24c60ece3c00b2b4b70aec03aa71eae7bcef091
-  data.tar.gz: f39b067a52e6724302dc92d1eeb4dac3c22ce07e25c4e986d9a3018c92dcfdd79441d23bd2c9f0dca90a7a39d6ca95bc52fab92b0b4999cc003e48767e26bce8
+  metadata.gz: 679d05d7ae724825e18a2ce12862d08a3e75179f575a19ff960f4f379be6f2ce844326beca6e47d4bce3f585a4e637c71164b8642a2d61a85b2a54236f345153
+  data.tar.gz: c952339dfaa42d948a297a7f2c456d63bb4624d0ec71ee26499b79a2a25be8f32ec8c2b5ea2bd9cc17a664c50edb8f6cf22dec458ea103eb6e82614774973bdc

data/.vimproject

@@ -1,5 +1,15 @@
 scout-gear=/$PWD filter="*.rb *.yaml" {
  Rakefile
+ README.md
+ chats=chats filter="*"{
+  debug
+  document.rb
+  pipes
+  test_doc
+  doc=doc{
+   documenter.rb
+  }
+ }
  bin=bin filter="*"{
   scout
  }
@@ -137,7 +147,9 @@ scout-gear=/$PWD filter="*.rb *.yaml" {
  scout_commands=scout_commands filter="*"{
   rbbt
   alias
+  entity
   find
+  cat
   glob
   log
   doc
@@ -174,6 +186,7 @@ scout-gear=/$PWD filter="*.rb *.yaml" {
   }
   system=system{
    clean
+   status
   }
  }
  test=test {

data/README.md

@@ -0,0 +1,352 @@
+# Scout Gear
+
+Scout Gear is the core, higher-level module set of the Scout framework. It bundles rich, production-grade data and workflow tooling built on top of the lower-level primitives in scout-essentials, and adds domain abstractions such as TSV processing, workflows, knowledge bases, entity typing, parallel work queues, and more.
+
+Layering:
+- scout-essentials: foundational utilities used everywhere (Path, Open, CMD, IndiferentHash, Persist, Resource, etc.)
+- scout-gear (this repo): TSV, Workflow, KnowledgeBase, Entity/Association, WorkQueue, Semaphore, and glue code
+- Additional packages:
+  - scout-camp: remote servers, cloud deployments, web interfaces, cross-site operations
+  - scout-ai: model training and chat agents
+  - scout-rig: connect with other languages (e.g., Python)
+
+Related ecosystem:
+- Rbbt (Ruby bioinformatics): Many of Scout’s ideas and utilities originated in Rbbt. It still provides a broad set of bioinformatics workflows and tools. See the Rbbt-Workflows organization for many real-world examples and usage patterns:
+  - https://github.com/Rbbt-Workflows
+
+For module-specific guides, see doc/*.md in this repository (linked below).
+
+- TSV: doc/TSV.md
+- Workflow: doc/Workflow.md
+- KnowledgeBase: doc/KnowledgeBase.md
+- Association: doc/Association.md
+- Entity: doc/Entity.md
+- WorkQueue: doc/WorkQueue.md
+- Semaphore: doc/Semaphore.md
+
+Additionally, Scout Gear reuses and exposes core facilities from scout-essentials. Summaries of those core modules are included below for convenience.
+
+---
+
+## How command-line interfaces work (scout …)
+
+Scout provides a single “scout” command that discovers and runs nested subcommands from any installed Scout package. Scripts are discovered using the Path subsystem across PATH-like roots, enabling workflows or packages to inject their own commands.
+
+Basics:
+- The CLI resolves terms left-to-right until a file is found under a scout_commands tree.
+  - Example: scout workflow task runs scout_commands/workflow/task
+  - Example: all TSV-related scripts are under scout_commands/tsv and can be listed with scout tsv
+- If the path resolves to a directory instead of a script, a list of available subcommands in that directory is shown.
+- Remaining ARGV is parsed by the selected script using SimpleOPT (SOPT) or compatible parsers.
+- Because discovery uses Path maps, commands contributed by other packages or installed workflows are automatically found.
+
+See the per-module CLI sections below for TSV, Workflow, and KnowledgeBase.
+
+---
+
+## Scout Essentials: Core building blocks
+
+Scout Gear depends on the following main modules from scout-essentials. You’ll use these directly for filesystem/resource orchestration, external command execution, caching, and options handling.
+
+### Path
+
+doc/Path.md
+
+Path is a lightweight, annotation-enabled “smart string” for composing and locating project resources across multiple search maps (current/user/global/lib/tmp, etc.). It integrates with Open and Persist.
+
+Highlights:
+- Path.setup("str") turns a String into a Path with join via [], /, or method_missing (path.foo.bar)
+- Map logical locations to physical roots with path maps; find the first match across map order with path.find (and path.find_all)
+- Filename helpers: get/set/replace/unset extensions; sanitize filenames; relative paths
+- Directory helpers: glob and glob_all over maps; dirname/basename; realpath; newer?
+- Digest summaries: path.digest_str summarizes files/dirs for logging/debugging
+
+Usage:
+```ruby
+p = Path.setup('share/data/myfile')
+p.find             # resolve across configured maps
+p[:subdir, :file]  # joins => share/data/subdir/file
+```
+
+### Open
+
+doc/Open.md
+
+Open unifies file/stream/remote I/O, atomic writes, pipes/tees/FIFOs, (bg)zip helpers, rsync/sync, and lock handling.
+
+Highlights:
+- Open.open/read/write with auto-(de)compression for .gz/.bgz/.zip and remote urls (wget/ssh)
+- Streams: open_pipe, tee_stream, consume_stream, with_fifo
+- Safe writes: sensible_write (tmp + atomic rename + optional locks)
+- Remote: wget with caching, ssh/scp, digest_url, remote cache
+- Filesystem: mkdir/mkfiledir, mv/cp/ln/link_dir, rm/rm_rf, same_file?, exists?, writable?
+- Locking: Open.lock wraps a robust Lockfile (NFS-safe) with refresh/timeout/steal
+
+Example:
+```ruby
+Open.sensible_write("out.txt", Open.open("http://example.com"))
+Open.with_fifo { |fifo| ... }
+Open.rsync("src/", "user@server:dst/", delete: true)
+```
+
+### CMD
+
+doc/CMD.md
+
+CMD wraps Open3.popen3 with robust patterns for streaming, stderr logging, stdin feeding, auto-join of producers, and tool discovery/installation.
+
+Highlights:
+- CMD.cmd("tool args", pipe: true, in: io_or_string, stderr: Log::HIGH, autojoin: true)
+- ConcurrentStream-enabled stdout with join/error propagation
+- Convenience: CMD.bash("bash -l -c '...'"), cmd_pid/cmd_log
+- Tool registry: CMD.tool, CMD.get_tool (auto-install via conda or producers), version scanning
+
+Example:
+```ruby
+io = CMD.cmd("cut", "-f" => 2, "-d" => " ", in: "a b", pipe: true)
+io.read # => "b\n"; io.join
+```
+
+### IndiferentHash
+
+doc/IndiferentHash.md
+
+Hash mixin for indifferent access (string/symbol keys equal), deep-merge, options parsing, and string<->hash conversions.
+
+Highlights:
+- IndiferentHash.setup(hash) to extend a single hash instance
+- Access with h[:a] == h["a"]; delete/include? are indifferent
+- Helpers: deep_merge, values_at with indifferent keys, slice, except
+- Options utilities: parse_options, process_options, positional2hash, hash2string/string2hash
+
+Example:
+```ruby
+opts = IndiferentHash.parse_options('limit=10 title="A title"')
+opts[:title] # => "A title"
+```
+
+### Persist (core serialization/caching)
+
+doc/Persist.md (essentials)
+
+Typed serialization (json/yaml/marshal/binary/arrays), atomic saves, and the high-level persist pattern with locking and streaming.
+
+Highlights:
+- Persist.save/load(obj, file, type)
+- Persist.persist(name, type, dir: ...) { compute_or_stream }
+  - Locking and tmp-to-final atomic writes
+  - Streaming tee: one copy to file, one to caller
+- Memory cache: Persist.memory(name) { ... }
+- Helpers to parse YAML/JSON/Marshal via Open
+
+Example:
+```ruby
+val = Persist.persist("expensive", :json) { compute_hash }
+# subsequent calls load cached JSON unless :update or stale
+```
+
+### Resource
+
+doc/Resource.md
+
+Resource system to claim and produce files on demand (string/proc/url/rake/installers), integrated with Path/Open and locking.
+
+Highlights:
+- claim path => (:string, :proc, :url, :rake, :install)
+- Produce on demand via path.produce and path.open/read
+- Rake integration: drive file tasks/rules to generate outputs
+- Install software into a per-resource “software” dir and update env
+
+Example:
+```ruby
+module MyPkg
+  extend Resource
+  claim root.tmp.test.hello, :string, "Hello"
+end
+MyPkg.tmp.test.hello.read # produces if missing, then reads
+```
+
+Other essentials you’ll encounter:
+- Annotation / AnnotatedArray / NamedArray: lightweight typed attributes on objects and arrays; named tuple-style rows
+- ConcurrentStream: concurrency-aware streams with join/abort/callbacks
+- SimpleOPT (SOPT): tiny CLI option DSL/parser; used by scout commands
+- Log: leveled, colored logging; progress bars; fingerprint utilities
+- TmpFile: temp files/dirs and stable tmp path generator for caches
+
+---
+
+## Scout Gear modules
+
+Scout Gear builds on essentials to deliver domain abstractions and engines.
+
+### TSV
+
+doc/TSV.md
+
+A flexible, typed table abstraction with robust parser, streaming dumper/transformer, parallel traversal, joins/attachments, identifier translation, on-disk persistence (TokyoCabinet/Tkrzw), and range/position indices.
+
+Highlights:
+- Shapes: :double, :list, :flat, :single; key_field + fields
+- Parse TSV/CSV from files/streams/strings with rich header options (sep, type, cast, merge)
+- Dumper/Transformer for streaming pipelines
+- TSV.traverse(obj, cpus: N, into: …) for parallel iteration
+- Attach, change_key, change_id, translate via identifier indices
+- Persistence via TSVAdapter over HDB/BDB/Tkrzw/FWT/PKI/Sharder
+- Streaming paste/concat/collapse utilities; filters with persisted sets
+
+Example:
+```ruby
+tsv = TSV.open(path, persist: true, type: :double)
+tsv.attach(other, complete: true)
+index = TSV.index(tsv, target: "FieldA")
+```
+
+CLI (scout tsv):
+- Scripts live under scout_commands/tsv; list with scout tsv
+- Run a specific subcommand: scout tsv <subcommand> [options] [args...]
+- If you hit a directory, available subcommands are listed
+- Subcommands parse options with SOPT (see each script’s help)
+
+### Workflow
+
+doc/Workflow.md
+
+A lightweight workflow engine. Define tasks with typed inputs and dependencies, create jobs (Steps), and run them with persistence, streaming, provenance, and orchestration under resource rules.
+
+Highlights:
+- input/dep/task DSL with helper methods; task_alias and overrides
+- Jobs (Step): run/load/stream/join, info files, files_dir, provenance
+- Orchestrator: schedule dependent jobs under cpus/IO constraints; retry recoverable errors; archive/erase deps per rules
+- EntityWorkflow: entity-centric tasks and properties
+- Queue helpers to enqueue and process jobs
+
+Example:
+```ruby
+module Baking
+  extend Workflow
+  task :say => :string do |name| "Hi #{name}" end
+end
+
+Baking.job(:say, "Miguel").run # => "Hi Miguel"
+```
+
+CLI (scout workflow):
+- List workflows: scout workflow list
+- Run a task: scout workflow task <workflow> <task> [--jobname NAME] [input options...]
+  - Options include --fork, --nostream, --update, --printpath, --provenance, --clean, --recursive_clean, --override_deps, --deploy (serial|local|queue|SLURM|server)
+- Show job info: scout workflow info <step_path> [--inputs|--recursive_inputs]
+- Provenance: scout workflow prov <step_path> [--plot file.png] […]
+- Trace execution: scout workflow trace <job-result> [options]
+- Process queue: scout workflow process [filters] [--continuous] [--produce_cpus N] […]
+
+You can also dispatch workflow-specific custom commands via:
+- scout workflow cmd <workflow> <subcommand> … (discovers scripts under <workflow>/share/scout_commands/workflow)
+
+### KnowledgeBase
+
+doc/KnowledgeBase.md
+
+A thin orchestrator around Association, TSV, Entity, and Persist to register multiple association databases, normalize/index them, query/traverse across them, manage entity lists, and generate markdown descriptions.
+
+Highlights:
+- Register databases with source/target specs and identifier files
+- get_database/get_index (BDB-backed) with undirected options
+- Query: all, subset (children/parents/neighbours), identify/translate entities
+- Lists: save/load/delete/enumerate typed lists
+- Traversal DSL: multi-hop path finding with wildcards/conditions
+- Markdown descriptions from registry/README files
+
+Example:
+```ruby
+kb = KnowledgeBase.new(Path.setup("var/kb"), "Hsa")
+kb.register :brothers, datafile_test(:person).brothers, undirected: true
+kb.children(:brothers, "Miki") # => ["Miki~Isa", ...]
+```
+
+CLI (scout kb):
+- Configure KB: scout kb config [options] <name>
+- Register DB: scout kb register [options] <name> <filename>
+- Declare entities: scout kb entities <entity> <identifier_files>
+- Show info: scout kb show [<name>]
+- Query: scout kb query <name> <entity_spec>
+- Lists: scout kb list [<list_name>]
+- Traverse: scout kb traverse [options] "<rules,comma,separated>"
+
+### Association
+
+doc/Association.md
+
+Utilities to normalize source/target field specifications from TSVs, open normalized association databases with optional identifier translation, and build pairwise “source~target” indices (optionally undirected). Also includes AssociationItem for entity-like behavior over pair strings and utilities to build incidence/adjacency matrices.
+
+Example:
+```ruby
+idx = Association.index(file, source: "=>Name", target: "Parent=>Name", undirected: true)
+idx.match("Clei")       # => ["Clei~Guille"]
+idx.to_matrix           # boolean incidence matrix
+```
+
+### Entity
+
+doc/Entity.md
+
+Annotate plain values or arrays as entities with behavior-rich “properties”, automatic format mapping, identifier translation (Entity::Identified), array-aware property batching/caching, and persistence for property results via Persist.
+
+Example:
+```ruby
+module Person
+  extend Entity
+  property :greet => :single do "Hi #{self}" end
+end
+Person.setup("Miki").greet
+```
+
+### WorkQueue
+
+doc/WorkQueue.md
+
+A multi-process work pipeline (forked workers + semaphore-guarded sockets) to parallelize processing over a stream of inputs, with robust error propagation.
+
+Example:
+```ruby
+q = WorkQueue.new(4){|x| x * 2}
+out = []; q.process{|y| out << y}
+(1..100).each{|i| q.write i}; q.close; q.join
+```
+
+### Semaphore (ScoutSemaphore)
+
+doc/Semaphore.md
+
+Concurrency helpers based on POSIX named semaphores (via RubyInline C bindings), plus higher-level helpers to bound concurrency with forks/threads.
+
+Example:
+```ruby
+ScoutSemaphore.with_semaphore(2) do |sem|
+  ScoutSemaphore.synchronize(sem){ critical_work }
+end
+```
+
+---
+
+## Examples and further reading
+
+- This repository’s docs directory provides in-depth guides for each module:
+  - TSV: doc/TSV.md
+  - Workflow: doc/Workflow.md
+  - KnowledgeBase: doc/KnowledgeBase.md
+  - Association: doc/Association.md
+  - Entity: doc/Entity.md
+  - WorkQueue: doc/WorkQueue.md
+  - Semaphore: doc/Semaphore.md
+- For numerous end-to-end examples and real datasets, explore the Rbbt-Workflows organization:
+  - https://github.com/Rbbt-Workflows
+- For foundational utilities (Path, Open, CMD, IndiferentHash, Persist, Resource, etc.), consult the scout-essentials documentation:
+  - Those modules are summarized above and used pervasively across Scout Gear.
+
+---
+
+## Notes
+
+- Streaming everywhere: many APIs return ConcurrentStream-enabled IOs. Always read to EOF and join (or rely on autojoin) to ensure producers exit and errors are surfaced.
+- Atomicity and locking: Open.sensible_write and Persist.persist use tmp+mv and lockfiles to provide robust cross-process behavior.
+- Discovery and composition: the Path subsystem and Resource claims make it easy to build portable projects with on-demand production of resources and discoverable commands.

data/VERSION

@@ -1 +1 @@
-10.8.4
+10.9.0

data/doc/Association.md

@@ -0,0 +1,288 @@
+# Association
+
+Association provides a compact toolkit to open, normalize, and index pairwise relationships from TSV-like sources. With it you can:
+
+- Parse declarative source/target field specifications (including format remapping).
+- Open an “association database” (TSV) that standardizes keys/fields and optional identifier translation via Entity/TSV indices.
+- Build a fast BDB-backed index over pair “edges” using “source~target” keys, optionally undirected.
+- Work with association “items” (pairs) as Entities with useful properties and conversions.
+- Produce incidence/adjacency matrices and perform filtering/subsetting over pairs.
+
+It integrates with:
+- TSV (parsing, reordering, indices)
+- Entity (format registry and identifier translation)
+- Persist (caching/DB backends)
+
+Sections:
+- Field specification syntax and normalization
+- Opening association databases
+- Building and using association indices
+- AssociationItem: entity properties over pairs
+- Matrix utilities
+- Examples
+
+---
+
+## Field specification syntax and normalization
+
+Association accepts flexible “field specs” to declare which columns are source and target, optionally including header aliases and format conversions.
+
+Syntax patterns (strings):
+
+- "FieldName"
+  - Use the column named FieldName.
+- "FieldName=~Header"
+  - Use field FieldName but present it as Header in outputs.
+- "=~Header"
+  - No explicit field (infer from header or Entity format), but present as Header.
+- "FieldName=>TargetFormat"
+  - Use FieldName and translate identifiers to TargetFormat (via TSV.translation_index / Entity identifiers).
+- "FieldName=~Header=>TargetFormat"
+  - Full form; pick field, rename header, and convert identifiers.
+
+Parsing and normalization helpers:
+- Association.parse_field_specification(spec) -> [field, header, final_format]
+- Association.normalize_specs(spec, all_fields=nil) -> normalized [field, header, format]
+  - If a field is not directly present but is a recognized Entity format, it tries to find a matching column within all_fields by that Entity.
+
+Extract source/target specs:
+- specs = Association.extract_specs(all_fields, options)
+  - options keys: :source, :target, :source_format, :target_format, :format (hash of entity_type -> default_target_format)
+  - Returns a Hash with:
+    - :source => [field, header, final_format]
+    - :target => [field, header, final_format]
+  - Infers default source/target when not provided:
+    - If both nil → source := key_field; target := first data field
+    - If source nil but target is key → source := first data field; and vice versa
+
+Resolve headers and positions:
+- Association.headers(all_fields, info_fields=nil, options)
+  - all_fields: [key_field, field1, ...]
+  - info_fields: extra value columns to keep besides target (defaults to “all” except source and target).
+  - Returns:
+    - [source_pos, field_pos, source_header, field_headers, source_format, target_format]
+  - Handles :format hash defaults per entity type, and honors explicit source/target formats.
+
+---
+
+## Opening association databases
+
+Association.open coerces a TSV (file/Path/TSV) into a normalized association database with optional identifier translation.
+
+```ruby
+db = Association.open(
+  file_or_tsv,
+  source: "Wife (ID)=>Alias",
+  target: "Husband (ID)=>Name",
+  namespace: "person",   # optional; replaces NAMESPACE placeholders in paths
+  type: :list            # optional TSV type; inferred when not set
+)
+```
+
+Behavior:
+- Reads header and infers positions via headers(...).
+- If target/source formats are specified:
+  - Builds translation indices from:
+    - TSV.identifier_files(file), Entity.identifier_files(format), and options[:identifiers].
+  - Rewrites keys/values to requested formats (e.g., “(ID)=>Name”).
+- Produces a TSV with:
+  - key_field: resolved source field name (with “(format)” suffix if translated).
+  - fields: [resolved target field (with “(format)” if translated), plus remaining info_fields].
+  - type: inherited/passed (:double, :list, :flat, :single).
+
+Namespace placeholder:
+- When opening from a path string containing “NAMESPACE”, passing namespace: will substitute it:
+  - Example: ".../NAMESPACE/identifiers.tsv" -> ".../person/identifiers.tsv"
+
+Persisted variant:
+- Association.database(file, ...) wraps Association.open with Persist.tsv and a “BDB” engine:
+  - Returns a persistence-backed TSV (keys/fields/type saved with TSVAdapter).
+  - Options: any Association.open options plus :persist / persist_* (via IndiferentHash).
+
+Examples:
+- Simple open:
+  ```ruby
+  db = Association.database(datadir.person.marriages,
+                            source: "Wife", target: "Husband", persist: true)
+  db["Clei"]["Husband"]  # => "Miguel"
+  db["Clei"]["Date"]     # => "2021"
+  ```
+
+- Partial field + format:
+  ```ruby
+  db = Association.database(datadir.person.marriages,
+                            source: "Wife=>Alias", target: "Husband=>Name")
+  ```
+
+- Flat TSV:
+  ```ruby
+  flat = datadir.person.parents.tsv(type: :flat, fields: ["Parent"])
+  db = Association.database(flat)
+  db["Miki"]  # => %w(Juan Mariluz)
+  ```
+
+---
+
+## Building and using association indices
+
+Association.index materializes a BDB index over pairwise relations with keys of the form “source~target”. The index entries store the “info fields” (everything but the two endpoints) as a :list TSV.
+
+```ruby
+idx = Association.index(file_or_tsv,
+                        source: "=>Name",
+                        target: "Parent=>Name",
+                        undirected: false,  # true duplicates (source~target) and (target~source)
+                        persist: true)
+```
+
+- Under the hood:
+  - Opens/normalizes the database with Association.open (or uses provided DB).
+  - Builds keys “[source]~[target]” and writes values (info fields) as a list.
+  - If undirected true (or same source/target column), writes both “[s]~[t]” and “[t]~[s]”.
+
+- Return value:
+  - A BDB TSV extended with Association::Index, annotated with:
+    - source_field, target_field, undirected
+  - The index sets key_field to “SourceField~TargetField[~undirected]”.
+
+- Methods on Association::Index:
+  - parse_key_field → sets source_field/target_field/undirected from key_field.
+  - match(entity) → returns all “source~target” keys whose source starts with entity (prefix-based).
+  - subset(source_list, target_spec)
+    - source_list: list of source entities or :all.
+    - target_spec: :all or list to filter by target side.
+    - Returns matching keys, handling undirected symmetry.
+  - reverse → returns a reversed index (keys swapped to “target~source”) persisted in a side file (.reverse).
+  - filter(value_field=nil, target_value=nil, &block)
+    - Without block: filter keys whose value_field is present (or equals target_value).
+    - With block: custom predicate over values (or key+values if value_field nil).
+  - to_matrix(value_field=nil) { |values| ... }
+    - Produces an incidence matrix TSV (rows: sources, columns: targets):
+      - If value_field provided, uses that column (or block mapping).
+      - Else boolean incidence.
+
+Note:
+- reverse persists its own DB with swapped key_field; it carries over annotations, unnamed flag, and undirected.
+
+Example:
+```ruby
+idx = Association.index(datadir.person.brothers, undirected: true)
+idx.match("Clei")           # => ["Clei~Guille"]
+idx.reverse.match("Clei")   # => ["Clei~Guille"] (same when undirected)
+idx.filter("Type", "mother")
+idx.subset(["Miki","Guille"], :all) # some “source~target” keys
+```
+
+---
+
+## AssociationItem: entity properties over pairs
+
+AssociationItem is an Entity module that represents “pairs” as annotated strings “source~target”. You typically obtain such lists from index.keys, and then call properties on the annotated list.
+
+Annotate:
+- Association.index(file).keys returns raw strings; annotate them with AssociationItem.setup if needed, or use Index helpers that return annotated where applicable.
+
+Properties (selected):
+- name (single): "source~target" (returns friendly names using entity .name where available).
+- full_name: database-prefixed “db:source~target” when database set.
+- invert: swap endpoints (works on single or array); toggles reverse flag.
+- namespace: forwarded from knowledge_base (if present).
+- part (array2single): returns [source, "~", target] tuples for each pair.
+- target / source (array2single): returns just target or source identifiers.
+- target_type / source_type (both): resolve entity type names via knowledge_base target/source (requires a KnowledgeBase integration providing #source/#target/#undirected/#get_index/#index_fields).
+- target_entity / source_entity: wrap target/source into Entity-typed values according to knowledge_base types.
+- index(database=nil): resolve underlying index (delegates to knowledge_base.get_index).
+- value (array2single): fetch info values for each pair from the index; returns NamedArrays.
+- info_fields / info: helper for value lookups; info builds a Hash for each pair.
+- tsv (array): emit a TSV for the pair list with columns: source_type, target_type, info_fields.
+- filter(*args, &block): filter this pair list using the generated tsv.select.
+
+Utilities:
+- AssociationItem.incidence(pairs, key_field="Source") { |pair| optional_value }
+  - Returns TSV (list) with rows as sources and columns as targets; cells are blocks’ value or booleans.
+- AssociationItem.adjacency(pairs, key_field="Source") { |pair| value }
+  - Returns TSV (double) mapping source -> [Target, values].
+
+Convenience:
+- TSV.incidence(tsv, **kwargs) delegates to Association.index(...).keys -> AssociationItem.incidence
+
+---
+
+## Matrix utilities
+
+Given an index:
+- idx.to_matrix(value_field=nil) { |values| ... } → TSV list
+  - value_field omitted and no block → boolean incidence.
+  - With value_field → use that column (vector) as the cell value.
+  - With block → compute cell values programmatically.
+
+Standalone:
+- AssociationItem.incidence/pairs as above.
+- AssociationItem.adjacency for adjacency list.
+
+---
+
+## Examples
+
+Parse specs:
+```ruby
+Association.parse_field_specification("=~Associated Gene Name=>Ensembl Gene ID")
+# => [nil, "Associated Gene Name", "Ensembl Gene ID"]
+
+Association.normalize_specs("TG=~Associated Gene Name=>Ensembl Gene ID", %w(SG TG Effect))
+# => ["TG", "Associated Gene Name", "Ensembl Gene ID"]
+
+Association.extract_specs(%w(SG TG Effect), source: "SG", target: "TG")
+# => { source: ["SG", nil, nil], target: ["TG", nil, nil] }
+```
+
+Open database (translate to human-readable names):
+```ruby
+db = Association.database(datadir.person.marriages,
+                          source: "Wife (ID)=>Alias",
+                          target: "Husband (ID)=>Name")
+db["Clei"]["Husband"]  # => "Miguel"
+db["Clei"]["Date"]     # => "2021"
+```
+
+Index and match:
+```ruby
+idx = Association.index(datadir.person.brothers, undirected: true)
+idx.match("Clei")  # => ["Clei~Guille"]
+idx.subset(["Clei"], :all) # => ["Clei~Guille"]
+idx.reverse.subset(["Guille"], :all) # => ["Guille~Clei"]
+```
+
+Filter:
+```ruby
+idx = Association.index(datadir.person.parents)
+idx.filter('Type of parent', 'mother') # keys whose info field contains 'mother'
+```
+
+Incidence matrix:
+```ruby
+pairs = Association.index(datadir.person.brothers, undirected: true).keys
+inc = AssociationItem.incidence(pairs)
+inc["Clei"]["Guille"] # => true
+```
+
+List serializer handling:
+```ruby
+tsv = TSV.open <<~EOF
+#: :sep=,#:type=:list
+#lowcase,upcase,double,triple
+a,A,aa,aaa
+b,B,bb,bbb
+EOF
+i = Association.index(tsv)
+i["a~A"] # => ['aa', 'aaa']
+```
+
+---
+
+## Notes and edge cases
+
+- undirected default: if source_field == target_field, undirected is assumed true; else false unless set.
+- When specifying formats, ensure identifier TSVs are reachable. You can pass :identifiers (TSV/Path) or rely on TSV.identifier_files(file) and Entity.identifier_files(format).
+- Association.index returns a BDB-backed TSV; reverse indexing persists to a side .reverse database next to the main DB.
+- Paths containing [NAMESPACE] or NAMESPACE are substituted with options[:namespace].