scout-gear 10.8.3 → 10.9.0

data/doc/Association.md

@@ -0,0 +1,288 @@
+# Association
+
+Association provides a compact toolkit to open, normalize, and index pairwise relationships from TSV-like sources. With it you can:
+
+- Parse declarative source/target field specifications (including format remapping).
+- Open an “association database” (TSV) that standardizes keys/fields and optional identifier translation via Entity/TSV indices.
+- Build a fast BDB-backed index over pair “edges” using “source~target” keys, optionally undirected.
+- Work with association “items” (pairs) as Entities with useful properties and conversions.
+- Produce incidence/adjacency matrices and perform filtering/subsetting over pairs.
+
+It integrates with:
+- TSV (parsing, reordering, indices)
+- Entity (format registry and identifier translation)
+- Persist (caching/DB backends)
+
+Sections:
+- Field specification syntax and normalization
+- Opening association databases
+- Building and using association indices
+- AssociationItem: entity properties over pairs
+- Matrix utilities
+- Examples
+
+---
+
+## Field specification syntax and normalization
+
+Association accepts flexible “field specs” to declare which columns are source and target, optionally including header aliases and format conversions.
+
+Syntax patterns (strings):
+
+- "FieldName"
+  - Use the column named FieldName.
+- "FieldName=~Header"
+  - Use field FieldName but present it as Header in outputs.
+- "=~Header"
+  - No explicit field (infer from header or Entity format), but present as Header.
+- "FieldName=>TargetFormat"
+  - Use FieldName and translate identifiers to TargetFormat (via TSV.translation_index / Entity identifiers).
+- "FieldName=~Header=>TargetFormat"
+  - Full form; pick field, rename header, and convert identifiers.
+
+Parsing and normalization helpers:
+- Association.parse_field_specification(spec) -> [field, header, final_format]
+- Association.normalize_specs(spec, all_fields=nil) -> normalized [field, header, format]
+  - If a field is not directly present but is a recognized Entity format, it tries to find a matching column within all_fields by that Entity.
+
+Extract source/target specs:
+- specs = Association.extract_specs(all_fields, options)
+  - options keys: :source, :target, :source_format, :target_format, :format (hash of entity_type -> default_target_format)
+  - Returns a Hash with:
+    - :source => [field, header, final_format]
+    - :target => [field, header, final_format]
+  - Infers default source/target when not provided:
+    - If both nil → source := key_field; target := first data field
+    - If source nil but target is key → source := first data field; and vice versa
+
+Resolve headers and positions:
+- Association.headers(all_fields, info_fields=nil, options)
+  - all_fields: [key_field, field1, ...]
+  - info_fields: extra value columns to keep besides target (defaults to “all” except source and target).
+  - Returns:
+    - [source_pos, field_pos, source_header, field_headers, source_format, target_format]
+  - Handles :format hash defaults per entity type, and honors explicit source/target formats.
+
+---
+
+## Opening association databases
+
+Association.open coerces a TSV (file/Path/TSV) into a normalized association database with optional identifier translation.
+
+```ruby
+db = Association.open(
+  file_or_tsv,
+  source: "Wife (ID)=>Alias",
+  target: "Husband (ID)=>Name",
+  namespace: "person",   # optional; replaces NAMESPACE placeholders in paths
+  type: :list            # optional TSV type; inferred when not set
+)
+```
+
+Behavior:
+- Reads header and infers positions via headers(...).
+- If target/source formats are specified:
+  - Builds translation indices from:
+    - TSV.identifier_files(file), Entity.identifier_files(format), and options[:identifiers].
+  - Rewrites keys/values to requested formats (e.g., “(ID)=>Name”).
+- Produces a TSV with:
+  - key_field: resolved source field name (with “(format)” suffix if translated).
+  - fields: [resolved target field (with “(format)” if translated), plus remaining info_fields].
+  - type: inherited/passed (:double, :list, :flat, :single).
+
+Namespace placeholder:
+- When opening from a path string containing “NAMESPACE”, passing namespace: will substitute it:
+  - Example: ".../NAMESPACE/identifiers.tsv" -> ".../person/identifiers.tsv"
+
+Persisted variant:
+- Association.database(file, ...) wraps Association.open with Persist.tsv and a “BDB” engine:
+  - Returns a persistence-backed TSV (keys/fields/type saved with TSVAdapter).
+  - Options: any Association.open options plus :persist / persist_* (via IndiferentHash).
+
+Examples:
+- Simple open:
+  ```ruby
+  db = Association.database(datadir.person.marriages,
+                            source: "Wife", target: "Husband", persist: true)
+  db["Clei"]["Husband"]  # => "Miguel"
+  db["Clei"]["Date"]     # => "2021"
+  ```
+
+- Partial field + format:
+  ```ruby
+  db = Association.database(datadir.person.marriages,
+                            source: "Wife=>Alias", target: "Husband=>Name")
+  ```
+
+- Flat TSV:
+  ```ruby
+  flat = datadir.person.parents.tsv(type: :flat, fields: ["Parent"])
+  db = Association.database(flat)
+  db["Miki"]  # => %w(Juan Mariluz)
+  ```
+
+---
+
+## Building and using association indices
+
+Association.index materializes a BDB index over pairwise relations with keys of the form “source~target”. The index entries store the “info fields” (everything but the two endpoints) as a :list TSV.
+
+```ruby
+idx = Association.index(file_or_tsv,
+                        source: "=>Name",
+                        target: "Parent=>Name",
+                        undirected: false,  # true duplicates (source~target) and (target~source)
+                        persist: true)
+```
+
+- Under the hood:
+  - Opens/normalizes the database with Association.open (or uses provided DB).
+  - Builds keys “[source]~[target]” and writes values (info fields) as a list.
+  - If undirected true (or same source/target column), writes both “[s]~[t]” and “[t]~[s]”.
+
+- Return value:
+  - A BDB TSV extended with Association::Index, annotated with:
+    - source_field, target_field, undirected
+  - The index sets key_field to “SourceField~TargetField[~undirected]”.
+
+- Methods on Association::Index:
+  - parse_key_field → sets source_field/target_field/undirected from key_field.
+  - match(entity) → returns all “source~target” keys whose source starts with entity (prefix-based).
+  - subset(source_list, target_spec)
+    - source_list: list of source entities or :all.
+    - target_spec: :all or list to filter by target side.
+    - Returns matching keys, handling undirected symmetry.
+  - reverse → returns a reversed index (keys swapped to “target~source”) persisted in a side file (.reverse).
+  - filter(value_field=nil, target_value=nil, &block)
+    - Without block: filter keys whose value_field is present (or equals target_value).
+    - With block: custom predicate over values (or key+values if value_field nil).
+  - to_matrix(value_field=nil) { |values| ... }
+    - Produces an incidence matrix TSV (rows: sources, columns: targets):
+      - If value_field provided, uses that column (or block mapping).
+      - Else boolean incidence.
+
+Note:
+- reverse persists its own DB with swapped key_field; it carries over annotations, unnamed flag, and undirected.
+
+Example:
+```ruby
+idx = Association.index(datadir.person.brothers, undirected: true)
+idx.match("Clei")           # => ["Clei~Guille"]
+idx.reverse.match("Clei")   # => ["Clei~Guille"] (same when undirected)
+idx.filter("Type", "mother")
+idx.subset(["Miki","Guille"], :all) # some “source~target” keys
+```
+
+---
+
+## AssociationItem: entity properties over pairs
+
+AssociationItem is an Entity module that represents “pairs” as annotated strings “source~target”. You typically obtain such lists from index.keys, and then call properties on the annotated list.
+
+Annotate:
+- Association.index(file).keys returns raw strings; annotate them with AssociationItem.setup if needed, or use Index helpers that return annotated where applicable.
+
+Properties (selected):
+- name (single): "source~target" (returns friendly names using entity .name where available).
+- full_name: database-prefixed “db:source~target” when database set.
+- invert: swap endpoints (works on single or array); toggles reverse flag.
+- namespace: forwarded from knowledge_base (if present).
+- part (array2single): returns [source, "~", target] tuples for each pair.
+- target / source (array2single): returns just target or source identifiers.
+- target_type / source_type (both): resolve entity type names via knowledge_base target/source (requires a KnowledgeBase integration providing #source/#target/#undirected/#get_index/#index_fields).
+- target_entity / source_entity: wrap target/source into Entity-typed values according to knowledge_base types.
+- index(database=nil): resolve underlying index (delegates to knowledge_base.get_index).
+- value (array2single): fetch info values for each pair from the index; returns NamedArrays.
+- info_fields / info: helper for value lookups; info builds a Hash for each pair.
+- tsv (array): emit a TSV for the pair list with columns: source_type, target_type, info_fields.
+- filter(*args, &block): filter this pair list using the generated tsv.select.
+
+Utilities:
+- AssociationItem.incidence(pairs, key_field="Source") { |pair| optional_value }
+  - Returns TSV (list) with rows as sources and columns as targets; cells are blocks’ value or booleans.
+- AssociationItem.adjacency(pairs, key_field="Source") { |pair| value }
+  - Returns TSV (double) mapping source -> [Target, values].
+
+Convenience:
+- TSV.incidence(tsv, **kwargs) delegates to Association.index(...).keys -> AssociationItem.incidence
+
+---
+
+## Matrix utilities
+
+Given an index:
+- idx.to_matrix(value_field=nil) { |values| ... } → TSV list
+  - value_field omitted and no block → boolean incidence.
+  - With value_field → use that column (vector) as the cell value.
+  - With block → compute cell values programmatically.
+
+Standalone:
+- AssociationItem.incidence/pairs as above.
+- AssociationItem.adjacency for adjacency list.
+
+---
+
+## Examples
+
+Parse specs:
+```ruby
+Association.parse_field_specification("=~Associated Gene Name=>Ensembl Gene ID")
+# => [nil, "Associated Gene Name", "Ensembl Gene ID"]
+
+Association.normalize_specs("TG=~Associated Gene Name=>Ensembl Gene ID", %w(SG TG Effect))
+# => ["TG", "Associated Gene Name", "Ensembl Gene ID"]
+
+Association.extract_specs(%w(SG TG Effect), source: "SG", target: "TG")
+# => { source: ["SG", nil, nil], target: ["TG", nil, nil] }
+```
+
+Open database (translate to human-readable names):
+```ruby
+db = Association.database(datadir.person.marriages,
+                          source: "Wife (ID)=>Alias",
+                          target: "Husband (ID)=>Name")
+db["Clei"]["Husband"]  # => "Miguel"
+db["Clei"]["Date"]     # => "2021"
+```
+
+Index and match:
+```ruby
+idx = Association.index(datadir.person.brothers, undirected: true)
+idx.match("Clei")  # => ["Clei~Guille"]
+idx.subset(["Clei"], :all) # => ["Clei~Guille"]
+idx.reverse.subset(["Guille"], :all) # => ["Guille~Clei"]
+```
+
+Filter:
+```ruby
+idx = Association.index(datadir.person.parents)
+idx.filter('Type of parent', 'mother') # keys whose info field contains 'mother'
+```
+
+Incidence matrix:
+```ruby
+pairs = Association.index(datadir.person.brothers, undirected: true).keys
+inc = AssociationItem.incidence(pairs)
+inc["Clei"]["Guille"] # => true
+```
+
+List serializer handling:
+```ruby
+tsv = TSV.open <<~EOF
+#: :sep=,#:type=:list
+#lowcase,upcase,double,triple
+a,A,aa,aaa
+b,B,bb,bbb
+EOF
+i = Association.index(tsv)
+i["a~A"] # => ['aa', 'aaa']
+```
+
+---
+
+## Notes and edge cases
+
+- undirected default: if source_field == target_field, undirected is assumed true; else false unless set.
+- When specifying formats, ensure identifier TSVs are reachable. You can pass :identifiers (TSV/Path) or rely on TSV.identifier_files(file) and Entity.identifier_files(format).
+- Association.index returns a BDB-backed TSV; reverse indexing persists to a side .reverse database next to the main DB.
+- Paths containing [NAMESPACE] or NAMESPACE are substituted with options[:namespace].

data/doc/Entity.md

@@ -0,0 +1,296 @@
+# Entity
+
+Entity is a lightweight system to turn plain Ruby values (strings, arrays, numerics) into annotated, behavior-rich “entities.” It layers on top of Annotation and provides:
+
+- A module-level DSL to define “properties” (methods) for entities and arrays of entities.
+- Format mapping and identifier translation between formats (via TSV indices).
+- Automatic conversion of NamedArray field values into the appropriate entity type.
+- Optional persistence for property results (including annotation lists) using Persist.
+- Array-aware property execution with smart caching and support for multi-return computations.
+
+Sections:
+- Getting started and core concepts
+- Formats and automatic conversion
+- Properties: types, array semantics and persistence
+- Identifier translation (Entity::Identified)
+- Integration with NamedArray and TSV
+- Introspection helpers
+- Examples
+
+---
+
+## Getting started and core concepts
+
+Define a new entity type by extending Entity in a module. The module becomes the “entity class” for values you annotate with it.
+
+```ruby
+module ReversableString
+  extend Entity
+
+  property :reverse_text => :single do
+    self.reverse
+  end
+end
+
+s = ReversableString.setup("String1")
+s.reverse_text  # => "1gnirtS"
+```
+
+Key facts:
+- Extending Entity decorates the module with Annotation and Entity::Property capabilities.
+- Entity.setup(value, format: ..., namespace: ...) annotates the value with this entity module (and any extra metadata).
+- Entities can also be arrays: pass an array to setup to make an AnnotatedArray; properties can be defined to act on the array or per-item.
+
+---
+
+## Formats and automatic conversion
+
+Entity supports “formats” to describe the logical identifier type of a value (e.g., “Ensembl Gene ID”, “Name”). Formats are globally mapped to entity modules using a tolerant index:
+
+- Set formats accepted by the entity:
+  ```ruby
+  module Gene
+    extend Entity
+    self.format = ["Ensembl Gene ID", "Alias", "Name"]
+  end
+  ```
+
+- Global registry:
+  - Entity.formats is a FormatIndex (case-aware, tolerant finder). It can match strings like “Transcription Factor (Ensembl Gene ID)” to “Ensembl Gene ID”.
+  - Entity.formats[format_name] ⇒ entity module.
+
+Automatic conversion when reading from tables:
+- NamedArray fields return values wrapped as entities if there is a matching format. See Integration with NamedArray.
+
+Manual preparation:
+- Entity.prepare_entity(value, field, options = {}) returns a value annotated with the entity for that field if a matching format is known:
+  ```ruby
+  Entity.prepare_entity("ENSG000001", "Ensembl Gene ID")  # wraps into the entity registered for that format
+  ```
+
+---
+
+## Properties: types, array semantics and persistence
+
+Define behaviors (methods) using the property DSL. A property can target:
+- :single — defined for a single entity.
+- :array — defined for an array of entities (takes the array as self).
+- :multiple — batch property for arrays that computes all missing per-item results at once and returns a mapping/array; Entity handles filling per-item caches.
+- :both — define a method directly that should work for both single and array (default).
+- Interface adapters:
+  - :single2array — defined for single values, but expose an array facade.
+  - :array2single — defined for arrays, but expose single-return facade.
+
+Examples:
+
+```ruby
+module ReversableString
+  extend Entity
+
+  # Operates on single entity
+  property :reverse_text_single => :single do
+    self.reverse
+  end
+
+  # Operates on an array and returns per-item values
+  property :reverse_text_ary => :array do
+    self.collect { |s| s.reverse }
+  end
+
+  # Both single and array supported by a single method
+  property :reverse_both => :both do
+    if Array === self
+      self.collect(&:reverse)
+    else
+      self.reverse
+    end
+  end
+
+  # Batch compute for arrays (multiple)
+  property :multiple_annotation_list => :multiple do
+    # Return either an Array aligned with input indices or a Hash {item => result}
+    self.collect { |e| e.chars } # e.g., list of char arrays
+  end
+end
+```
+
+Array semantics and caching:
+- When you call an array property from an element (item.reverse_text_ary), Entity uses the container’s cached result via an internal _ary_property_cache to avoid recomputing per element.
+- For :multiple, Entity runs the computation once for the whole array, caches, and dispatches results to the items that requested it (even across partially overlapping arrays).
+
+Persistence for properties:
+- Mark any property as persisted to cache its result across runs/filesystems:
+
+```ruby
+ReversableString.persist :reverse_text_single, :marshal
+ReversableString.persist :reverse_text_ary, :array, dir: "/tmp/entity_cache"
+ReversableString.persist :annotation_list, :annotation, annotation_repo: "/path/to/repo.tch"
+```
+
+- persist(name, type=:marshal, options={})
+  - type can be any Persist serializer or special:
+    - :annotation or :annotations — store annotation objects via Persist.annotation_repo_persist (Tokyo Cabinet repo), with option :annotation_repo pointing to the repo path.
+    - :array, :marshal, etc.
+  - options default to:
+    - persist: true
+    - dir: Entity.entity_property_cache[self.to_s][name] (default cache under var/entity_property/<Entity>/<property>)
+- persisted?(name), unpersist(name) — manage persisted registration.
+- Internally, Entity::Property.persist wraps property execution inside Persist.persist (or annotation_repo_persist) and keys it by entity id.
+
+Notes:
+- Entity.ids are derived from Annotation ids (Annotation::AnnotatedObject#id).
+- Persisted array returns are validated against the current array call sites to extract per-item results correctly.
+
+---
+
+## Identifier translation (Entity::Identified)
+
+For entities that can translate between identifier formats, include Entity::Identified and register identifier sources.
+
+Register identifier files:
+- add_identifiers(file_or_tsv, default_format=nil, name_format=nil, description_format=nil)
+  - file can be a Path/filename (with optional NAMESPACE placeholders) or a TSV instance.
+  - This sets:
+    - identity formats on the entity (formats accepted),
+    - default format (:default),
+    - name format (:name),
+    - description format (not used directly in core, but available).
+
+Namespace placeholder:
+- Use “NAMESPACE” in file paths to be replaced dynamically using the entity instance’s namespace annotation.
+  - If your files include NAMESPACE and the value is not provided on the entity, those files are skipped with a warning.
+
+Translate between formats:
+- to(target_format) property is auto-defined for Identified entities.
+  - target_format can be a literal format name, :name (-> name_format), or :default.
+  - Works on single entities or arrays; on arrays returns an array aligned with input order.
+  - Example:
+    ```ruby
+    module Person
+      extend Entity
+    end
+    Person.add_identifiers("/data/#{Entity::Identified::NAMESPACE_TAG}/identifiers", "Name", "Alias")
+
+    miguel = Person.setup("001", format: "ID", namespace: :person)
+    miguel.to("Alias")   # => "Miki"
+    miguel.to(:name)     # => "Miguel"
+    ```
+
+Identifier indexes:
+- Entity builds and caches TSV.translation_index from identifier files via Persist.memory, keyed by [entity_type, source_format, target_format].
+- Call identifier_index(target_format, source_format=nil) to get the TSV index.
+  - source_format defaults to the entity’s current format; if not found, Entity retries without specifying source.
+
+Introspection:
+- Entity.identifier_files(field) — class method returning the list of TSVs involved in a format for entities that include Identified.
+
+---
+
+## Integration with NamedArray and TSV
+
+Entity values are automatically prepared when accessing NamedArray fields:
+
+- NamedArray#[](key) is overridden to call Entity.prepare_entity(v, key), so if a field name is a recognized format (or carries it in parentheses, e.g., “Gene Name (Ensembl Gene ID)”), the returned cell value is wrapped as an entity.
+
+Example:
+```ruby
+module SomeEntity; extend Entity; self.format = "SomeEntity"; end
+
+row = NamedArray.setup(["a", "b"], %w(SomeEntity Other))
+row["SomeEntity"].respond_to?(:all_properties)  # => true
+```
+
+This makes TSV rows entity-aware when you deserialize via TSV.open; NamedArray instances become rich objects with entity behaviors available per column.
+
+---
+
+## Introspection helpers
+
+Entity::Object adds convenience to every annotated entity:
+
+- entity_classes — list of Entity modules applied (from Annotation).
+- base_entity — the last Entity in annotation_types, i.e., the primary one.
+- all_properties — list of property names available across entity modules.
+- _ary_property_cache — internal cache used to memoize array property evaluations for items.
+
+The Entity module itself exposes:
+- Entity.formats — global FormatIndex of format name → entity module, with tolerant lookup (find handles strings with extra decorations).
+- Entity.prepare_entity(value, field, options={}) — utility to wrap a value or array into an entity based on format mapping.
+
+---
+
+## Examples
+
+Define a property-rich entity and use it on values and arrays:
+
+```ruby
+module ReversableString
+  extend Entity
+
+  property :reverse_text_single => :single do
+    self.reverse
+  end
+
+  property :reverse_text_ary => :array do
+    self.collect { |s| s.reverse }
+  end
+
+  # Persist selected properties
+  persist :reverse_text_single, :marshal
+  persist :reverse_text_ary, :array
+end
+
+# Single
+s = ReversableString.setup("String1")
+s.reverse_text_single  # => "1gnirtS"
+
+# Array
+arr = ReversableString.setup(["String1", "String2"])
+arr.reverse_text_ary       # => ["1gnirtS", "2gnirtS"]
+arr[1].reverse_text_ary    # uses cached array result; returns "2gnirtS"
+```
+
+Translate identifiers:
+
+```ruby
+module Person
+  extend Entity
+end
+
+# Identify formats and sources
+Person.add_identifiers("/data/#{Entity::Identified::NAMESPACE_TAG}/identifiers.tsv",
+                       "Name", "Alias")
+
+Person.setup("001", format: "ID", namespace: :person).to("Alias")   # => "Miki"
+Person.setup("001", format: "ID", namespace: :person).to(:name)     # => "Miguel"
+
+list = Person.setup(["001"], format: "ID", namespace: :person)
+list.to("Name")  # => ["Miguel"]
+```
+
+Automatic entity wrapping from NamedArray/TSV:
+
+```ruby
+module Gene; extend Entity; self.format = "Ensembl Gene ID"; end
+
+tsv = TSV.open <<~EOF
+#: :sep=" " #:type=:list
+#Id Ensembl Gene ID Other
+row1 ENSG0001 X
+EOF
+
+row = tsv["row1"]
+g = row["Ensembl Gene ID"]   # => wrapped into Gene entity (if format registered)
+g.all_properties             # => property list for Gene
+```
+
+---
+
+## Notes and edge cases
+
+- Entity.prepare_entity duplicates input strings/arrays to avoid mutating caller state; array duplication can be forced per call via dup_array:true.
+- For arrays, properties marked :array2single or :single2array adapt their interface between collection and element call sites.
+- When using identifiers with NAMESPACE placeholders, ensure you set namespace on entities (Person.setup("001", namespace: :person)) or those files will be ignored.
+- Persisted annotation properties (type :annotation) use a Tokyo Cabinet repo; you can supply a repo path via annotation_repo:, or let Persist.annotation_repo_persist create/use a repo by path.
+
+Entity turns plain values into meaningful, behavior-rich objects tailored to your domain (genes, samples, users, etc.), with robust identifier translation and scalable property evaluation/persistence built-in.