legion-data 1.7.3 → 1.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a12da2249f20eb148411c6e052ef5b0bbdef86579907a84feab15d713f7b51f9
-  data.tar.gz: 433ea32fa8120428137d2114e54f5285653acc872956d04df3520cc996d277bc
+  metadata.gz: 86680ede4352bc3726face7c12c9b574212747c308a3a90aae923a48d12f6c24
+  data.tar.gz: 97e4c063fc98c918ddccd34e39c84756487f7e88c227f91b2c35e337c2d5045c
 SHA512:
-  metadata.gz: 8f8c62cfbe83e98ccb220ab02432ed7f2e02837350b4a7eb30ac24e51de6486c128e9f17846ac89dcd67fb835b8f292ded46d61830a2aa037f3f61e1734ee288
-  data.tar.gz: e73434d2a098fa7b57d5c6dea6e75daf43ca6585492984f3c225449c7819ffde7e796e6ac113d3ee4d5caf30bb627085fc2cfec3400ec4c7ca43a9e980b5cb8c
+  metadata.gz: 6286777c6f31b2eef443c76227dd4404bf8cc0509ab385210d62a8c2860d6449586ae54a6c56c7c5c419c46aed92b4409a63db9e5be1e3c0f77290520ff3d423
+  data.tar.gz: bc66b1510ca21645de0440ecbbe8ae543139e96e6d393a37d0a47613e31610cb6fc737bf8e104d31b8c61f8bcb4a14f9e7609a9106fdf26384e37bc4a347edb9

data/.pre-commit-config.yaml

@@ -0,0 +1,29 @@
+# Standard LegionIO pre-commit configuration
+# Install: pre-commit install
+# Manual:  pre-commit run --all-files
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-json
+        exclude: Gemfile\.lock
+      - id: check-merge-conflict
+
+  - repo: local
+    hooks:
+      - id: rubocop
+        name: RuboCop (autofix)
+        entry: scripts/pre-commit-rubocop.sh
+        language: script
+        types: [ruby]
+        pass_filenames: true
+
+      - id: ruby-syntax
+        name: Ruby syntax check
+        entry: bash -c 'status=0; for file in "$@"; do ruby -c "$file" || status=$?; done; exit $status' --
+        language: system
+        types: [ruby]
+        pass_filenames: true

data/AGENTS.md

@@ -1,24 +1,77 @@
 Always run a full `bundle exec rspec` and `bundle exec rubocop -A` and fix all errors before committing.
 
-# legion-data
+# AGENTS.md - legion-data
 
-`legion-data` is the persistent database storage gem for the LegionIO async job engine framework. It provides database connectivity via the Sequel ORM, automatic schema migrations (70+ numbered migrations), and Sequel models for the full LegionIO control plane: extensions, functions, runners, nodes, tasks, settings, digital workers, task relationships, Apollo shared knowledge tables (PostgreSQL only), RBAC, tenants, audit log, governance events, and archive tables.
+## Repo Role
 
-It also ships a parallel local SQLite database (`Legion::Data::Local`) for on-node agentic cognitive state persistence (memory traces, trust scores, etc.), independent of the shared database.
+`legion-data` owns persistent storage for LegionIO. Keep this repo focused on database connectivity, Sequel migrations, Sequel models, local SQLite state, extraction persistence, audit/governance storage, identity/RBAC storage, Apollo storage, and the LLM lifecycle ledger.
 
-## Key entry points
+HTTP APIs, runtime orchestration, extension behavior, and UI concerns belong in their owning repos. This repo should expose clean model contracts that those layers can call.
 
-- `Legion::Data.setup` — connect, migrate, load models, set up local DB
-- `Legion::Data::Model::*` — Sequel model classes
-- `Legion::Data::Local` — local SQLite for agentic state
-- `Legion::Data::Extract` — text extraction from documents (pdf, docx, csv, etc.)
-- `Legion::Data::Spool` — filesystem write buffer for DB-unavailable scenarios
+## Required Commands
 
-## Testing
+Run from the repo root:
 
 ```bash
-cd /path/to/legion-data
-bundle install
-bundle exec rspec
 bundle exec rubocop -A
+bundle exec rspec --format json --out tmp/rspec_results.json --format progress --out tmp/rspec_progress.txt
 ```
+
+If RSpec fails, extract failures with:
+
+```bash
+jq '[.examples[] | select(.status != "passed") | {file_path, line_number, full_description, status, exception: .exception}]' tmp/rspec_results.json > tmp/rspec_failures.json
+```
+
+Do not run partial RSpec or partial RuboCop for release validation.
+
+## Migration Rules
+
+- Never edit published migrations. Add a new migration instead.
+- Do not guard migrations with `create_table?`, `drop_table?`, `table_exists?`, `if_exists`, `if_not_exists`, `next if`, or `next unless`.
+- Keep migrations split by domain and dependency. Do not hide a whole schema rewrite in one large migration.
+- Use portable Sequel DSL by default. Adapter-specific code is acceptable only for adapter-specific features, such as PostgreSQL vector columns.
+- Prefer `id` integer primary keys for joins and `uuid` public identifiers for APIs, logs, and external references.
+- Avoid JSON columns unless the data is genuinely dynamic provider evidence or cannot be normalized without losing meaning.
+
+## Sequel Association Rules
+
+Use the official Sequel association APIs as the model contract:
+
+- Association API reference: https://sequel.jeremyevans.net/rdoc/classes/Sequel/Model/Associations/ClassMethods.html
+- Association basics: https://github.com/jeremyevans/sequel/blob/master/doc/association_basics.rdoc
+
+Required mapping:
+
+| Schema shape | Sequel association |
+|--------------|--------------------|
+| This table has the foreign key | `many_to_one` |
+| Other table has the foreign key | `one_to_many` or `one_to_one` |
+| Join table connects both sides | `many_to_many` |
+| One associated row through a join table | `one_through_one` |
+
+Rules:
+
+- Define associations for real foreign-key relationships when adding or changing models.
+- Prefer association methods and association datasets over ad hoc `where(foreign_key: ...)` lookups in model helpers.
+- When names are not inferable, explicitly set `:class`, `:key`, `:primary_key`, `:join_table`, `:left_key`, and `:right_key`.
+- Do not create association names that collide with actual column names; Sequel creates methods using the association name.
+- Keep namespace models aligned with API/domain shape, for example `Legion::Data::Model::Identity::*`, `LLM::*`, `Apollo::*`, and `RBAC::*`.
+
+## Current Schema Notes
+
+- Migrations currently run through `096`.
+- `074`-`076` are mainline Apollo/task/extract migrations.
+- `077`-`090` define the LLM lifecycle ledger.
+- `091`-`096` define portable identity companion tables.
+- Published PostgreSQL identity migrations remain in place; portable identity tables are additive.
+
+## Release Hygiene
+
+For behavior, model, migration, or Ruby code changes:
+
+- Update `lib/legion/data/version.rb`.
+- Update `CHANGELOG.md`.
+- Update `README.md` when public behavior, schema, configuration, or model surface changes.
+- Keep `.gitignore` ignoring `/Gemfile.lock` and `*.gem`.
+- Do not include generated DBs, logs, coverage output, built gems, or repo-external `/docs` workspace files in commits.

data/CHANGELOG.md

@@ -2,6 +2,35 @@
 
 ## [Unreleased]
 
+## [1.8.0] - 2026-05-06
+
+### Added
+- Migration 097 adds official LLM dispatch fields for fleet operation, correlation, idempotency, provider instance, and dispatch path tracking.
+
+### Changed
+- LLM lifecycle Sequel models now live under `Legion::Data::Models::LLM` to match the official data model namespace.
+
+## [1.7.5] - 2026-05-05
+
+### Added
+- Migrations 077-090: portable LLM lifecycle schema covering conversations, messages, message inference requests/responses, route attempts, inference metrics, provider-requested tool calls, tool call attempts, conversation compactions, policy evaluations, security events, and registry events.
+- Migrations 091-096: portable identity companion schema with integer primary keys, public UUIDs, normalized provider capabilities, principals, identities, groups, memberships, and audit events.
+- Sequel models and associations for the new `Legion::Data::Models::LLM` lifecycle tables.
+- Nested Sequel model namespaces for Identity, Apollo, and RBAC tables.
+- Lookup helpers for nested and legacy identity models.
+- LLM reconstruction query helpers for audit lineage, finance rollups, security incident lineage, and message-to-tool incident flow.
+- Additional Sequel associations for core execution and identity models, including function/task, relationship/chain, task/worker, task log aliases, and principal/group many-to-many membership helpers.
+
+## [1.7.4] - 2026-04-28
+
+### Fixed
+- Pre-commit RuboCop hook now distinguishes missing tools from real RuboCop failures and propagates failures instead of silently passing.
+- Ruby syntax pre-commit hook now checks every staged Ruby file instead of only the first argument.
+- Connection setup now refreshes the configured adapter before each setup call and clears fallback state on shutdown so fallback health checks do not stay stale across reconnects.
+
+### Changed
+- README refreshed for the current migration count, version line, fallback diagnostics, pre-commit workflow, and recent model surface.
+
 ## [1.7.3] - 2026-04-27
 
 ### Added

data/CLAUDE.md

@@ -1,330 +1,67 @@
 Always run a full `bundle exec rspec` and `bundle exec rubocop -A` and fix all errors before committing.
 
-# legion-data: Persistent Storage for LegionIO
+# legion-data
 
-**Repository Level 3 Documentation**
-- **Parent**: `/Users/miverso2/rubymine/legion/CLAUDE.md`
+`legion-data` is the persistent storage gem for LegionIO. It owns Sequel database connections, numbered migrations, Sequel models, local SQLite state, extract timing persistence, audit/governance storage, identity/RBAC storage, Apollo storage, and the LLM lifecycle ledger.
 
-## Purpose
-
-Manages persistent database storage for the LegionIO framework. Supports SQLite (default), MySQL, and PostgreSQL via Sequel ORM. Provides automatic schema migrations and data models for extensions, functions, runners, nodes, tasks, settings, digital workers, task relationships, Apollo shared knowledge tables (PostgreSQL only), tenants, webhooks, audit log, and archive tables. Also provides a parallel local SQLite database (`Legion::Data::Local`) for agentic cognitive state persistence.
-
-**GitHub**: https://github.com/LegionIO/legion-data
-**Version**: 1.6.21
-**License**: Apache-2.0
-
-## Supported Databases
-
-| Database | Adapter | Gem | Use Case |
-|----------|---------|-----|----------|
-| SQLite | `sqlite` | `sqlite3` (bundled) | Default, dev/test, single-node |
-| MySQL | `mysql2` | `mysql2` (optional) | Production |
-| PostgreSQL | `postgres` | `pg` (optional) | Production |
-
-Adapter is set via `Legion::Settings[:data][:adapter]`. All migrations use Sequel DSL for cross-database compatibility.
-
-## Architecture
+## Commands
 
+```bash
+bundle install
+bundle exec rubocop -A
+bundle exec rspec --format json --out tmp/rspec_results.json --format progress --out tmp/rspec_progress.txt
 ```
-Legion::Data (singleton module)
-├── .setup             # Connect, migrate, load models, setup cache, setup local
-├── .connection        # Sequel database handle (shared/central)
-├── .local             # Legion::Data::Local accessor
-├── .stats             # Combined { shared: Connection.stats, local: Local.stats }
-├── .reload_static_cache  # Refresh in-memory StaticCache after hot-loading extensions
-├── .shutdown          # Close both connections
-│
-├── Connection         # Sequel database connection management (shared)
-│   ├── .adapter       # Reads from settings (sqlite, mysql2, postgres)
-│   ├── .setup         # Establish connection (dev_mode fallback to SQLite if network DB unreachable)
-│   ├── .sequel        # Raw Sequel::Database accessor
-│   ├── .stats         # Pool metrics, tuning snapshot, adapter-specific DB stats
-│   ├── .pool_stats    # Connection pool usage (size, available, in_use, waiting)
-│   ├── .shutdown      # Close connection
-│   ├── GENERIC_KEYS   # Pool options forwarded to Sequel (:max_connections, :pool_timeout, etc.)
-│   ├── ADAPTER_KEYS   # Per-adapter option whitelists (sqlite, postgres, mysql2)
-│   ├── ADAPTER_DEFAULTS # Built-in defaults per adapter when user hasn't set a value
-│   ├── SlowQueryLogger    # Wraps Legion::Logging with [slow-query] prefix for Sequel warn
-│   └── QueryFileLogger    # Thread-safe file logger for query_log mode (~/.legionio/logs/)
-│
-├── Local              # Local SQLite database for agentic cognitive state
-│   ├── .setup         # Lazy init — creates legionio_local.db on first access
-│   ├── .connection    # Sequel::SQLite::Database handle
-│   ├── .connected?    # Whether local DB is active
-│   ├── .db_path       # Path to the local SQLite file
-│   ├── .model(:table) # Create Sequel::Model bound to local connection
-│   ├── .register_migrations(name:, path:) # Extensions register their migration dirs
-│   ├── .stats         # Local SQLite metrics (PRAGMAs, file size, registered migrations)
-│   ├── .shutdown      # Close local connection
-│   └── .reset!        # Clear all state (testing)
-│
-├── Migration          # Auto-migration system (58 migrations, Sequel DSL)
-│   └── migrations/
-│       ├── 001_add_schema_columns
-│       ├── 002_add_nodes
-│       ├── 003_add_settings
-│       ├── 004_add_extensions
-│       ├── 005_add_runners
-│       ├── 006_add_functions
-│       ├── 007_add_default_extensions
-│       ├── 008_add_tasks
-│       ├── 009_add_digital_workers
-│       ├── 010_add_value_metrics
-│       ├── 011_add_extensions_registry
-│       ├── 012_add_apollo_tables      # postgres-only: pgvector, uuid-ossp, 4 apollo tables
-│       ├── 013_add_relationships      # relationships table with trigger/action FK to functions
-│       ├── 014_add_relationship_columns  # delay, chain_id, debug, conditions, transformation, active, allow_new_chains
-│       ├── 015_add_rbac_tables
-│       ├── 016_add_worker_health
-│       ├── 017_add_audit_log
-│       ├── 018_add_governance_events    # append-only event store with hash chain
-│       ├── 019_add_audit_hash_chain
-│       ├── 020_add_webhooks
-│       ├── 021_add_archive_tables
-│       ├── 022_add_memory_traces
-│       ├── 023_add_data_archive
-│       ├── 024_add_tenant_partition_columns
-│       ├── 025_add_tenants_table
-│       ├── 026_add_function_embeddings  # description + embedding (TEXT) on functions; postgres: embedding_vector vector(1536) with HNSW cosine index
-│       ├── 027_add_apollo_source_provider
-│       ├── 028_add_agent_cluster
-│       ├── 029_add_agent_cluster_tasks
-│       ├── 030_add_approval_queue
-│       ├── 031_add_task_depth
-│       ├── 032_add_task_cancelled_at
-│       ├── 033_add_task_delay
-│       ├── 034_add_archive_manifest
-│       ├── 035_add_apollo_source_channel
-│       ├── 036_add_audit_context_snapshot
-│       ├── 037_add_apollo_knowledge_domain
-│       ├── 038_add_conversations
-│       ├── 039_add_audit_archive_manifest  # 7-year tiered audit retention
-│       ├── 040_add_slow_query_indexes       # tasks table performance indexes
-│       ├── 041_resize_vector_columns
-│       ├── 042_add_tenant_to_registry_tables
-│       ├── 043_add_rls_placeholder          # PostgreSQL row-level security
-│       ├── 044_expand_memory_traces
-│       ├── 045_add_memory_associations
-│       ├── 046_add_metering_hourly_rollup
-│       ├── 047_apollo_knowledge_capture     # identity cols, ops table, archive table, 25+ indexes
-│       ├── 048_add_financial_logging        # 7 UAIS cost recovery tables (identity, asset, environment, accounting, execution, tags, usage)
-│       ├── 049_add_remote_invocable_to_functions  # remote_invocable boolean on functions (v3.0)
-│       ├── 050_add_missing_indexes         # critical indexes across 13 tables
-│       ├── 051_fix_tasks_created_at        # created_at alias for archival (PG generated, SQLite backfill)
-│       ├── 052_drop_redundant_apollo_indexes   # PG only: remove duplicate auto-named indexes
-│       ├── 053_add_tasks_relationship_fk   # PG only: FK constraint on tasks.relationship_id
-│       ├── 054_add_component_type_to_functions  # component_type on functions (runner/hook/absorber, v3.0)
-│       ├── 055_add_definition_to_functions      # definition text column on functions (v3.0)
-│       ├── 056_add_absorber_patterns       # absorber_patterns table for pattern-matched acquisition
-│       ├── 057_add_routing_key_to_runners  # routing_key on runners (v3.0 AMQP)
-│       └── 058_add_tool_embedding_cache    # tool_embedding_cache table for global embedding cache tier (Tools::EmbeddingCache L4)
-│
-├── Model              # Sequel model loader
-│   └── Models/
-│       ├── Extension      # Installed LEX extensions
-│       ├── Function       # Available functions per extension (with trigger/action relationship associations)
-│       ├── Runner         # Runner definitions (extension + function bindings)
-│       ├── Node           # Cluster node registry
-│       ├── Task           # Task instances (belongs_to Relationship, belongs_to DigitalWorker)
-│       ├── TaskLog        # Task execution logs
-│       ├── Setting        # Persistent settings store
-│       ├── DigitalWorker  # Digital worker registry (lifecycle: bootstrap/active/paused/retired/terminated)
-│       ├── Relationship   # Task trigger/action relationships between functions (migration 013/014)
-│       ├── ApolloEntry    # Apollo knowledge entries — postgres only (pgvector embedding, confidence lifecycle)
-│       ├── ApolloRelation # Weighted relations between Apollo entries — postgres only
-│       ├── ApolloExpertise  # Per-agent domain expertise tracking — postgres only
-│       ├── ApolloAccessLog  # Apollo entry access audit log — postgres only
-│       ├── AuditLog       # Audit trail entries (AMQP + query layer)
-│       ├── RbacRoleAssignment  # RBAC principal -> role mappings
-│       ├── RbacRunnerGrant     # RBAC per-runner permission grants
-│       └── RbacCrossTeamGrant  # RBAC cross-team access grants
-│   Note: value_metrics table (migration 010) is accessed via raw Sequel dataset,
-│         not via a named Sequel::Model subclass.
-│   Note: Apollo models are guarded with `return unless adapter == :postgres` at load time.
-│
-├── Settings           # Default DB config with per-adapter credential presets
-└── Version
-```
-
-### Key Design Patterns
 
-- **Two-Database Architecture**: Shared (MySQL/PG/SQLite) for control plane data + Local (always SQLite) for agentic cognitive state. Two files, always separate, no cross-database joins.
-- **Adapter-Driven**: `Connection.adapter` reads from settings; all adapters (including SQLite) use `Sequel.connect` so all options flow through uniformly
-- **Flat Settings**: all connection/pool/adapter options live directly on `data.*` — legion-data resolves which options apply to the current adapter via `ADAPTER_KEYS` whitelists
-- **Per-Adapter Defaults**: `ADAPTER_DEFAULTS` provides built-in defaults (e.g., sqlite timeout 5000, postgres connect_timeout 20) when user hasn't set a value; nil in settings means "use adapter default"
-- **Dev Mode Fallback**: When `dev_mode: true` and network DB unreachable, shared connection falls back to SQLite (`legionio.db`) with warning log
-- **Connection Health**: `connection_validator` (pings idle connections) and `connection_expiration` (retires old connections) extensions auto-enabled for non-SQLite adapters
-- **Cross-DB Migrations**: Shared migrations use IntegerMigrator (Sequel DSL), local migrations use TimestampMigrator (per-extension registration)
-- **Auto-Migration**: Runs Sequel migrations on startup (`auto_migrate: true` by default)
-- **Sequel ORM**: Shared models are `Sequel::Model` subclasses (inherit global connection). Local models use `Legion::Data::Local.model(:table)` (explicit connection binding).
-- **Two-Tier Caching**: StaticCache (in-process frozen hash, no external deps) for lookup models (Extension, Runner, Function) + external Caching plugin (via `Legion::Cache` — Redis/Memcached/Memory) for dynamic models (Relationship, Node, Setting). Both disabled by default.
-- **Query Log Isolation**: `query_log` flag pipes all SQL to dedicated files (`~/.legionio/logs/data-shared-query.log`, `data-local-query.log`) via `QueryFileLogger` — completely isolated from the `Legion::Logging` domain
-- **Cryptographic Erasure**: Deleting `legionio_local.db` is a hard guarantee — no residual data. Used by `lex-privatecore`.
-- **CLI Executable**: Ships with `legionio_migrate` executable in `exe/` for running database migrations standalone
+RSpec output belongs in `tmp/`. On failure, extract only failures:
 
-## Default Settings
-
-```json
-{
-  "adapter": "sqlite",
-  "connected": false,
-  "dev_mode": false,
-  "dev_fallback": true,
-  "connect_on_start": true,
-
-  "max_connections": 25,
-  "pool_timeout": 5,
-  "preconnect": "concurrently",
-  "single_threaded": false,
-  "test": true,
-  "name": null,
-
-  "log": false,
-  "query_log": false,
-  "log_connection_info": false,
-  "log_warn_duration": 1,
-  "sql_log_level": "debug",
-
-  "connection_validation": true,
-  "connection_validation_timeout": 600,
-  "connection_expiration": true,
-  "connection_expiration_timeout": 14400,
-
-  "connect_timeout": null,
-  "read_timeout": null,
-  "write_timeout": null,
-  "encoding": null,
-  "sql_mode": null,
-  "sslmode": null,
-  "sslrootcert": null,
-  "search_path": null,
-  "timeout": null,
-  "readonly": null,
-  "disable_dqs": null,
-
-  "read_replica_url": null,
-  "replicas": [],
-
-  "creds": {
-    "database": "legionio.db"
-  },
-  "migrations": {
-    "continue_on_fail": false,
-    "auto_migrate": true,
-    "ran": false,
-    "version": null
-  },
-  "models": {
-    "continue_on_load_fail": false,
-    "autoload": true
-  },
-  "local": {
-    "enabled": true,
-    "database": "legionio_local.db",
-    "query_log": false,
-    "migrations": {
-      "auto_migrate": true
-    }
-  },
-  "cache": {
-    "connected": false,
-    "auto_enable": false,
-    "static_cache": false,
-    "ttl": 60
-  },
-  "archival": {
-    "retention_days": 90,
-    "batch_size": 1000,
-    "storage_backend": null
-  }
-}
+```bash
+jq '[.examples[] | select(.status != "passed") | {file_path, line_number, full_description, status, exception: .exception}]' tmp/rspec_results.json > tmp/rspec_failures.json
 ```
 
-Settings are **flat** — all pool, logging, health, and adapter-specific options live directly on `data.*`. Adapter-specific options (e.g., `connect_timeout`, `encoding`, `sslmode`) default to `null` and resolve to per-adapter built-in defaults at connection time:
-
-| Adapter | Applied Options | Defaults |
-|---------|----------------|----------|
-| sqlite | `timeout`, `readonly`, `disable_dqs` | `timeout: 5000`, `readonly: false`, `disable_dqs: true` |
-| postgres | `connect_timeout`, `sslmode`, `sslrootcert`, `search_path` | `connect_timeout: 20`, `sslmode: "disable"` |
-| mysql2 | `connect_timeout`, `read_timeout`, `write_timeout`, `encoding`, `sql_mode` | `connect_timeout: 120`, `encoding: "utf8mb4"` |
+## Architecture
 
-### Caching
+- `lib/legion/data/connection.rb`: shared Sequel connection setup, diagnostics, fallback handling, query logging.
+- `lib/legion/data/migration.rb`: numbered Sequel migrations.
+- `lib/legion/data/model.rb`: shared model loader.
+- `lib/legion/data/models/`: flat and namespaced Sequel model classes.
+- `lib/legion/data/local.rb`: local SQLite database for on-node state.
+- `lib/legion/data/extract.rb`: text extraction and persisted extract step timings.
+- `lib/legion/data/spool.rb`: filesystem write buffer when DB writes are unavailable.
 
-Two independent caching tiers, both disabled by default:
+## Migration Rules
 
-| Tier | Setting | Models | Backend | Use Case |
-|------|---------|--------|---------|----------|
-| **StaticCache** | `data.cache.static_cache: true` | Extension, Runner, Function | In-process frozen Ruby hash | Zero-DB-hit reads for lookup tables. No external deps. Call `Legion::Data.reload_static_cache` after hot-loading extensions. |
-| **External Cache** | `data.cache.auto_enable: true` + `Legion::Cache` loaded | Relationship (10s), Node (10s), Setting (ttl) | `Legion::Cache` (Redis/Memcached/Memory) | Cross-process cache sharing for dynamic models. Requires `legion-cache` gem connected. |
+- Never edit published migrations. Add a new migration.
+- Do not guard migrations with `create_table?`, `drop_table?`, `table_exists?`, `if_exists`, `if_not_exists`, `next if`, or `next unless`.
+- Keep migrations small enough to diagnose and roll back. Split by domain and dependency.
+- Use portable Sequel DSL unless the feature truly requires adapter-specific behavior.
+- Use integer `id` primary keys for joins and public `uuid` columns for APIs/logs/external references.
+- Normalize stable fields. Use JSON only for genuinely dynamic provider payloads or evidence.
 
-For thousands of agents, enable `static_cache` first — biggest impact, zero dependencies. External cache only adds value when you need cross-process sharing via Redis/Memcached.
+## Sequel ORM Rules
 
-Per-adapter credential defaults are defined in `Settings::CREDS`:
-- **sqlite**: `{ database: "legionio.db" }`
-- **mysql2**: `{ username: "legion", password: "legion", database: "legionio", host: "127.0.0.1", port: 3306 }`
-- **postgres**: `{ user: "legion", password: "legion", database: "legionio", host: "127.0.0.1", port: 5432 }`
+Use Sequel associations as the object graph. References:
 
-## Dependencies
+- https://sequel.jeremyevans.net/rdoc/classes/Sequel/Model/Associations/ClassMethods.html
+- https://github.com/jeremyevans/sequel/blob/master/doc/association_basics.rdoc
 
-| Gem | Purpose |
-|-----|---------|
-| `sequel` (>= 5.70) | ORM and migration framework |
-| `sqlite3` (>= 2.0) | SQLite adapter (default, bundled) |
-| `mysql2` (>= 0.5.5) | MySQL adapter (optional) |
-| `pg` (>= 1.5) | PostgreSQL adapter (optional) |
-| `legion-logging` | Logging |
-| `legion-settings` | Configuration |
+Association mapping:
 
-## File Map
+- Foreign key on this model: `many_to_one`.
+- Foreign key on the associated model: `one_to_many` or `one_to_one`.
+- Join table between models: `many_to_many`.
+- Single associated record through a join table: `one_through_one`.
 
-| Path | Purpose |
-|------|---------|
-| `lib/legion/data.rb` | Module entry, setup/shutdown lifecycle |
-| `lib/legion/data/connection.rb` | Sequel database connection (adapter selection) |
-| `lib/legion/data/migration.rb` | Migration runner |
-| `lib/legion/data/migrations/` | 58 numbered migration files (Sequel DSL) |
-| `lib/legion/data/model.rb` | Model autoloader |
-| `lib/legion/data/local.rb` | Local SQLite module for agentic cognitive state |
-| `lib/legion/data/models/` | Sequel models (Extension, Function, Runner, Node, Task, TaskLog, Setting, DigitalWorker, Relationship, ApolloEntry, ApolloRelation, ApolloExpertise, ApolloAccessLog, AuditLog, RbacRoleAssignment, RbacRunnerGrant, RbacCrossTeamGrant) |
-| `lib/legion/data/encryption/cipher.rb` | AES-256-GCM encrypt/decrypt with versioned binary format and AAD |
-| `lib/legion/data/encryption/key_provider.rb` | Vault-backed key derivation with per-tenant scope and local fallback |
-| `lib/legion/data/encryption/sequel_plugin.rb` | Transparent `encrypted_column` DSL for Sequel models |
-| `lib/legion/data/event_store.rb` | Append-only governance event store with hash chain integrity |
-| `lib/legion/data/event_store/projection.rb` | Projection base class, ConsentState, GovernanceTimeline |
-| `lib/legion/data/vector.rb` | Reusable pgvector helpers: `available?`, `cosine_search`, `l2_search`, `ensure_extension!` |
-| `lib/legion/data/storage_tiers.rb` | Hot/warm/cold archival lifecycle: `archive_to_warm`, `export_to_cold`, `stats` |
-| `lib/legion/data/archival.rb` | Archival module entry point and configuration |
-| `lib/legion/data/archival/` | Archival strategy implementations |
-| `lib/legion/data/extract.rb` | 10-handler text extraction registry (txt/md/csv/json/jsonl/html/xlsx/docx/pdf/pptx) |
-| `lib/legion/data/extract/handlers/` | Per-format extraction handlers (base, csv, docx, html, json, jsonl, markdown, pdf, pptx, text, xlsx) |
-| `lib/legion/data/extract/type_detector.rb` | MIME type detection for extract registry |
-| `lib/legion/data/rls.rb` | PostgreSQL row-level security helpers (tenant isolation, session variable) |
-| `lib/legion/data/partition_manager.rb` | Tenant partition management |
-| `lib/legion/data/retention.rb` | Audit retention and archival lifecycle |
-| `lib/legion/data/settings.rb` | Default configuration with per-adapter credential presets |
-| `lib/legion/data/version.rb` | VERSION constant |
-| `exe/legionio_migrate` | CLI executable for running database migrations standalone |
+When Sequel cannot infer names, set `:class`, `:key`, `:primary_key`, `:join_table`, `:left_key`, and `:right_key` explicitly. Do not create association names that collide with real columns.
 
-## Role in LegionIO
+## Current Schema Landmarks
 
-Optional persistent storage initialized during `Legion::Service` startup (after transport). Provides:
-1. Extension and function registry (which LEXs are installed, what functions they expose)
-2. Task scheduling and logging
-3. Node cluster membership tracking
-4. Persistent settings storage
-5. Digital worker registry (AI-as-labor platform)
-6. Task relationship graph (trigger/action chains)
-7. Apollo shared knowledge store (PostgreSQL + pgvector only, used by lex-apollo)
-8. Local SQLite for agentic cognitive state (memory traces, trust scores, dream journals) — always on-node, independent of shared DB
-9. RBAC assignment tables (migrations 015 — role assignments, runner grants, cross-team grants)
-10. Audit log with tamper-evident hash chain (migrations 017, 019)
-11. Governance event store with append-only integrity (migration 018)
-12. Webhook subscription storage (migration 020)
-13. Archive, memory traces, and tenant partition tables (migrations 021–025)
-14. Function embeddings for semantic runner discovery (migration 026 — description + vector columns on functions table)
-15. Financial logging for UAIS cost recovery (migration 048 — 7 tables: identity, asset, environment, accounting, execution, tags, usage rollup)
-16. Global tool embedding cache (migration 058 — `tool_embedding_cache` table, L4 tier for `Legion::Tools::EmbeddingCache`)
+- `074`-`076`: Apollo field width, task idempotency, extract step timings.
+- `077`-`090`: LLM lifecycle ledger.
+- `091`-`096`: portable identity companion tables.
+- Namespaced models exist for `Identity::*`, `Apollo::*`, `RBAC::*`, and `LLM::*`.
 
----
+## Boundaries
 
-**Maintained By**: Matthew Iverson (@Esity)
+- REST APIs belong in LegionIO, not this gem.
+- Extension runtime behavior belongs in the owning extension repos.
+- Do not commit generated DBs, logs, coverage output, built gems, or workspace `/docs` files from outside this repo.