legion-data 1.7.3 → 1.8.0

data/README.md

@@ -1,11 +1,28 @@
 # legion-data
 
-Persistent database storage for the [LegionIO](https://github.com/LegionIO/LegionIO) async job engine and AI coding assistant platform. Provides database connectivity via the [Sequel ORM](https://sequel.jeremyevans.net/), automatic schema migrations (71 numbered migrations), Sequel models for the full LegionIO control plane, and a parallel local SQLite database for on-node agentic cognitive state.
+Persistent database storage for the [LegionIO](https://github.com/LegionIO/LegionIO) async job engine and AI coding assistant platform. Provides database connectivity via the [Sequel ORM](https://sequel.jeremyevans.net/), automatic schema migrations (97 numbered migrations), Sequel models for the full LegionIO control plane, and a parallel local SQLite database for on-node agentic cognitive state.
 
-**Version**: 1.6.25 | **Ruby**: >= 3.4 | **License**: Apache-2.0
+**Version**: 1.8.0 | **Ruby**: >= 3.4 | **License**: Apache-2.0
 
 ---
 
+## What It Owns
+
+`legion-data` is the data contract for LegionIO. It owns database connectivity, migrations, model loading, and portable Sequel model definitions for shared platform state. HTTP routes, runtime orchestration, and extension behavior live in other LegionIO repos and call into these models.
+
+Core responsibilities:
+
+| Area | Tables and models |
+|------|-------------------|
+| Control plane | extensions, functions, runners, nodes, tasks, settings, workers, relationships, chains |
+| Audit and governance | `audit_log`, `audit_records`, `governance_events`, archive manifests |
+| Identity and RBAC | providers, principals, identities, groups, memberships, role grants, runner grants |
+| LLM ledger | conversations, model-visible messages, inference requests/responses, routing, metrics, tool calls, policy/security events |
+| Apollo knowledge | PostgreSQL `pgvector` knowledge entries, relations, expertise, access logs |
+| Local state | on-node SQLite cognitive state, independent of the shared database |
+
+The schema is portable by default across SQLite, MySQL, and PostgreSQL. PostgreSQL-only behavior is isolated to features that need PostgreSQL, such as Apollo vector columns.
+
 ## Supported Databases
 
 | Database | Adapter | Gem | Default |
@@ -51,18 +68,25 @@ Legion::Data (singleton module)
 │   ├── .adapter        # Reads adapter from settings (:sqlite, :mysql2, :postgres)
 │   ├── .setup          # Establish connection (dev_mode fallback to SQLite if unreachable)
 │   ├── .sequel         # Raw Sequel::Database accessor
+│   ├── .connection_info  # Adapter, liveness, and fallback diagnostics
+│   ├── .fallback_active? # True when dev fallback moved a network DB to SQLite
 │   ├── .stats          # Pool metrics, tuning snapshot, adapter-specific DB stats
 │   └── .shutdown       # Disconnect and close query file logger
 │
-├── Migration           # Auto-migration system (71 numbered Sequel DSL migrations)
+├── Migration           # Auto-migration system (97 numbered Sequel DSL migrations)
 │
 ├── Model               # Sequel model autoloader
 │   └── Models: Extension, Function, Runner, Node, Task, TaskLog, Setting,
 │               DigitalWorker, Relationship, AuditLog, AuditRecord, Chain,
 │               RbacRoleAssignment, RbacRunnerGrant, RbacCrossTeamGrant,
 │               IdentityProvider, Principal, Identity, IdentityGroup,
-│               IdentityGroupMembership,
-│               ApolloEntry, ApolloRelation, ApolloExpertise, ApolloAccessLog (PG only)
+│               IdentityGroupMembership, IdentityAuditLog, ExtractStepTiming,
+│               ApolloEntry, ApolloRelation, ApolloExpertise, ApolloAccessLog (PG only),
+│               LLM::Conversation, LLM::Message, LLM::MessageInferenceRequest,
+│               LLM::MessageInferenceResponse, LLM::RouteAttempt,
+│               LLM::MessageInferenceMetric, LLM::ToolCall, LLM::ToolCallAttempt,
+│               LLM::ConversationCompaction, LLM::PolicyEvaluation,
+│               LLM::SecurityEvent, LLM::RegistryEvent
 │
 ├── Local               # Parallel local SQLite for agentic cognitive state
 │   ├── .setup          # Lazy init — creates legionio_local.db on first access
@@ -117,10 +141,42 @@ Legion::Data.local.db_path           # => "legionio_local.db"
 Legion::Data.connected?              # => true
 Legion::Data.stats                   # => { shared: {...}, local: {...} }
 
+# Inspect shared DB diagnostics, including dev fallback state
+Legion::Data::Connection.connection_info
+# => { adapter: :sqlite, connected: true, fallback_active: false, ... }
+
 # Shut down both connections
 Legion::Data.shutdown
 ```
 
+### Model Associations
+
+Models use Sequel associations as the public object graph. Prefer association methods and association datasets over hand-written foreign-key lookups when the relationship is part of the schema contract.
+
+```ruby
+task = Legion::Data::Model::Task.first(id: 42)
+task.function             # many_to_one :function
+task.relationship         # many_to_one :relationship
+task.task_logs_dataset    # further filter/order without losing the relationship
+
+conversation = Legion::Data::Models::LLM::Conversation.first(uuid: conversation_uuid)
+conversation.messages_dataset.order(:seq).all
+conversation.security_incident_lineage
+```
+
+Official LLM lifecycle data lives under `Legion::Data::Models::LLM`. `legion-llm` and `lex-llm-ledger` should use these models and the `llm_*` migration tables for conversations, model-visible messages, inference requests, responses, route attempts, metrics, tool calls, policy decisions, security events, and registry events. Legacy ledger-only tables are not the canonical schema.
+
+Association rules used in this repo follow Sequel's own association model:
+
+| Relationship | Use this Sequel association |
+|--------------|-----------------------------|
+| Current table has the foreign key | `many_to_one` |
+| Associated table has the foreign key | `one_to_many` or `one_to_one` |
+| Join table connects both sides | `many_to_many` |
+| One associated record through a join table | `one_through_one` |
+
+When Sequel cannot infer names from the schema, models must be explicit with `:class`, `:key`, `:primary_key`, `:join_table`, `:left_key`, and `:right_key`. Association names must not collide with real column names because Sequel creates methods with the association name.
+
 ### Local Database (Agentic Cognitive State)
 
 Extensions register their own migration directories and create models bound to the local connection:
@@ -314,6 +370,8 @@ When `dev_mode: true` and a network database is unreachable, the shared connecti
 { "data": { "dev_mode": true, "dev_fallback": true } }
 ```
 
+Fallback is intentionally loud. `Connection.setup` logs the degraded mode at error level, `Connection.fallback_active?` returns `true`, and `Connection.connection_info` reports the configured adapter, actual adapter, connection state, and Sequel liveness. Data written during fallback is local-only SQLite data and will not appear in the configured network database after reconnect.
+
 ### HashiCorp Vault Integration
 
 When Vault is connected, credentials are fetched dynamically from `database/creds/legion`, overriding any static `creds` block.
@@ -377,6 +435,37 @@ Legion::Data.reload_static_cache
 
 Apollo models require PostgreSQL with the `pgvector` extension. They are skipped silently on SQLite and MySQL.
 
+The `Legion::Data::Model::Identity::*`, `Apollo::*`, and `RBAC::*` namespaces provide cleaner Sequel model names for API-facing code while preserving the legacy flat model classes. Official LLM lifecycle models live under `Legion::Data::Models::LLM`.
+
+### Identity Namespace Models
+
+| Model | Table | Description |
+|-------|-------|-------------|
+| `Identity::Provider` | `portable_identity_providers` | Portable provider records with integer primary keys and public UUIDs |
+| `Identity::ProviderCapability` | `portable_identity_provider_capabilities` | Normalized provider capability declarations |
+| `Identity::Principal` | `portable_identity_principals` | Human, service, worker, or system principals |
+| `Identity::Identity` | `portable_identities` | Provider-bound identities for principals |
+| `Identity::Group` | `portable_identity_groups` | Identity groups |
+| `Identity::GroupMembership` | `portable_identity_group_memberships` | Principal and identity group membership rows |
+| `Identity::AuditLog` | `portable_identity_audit_log` | Identity lifecycle and lookup audit events |
+
+### LLM Lifecycle Models
+
+| Model | Table | Description |
+|-------|-------|-------------|
+| `LLM::Conversation` | `llm_conversations` | Conversation container tied to the base user identity |
+| `LLM::Message` | `llm_messages` | Model-visible conversation transcript messages |
+| `LLM::MessageInferenceRequest` | `llm_message_inference_requests` | Provider request assembled from message, context, tools, policy, and routing inputs |
+| `LLM::MessageInferenceResponse` | `llm_message_inference_responses` | Provider/runtime response for one inference request |
+| `LLM::RouteAttempt` | `llm_route_attempts` | Provider/model/runner routing attempts, including failures and escalations |
+| `LLM::MessageInferenceMetric` | `llm_message_inference_metrics` | Token, latency, cost, and finance usage metrics for an inference pair |
+| `LLM::ToolCall` | `llm_tool_calls` | Tool calls requested by an LLM provider response |
+| `LLM::ToolCallAttempt` | `llm_tool_call_attempts` | Execution attempts, retries, failures, and results for provider-requested tool calls |
+| `LLM::ConversationCompaction` | `llm_conversation_compactions` | Conversation-scoped compaction events |
+| `LLM::PolicyEvaluation` | `llm_policy_evaluations` | Policy, classification, RBAC, and enforcement decisions for inference requests |
+| `LLM::SecurityEvent` | `llm_security_events` | Security-relevant events tied to conversation, inference, response, or tool attempts |
+| `LLM::RegistryEvent` | `llm_registry_events` | Provider/model registry availability and health events |
+
 ---
 
 ## Dependencies
@@ -396,7 +485,7 @@ Apollo models require PostgreSQL with the `pgvector` extension. They are skipped
 
 ## Migrations
 
-76 numbered Sequel DSL migrations run automatically on startup (`auto_migrate: true`). Key milestones:
+97 numbered Sequel DSL migrations run automatically on startup (`auto_migrate: true`). Key milestones:
 
 | Range | What was added |
 |-------|---------------|
@@ -412,7 +501,11 @@ Apollo models require PostgreSQL with the `pgvector` extension. They are skipped
 | 050 | Critical indexes across 13 tables |
 | 058–067 | Audit records, chains, knowledge tiers, tool embedding cache, identity system (providers, principals, identities, groups) |
 | 068–071 | Entity type on audit records, principal on nodes, approval queue resume, engine on relationships |
-| 072–076 | Identity audit/multi-instance columns, Apollo identifier widening, task idempotency, Extract step timings |
+| 072–073 | Identity audit log and multi-instance identity columns |
+| 074–076 | Apollo field width fixes, task idempotency columns, and Extract step timing rows |
+| 077–090 | Portable LLM lifecycle schema: conversations, messages, inference requests/responses, route attempts, inference metrics, provider-requested tool calls, compactions, policy/security, and registry events |
+| 091–096 | Portable identity companion schema with integer primary keys, public UUIDs, provider capabilities, principals, identities, groups, memberships, and audit log |
+| 097 | LLM dispatch identifiers for fleet operation, correlation, idempotency, provider instance, and dispatch path |
 
 Run migrations standalone:
 
@@ -420,6 +513,15 @@ Run migrations standalone:
 bundle exec legionio_migrate
 ```
 
+Migration rules:
+
+- Do not edit published migrations.
+- Do not guard migrations with `create_table?`, `table_exists?`, `if_not_exists`, or similar conditional schema logic.
+- Add new migrations in the next available number and keep domains split by dependency and rollback risk.
+- Use portable Sequel DSL unless a feature truly requires adapter-specific behavior.
+- Prefer integer `id` primary keys for joins plus public `uuid` columns for APIs, logs, and external references.
+- Avoid JSON columns unless the shape is genuinely provider-specific or dynamic evidence.
+
 ---
 
 ## CLI Executable
@@ -449,6 +551,7 @@ bundle exec legionio_migrate
 11. Financial logging for UAIS cost recovery
 12. Global tool embedding cache (L4 tier for `Legion::Tools::EmbeddingCache`)
 13. Unified identity system (providers, principals, identities, groups)
+14. LLM lifecycle ledger for audit, finance metrics, routing reconstruction, tool calls, and security incident lineage
 
 ---
 
@@ -462,6 +565,15 @@ bundle exec rspec        # all tests must pass
 bundle exec rubocop -A   # zero offenses expected
 ```
 
+This repo also includes a pre-commit configuration:
+
+```bash
+pre-commit install
+pre-commit run --all-files
+```
+
+The local RuboCop hook auto-corrects staged Ruby files when RuboCop is available and fails the commit when RuboCop reports real offenses. The Ruby syntax hook checks every staged Ruby file.
+
 Follow the [LegionIO contribution guide](https://github.com/LegionIO/.github/blob/main/CONTRIBUTING.md). Open a PR against `main`.
 
 ---

data/lib/legion/data/connection.rb

@@ -161,6 +161,7 @@ module Legion
         end
 
         def setup
+          @adapter = Legion::Settings[:data][:adapter]&.to_sym || :sqlite
           opts = sequel_opts
           log.info("Legion::Data::Connection setup adapter=#{adapter}")
           @fallback_active = false
@@ -173,7 +174,7 @@ module Legion
                       rescue StandardError => e
                         raise unless dev_fallback?
 
-                        log.error("Legion::Data FALLING BACK TO SQLITE — #{attempted_adapter} connection failed: #{e.message}")
+                        log.error("Legion::Data FALLING BACK TO SQLITE — #{attempted_adapter} network DB connection failed: #{e.message}")
                         log.error("Legion::Data WARNING: Data written to SQLite will NOT be visible when #{attempted_adapter} reconnects. " \
                                   'Apollo knowledge, audit logs, and other DB-backed services will use a local-only store.')
                         handle_exception(e, level: :error, handled: true, operation: :shared_connect, fallback: :sqlite)
@@ -261,6 +262,7 @@ module Legion
           @sequel&.disconnect
           @query_file_logger&.close
           @query_file_logger = nil
+          @fallback_active = false
           Legion::Settings[:data][:connected] = false
           log.info 'Legion::Data connection closed'
         end

data/lib/legion/data/migrations/077_create_llm_conversations.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    create_table(:llm_conversations) do
+      primary_key :id
+      String :uuid, size: 36, null: false, unique: true
+      Integer :principal_id
+      Integer :identity_id
+      String :title, size: 255
+      String :status, size: 32, null: false, default: 'active'
+      String :system_prompt_key, size: 255
+      String :system_prompt_hash, size: 128
+      String :classification_level, size: 64
+      TrueClass :contains_phi, null: false, default: false
+      TrueClass :contains_pii, null: false, default: false
+      String :retention_policy, size: 64, null: false, default: 'default'
+      DateTime :expires_at
+      DateTime :recorded_at
+      DateTime :inserted_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+      DateTime :created_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+      DateTime :updated_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+
+      index :uuid
+      index :principal_id
+      index :identity_id
+      index :status
+      index :retention_policy
+      index :expires_at
+    end
+  end
+end

data/lib/legion/data/migrations/078_create_llm_messages.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    create_table(:llm_messages) do
+      primary_key :id
+      String :uuid, size: 36, null: false, unique: true
+      foreign_key :conversation_id, :llm_conversations, null: false, on_delete: :cascade
+      foreign_key :parent_message_id, :llm_messages, null: true, on_delete: :set_null
+      Integer :message_inference_request_id
+      Integer :message_inference_response_id
+      Integer :tool_call_id
+      Integer :seq, null: false
+      String :role, size: 32, null: false
+      String :content_type, size: 64, null: false, default: 'text'
+      String :content, text: true
+      Integer :input_tokens, null: false, default: 0
+      Integer :output_tokens, null: false, default: 0
+      DateTime :created_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+      DateTime :inserted_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+
+      unique %i[conversation_id seq]
+      index :uuid
+      index :conversation_id
+      index :parent_message_id
+      index :message_inference_request_id
+      index :message_inference_response_id
+      index :tool_call_id
+      index %i[conversation_id role]
+      index :created_at
+    end
+  end
+end

data/lib/legion/data/migrations/079_create_llm_message_inference_requests.rb

@@ -0,0 +1,47 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    create_table(:llm_message_inference_requests) do
+      primary_key :id
+      String :uuid, size: 36, null: false, unique: true
+      foreign_key :conversation_id, :llm_conversations, null: false, on_delete: :cascade
+      foreign_key :latest_message_id, :llm_messages, null: true, on_delete: :set_null
+      Integer :caller_principal_id
+      Integer :caller_identity_id
+      String :runtime_caller_type, size: 64
+      String :request_ref, size: 128
+      String :correlation_ref, size: 128
+      String :exchange_ref, size: 128
+      String :request_type, size: 64, null: false, default: 'chat'
+      String :status, size: 64, null: false, default: 'created'
+      Integer :context_message_count, null: false, default: 0
+      Integer :context_tokens, null: false, default: 0
+      Integer :token_budget, null: false, default: 0
+      String :curation_strategy, size: 128
+      Integer :injected_tool_count, null: false, default: 0
+      String :tool_policy, size: 128
+      String :request_capture_mode, size: 64, null: false, default: 'metadata_only'
+      String :request_content_hash, size: 128
+      String :request_json, text: true
+      String :classification_level, size: 64
+      String :rbac_decision, size: 64
+      String :cost_center, size: 128
+      String :budget_key, size: 128
+      DateTime :requested_at
+      DateTime :inserted_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+
+      index :uuid
+      index :conversation_id
+      index :latest_message_id
+      index :caller_principal_id
+      index :caller_identity_id
+      index :request_ref
+      index :correlation_ref
+      index :exchange_ref
+      index :status
+      index %i[cost_center requested_at]
+      index :requested_at
+    end
+  end
+end

data/lib/legion/data/migrations/080_create_llm_message_inference_responses.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    create_table(:llm_message_inference_responses) do
+      primary_key :id
+      String :uuid, size: 36, null: false, unique: true
+      foreign_key :message_inference_request_id, :llm_message_inference_requests, null: false, on_delete: :cascade
+      foreign_key :response_message_id, :llm_messages, null: true, on_delete: :set_null
+      String :provider, size: 128
+      String :model_key, size: 255
+      String :tier, size: 64
+      String :runner_ref, size: 128
+      String :provider_response_ref, size: 255
+      String :status, size: 64, null: false, default: 'created'
+      String :finish_reason, size: 128
+      String :error_category, size: 128
+      String :error_code, size: 128
+      String :error_message, text: true
+      Integer :latency_ms, null: false, default: 0
+      Integer :wall_clock_ms, null: false, default: 0
+      String :response_capture_mode, size: 64, null: false, default: 'metadata_only'
+      String :response_content_hash, size: 128
+      String :response_json, text: true
+      String :response_thinking_json, text: true
+      DateTime :responded_at
+      DateTime :inserted_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+
+      index :uuid
+      index :message_inference_request_id
+      index :response_message_id
+      index %i[provider model_key]
+      index :runner_ref
+      index :provider_response_ref
+      index :status
+      index :responded_at
+    end
+  end
+end

data/lib/legion/data/migrations/081_add_llm_message_inference_foreign_keys.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  up do
+    alter_table(:llm_messages) do
+      add_foreign_key [:message_inference_request_id], :llm_message_inference_requests, key: :id, on_delete: :set_null
+      add_foreign_key [:message_inference_response_id], :llm_message_inference_responses, key: :id, on_delete: :set_null
+    end
+  end
+
+  down do
+    alter_table(:llm_messages) do
+      drop_foreign_key [:message_inference_response_id]
+      drop_foreign_key [:message_inference_request_id]
+    end
+  end
+end

data/lib/legion/data/migrations/082_create_llm_route_attempts.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    create_table(:llm_route_attempts) do
+      primary_key :id
+      String :uuid, size: 36, null: false, unique: true
+      foreign_key :message_inference_request_id, :llm_message_inference_requests, null: false, on_delete: :cascade
+      foreign_key :message_inference_response_id, :llm_message_inference_responses, null: true, on_delete: :set_null
+      Integer :attempt_no, null: false
+      String :provider, size: 128
+      String :model_key, size: 255
+      String :tier, size: 64
+      String :route_target, size: 255
+      String :status, size: 64, null: false
+      String :failure_reason, text: true
+      Integer :latency_ms, null: false, default: 0
+      DateTime :started_at
+      DateTime :ended_at
+      DateTime :inserted_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+
+      unique %i[message_inference_request_id attempt_no]
+      index :uuid
+      index :message_inference_request_id
+      index :message_inference_response_id
+      index %i[provider model_key]
+      index :status
+      index :started_at
+    end
+  end
+end

data/lib/legion/data/migrations/083_create_llm_message_inference_metrics.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    create_table(:llm_message_inference_metrics) do
+      primary_key :id
+      String :uuid, size: 36, null: false, unique: true
+      foreign_key :message_inference_request_id, :llm_message_inference_requests, null: false, on_delete: :cascade
+      foreign_key :message_inference_response_id, :llm_message_inference_responses, null: true, on_delete: :set_null
+      String :provider, size: 128
+      String :model_key, size: 255
+      String :tier, size: 64
+      Integer :input_tokens, null: false, default: 0
+      Integer :output_tokens, null: false, default: 0
+      Integer :thinking_tokens, null: false, default: 0
+      Integer :total_tokens, null: false, default: 0
+      Integer :latency_ms, null: false, default: 0
+      Integer :wall_clock_ms, null: false, default: 0
+      BigDecimal :cost_usd, size: [20, 8], null: false, default: 0
+      String :currency, size: 3, null: false, default: 'USD'
+      String :cost_center, size: 128
+      String :budget_key, size: 128
+      DateTime :recorded_at
+      DateTime :inserted_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+
+      index :uuid
+      index :message_inference_request_id
+      index :message_inference_response_id
+      index %i[provider model_key]
+      index :cost_center
+      index :budget_key
+      index :recorded_at
+      index %i[cost_center recorded_at]
+    end
+  end
+end

data/lib/legion/data/migrations/084_create_llm_tool_calls.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    create_table(:llm_tool_calls) do
+      primary_key :id
+      String :uuid, size: 36, null: false, unique: true
+      foreign_key :message_inference_response_id, :llm_message_inference_responses, null: false, on_delete: :cascade
+      foreign_key :requested_by_message_id, :llm_messages, null: true, on_delete: :set_null
+      foreign_key :result_message_id, :llm_messages, null: true, on_delete: :set_null
+      Integer :tool_call_index, null: false, default: 0
+      String :provider_tool_call_ref, size: 255
+      String :tool_name, size: 255, null: false
+      String :tool_source_type, size: 128
+      String :tool_source_server, size: 255
+      String :status, size: 64, null: false, default: 'requested'
+      DateTime :requested_at
+      DateTime :completed_at
+      DateTime :inserted_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+
+      unique %i[message_inference_response_id tool_call_index]
+      index :uuid
+      index :message_inference_response_id
+      index :requested_by_message_id
+      index :result_message_id
+      index :provider_tool_call_ref
+      index :tool_name
+      index :status
+      index :requested_at
+    end
+  end
+end

data/lib/legion/data/migrations/085_add_llm_message_tool_call_foreign_key.rb

@@ -0,0 +1,15 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  up do
+    alter_table(:llm_messages) do
+      add_foreign_key [:tool_call_id], :llm_tool_calls, key: :id, on_delete: :set_null
+    end
+  end
+
+  down do
+    alter_table(:llm_messages) do
+      drop_foreign_key [:tool_call_id]
+    end
+  end
+end

data/lib/legion/data/migrations/086_create_llm_tool_call_attempts.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    create_table(:llm_tool_call_attempts) do
+      primary_key :id
+      String :uuid, size: 36, null: false, unique: true
+      foreign_key :tool_call_id, :llm_tool_calls, null: false, on_delete: :cascade
+      Integer :attempt_no, null: false
+      String :runner_ref, size: 128
+      String :status, size: 64, null: false
+      String :error_category, size: 128
+      String :error_code, size: 128
+      String :error_message, text: true
+      Integer :duration_ms, null: false, default: 0
+      String :arguments_ref, size: 255
+      String :result_ref, size: 255
+      DateTime :started_at
+      DateTime :ended_at
+      DateTime :inserted_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+
+      unique %i[tool_call_id attempt_no]
+      index :uuid
+      index :tool_call_id
+      index :runner_ref
+      index :status
+      index :started_at
+    end
+  end
+end

data/lib/legion/data/migrations/087_create_llm_conversation_compactions.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    create_table(:llm_conversation_compactions) do
+      primary_key :id
+      String :uuid, size: 36, null: false, unique: true
+      foreign_key :conversation_id, :llm_conversations, null: false, on_delete: :cascade
+      foreign_key :triggered_by_message_inference_request_id, :llm_message_inference_requests, null:      true,
+                                                                                               on_delete: :set_null
+      foreign_key :replaces_message_from_id, :llm_messages, null: true, on_delete: :set_null
+      foreign_key :replaces_message_to_id, :llm_messages, null: true, on_delete: :set_null
+      String :strategy, size: 128
+      String :status, size: 64, null: false, default: 'created'
+      Integer :source_message_count, null: false, default: 0
+      Integer :source_token_count, null: false, default: 0
+      Integer :compacted_token_count, null: false, default: 0
+      String :content_hash, size: 128
+      String :summary, text: true
+      String :error_message, text: true
+      DateTime :compacted_at
+      DateTime :inserted_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+
+      index :uuid
+      index :conversation_id
+      index :triggered_by_message_inference_request_id
+      index :status
+      index :compacted_at
+    end
+  end
+end

data/lib/legion/data/migrations/088_create_llm_policy_evaluations.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    create_table(:llm_policy_evaluations) do
+      primary_key :id
+      String :uuid, size: 36, null: false, unique: true
+      foreign_key :conversation_id, :llm_conversations, null: true, on_delete: :set_null
+      foreign_key :message_inference_request_id, :llm_message_inference_requests, null: true, on_delete: :set_null
+      foreign_key :message_inference_response_id, :llm_message_inference_responses, null: true, on_delete: :set_null
+      String :policy_key, size: 128, null: false
+      String :policy_version, size: 64
+      String :evaluation_type, size: 64, null: false
+      String :decision, size: 64, null: false
+      String :enforcement_action, size: 64
+      String :classification_level, size: 64
+      TrueClass :contains_phi, null: false, default: false
+      TrueClass :contains_pii, null: false, default: false
+      String :reason_code, size: 128
+      String :reason, text: true
+      DateTime :evaluated_at
+      DateTime :inserted_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+
+      index :uuid
+      index :conversation_id
+      index :message_inference_request_id
+      index :message_inference_response_id
+      index :policy_key
+      index :decision
+      index :evaluated_at
+    end
+  end
+end

data/lib/legion/data/migrations/089_create_llm_security_events.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  change do
+    create_table(:llm_security_events) do
+      primary_key :id
+      String :uuid, size: 36, null: false, unique: true
+      foreign_key :conversation_id, :llm_conversations, null: true, on_delete: :set_null
+      foreign_key :message_inference_request_id, :llm_message_inference_requests, null: true, on_delete: :set_null
+      foreign_key :message_inference_response_id, :llm_message_inference_responses, null: true, on_delete: :set_null
+      foreign_key :tool_call_id, :llm_tool_calls, null: true, on_delete: :set_null
+      foreign_key :tool_call_attempt_id, :llm_tool_call_attempts, null: true, on_delete: :set_null
+      foreign_key :policy_evaluation_id, :llm_policy_evaluations, null: true, on_delete: :set_null
+      String :event_type, size: 128, null: false
+      String :severity, size: 32, null: false, default: 'info'
+      String :status, size: 64, null: false, default: 'open'
+      String :description, text: true
+      DateTime :detected_at
+      DateTime :inserted_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+
+      index :uuid
+      index :conversation_id
+      index :message_inference_request_id
+      index :message_inference_response_id
+      index :tool_call_id
+      index :tool_call_attempt_id
+      index :policy_evaluation_id
+      index :event_type
+      index :severity
+      index :detected_at
+    end
+  end
+end