lex-llm-ledger 0.1.11 → 0.2.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 95df124106575b40d9e7bd4534782f90433c3f8bb12cd3473ca61b359f9f22ad
-  data.tar.gz: f8ac1dbc0227c6c1b231406db8e11270ea846bb8a25c1f9a2c70c05dc51631ce
+  metadata.gz: 83376ade1924f74e2c734ccf0b9e67b1a1db0ad0494a883469fb86a4eb1763a1
+  data.tar.gz: 83c5f37ff7b8d63c2865e3f758f0ecbb116746d93817a2b67aadc7a14f66f503
 SHA512:
-  metadata.gz: e6af8de42045313fee0582ee1ee58dbdcdc0d9e9cfa85d5af94c9ca87ae6358a2a5e019aa035e588cf9d8ea1c883b9635ffb09c55bb9bcf90414eda7a7d60de2
-  data.tar.gz: 13bcf5ba5f6e0d64254e04e523758fd85f5f050a316b497013769eefde91877d995b8361f18bde8270f8f4b04b037cc6f988455bbcc33148f89c564a7bded585
+  metadata.gz: dcd276003a26041f9b0e5aa373547d7d03f74010c1a7fd011cd0a6fd1a9ded022425476ff95e7efc5a466c94c8362b699e5d1e3f8468609eec2bc6391e02b016
+  data.tar.gz: 0b0168807bc938bdcba4d3a2955c3d5e86f58a6138a93970fea1addbd2bc1f38fef44ed5bbedfcd7d01c3d26e23c451920c645d79acce603b26b98ae7bfffddc

data/CHANGELOG.md

@@ -1,5 +1,59 @@
 # Changelog
 
+## [0.2.5] - 2026-05-06
+
+### Fixed
+- Log every successful ledger audit and metric database insert at `info` with safe row context.
+- Log duplicate insert failures at `warn` and unexpected insert failures at `error` before returning or re-raising.
+
+## [0.2.4] - 2026-05-06
+
+### Fixed
+- Replace generated runner subscription actors with runner-named ledger-owned subscription actors so audit queues are consumed through the ledger decoder.
+- Route ledger subscription actor payload decoding through the ledger decoder so encrypted audit messages preserve metadata and missing-IV messages dead-letter before core decryption.
+
+## [0.2.3] - 2026-05-06
+
+### Fixed
+- Use the real `legion-json` load contract for ledger JSON parsing and remove root `JSON` fallbacks from runtime code.
+- Route retention TTL overrides through extension-scoped Legion settings and add default retention settings metadata.
+- Send handled runner/backfill errors through `handle_exception` for structured Legion logging.
+- Reject encrypted audit payloads that are missing the required `iv` header before attempting decryption.
+
+## [0.2.2] - 2026-05-06
+
+### Fixed
+- Persist official response-message foreign keys, keep generated request references stable within a write, and remove raw payload logging from ledger runners.
+- Make legacy backfill counts idempotent and attach legacy tool rows only to existing official inference responses.
+- Clarify README cutover status for tool and registry projection tables.
+
+## [0.2.1] - 2026-05-06
+
+### Fixed
+- Preserve namespaced caller identities from current LLM audit and metering envelopes instead of storing ambiguous display identities such as `system`.
+
+## [0.2.0] - 2026-05-06
+
+### Changed
+- Write prompt audit and metering events into the official `legion-data` LLM lifecycle schema instead of legacy ledger-only tables.
+- Move provider stats and usage reporting to official inference request, response, and metric tables grouped by provider, provider instance, model, and operation.
+- Bumped the transport dependency floor to `legion-transport >= 1.4.14` for the coordinated fleet envelope sweep.
+
+### Added
+- Add official prompt and metering writers plus legacy LLM ledger backfill for prompt, metering, tool, and registry availability records.
+- Add a hard stop for legacy-only writer mode after official cutover.
+
+## [0.1.13] - 2026-05-03
+
+### Added
+- Add `response_thinking_json` to prompt audit records so provider thinking payloads are stored separately from assistant response content.
+
+## [0.1.12] - 2026-04-28
+
+### Added
+- Persist provider-neutral `llm.registry` availability event envelopes for offering, worker, lane, model, runtime, capacity, and health diagnostics
+- Add ledger-owned passive `llm.registry` transport exchange, durable registry availability queue, and subscription actor
+
 ## [0.1.11] - 2026-04-28
 
 ### Fixed

data/Gemfile

@@ -3,7 +3,9 @@
 source 'https://rubygems.org'
 
 group :test do
+  legion_data_path = ENV.fetch('LEGION_DATA_PATH', File.expand_path('../../legion-data', __dir__))
   llm_base_path = ENV.fetch('LEX_LLM_PATH', File.expand_path('../lex-llm', __dir__))
+  gem 'legion-data', path: legion_data_path if File.directory?(legion_data_path)
   gem 'lex-llm', path: llm_base_path if File.directory?(llm_base_path)
 end
 

data/README.md

@@ -1,8 +1,8 @@
 # lex-llm-ledger
 
 LLM observability persistence for LegionIO. Consumes metering and audit messages from
-AMQP queues, decrypts audit payloads, enforces retention policies, and writes records
-to a database for usage reporting and compliance.
+AMQP queues, decrypts audit payloads, enforces retention policies, and writes official
+`legion-data` LLM lifecycle records for usage reporting and compliance.
 
 ## Queues Consumed
 
@@ -12,11 +12,61 @@ to a database for usage reporting and compliance.
 | `llm.audit.prompts` | `llm.audit` (topic) | `audit.prompt.#` | Encrypted prompt+response pairs |
 | `llm.audit.tools` | `llm.audit` (topic) | `audit.tool.#` | Encrypted tool call records |
 
-## Tables
+## Official Tables
 
-- `metering_records` - One row per LLM inference (tokens, cost, latency, routing)
-- `prompt_records` - Full prompt/response audit with retention TTL and PHI classification
-- `tool_records` - Tool call audit linked to parent prompt via correlation_id
+- `llm_conversations` - Conversation container and retention/classification metadata
+- `llm_messages` - Model-visible user and assistant messages
+- `llm_message_inference_requests` - Operation, correlation, request payload, and policy context
+- `llm_message_inference_responses` - Provider, provider instance, model, dispatch path, visible response, and thinking payload
+- `llm_message_inference_metrics` - Tokens, latency, cost, and finance allocation
+- `llm_tool_calls` - Provider-requested tool call lineage
+- `llm_registry_events` - Provider/model availability events
+
+Prompt and metering consumers write the official lifecycle tables directly.
+`llm_tool_records` and `llm_registry_availability_records` remain operational
+projection tables while the official tool/registry event cutover continues.
+The legacy backfill reconciles those rows into `llm_tool_calls` and
+`llm_registry_events` when they can be linked to official inference responses.
+Legacy-only prompt/metering writer mode hard-stops instead of silently writing
+stale projections.
+
+## Event Spine Target
+
+The existing tables are useful reporting projections, but the uplift target is end-to-end visibility for every LLM-related lifecycle event. Ledger should add a canonical `llm_events` stream/table and keep `metering_records`, `prompt_records`, and `tool_records` as specialized query views or companion tables.
+
+Every event should share these correlation keys:
+
+- `conversation_id`
+- `request_id`
+- `exchange_id`
+- `message_id`
+- `parent_message_id`
+- `message_seq`
+- `correlation_id`
+- `trace_id`
+- `span_id`
+- `event_id`
+- `event_seq`
+
+Event types should cover at least:
+
+- request received, normalized, classified, enriched, and context-assembled
+- routing candidates built, candidates excluded, offering selected, failover attempted, escalation attempted
+- provider request started, provider response received, provider error/timeout/cancel
+- response normalized, streamed chunk emitted, final response returned
+- MCP/tool call planned, started, completed, failed, denied, or timed out
+- fleet request published, broker accepted/unroutable, worker accepted, worker rejected, fleet response received
+- metering emitted, audit emitted, ledger write queued, ledger write succeeded/failed/spooled
+
+This lets operators reconstruct a conversation without replaying prompt bodies. Example: conversation `123` had 32 messages, one failed, five executed on Anthropic direct, four locally, the rest on GPU fleet, with per-step response time, token totals, cost allocation, and failover history.
+
+Ledger has three distinct outputs:
+
+1. **Legal/evidence reconstruction** - immutable, correlated, retention-controlled event evidence sufficient to answer a legal or security request. This favors completeness, ordering, integrity, and capture-mode correctness.
+2. **Operational analytics** - structured projections for high-level patterns, cost, latency, quality, routing behavior, fleet utilization, tool usage, and failure rates. This favors queryability and aggregation without requiring raw prompt bodies.
+3. **Governed training/evaluation datasets** - policy-approved derived datasets for model improvement, team/org use-case tuning, eval generation, routing-quality analysis, and tool-use learning. This must be derived from ledger events through explicit consent, classification, redaction/de-identification, retention, and export controls.
+
+Training/eval export is not automatic reuse of raw audit. A future dataset builder should select eligible events, apply redaction and capture-mode policy, preserve provenance back to `event_id`/`conversation_id`, and write a dataset manifest that records data classes, consent basis, source filters, transform versions, and approval state.
 
 ## Key Design Decisions
 
@@ -24,29 +74,94 @@ to a database for usage reporting and compliance.
 - **Passive exchange references** - does not declare `llm.metering` or `llm.audit` (owned by legion-llm)
 - **DecryptionUnavailable causes NACK** - messages requeue until the node has Vault credentials
 - **PHI TTL cap** - records flagged `contains_phi` are capped at 30 days regardless of retention label
-- **Idempotent writes** - duplicate message_id inserts are silently dropped
+- **Idempotent official writes** - duplicate request/response/message references resolve to existing official rows
+
+## Routing Uplift Target
+
+The 2026-04-25 `legion-llm` routing redesign moves routing to operation-aware model offerings. Ledger should persist the enriched metadata published by `legion-llm` without owning routing policy.
+
+Target metering, prompt, and tool records should be able to store:
+
+- selected offering identity: `offering_id`, `provider_family`, `instance_id`, `canonical_model`, `provider_model`, `operation`, `transport`, `region`, `endpoint_hash`
+- routing details: requested route, selected route, excluded candidates, lateral failover chain, vertical escalation chain, and policy decisions
+- identity details: caller principal/canonical name/kind/source, accepting runtime identity, executing runtime identity for fleet requests, fleet lane, fleet class, network boundary, placement policy, fleet correlation ID, hashed reply target, and credential lease/grant metadata
+- token and cost allocation: conversation ID, input/output/total tokens, selected-offering cost, pricing tier, configured baseline/comparable provider cost, avoided cost, and aggregation keys for tier, fleet class, provider family, instance, model, transport, and lane
+- compliance details: `contains_pii`, `contains_phi`, `contains_pci`, `data_classes`, `jurisdictions`, `retention_policy`, and `capture_mode`
+- model provenance: management state, model depot registry ID, artifact digest, signature verification status, rollout ring, and approval state
+- tool provenance: source type/server, policy tags, approval/denial state, redacted or hashed resource identifiers, and input/output classification flags
+- registry/availability events: worker heartbeat, lane availability, offering availability, model sync state, degraded/draining/blocked transitions, and capacity changes from `llm.registry`
+
+The uplift must validate the existing runners and migrations against this target. Current tables already capture core metering, prompt audit, and tool audit, but they need additional correlation fields, routing/offering fields, token context fields, cost allocation fields, identity/fleet fields, and event-spine coverage for request/response/MCP lifecycle events that are not prompt or tool records.
+
+Audit capture modes expected from `legion-llm`:
+
+- `none` - do not publish prompt/tool body audit
+- `metadata_only` - store routing/classification/token/cost metadata only
+- `redacted` - store redacted bodies plus redaction metadata
+- `encrypted_raw` - store encrypted full payloads for approved consumers
+- `raw` - plaintext full payloads for local/dev or explicitly approved environments
+
+Prompt/tool audit should be durable. If transport is unavailable, `legion-llm` should spool audit records or use a durable local audit queue unless capture mode is `none` or policy explicitly allows best-effort audit.
+
+For async `:fleet` inference, ledger records should preserve the original caller identity and record both runtimes: the process that accepted/enqueued the request and the worker process that executed the provider call. Fleet records should also persist the selected lane, worker fleet class (`endpoint`, `datacenter`, `cloud_vpc`, etc.), placement policy, and model provenance so investigators can tell whether a request ran on the caller's own machine, another endpoint, a datacenter GPU, or a cloud-adjacent worker. The raw RabbitMQ `reply_to` queue should remain transport-only; persisted records should use a stable hash plus the `correlation_id` for reconstruction.
+
+Fleet registry history should arrive through RabbitMQ rather than endpoint workers writing directly to the database. `legion-llm` and provider workers publish availability events to `llm.registry`; ledger consumes those events and persists durable history for operator diagnostics, audit, and legal reconstruction.
+
+Ledger should be able to answer spend-allocation questions without replaying raw prompts: how many input/output tokens a conversation used, how tokens split across Anthropic direct versus fleet GPU versus endpoint MacBook fleet, and estimated dollars saved by local/fleet execution compared with a configured cloud/frontier baseline.
+
+Ledger is not on the LLM execution critical path. If the database is unavailable, ledger consumers should retry, requeue, DLQ, or spool according to transport policy while `legion-llm` continues routing and executing requests. Compliance profiles that require durable audit before response are the explicit exception and should fail closed upstream with a clear policy error.
 
 ## Requirements
 
-- `legion-data` >= 1.6 (Sequel DB connection)
+- `legion-data` >= 1.8.0 (official LLM lifecycle schema)
 - `legion-json` >= 1.2 (JSON serialization)
-- `legion-transport` >= 1.4 (AMQP transport)
+- `legion-logging` >= 1.3 (structured exception logging)
+- `legion-settings` >= 1.3 (extension-scoped retention settings)
+- `legion-transport` >= 1.4.14 (AMQP transport)
 - `legion-crypt` >= 1.5 (for decrypting audit messages, optional at runtime)
 
+## Configuration
+
+Ledger runs with safe defaults and reads extension settings from
+`extensions.llm.ledger`:
+
+```json
+{
+  "extensions": {
+    "llm": {
+      "ledger": {
+        "retention": {
+          "default_days": 90,
+          "phi_ttl_days": 30
+        }
+      }
+    }
+  }
+}
+```
+
+`default_days` controls records with the `default` retention label. `phi_ttl_days`
+caps PHI records even when the event asks for longer or permanent retention.
+Encrypted audit messages must include an `iv` header; missing-IV messages are
+rejected as malformed encrypted audit records rather than retried.
+
 ## Usage
 
 ```ruby
-# Metering write (called by MeteringWriter actor)
-Legion::Extensions::LLM::Ledger::Runners::Metering.write_metering_record(payload, metadata)
+# Metering write (called by Metering actor)
+Legion::Extensions::Llm::Ledger::Runners::Metering.write_metering_record(payload, metadata)
 
 # Usage summary
-Legion::Extensions::LLM::Ledger::Runners::UsageReporter.summary(period: 'day', group_by: 'provider')
+Legion::Extensions::Llm::Ledger::Runners::UsageReporter.summary(period: 'day', group_by: 'provider_instance')
 
 # Budget check
-Legion::Extensions::LLM::Ledger::Runners::UsageReporter.budget_check(budget_id: 'budget_q1', budget_usd: 100.0)
+Legion::Extensions::Llm::Ledger::Runners::UsageReporter.budget_check(budget_id: 'budget_q1', budget_usd: 100.0)
 
 # Provider health
-Legion::Extensions::LLM::Ledger::Runners::ProviderStats.health_report
+Legion::Extensions::Llm::Ledger::Runners::ProviderStats.health_report
+
+# One-time legacy reconciliation
+Legion::Extensions::Llm::Ledger::Backfill::LegacyLlmRecords.run
 ```
 
 ## Development

data/lex-llm-ledger.gemspec

@@ -28,12 +28,12 @@ Gem::Specification.new do |spec|
   end
   spec.require_paths = ['lib']
 
-  spec.add_dependency 'legion-data',      '>= 1.6'
+  spec.add_dependency 'legion-data',      '>= 1.8.0'
   spec.add_dependency 'legion-json',      '>= 1.2'
   spec.add_dependency 'legion-logging',   '>= 1.3'
   spec.add_dependency 'legion-settings',  '>= 1.3'
-  spec.add_dependency 'legion-transport', '>= 1.4'
-  spec.add_dependency 'lex-llm',          '>= 0.1.5'
+  spec.add_dependency 'legion-transport', '>= 1.4.14'
+  spec.add_dependency 'lex-llm',          '>= 0.4.0'
 
   spec.add_development_dependency 'rspec'
   spec.add_development_dependency 'rubocop'

data/lib/legion/extensions/llm/ledger/actors/{metering_writer.rb → metering.rb}

@@ -1,13 +1,16 @@
 # frozen_string_literal: true
 
 require 'legion/extensions/actors/subscription'
+require_relative '../helpers/subscription_actor'
 
 module Legion
   module Extensions
     module Llm
       module Ledger
         module Actor
-          class MeteringWriter < Legion::Extensions::Actors::Subscription
+          class Metering < Legion::Extensions::Actors::Subscription
+            include Helpers::SubscriptionActor
+
             def runner_class = Legion::Extensions::Llm::Ledger::Runners::Metering
 
             def runner_function
@@ -17,6 +20,10 @@ module Legion
             def use_runner?
               false
             end
+
+            def queue
+              Legion::Extensions::Llm::Ledger::Transport::Queues::MeteringWrite
+            end
           end
         end
       end

data/lib/legion/extensions/llm/ledger/actors/{prompt_writer.rb → prompts.rb}

@@ -1,14 +1,16 @@
 # frozen_string_literal: true
 
 require 'legion/extensions/actors/subscription'
-require_relative '../helpers/subscription_message'
+require_relative '../helpers/subscription_actor'
 
 module Legion
   module Extensions
     module Llm
       module Ledger
         module Actor
-          class PromptWriter < Legion::Extensions::Actors::Subscription
+          class Prompts < Legion::Extensions::Actors::Subscription
+            include Helpers::SubscriptionActor
+
             def runner_class = Legion::Extensions::Llm::Ledger::Runners::Prompts
 
             def runner_function
@@ -19,8 +21,8 @@ module Legion
               false
             end
 
-            def process_message(message, metadata, delivery_info)
-              Helpers::SubscriptionMessage.decode_payload(message, metadata, delivery_info)
+            def queue
+              Legion::Extensions::Llm::Ledger::Transport::Queues::AuditPrompts
             end
           end
         end

data/lib/legion/extensions/llm/ledger/actors/registry_availability.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+require 'legion/extensions/actors/subscription'
+require_relative '../helpers/subscription_actor'
+
+module Legion
+  module Extensions
+    module Llm
+      module Ledger
+        module Actor
+          class RegistryAvailability < Legion::Extensions::Actors::Subscription
+            include Helpers::SubscriptionActor
+
+            def runner_class = Legion::Extensions::Llm::Ledger::Runners::RegistryAvailability
+
+            def runner_function
+              'write_registry_availability_record'
+            end
+
+            def use_runner?
+              false
+            end
+
+            def queue
+              Legion::Extensions::Llm::Ledger::Transport::Queues::RegistryAvailability
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/legion/extensions/llm/ledger/actors/spool_flush.rb

@@ -24,7 +24,7 @@ module Legion
 
               Legion::LLM::Metering.flush_spool
             rescue StandardError => e
-              Legion::Logging.warn("[lex-llm-ledger] SpoolFlush error: #{e.message}") # rubocop:disable Legion/HelperMigration/DirectLogging
+              handle_exception(e, level: :warn, handled: true, operation: 'spool_flush')
             end
 
             def run_now?

data/lib/legion/extensions/llm/ledger/actors/{tool_writer.rb → tools.rb}

@@ -1,14 +1,16 @@
 # frozen_string_literal: true
 
 require 'legion/extensions/actors/subscription'
-require_relative '../helpers/subscription_message'
+require_relative '../helpers/subscription_actor'
 
 module Legion
   module Extensions
     module Llm
       module Ledger
         module Actor
-          class ToolWriter < Legion::Extensions::Actors::Subscription
+          class Tools < Legion::Extensions::Actors::Subscription
+            include Helpers::SubscriptionActor
+
             def runner_class = Legion::Extensions::Llm::Ledger::Runners::Tools
 
             def runner_function
@@ -19,8 +21,8 @@ module Legion
               false
             end
 
-            def process_message(message, metadata, delivery_info)
-              Helpers::SubscriptionMessage.decode_payload(message, metadata, delivery_info)
+            def queue
+              Legion::Extensions::Llm::Ledger::Transport::Queues::AuditTools
             end
           end
         end

data/lib/legion/extensions/llm/ledger/backfill/legacy_llm_records.rb

@@ -0,0 +1,223 @@
+# frozen_string_literal: true
+
+require 'legion/logging'
+require_relative '../helpers/json'
+require_relative '../helpers/persistence_logging'
+
+module Legion
+  module Extensions
+    module Llm
+      module Ledger
+        module Backfill
+          module LegacyLlmRecords
+            extend Legion::Logging::Helper
+
+            LEGACY_TABLES = %i[
+              llm_prompt_records
+              llm_metering_records
+              llm_tool_records
+              llm_registry_availability_records
+            ].freeze
+
+            module_function
+
+            def run(limit: nil, writer_mode: :official)
+              ensure_no_legacy_writer_mode!(writer_mode)
+
+              LEGACY_TABLES.to_h do |table|
+                [table, table_present?(table) ? backfill_table(table, limit:) : 0]
+              end
+            end
+
+            def ensure_no_legacy_writer_mode!(mode)
+              return unless %i[legacy legacy_only legacy_table_only].include?(mode.to_sym)
+
+              raise ArgumentError, 'Legacy LLM writer mode is disabled after official backfill; configure official LLM writers.'
+            end
+
+            def backfill_table(table, limit:)
+              dataset = db[table].order(:id)
+              dataset = dataset.limit(limit) if limit
+              dataset.all.sum { |row| backfill_row(table, row) }
+            end
+
+            def backfill_row(table, row)
+              case table
+              when :llm_prompt_records
+                backfill_prompt(row)
+              when :llm_metering_records
+                backfill_metering(row)
+              when :llm_tool_records
+                backfill_tool(row)
+              when :llm_registry_availability_records
+                backfill_registry(row)
+              end
+            rescue Sequel::UniqueConstraintViolation => e
+              handle_exception(e, level: :warn, handled: true, operation: 'legacy_llm_backfill.duplicate')
+              0
+            end
+
+            def backfill_prompt(row)
+              payload = prompt_payload(row)
+              return 0 if official_metric_exists?(payload)
+
+              Writers::OfficialPromptWriter.write(payload)
+              1
+            end
+
+            def backfill_metering(row)
+              payload = metering_payload(row)
+              return 0 if official_metric_exists?(payload)
+
+              Writers::OfficialMeteringWriter.write(payload)
+              1
+            end
+
+            def prompt_payload(row)
+              {
+                message_id:           row[:message_id],
+                correlation_id:       row[:correlation_id],
+                conversation_id:      row[:conversation_id],
+                response_message_id:  row[:response_message_id],
+                request_id:           row[:request_id],
+                exchange_id:          row[:exchange_id],
+                operation:            row[:request_type],
+                provider:             row[:provider],
+                model_id:             row[:model_id],
+                tier:                 row[:tier],
+                request:              json_load(row[:request_json]),
+                response:             json_load(row[:response_json]),
+                response_thinking:    json_load(row[:response_thinking_json]),
+                input_tokens:         row[:input_tokens],
+                output_tokens:        row[:output_tokens],
+                total_tokens:         row[:total_tokens],
+                cost_usd:             row[:cost_usd],
+                classification_level: row[:classification_level],
+                contains_phi:         row[:contains_phi],
+                contains_pii:         row[:contains_pii],
+                retention_policy:     row[:retention_policy],
+                expires_at:           row[:expires_at],
+                recorded_at:          row[:recorded_at]
+              }
+            end
+
+            def metering_payload(row)
+              {
+                message_id:        row[:message_id],
+                correlation_id:    row[:correlation_id],
+                conversation_id:   row[:conversation_id],
+                request_id:        row[:request_id],
+                exchange_id:       row[:exchange_id],
+                operation:         row[:request_type],
+                provider:          row[:provider],
+                provider_instance: row[:worker_id],
+                model_id:          row[:model_id],
+                tier:              row[:tier],
+                input_tokens:      row[:input_tokens],
+                output_tokens:     row[:output_tokens],
+                thinking_tokens:   row[:thinking_tokens],
+                total_tokens:      row[:total_tokens],
+                latency_ms:        row[:latency_ms],
+                wall_clock_ms:     row[:wall_clock_ms],
+                cost_usd:          row[:cost_usd],
+                recorded_at:       row[:recorded_at],
+                billing:           {
+                  cost_center: row[:cost_center],
+                  budget_id:   row[:budget_id]
+                }
+              }
+            end
+
+            def backfill_tool(row)
+              response = response_for_request(row[:request_id])
+              return 0 unless response
+
+              tool_uuid = Writers::OfficialRecordWriter.stable_uuid(row[:tool_call_id] || row[:message_id])
+              return 0 if db[:llm_tool_calls].where(uuid: tool_uuid).first
+
+              insert_row(:llm_tool_calls, {
+                           uuid:                          tool_uuid,
+                           message_inference_response_id: response[:id],
+                           tool_call_index:               next_tool_index(response[:id]),
+                           provider_tool_call_ref:        row[:tool_call_id],
+                           tool_name:                     row[:tool_name],
+                           tool_source_type:              row[:tool_source_type],
+                           tool_source_server:            row[:tool_source_server],
+                           status:                        row[:tool_status],
+                           requested_at:                  row[:tool_start_at],
+                           completed_at:                  row[:tool_end_at],
+                           inserted_at:                   Time.now.utc
+                         }, operation: 'legacy_llm_backfill.tool_call')
+              1
+            end
+
+            def backfill_registry(row)
+              uuid = Writers::OfficialRecordWriter.stable_uuid(row[:event_id] || row[:message_id])
+              return 0 if db[:llm_registry_events].where(uuid: uuid).first
+
+              insert_row(:llm_registry_events, {
+                           uuid:        uuid,
+                           provider:    row[:provider_family],
+                           model_key:   row[:model_id],
+                           event_type:  row[:event_type],
+                           status:      registry_status(row),
+                           reason:      row[:metadata_json],
+                           recorded_at: row[:occurred_at],
+                           inserted_at: Time.now.utc
+                         }, operation: 'legacy_llm_backfill.registry_event')
+              1
+            end
+
+            def insert_row(table, attributes, operation:)
+              Helpers::PersistenceLogging.insert_row(db, table, attributes, operation: operation)
+            end
+
+            def response_for_request(request_id)
+              request = db[:llm_message_inference_requests].where(request_ref: request_id).first
+              return nil unless request
+
+              db[:llm_message_inference_responses].where(message_inference_request_id: request[:id]).first
+            end
+
+            def official_metric_exists?(payload)
+              db[:llm_message_inference_metrics].where(uuid: official_metric_uuid(payload)).first
+            end
+
+            def official_metric_uuid(payload)
+              ref = payload[:message_id] || "metric:#{Writers::OfficialRecordWriter.request_ref(payload)}"
+              Writers::OfficialRecordWriter.stable_uuid(ref)
+            end
+
+            def next_tool_index(response_id)
+              db[:llm_tool_calls].where(message_inference_response_id: response_id).max(:tool_call_index).to_i + 1
+            end
+
+            def registry_status(row)
+              health = json_load(row[:health_json])
+              health[:status] || health['status'] || row[:event_type] || 'unknown'
+            end
+
+            def table_present?(table)
+              db.table_exists?(table)
+            end
+
+            def db
+              ::Legion::Data.connection
+            end
+
+            def json_load(value)
+              return {} if value.nil? || value.to_s.empty?
+
+              Helpers::Json.load(value)
+            rescue StandardError => e
+              raise unless Helpers::Json.parse_error?(e)
+
+              handle_exception(e, level: :warn, handled: true, operation: 'legacy_llm_backfill.json_load')
+              { content: value }
+            end
+          end
+        end
+      end
+    end
+  end
+end