legion-data 1.6.30 → 1.7.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5736a13f366ea61837d0f2a27e51011b3e3729d4967f0434ca4cae245c4591c8
-  data.tar.gz: 1bc10f1333882ee230dd23a782d14603117ea1f141d35586ea81c5c713504795
+  metadata.gz: a12da2249f20eb148411c6e052ef5b0bbdef86579907a84feab15d713f7b51f9
+  data.tar.gz: 433ea32fa8120428137d2114e54f5285653acc872956d04df3520cc996d277bc
 SHA512:
-  metadata.gz: 29a76a92e428b6b255e20ade26b6323996762301f308fb711f7d02d47cc991e41e5f561babc0c068e1971d7453d3ef8fb52067ddf0d9f31093c8ca90a29966b1
-  data.tar.gz: f5a8d323ae53427f73bf194c7868ee4854b330b83329b825664b2a27d20ae7f7e53b0225399b07e12f8946a9c6bdc1b4e2b3038e62d48df65dc461d6b6c4681c
+  metadata.gz: 8f8c62cfbe83e98ccb220ab02432ed7f2e02837350b4a7eb30ac24e51de6486c128e9f17846ac89dcd67fb835b8f292ded46d61830a2aa037f3f61e1734ee288
+  data.tar.gz: e73434d2a098fa7b57d5c6dea6e75daf43ca6585492984f3c225449c7819ffde7e796e6ac113d3ee4d5caf30bb627085fc2cfec3400ec4c7ca43a9e980b5cb8c

data/CHANGELOG.md

@@ -2,6 +2,40 @@
 
 ## [Unreleased]
 
+## [1.7.3] - 2026-04-27
+
+### Added
+- Migration 074: widens Apollo `content_hash` to 64 fixed characters and `knowledge_domain` / `source_provider` / `source_agent` to 255 characters so SHA-256 hashes and real-world identifiers fit without ingestion truncation failures. (Fixes #33, #34)
+- Migration 075: adds task `idempotency_key` and `idempotency_expires_at` columns plus indexes for SHA-256 payload deduplication windows. (Fixes #14)
+- Migration 076: adds `extract_step_timings` for per-step Extract pipeline timing visibility. (Fixes #15)
+- `Task.idempotency_key_for`, `Task.find_active_by_idempotency_key`, and `Task.create_idempotent` for stable content-addressed task dispatch deduplication. (Fixes #14)
+- Extract results now include `extract_id` and `step_timings`, and persist timing rows when the migration is present. (Fixes #15)
+- `AuditLogHashChain` plus `AuditLog.compute_hash` / `AuditLog.verify_chain` as the canonical data-side audit log hash-chain implementation for standard write paths to share. (Refs #13)
+
+### Fixed
+- Migration 051 now adds SQLite/MySQL `tasks.created_at` without a non-constant default before backfilling from `created`, allowing later migration specs and fresh SQLite databases to migrate cleanly.
+
+## [1.7.2] - 2026-04-27
+
+### Fixed
+- Dev-fallback to SQLite now logs at `:error` level with explicit warnings that data written to SQLite will not be visible when the configured network database reconnects.
+
+### Added
+- `Connection.connection_info` — returns adapter, connection state, and fallback status for health checks and diagnostics
+- `Connection.fallback_active?` — returns true when the data layer fell back to SQLite from a configured network database; Apollo and other services can check this to detect degraded mode and log appropriate warnings
+
+## [1.7.1] - 2026-04-27
+
+### Fixed
+- `QueryFileLogger` now treats writes after `close` as no-ops, preventing repeated `IOError: closed stream` warnings from late Sequel query callbacks during shutdown. (Fixes #35)
+
+## [1.7.0] - 2026-04-24
+
+### Added
+- Migration 072: `identity_audit_log` table (Postgres-only) with indexes
+- Migration 073: `employee_id` on principals, `account_type`/`qualifier`/`is_default`/`link_evidence` on identities, partial unique index for one-default-per-provider
+- `IdentityAuditLog` model added to model loader
+
 ## [1.6.30] - 2026-04-22
 
 ### Fixed

data/README.md

@@ -139,12 +139,30 @@ MyMemoryTrace.all  # queries legionio_local.db, never the shared DB
 `Legion::Data::Extract` provides a handler registry for extracting text from documents, used by `lex-knowledge` for corpus ingestion:
 
 ```ruby
-text = Legion::Data::Extract.extract('/path/to/document.pdf')
-text = Legion::Data::Extract.extract('/path/to/data.csv')
+result = Legion::Data::Extract.extract('/path/to/document.pdf')
+text = result[:text]
+result[:step_timings] # per-step name, start_time, end_time, status, error, duration_ms
 ```
 
 Supported formats: `.txt`, `.md`, `.csv`, `.json`, `.jsonl`, `.html`, `.xlsx`, `.docx`, `.pdf`, `.pptx`, `.vtt`
 
+When migration 076 is present, Extract also persists the same per-step timing rows to `extract_step_timings`
+under the returned `extract_id`.
+
+### Task Idempotency
+
+`Task.idempotency_key_for` computes a stable SHA-256 key from canonical JSON payloads. `Task.create_idempotent`
+returns an existing non-terminal task for the same key inside the optional TTL window, or creates a new task
+with `idempotency_key` and `idempotency_expires_at` populated:
+
+```ruby
+task = Legion::Data::Model::Task.create_idempotent(
+  { status: 'pending', payload: Legion::JSON.dump(payload) },
+  payload: payload,
+  ttl: 300
+)
+```
+
 ### Filesystem Spool (Write Buffer)
 
 When the database is unavailable, `Legion::Data::Spool` buffers writes to `~/.legionio/data/spool/` and replays once the connection is restored:
@@ -343,6 +361,7 @@ Legion::Data.reload_static_cache
 | `Chain` | `chains` | Task execution chains |
 | `AuditLog` | `audit_log` | Tamper-evident audit trail with hash chain |
 | `AuditRecord` | `audit_records` | Structured audit records |
+| `ExtractStepTiming` | `extract_step_timings` | Per-step Extract pipeline timing metadata |
 | `RbacRoleAssignment` | `rbac_role_assignments` | RBAC principal -> role mappings |
 | `RbacRunnerGrant` | `rbac_runner_grants` | Per-runner permission grants |
 | `RbacCrossTeamGrant` | `rbac_cross_team_grants` | Cross-team access grants |
@@ -377,7 +396,7 @@ Apollo models require PostgreSQL with the `pgvector` extension. They are skipped
 
 ## Migrations
 
-71 numbered Sequel DSL migrations run automatically on startup (`auto_migrate: true`). Key milestones:
+76 numbered Sequel DSL migrations run automatically on startup (`auto_migrate: true`). Key milestones:
 
 | Range | What was added |
 |-------|---------------|
@@ -393,6 +412,7 @@ Apollo models require PostgreSQL with the `pgvector` extension. They are skipped
 | 050 | Critical indexes across 13 tables |
 | 058–067 | Audit records, chains, knowledge tiers, tool embedding cache, identity system (providers, principals, identities, groups) |
 | 068–071 | Entity type on audit records, principal on nodes, approval queue resume, engine on relationships |
+| 072–076 | Identity audit/multi-instance columns, Apollo identifier widening, task idempotency, Extract step timings |
 
 Run migrations standalone:
 

data/lib/legion/data/audit_log_hash_chain.rb

@@ -0,0 +1,85 @@
+# frozen_string_literal: true
+
+require 'digest'
+require 'legion/json'
+require 'time'
+
+module Legion
+  module Data
+    module AuditLogHashChain
+      GENESIS_HASH = ('0' * 64).freeze
+      CANONICAL_FIELDS = %i[
+        principal_id action resource source status detail created_at previous_hash
+      ].freeze
+
+      class << self
+        def compute_hash(record)
+          Digest::SHA256.hexdigest(canonical_payload(record))
+        end
+
+        def verify(records)
+          previous_hash = GENESIS_HASH
+          records.each do |record|
+            return invalid(record, :parent_mismatch) unless value_for(record, :previous_hash).to_s == previous_hash
+
+            expected = compute_hash(record)
+            return invalid(record, :hash_mismatch) unless value_for(record, :record_hash).to_s == expected
+
+            previous_hash = expected
+          end
+
+          { valid: true, length: records.size }
+        end
+
+        def canonical_payload(record)
+          CANONICAL_FIELDS.map do |field|
+            "#{field}:#{canonical_value(value_for(record, field))}"
+          end.join('|')
+        end
+
+        private
+
+        def invalid(record, reason)
+          { valid: false, broken_at: value_for(record, :id), reason: reason }
+        end
+
+        def canonical_value(value)
+          case value
+          when Time
+            value.utc.iso8601(6)
+          when DateTime
+            value.to_time.utc.iso8601(6)
+          when Hash
+            Legion::JSON.dump(canonical_hash(value))
+          when Array
+            Legion::JSON.dump(value.map { |item| canonical_json_value(item) })
+          else
+            value.to_s
+          end
+        end
+
+        def canonical_json_value(value)
+          case value
+          when Hash then canonical_hash(value)
+          when Array then value.map { |item| canonical_json_value(item) }
+          else value
+          end
+        end
+
+        def canonical_hash(hash)
+          hash.keys.map(&:to_s).sort.to_h do |key|
+            [key, canonical_json_value(hash.fetch(key) { hash.fetch(key.to_sym) })]
+          end
+        end
+
+        def value_for(record, field)
+          return record[field] if record.respond_to?(:[]) && !record[field].nil?
+          return record[field.to_s] if record.respond_to?(:[]) && !record[field.to_s].nil?
+          return record.public_send(field) if record.respond_to?(field)
+
+          nil
+        end
+      end
+    end
+  end
+end

data/lib/legion/data/connection.rb

@@ -103,12 +103,13 @@ module Legion
 
         def initialize(path)
           @path = path
+          @closed = false
+          @mutex = Mutex.new
           dir = File.dirname(path)
           FileUtils.mkdir_p(dir)
           FileUtils.chmod(0o700, dir) if File.directory?(dir)
           @file = File.open(path, File::WRONLY | File::APPEND | File::CREAT, 0o600)
           @file.sync = true
-          @mutex = Mutex.new
         end
 
         def debug(message)
@@ -128,16 +129,23 @@ module Legion
         end
 
         def close
-          @mutex.synchronize { @file.close unless @file.closed? }
+          @mutex.synchronize do
+            @closed = true
+            @file.close unless @file.closed?
+          end
         end
 
         private
 
         def write(level, message)
           @mutex.synchronize do
+            return if @closed || @file.closed?
+
             @file.puts "[#{Time.now.strftime('%Y-%m-%d %H:%M:%S.%L')}] #{level} #{message}"
           end
         rescue IOError => e
+          return nil if @closed || @file.closed?
+
           handle_exception(e, level: :warn, handled: true, operation: :query_file_write, path: @path)
           nil
         end
@@ -155,16 +163,22 @@ module Legion
         def setup
           opts = sequel_opts
           log.info("Legion::Data::Connection setup adapter=#{adapter}")
+          @fallback_active = false
           @sequel = if adapter == :sqlite
                       ::Sequel.connect(opts.merge(adapter: :sqlite, database: sqlite_path))
                     else
+                      attempted_adapter = adapter
                       begin
-                        ::Sequel.connect(connection_opts_for(adapter: adapter, opts: opts))
+                        ::Sequel.connect(connection_opts_for(adapter: attempted_adapter, opts: opts))
                       rescue StandardError => e
                         raise unless dev_fallback?
 
-                        handle_exception(e, level: :warn, handled: true, operation: :shared_connect, fallback: :sqlite)
+                        log.error("Legion::Data FALLING BACK TO SQLITE — #{attempted_adapter} connection failed: #{e.message}")
+                        log.error("Legion::Data WARNING: Data written to SQLite will NOT be visible when #{attempted_adapter} reconnects. " \
+                                  'Apollo knowledge, audit logs, and other DB-backed services will use a local-only store.')
+                        handle_exception(e, level: :error, handled: true, operation: :shared_connect, fallback: :sqlite)
                         @adapter = :sqlite
+                        @fallback_active = true
                         sqlite_opts = sequel_opts
                         ::Sequel.connect(sqlite_opts.merge(adapter: :sqlite, database: sqlite_path))
                       end
@@ -175,6 +189,25 @@ module Legion
           connect_with_replicas
         end
 
+        # Returns connection metadata for health checks and diagnostics.
+        # Apollo and other services can use this to detect silent fallback.
+        def connection_info
+          {
+            adapter:            adapter,
+            connected:          Legion::Settings[:data][:connected],
+            fallback_active:    @fallback_active || false,
+            configured_adapter: Legion::Settings[:data][:adapter]&.to_sym || :sqlite,
+            sequel_alive:       (begin; !@sequel&.test_connection.nil?; rescue StandardError; false; end)
+          }
+        end
+
+        # Returns true if the data layer fell back to SQLite from a configured
+        # network database (PostgreSQL/MySQL). Services should check this and
+        # log warnings when operating in degraded mode.
+        def fallback_active?
+          @fallback_active == true
+        end
+
         def stats
           return { connected: false } unless @sequel
 

data/lib/legion/data/extract.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'legion/logging/helper'
+require 'securerandom'
 require_relative 'extract/type_detector'
 require_relative 'extract/handlers/base'
 
@@ -11,29 +12,49 @@ module Legion
         include Legion::Logging::Helper
 
         def extract(source, type: :auto)
-          detected_type = type == :auto ? TypeDetector.detect(source) : type&.to_sym
-          return { success: false, text: nil, error: :unknown_type } unless detected_type
+          extract_id = SecureRandom.uuid
+          timings = []
+          detected_type = timed_step(:detect_type, timings) do
+            type == :auto ? TypeDetector.detect(source) : type&.to_sym
+          end
+          unless detected_type
+            result = { success: false, text: nil, error: :unknown_type, extract_id: extract_id,
+                       step_timings: timings }
+            persist_step_timings(extract_id, timings)
+            return result
+          end
 
-          handler = Handlers::Base.for_type(detected_type)
-          return { success: false, text: nil, error: :no_handler, type: detected_type } unless handler
+          handler = timed_step(:resolve_handler, timings) { Handlers::Base.for_type(detected_type) }
+          unless handler
+            result = { success: false, text: nil, error: :no_handler, type: detected_type, extract_id: extract_id,
+                       step_timings: timings }
+            persist_step_timings(extract_id, timings)
+            return result
+          end
 
-          unless handler.available?
+          available = timed_step(:check_availability, timings) { handler.available? }
+          unless available
             return { success: false, text: nil, error: :gem_not_installed,
-                     gem: handler.gem_name, type: detected_type }
+                     gem: handler.gem_name, type: detected_type, extract_id: extract_id,
+                     step_timings: timings }.tap { persist_step_timings(extract_id, timings) }
           end
 
           log.info "Extract starting type=#{detected_type} handler=#{handler.name}"
-          result = handler.extract(source)
+          result = timed_step(:handler_extract, timings) { handler.extract(source) }
           if result[:text]
             log.info "Extract succeeded type=#{detected_type}"
-            { success: true, text: result[:text], metadata: result[:metadata], type: detected_type }
+            { success: true, text: result[:text], metadata: result[:metadata], type: detected_type,
+              extract_id: extract_id, step_timings: timings }
           else
             log.warn "Extract failed type=#{detected_type} error=#{result[:error]}"
-            { success: false, text: nil, error: result[:error], type: detected_type }
-          end
+            { success: false, text: nil, error: result[:error], type: detected_type,
+              extract_id: extract_id, step_timings: timings }
+          end.tap { persist_step_timings(extract_id, timings) }
         rescue StandardError => e
           handle_exception(e, level: :error, handled: true, operation: :extract, type: detected_type)
-          { success: false, text: nil, error: e.message, type: detected_type }
+          persist_step_timings(extract_id, timings) if extract_id
+          { success: false, text: nil, error: e.message, type: detected_type, extract_id: extract_id,
+            step_timings: timings }
         end
 
         def supported_types
@@ -54,6 +75,48 @@ module Legion
 
         private
 
+        def timed_step(name, timings)
+          monotonic_start = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          start_time = Time.now.utc
+          result = yield
+          record_step_timing(timings, name: name, start_time: start_time, monotonic_start: monotonic_start,
+                                      status: :success)
+          result
+        rescue StandardError => e
+          record_step_timing(timings, name: name, start_time: start_time, monotonic_start: monotonic_start,
+                                      status: :error, error: "#{e.class}: #{e.message}")
+          raise
+        end
+
+        def record_step_timing(timings, name:, start_time:, monotonic_start:, status:, error: nil)
+          end_time = Time.now.utc
+          duration_ms = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - monotonic_start) * 1000).round
+          timings << {
+            name:        name.to_s,
+            start_time:  start_time,
+            end_time:    end_time,
+            status:      status.to_s,
+            error:       error,
+            duration_ms: duration_ms
+          }
+        end
+
+        def persist_step_timings(extract_id, timings)
+          return unless defined?(Legion::Data)
+
+          connection = Legion::Data.connection
+          return unless connection&.table_exists?(:extract_step_timings)
+
+          existing_steps = connection[:extract_step_timings].where(extract_id: extract_id).select_map(:name)
+          rows = timings.reject { |timing| existing_steps.include?(timing[:name]) }.map do |timing|
+            timing.merge(extract_id: extract_id)
+          end
+          connection[:extract_step_timings].multi_insert(rows) unless rows.empty?
+        rescue StandardError => e
+          handle_exception(e, level: :warn, handled: true, operation: :persist_extract_step_timings,
+                            extract_id: extract_id)
+        end
+
         def load_all_handlers
           return if @handlers_loaded
 

data/lib/legion/data/migrations/051_fix_tasks_created_at.rb

@@ -14,7 +14,7 @@ Sequel.migration do
     else
       # SQLite/MySQL: add real column and backfill from created
       alter_table(:tasks) do
-        add_column :created_at, DateTime, default: Sequel::CURRENT_TIMESTAMP
+        add_column :created_at, DateTime
       end
 
       run 'UPDATE tasks SET created_at = created WHERE created_at IS NULL'

data/lib/legion/data/migrations/072_create_identity_audit_log.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  up do
+    next unless adapter_scheme == :postgres
+
+    create_table(:identity_audit_log) do
+      column :id, :uuid, default: Sequel.lit('gen_random_uuid()'), primary_key: true
+      foreign_key :principal_id, :principals, type: :uuid, on_delete: :set_null
+      foreign_key :identity_id, :identities, type: :uuid, on_delete: :set_null
+      String :provider_name, null: false
+      String :event_type, null: false
+      String :trust_level
+      column :detail, :jsonb, null: false, default: Sequel.lit("'{}'")
+      String :node_id
+      String :session_id
+      DateTime :created_at, null: false, default: Sequel::CURRENT_TIMESTAMP
+    end
+
+    add_index :identity_audit_log, :principal_id
+    add_index :identity_audit_log, :event_type
+    add_index :identity_audit_log, :created_at
+    add_index :identity_audit_log, %i[principal_id event_type created_at]
+  end
+
+  down do
+    next unless adapter_scheme == :postgres
+
+    drop_table?(:identity_audit_log)
+  end
+end

data/lib/legion/data/migrations/073_add_identity_multi_instance_columns.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  up do
+    next unless adapter_scheme == :postgres
+
+    alter_table(:principals) do
+      add_column :employee_id, String
+    end
+    run 'CREATE INDEX idx_principals_employee_id ON principals (employee_id) WHERE employee_id IS NOT NULL'
+
+    alter_table(:identities) do
+      add_column :account_type, String, null: false, default: 'primary'
+      add_column :qualifier, String
+      add_column :is_default, TrueClass, null: false, default: false
+      add_column :link_evidence, String
+    end
+
+    run 'CREATE UNIQUE INDEX identities_one_default_per_provider ON identities (principal_id, provider_id) WHERE is_default = true AND active = true'
+  end
+
+  down do
+    next unless adapter_scheme == :postgres
+
+    run 'DROP INDEX IF EXISTS identities_one_default_per_provider'
+
+    alter_table(:identities) do
+      drop_column :link_evidence
+      drop_column :is_default
+      drop_column :qualifier
+      drop_column :account_type
+    end
+
+    run 'DROP INDEX IF EXISTS idx_principals_employee_id'
+    alter_table(:principals) do
+      drop_column :employee_id
+    end
+  end
+end

data/lib/legion/data/migrations/074_widen_apollo_entry_identifiers.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  up do
+    next unless adapter_scheme == :postgres
+    next unless table_exists?(:apollo_entries)
+
+    apollo_columns = schema(:apollo_entries).map(&:first)
+    alter_table(:apollo_entries) do
+      set_column_type :content_hash, String, fixed: true, size: 64 if apollo_columns.include?(:content_hash)
+      set_column_type :knowledge_domain, String, size: 255 if apollo_columns.include?(:knowledge_domain)
+      set_column_type :source_provider, String, size: 255 if apollo_columns.include?(:source_provider)
+      set_column_type :source_agent, String, size: 255 if apollo_columns.include?(:source_agent)
+    end
+
+    next unless table_exists?(:apollo_entries_archive)
+
+    archive_columns = schema(:apollo_entries_archive).map(&:first)
+    alter_table(:apollo_entries_archive) do
+      set_column_type :content_hash, String, fixed: true, size: 64 if archive_columns.include?(:content_hash)
+      set_column_type :knowledge_domain, String, size: 255 if archive_columns.include?(:knowledge_domain)
+      set_column_type :source_provider, String, size: 255 if archive_columns.include?(:source_provider)
+      set_column_type :source_agent, String, size: 255 if archive_columns.include?(:source_agent)
+    end
+  end
+
+  down do
+    next unless adapter_scheme == :postgres
+    next unless table_exists?(:apollo_entries)
+
+    apollo_columns = schema(:apollo_entries).map(&:first)
+    alter_table(:apollo_entries) do
+      set_column_type :content_hash, String, fixed: true, size: 32 if apollo_columns.include?(:content_hash)
+      set_column_type :knowledge_domain, String, size: 50 if apollo_columns.include?(:knowledge_domain)
+      set_column_type :source_provider, String, size: 50 if apollo_columns.include?(:source_provider)
+      set_column_type :source_agent, String, size: 50 if apollo_columns.include?(:source_agent)
+    end
+
+    next unless table_exists?(:apollo_entries_archive)
+
+    archive_columns = schema(:apollo_entries_archive).map(&:first)
+    alter_table(:apollo_entries_archive) do
+      set_column_type :content_hash, String, fixed: true, size: 32 if archive_columns.include?(:content_hash)
+      set_column_type :knowledge_domain, String, size: 50 if archive_columns.include?(:knowledge_domain)
+      set_column_type :source_provider, String, size: 50 if archive_columns.include?(:source_provider)
+      set_column_type :source_agent, String, size: 50 if archive_columns.include?(:source_agent)
+    end
+  end
+end

data/lib/legion/data/migrations/075_add_task_idempotency.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  up do
+    next unless table_exists?(:tasks)
+
+    existing_columns = schema(:tasks).map(&:first)
+    alter_table(:tasks) do
+      add_column :idempotency_key, String, size: 64 unless existing_columns.include?(:idempotency_key)
+      add_column :idempotency_expires_at, DateTime unless existing_columns.include?(:idempotency_expires_at)
+    end
+
+    add_index :tasks, :idempotency_key, name: :idx_tasks_idempotency_key, if_not_exists: true
+    add_index :tasks, :idempotency_expires_at, name: :idx_tasks_idempotency_expires_at, if_not_exists: true
+  end
+
+  down do
+    next unless table_exists?(:tasks)
+
+    existing_columns = schema(:tasks).map(&:first)
+    alter_table(:tasks) do
+      drop_index :idempotency_key, name: :idx_tasks_idempotency_key, if_exists: true
+      drop_index :idempotency_expires_at, name: :idx_tasks_idempotency_expires_at, if_exists: true
+      drop_column :idempotency_expires_at if existing_columns.include?(:idempotency_expires_at)
+      drop_column :idempotency_key if existing_columns.include?(:idempotency_key)
+    end
+  end
+end

data/lib/legion/data/migrations/076_create_extract_step_timings.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+Sequel.migration do
+  up do
+    create_table?(:extract_step_timings) do
+      primary_key :id
+      String :extract_id, size: 36, null: false
+      String :name, size: 100, null: false
+      DateTime :start_time, null: false
+      DateTime :end_time, null: false
+      String :status, size: 20, null: false
+      String :error, text: true
+      Integer :duration_ms, null: false, default: 0
+
+      index :extract_id, name: :idx_extract_step_timings_extract_id
+      index %i[extract_id name], name: :idx_extract_step_timings_extract_name
+      index :status, name: :idx_extract_step_timings_status
+    end
+  end
+
+  down do
+    drop_table?(:extract_step_timings)
+  end
+end

data/lib/legion/data/model.rb

@@ -14,7 +14,7 @@ module Legion
           %w[extension function relationship chain task runner node setting digital_worker
              apollo_entry apollo_relation apollo_expertise apollo_access_log audit_log
              audit_record identity_provider principal identity identity_group
-             identity_group_membership]
+             identity_group_membership identity_audit_log extract_step_timing]
         end
 
         def load

data/lib/legion/data/models/audit_log.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'legion/logging/helper'
+require 'legion/data/audit_log_hash_chain'
 
 module Legion
   module Data
@@ -33,6 +34,14 @@ module Legion
         def before_destroy
           raise 'audit_log records are immutable and cannot be deleted'
         end
+
+        def self.compute_hash(record)
+          Legion::Data::AuditLogHashChain.compute_hash(record)
+        end
+
+        def self.verify_chain(records = order(:created_at, :id).all)
+          Legion::Data::AuditLogHashChain.verify(records)
+        end
       end
     end
   end

data/lib/legion/data/models/extract_step_timing.rb

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+module Legion
+  module Data
+    module Model
+      class ExtractStepTiming < Sequel::Model(:extract_step_timings)
+      end
+    end
+  end
+end

data/lib/legion/data/models/identity_audit_log.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+return unless Legion::Data::Connection.adapter == :postgres
+
+module Legion
+  module Data
+    module Model
+      class IdentityAuditLog < Sequel::Model(:identity_audit_log)
+        many_to_one :principal, class: 'Legion::Data::Model::Principal'
+        many_to_one :identity, class: 'Legion::Data::Model::Identity'
+      end
+    end
+  end
+end

data/lib/legion/data/models/task.rb

@@ -1,9 +1,17 @@
 # frozen_string_literal: true
 
+require 'digest'
+require 'legion/json'
+require 'time'
+
 module Legion
   module Data
     module Model
       class Task < Sequel::Model
+        TERMINAL_STATUSES = %w[
+          completed complete failed error cancelled canceled timeout timed_out
+        ].freeze
+
         many_to_one :relationship
         one_to_many :task_log
         many_to_one :parent, class: self
@@ -11,9 +19,51 @@ module Legion
         many_to_one :master, class: self
         one_to_many :slave, key: :master_id, class: self
 
+        def self.idempotency_key_for(payload)
+          Digest::SHA256.hexdigest(Legion::JSON.dump(canonical_payload(payload)))
+        end
+
+        def self.find_active_by_idempotency_key(key, now: Time.now)
+          return nil if key.to_s.empty?
+          return nil unless columns.include?(:idempotency_key)
+
+          where(idempotency_key: key)
+            .exclude(status: TERMINAL_STATUSES)
+            .where { (idempotency_expires_at =~ nil) | (idempotency_expires_at > now) }
+            .reverse_order(:created, :id)
+            .first
+        end
+
+        def self.create_idempotent(values, payload: nil, idempotency_key: nil, ttl: nil)
+          key = idempotency_key || idempotency_key_for(payload || values)
+          existing = find_active_by_idempotency_key(key)
+          return existing if existing
+
+          expires_at = ttl ? Time.now + ttl : nil
+          create(values.merge(idempotency_key: key, idempotency_expires_at: expires_at))
+        end
+
         def cancelled?
           !cancelled_at.nil?
         end
+
+        def self.canonical_payload(value)
+          case value
+          when Hash
+            value.keys.map(&:to_s).sort.to_h do |key|
+              [key, canonical_payload(value.fetch(key) { value.fetch(key.to_sym) })]
+            end
+          when Array
+            value.map { |item| canonical_payload(item) }
+          when Time
+            value.utc.iso8601(6)
+          when DateTime
+            value.to_time.utc.iso8601(6)
+          else
+            value
+          end
+        end
+        private_class_method :canonical_payload
       end
     end
   end

data/lib/legion/data/version.rb

@@ -2,6 +2,6 @@
 
 module Legion
   module Data
-    VERSION = '1.6.30'
+    VERSION = '1.7.3'
   end
 end

data/lib/legion/data.rb

@@ -16,6 +16,7 @@ require_relative 'data/helper'
 require_relative 'data/rls'
 require_relative 'data/extract'
 require_relative 'data/audit_record'
+require_relative 'data/audit_log_hash_chain'
 
 unless Legion::Logging::Helper.method_defined?(:handle_exception)
   module Legion

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: legion-data
 version: !ruby/object:Gem::Version
-  version: 1.6.30
+  version: 1.7.3
 platform: ruby
 authors:
 - Esity
@@ -121,6 +121,7 @@ files:
 - lib/legion/data/archival.rb
 - lib/legion/data/archival/policy.rb
 - lib/legion/data/archiver.rb
+- lib/legion/data/audit_log_hash_chain.rb
 - lib/legion/data/audit_record.rb
 - lib/legion/data/connection.rb
 - lib/legion/data/encryption/cipher.rb
@@ -216,6 +217,11 @@ files:
 - lib/legion/data/migrations/069_add_principal_id_to_nodes.rb
 - lib/legion/data/migrations/070_add_approval_queue_resume.rb
 - lib/legion/data/migrations/071_add_engine_to_relationships.rb
+- lib/legion/data/migrations/072_create_identity_audit_log.rb
+- lib/legion/data/migrations/073_add_identity_multi_instance_columns.rb
+- lib/legion/data/migrations/074_widen_apollo_entry_identifiers.rb
+- lib/legion/data/migrations/075_add_task_idempotency.rb
+- lib/legion/data/migrations/076_create_extract_step_timings.rb
 - lib/legion/data/model.rb
 - lib/legion/data/models/apollo_access_log.rb
 - lib/legion/data/models/apollo_entry.rb
@@ -226,8 +232,10 @@ files:
 - lib/legion/data/models/chain.rb
 - lib/legion/data/models/digital_worker.rb
 - lib/legion/data/models/extension.rb
+- lib/legion/data/models/extract_step_timing.rb
 - lib/legion/data/models/function.rb
 - lib/legion/data/models/identity.rb
+- lib/legion/data/models/identity_audit_log.rb
 - lib/legion/data/models/identity_group.rb
 - lib/legion/data/models/identity_group_membership.rb
 - lib/legion/data/models/identity_provider.rb