legion-data 1.10.9 → 1.10.28

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8d929f0ed083e64145e6022ee40af61b677b852513e071ec9a986267837953a2
-  data.tar.gz: ed00e7cb0c7670378212cc1f8df39c893f0efbaf7a71961188a411a1956dcab9
+  metadata.gz: ec41889e66777b7940c00db2ed4f19ec704145ea27b149604ba82a168fed9e0b
+  data.tar.gz: a1d0eec9f3cd9e84d63a4b1eb4e145330b4cdd6354d742ca6f003ea78558ca4c
 SHA512:
-  metadata.gz: fdf3cf290669b970b523c3de8e5d6b42b9b14d877bd7260155c2067a708745cd59120605059dc1d5af85fd94f0a63d0c1e4e39ae4b7870409357572085b1d5f2
-  data.tar.gz: 3b1c0bb52ba3bfcc0249f6a6a4c0185b213f7e6cd8201989f5cd0279be1d9e1df1e364c0479d48920397f7e693ff5a63f557048acdaaefecdd32651e212922c7
+  metadata.gz: b62c13c70c6330f24c17c92c645887f7d506ac83b7978cd33226ca198b0294d49823c0143d778cb3cd03e73e508b87b4b90c1ed229f3fd558e7c5f4b7c9a8710
+  data.tar.gz: 3def1ace7b8172f1b8bd3b9ebd513fae7a65bc10cab304a5f6f7a593f6b44ef3431e93484c12af547b34961107fe02901d0930aefa6a8c8405501c95f7bf3de5

data/CHANGELOG.md

@@ -1,5 +1,106 @@
 # Legion::Data Changelog
 
+## [1.10.28] - 2026-10-30
+
+### Changed
+- Remove the unmerged task-idempotency claim migration and restore the existing task creation API without database claims. The concurrent-creation fix for #77 is withdrawn from this PR (#84).
+- Renumber the unmerged append-claims migration from 139 to 138, leaving a contiguous migration sequence without adding a compensating migration. Migrations already on main are unchanged.
+
+## [1.10.27] - 2026-10-30
+
+### Fixed
+- Remove unmerged migration 140 rather than conditionally taking ownership of a runtime-created archive. Verification handles hot-only history without creating schema and rechecks for an archive created during the first verification pass (#79, #84).
+- Preserve database errors other than a genuinely missing archive, including permission failures. Existing hot/warm verification remains a single ordered statement; migrations already on main are unchanged.
+
+## [1.10.26] - 2026-10-29
+
+### Fixed
+- Exercise concurrent SQLite audit/event appends with the same cooperative busy handler used by production connections, avoiding native busy-wait starvation in Ruby 3.4 CI while retaining lock-timeout behavior (#78).
+- Initialize concurrency-spec cleanup state before resource setup and guard an absent release queue so early setup failures are not masked by cleanup errors (#84).
+
+## [1.10.25] - 2026-10-29
+
+### Fixed
+- Archive generic retention backlogs in primary-key-ordered, bounded transactions. Copy locked rows and delete only their complete identities; verify deletion counts and roll back inconsistent batches. Independent workers cannot copy the same locked rows, and concurrent archive-table creation handles PostgreSQL catalog races (#82).
+- Add an optional positive `batch_size` (default 1000), require a declared source primary key, and support composite keys without exceeding SQLite predicate-depth limits. Completed batches survive later failures or process interruption; reruns move only remaining eligible records and report their aggregate count.
+
+## [1.10.24] - 2026-10-29
+
+### Fixed
+- Use the active adapter for privilege checks, including SQLite fallback. PostgreSQL uses native table privileges; MySQL uses nonmutating SELECT and EXPLAIN INSERT probes rather than PostgreSQL-only SQL. Failed queries return false without poisoning future checks (#81).
+- Cache successful privilege lookups per physical connection, invalidate on connection replacement and credential rotation, and prevent in-flight old sessions from overwriting replacement permissions.
+
+## [1.10.23] - 2026-10-29
+
+### Fixed
+- Claim spool entries with nonblocking cross-process file locks before reading, replaying, quarantining, or clearing them. Recheck descriptor identity after acquisition so an already-open deleted file cannot replay again. Reads and clears skip in-flight entries (#80).
+- Preserve callback exceptions and pending events for retry; operating-system lock release recovers crashed owners without stale claim files. Delivery remains at least once if a process exits after its side effect but before deleting the event.
+
+## [1.10.22] - 2026-10-29
+
+### Fixed
+- Verify the audit log across retained hot and warm rows in one ordered database statement, preserving genesis, predecessor, content-hash, and duplicate detection after repeated retention boundaries. Explicit record collections remain strictly verified as supplied; verification counts consumed rows without building an additional result array (#79).
+
+## [1.10.21] - 2026-10-29
+
+### Fixed
+- Serialize audit-chain and governance-stream appends with database claims, including first-entry races. Migration 138 adds scoped append claims. MySQL uses exclusive duplicate-key locking rather than shared-lock upgrades (#78).
+- Read locked tails on network adapters, retain database lock-timeout errors, and verify/walk audit records by append ID rather than writer-clock order. Unrelated insert failures roll back their claims and remain visible to callers.
+
+## [1.10.19] - 2026-10-29
+
+### Fixed
+- Drain database pools through tracked public checkout lifetimes across threaded and sharded pool implementations. Idle pools close immediately; shutdown returns false at its deadline while accepted work finishes, rejects new work, and closes connections and query logging only after the last checkout returns (#76).
+- Rebind loaded models and association caches to the current database on setup so a restart does not reopen a closed pool.
+
+## [1.10.18] - 2026-10-29
+
+### Fixed
+- Preserve task payloads, arguments, results, identities, links, and timestamps through scheduled/policy archival and restore. Copy and delete exact locked row IDs atomically; keep parents referenced by live tasks hot and roll back restores on missing dependencies or occupied IDs (#75).
+- Add migration 137 for missing task archive columns; existing migrations are unchanged. PostgreSQL generated timestamps are excluded from restore inserts.
+
+## [1.10.17] - 2026-10-29
+
+### Fixed
+- Cache the default encryption key under `nil`, separately from every tenant ID, including the literal `__default__`. Key selection no longer depends on which scope was requested first (#74).
+- Scope database spec cleanup to initialized resources so early setup errors are preserved and original state, including nil connection snapshots, is restored (#83).
+
+## [1.10.16] - 2026-10-29
+
+### Fixed
+- Generate each archive batch UUID before upload and use it in S3, Azure, and local object names and the manifest. Separate runs no longer overwrite earlier archives in the same month. Direct `upload_batch` callers now supply `batch_id:` instead of `batch_n:` (#73).
+- Load the archive compressor's StringIO dependency explicitly.
+
+## [1.10.15] - 2026-10-29
+
+### Fixed
+- Isolate query-log write and close I/O failures from database operations. Full disks no longer turn committed writes into apparent failures or replace the original SQL exception; logging failures remain reported through Legion::Logging (#72).
+
+## [1.10.14] - 2026-10-29
+
+### Fixed
+- Propagate fatal connection configuration, shared migration, model load, and local migration failures. Close failed setup connections, clear readiness, and allow setup to retry rather than continuing with partially initialized databases (#71).
+
+## [1.10.13] - 2026-10-29
+
+### Fixed
+- Build PostgreSQL replica pools before connecting, decode replica URLs into valid Sequel server options, and preserve inherited credentials and session timeouts. Clear replica routing on shutdown and reconfiguration (#70).
+
+## [1.10.12] - 2026-10-29
+
+### Fixed
+- Keep PostgreSQL tenant setup, scoped queries, and restoration on one checked-out connection. Preserve nested tenant scopes and clean up after exceptions without leaking tenant state to another borrower (#69).
+
+## [1.10.11] - 2026-10-29
+
+### Fixed
+- Apply PostgreSQL statement and lock timeouts before any connection is opened, including initial, preconnected, and replacement sessions (#68).
+
+## [1.10.10] - 2026-10-29
+
+### Fixed
+- Release Ruby threads while SQLite waits for write locks, with bounded per-connection busy handlers. Apply local SQLite tuning to every pooled connection and honor explicit `data.timeout` values (#67).
+
 ## [1.10.9] - 2026-07-31
 ### Fixed
 - **Postgres session timeouts**: Apply `statement_timeout` (default 5000ms) and `lock_timeout` (default 2000ms) via `after_connect` hook on every new Postgres connection. Prevents unbounded `SELECT` or lock-wait from hanging the process indefinitely — the root cause of ~1h50m connection wedges.

data/README.md

@@ -2,7 +2,7 @@
 
 Persistent database storage for the [LegionIO](https://github.com/LegionIO/LegionIO) async job engine and AI coding assistant platform. Provides database connectivity via the [Sequel ORM](https://sequel.jeremyevans.net/), automatic schema migrations (97 numbered migrations), Sequel models for the full LegionIO control plane, and a parallel local SQLite database for on-node agentic cognitive state.
 
-**Version**: 1.8.0 | **Ruby**: >= 3.4 | **License**: Apache-2.0
+**Version**: 1.10.18 | **Ruby**: >= 3.4 | **License**: Apache-2.0
 
 ---
 
@@ -23,6 +23,8 @@ Core responsibilities:
 
 The schema is portable by default across SQLite, MySQL, and PostgreSQL. PostgreSQL-only behavior is isolated to features that need PostgreSQL, such as Apollo vector columns.
 
+`Legion::Data.setup` raises on fatal connection configuration, migration, or model-loading failures rather than announcing successful startup. Failed setup closes its database connections and clears readiness; fix the underlying error before retrying. Local migrations registered after startup also propagate failures and leave the local database disconnected until setup succeeds.
+
 ## Supported Databases
 
 | Database | Adapter | Gem | Default |
@@ -71,7 +73,7 @@ Legion::Data (singleton module)
 │   ├── .connection_info  # Adapter, liveness, and fallback diagnostics
 │   ├── .fallback_active? # True when dev fallback moved a network DB to SQLite
 │   ├── .stats          # Pool metrics, tuning snapshot, adapter-specific DB stats
-│   └── .shutdown       # Disconnect and close query file logger
+│   └── .shutdown       # Drain accepted work; return true only after cleanup
 │
 ├── Migration           # Auto-migration system (97 numbered Sequel DSL migrations)
 │
@@ -149,6 +151,14 @@ Legion::Data::Connection.connection_info
 Legion::Data.shutdown
 ```
 
+### Graceful Database Shutdown
+
+`Legion::Data.shutdown` returns `true` when the shared pool has drained and its connections and query logger are closed. An idle pool closes immediately. Once shutdown starts, new checkouts are rejected; work already accepted by the pool may finish, including nested checkouts in the same execution context.
+
+If `shutdown_timeout` expires, shutdown returns `false` rather than claiming active connections were forcibly closed. The last accepted checkout performs deferred cleanup. `Connection.connection_info[:shutdown]` reports `state` (`open`, `draining`, `closing`, `closed`, or `failed`) and `active_requests`, which counts accepted execution contexts, including pool waiters—not physical connections. A cleanup failure is logged and can be retried by calling shutdown again.
+
+Shutdown does not cancel SQL, kill worker threads, or close borrowed connections. Configure database statement timeouts separately to bound query execution. To resume work after completed shutdown, call `Legion::Data.setup`; it creates a new pool and rebinds loaded models.
+
 ### Model Associations
 
 Models use Sequel associations as the public object graph. Prefer association methods and association datasets over hand-written foreign-key lookups when the relationship is part of the schema contract.
@@ -219,16 +229,48 @@ task = Legion::Data::Model::Task.create_idempotent(
 )
 ```
 
+Terminal tasks and expired TTLs allow a replacement. This database helper uses a lookup followed by an insert; concurrent callers are not guaranteed to receive the same task. It does not provide RabbitMQ delivery or execution deduplication.
+
+### Serialized Audit and Governance Appends
+
+After migration 138, `AuditRecord.append` serializes each chain and `EventStore.append` serializes each stream with database-backed claims, including empty chains. Different audit/event namespaces remain independent. Claim rows do not accumulate after transactions finish; failed appends roll back both claim and record.
+
+Audit tails, verification, and walks use append IDs rather than timestamps, preserving chain order across writer clock skew. Governance events retain their sequential stream numbering. PostgreSQL lock/statement timeouts and native adapter lock limits bound contention; timeout and unrelated database errors propagate rather than being retried indefinitely. MySQL claims use exclusive duplicate-key locking to avoid InnoDB shared-lock upgrade deadlocks.
+
+### Bounded Retention
+
+```ruby
+Legion::Data::Retention.archive_old_records(
+  table: :audit_log, archive_after_days: 90, batch_size: 1000
+) # => { archived: total_moved, table: :audit_log }
+```
+
+Source tables need a declared primary key, including composite keys. `batch_size` must be a positive integer and bounds each materialized batch. Rows are selected in primary-key order; PostgreSQL/MySQL lock them and SQLite uses an immediate write transaction. Each batch copies and deletes exactly the same identities atomically, including when other rows become eligible during the copy.
+
+Outside a caller-owned transaction, each batch commits independently. A later failure is raised without undoing earlier batches; rerun to archive the remaining eligible rows. A successful result sums that invocation's moved rows. If the caller wraps retention in its own transaction, that outer transaction still controls commit and lock lifetime.
+
+### Audit Verification After Retention
+
+`Legion::Data::Model::AuditLog.verify_chain` verifies hot `audit_log` rows together with retained `audit_log_archive` rows in one ordered SQL statement when the archive exists. Moving records between these tables does not discard their predecessors or create a mixed-snapshot boundary. Archived content is rehashed; missing predecessors, modified content, and duplicate copies remain verification failures.
+
+Passing an explicit record collection still verifies exactly that collection from genesis. Cold storage and purged history are not silently trusted: missing archived predecessors remain invalid until the complete history is supplied. This does not change LegionIO's separate tier-verification API.
+
+Before the first retention run, verification reads hot-only history without creating archive schema. It checks again afterward for concurrent archive creation and, if the archive appeared, verifies both tables together. Only an actual missing-table error means the archive is absent; permission, connection, and other database failures propagate.
+
 ### Filesystem Spool (Write Buffer)
 
 When the database is unavailable, `Legion::Data::Spool` buffers writes to `~/.legionio/data/spool/` and replays once the connection is restored:
 
 ```ruby
 spool = Legion::Data::Spool.for(Legion::Extensions::MyLex)
-spool.write({ task_id: SecureRandom.uuid, data: payload })
-spool.drain { |entry| process(entry) }
+spool.write(:events, { task_id: SecureRandom.uuid, data: payload })
+spool.flush(:events) { |entry| process(entry) }
 ```
 
+Replay claims each event with a nonblocking advisory file lock held through the callback and successful deletion. Concurrent processes cannot replay the same live claim. `read` and `clear` skip in-flight entries; `count` includes them until deletion. Corrupt files are quarantined under the same ownership protocol.
+
+Callback failure retains the event and releases its lock. Process exit also releases the lock automatically, so the next drainer can recover it without stale claim markers. Delivery is **at least once**: a crash after an external side effect but before deletion can replay that effect. Callbacks must provide their own idempotency. Single-drainer filename order is preserved, but concurrent completion order is not guaranteed. The spool filesystem must support cooperative advisory locking.
+
 ### Row-Level Security (PostgreSQL)
 
 `Legion::Data::Rls` provides tenant isolation via PostgreSQL session variables (migration 043):
@@ -239,6 +281,8 @@ Legion::Data::Rls.with_tenant(tenant_id) do
 end
 ```
 
+The scope holds one pooled connection until the block finishes and its prior tenant state has been restored, including when the block raises. Nested scopes restore the outer tenant. Run scoped database work in the calling thread; newly spawned threads do not inherit its tenant scope.
+
 ### Permission Checks
 
 ```ruby
@@ -247,6 +291,10 @@ Legion::Data.can_read?(:tasks)   # => true
 Legion::Data.reset_privileges!   # clear cached privilege checks
 ```
 
+Checks use the active database adapter, including SQLite fallback, and return false while disconnected. PostgreSQL checks table-level SELECT/INSERT grants. MySQL checks base tables with a zero-row SELECT and a nonexecuting `EXPLAIN INSERT`; checking a MySQL view also requires the privileges needed to explain that view, including SHOW VIEW for writes. Neither probe inserts data.
+
+Successful lookups are cached per physical connection. Connection replacement and credential rotation invalidate caches; query or connection failures return false without caching a denial. Call `reset_privileges!` after changing grants or roles on existing sessions.
+
 ---
 
 ## Configuration
@@ -266,6 +314,8 @@ All settings live under the `data` key. The adapter controls which options apply
 }
 ```
 
+SQLite lock waits use a Ruby busy handler that releases the GVL between retries, so other threads can finish transactions while a writer waits. `data.timeout` sets the wait limit in milliseconds for both databases; when unset, shared SQLite uses 5,000 ms and local SQLite uses 30,000 ms. Local connection-specific tuning is applied to every pooled connection. `PRAGMA busy_timeout` does not report the Ruby handler's deadline.
+
 ### MySQL
 
 ```json
@@ -300,6 +350,8 @@ All settings live under the `data` key. The adapter controls which options apply
 }
 ```
 
+`data.statement_timeout` (default 5,000 ms) and `data.lock_timeout` (default 2,000 ms) are applied before every PostgreSQL connection enters the pool, including startup and replacement connections. Set a value to `null` to leave that server setting unchanged, or `0` to explicitly disable the corresponding PostgreSQL timeout.
+
 PostgreSQL with `pgvector` is required for Apollo models:
 
 ```sql
@@ -390,6 +442,10 @@ Two independent caching tiers, both disabled by default:
 Legion::Data.reload_static_cache
 ```
 
+### Query File Logging
+
+Enable `data.query_log` (or `data.local.query_log`) to write SQL to a dedicated file. Write and close I/O failures, including a full disk, are reported through Legion::Logging without changing the database operation's result or replacing its exception. Opening the configured log file must still succeed during setup.
+
 ### Read Replicas (PostgreSQL)
 
 ```json
@@ -401,6 +457,22 @@ Legion::Data.reload_static_cache
 }
 ```
 
+Replica URLs initialize a sharded pool at startup. URL-specific options override primary connection options; omitted credentials and session timeouts are inherited. Duplicate URLs are registered once as `read_0`, `read_1`, and so on. Reads remain on the primary unless explicitly routed, for example `dataset.server(Legion::Data::Connection.read_server)`. With no replicas configured, `read_server` returns `:default`.
+
+### Task Database Archival
+
+`Archival.archive_completed_tasks` and task-policy archival retain every task field after migration 137. Copy/delete and restore are atomic; only copied task IDs are deleted. A task referenced as a parent or master by another live task remains hot, so linked task trees archive from leaves upward across batches.
+
+Restore preserves original IDs and restores parent/master links after inserting the requested rows. Include archived ancestors in the same restore request or restore them first. Occupied IDs or missing referenced functions/relationships cause rollback and leave the archive intact. Fields already discarded by older archival versions cannot be reconstructed by this migration.
+
+### Archive Objects
+
+Each archive batch receives a UUID before upload. S3, Azure, and local object paths use `legion-archive/<table>/<year>/<month>/batch_<uuid>.jsonl.gz`; the same UUID is stored in `archive_manifest.batch_id`. Repeated runs do not reuse numbered object names. Existing manifest paths remain valid; no stored objects are renamed. Direct `Archiver.upload_batch` callers must supply `batch_id:` instead of `batch_n:`.
+
+### Encryption Key Cache
+
+The default scope (`tenant_id: nil`) and tenant IDs have separate cache entries, including a tenant literally named `__default__`. Restart processes or clear existing key-provider caches when upgrading. This fix does not rewrite existing ciphertext: if that tenant name was used with the old cache, assess records encrypted with the wrong scope's key and re-encrypt them with the intended key. The reader does not fall back to another scope's key.
+
 ---
 
 ## Common Fields Standard

data/lib/legion/data/archival.rb

@@ -10,6 +10,11 @@ module Legion
         tasks:            :tasks_archive,
         metering_records: :metering_records_archive
       }.freeze
+      COLUMN_MAP = { id: :original_id, created_at: :original_created_at, updated_at: :original_updated_at }.freeze
+      TASK_COLUMN_MAP = {
+        id: :original_id, created: :original_created_at, updated: :original_updated_at,
+        function_args: :args, results: :result
+      }.freeze
 
       class << self
         include Legion::Logging::Helper
@@ -42,17 +47,17 @@ module Legion
           return 0 unless archive_table && db_ready?(archive_table)
 
           conn = Legion::Data.connection
-          restored = 0
-          conn.transaction do
-            conn[archive_table].where(original_id: ids).each do |row|
-              restore_row = row.except(:id, :archived_at, :original_id, :original_created_at, :original_updated_at)
-              restore_row[:id] = row[:original_id]
-              restore_row[:created_at] = row[:original_created_at]
-              restore_row[:updated_at] = row[:original_updated_at]
-              conn[source_table].insert(restore_row)
-              restored += 1
+          restored = conn.transaction(savepoint: true, **(conn.database_type == :sqlite ? { mode: :immediate } : {})) do
+            dataset = conn[archive_table].where(original_id: ids).order(:original_id)
+            dataset = dataset.for_update unless conn.database_type == :sqlite
+            archived = dataset.all
+            rows = restore_row_values(conn: conn, source: source_table, archived: archived)
+            rows.each do |row|
+              conn[source_table].insert(source_table == :tasks ? row.merge(parent_id: nil, master_id: nil) : row)
             end
-            conn[archive_table].where(original_id: ids).delete
+            rows.each { |row| conn[:tasks].where(id: row[:id]).update(row.slice(:parent_id, :master_id)) } if source_table == :tasks
+            conn[archive_table].where(id: archived.map { |row| row[:id] }).delete
+            rows.size
           end
           log.info "Restored #{restored} row(s) from #{archive_table} -> #{source_table}"
           restored
@@ -82,30 +87,8 @@ module Legion
 
           return { archived: 0, cutoff: cutoff.iso8601 } unless conn&.table_exists?(:tasks) && conn.table_exists?(:tasks_archive)
 
-          candidates = conn[:tasks]
-                       .where(status: %w[completed failed])
-                       .where(Sequel.lit('created < ?', cutoff))
-                       .limit(batch_size)
-
-          count = candidates.count
-          if count.positive?
-            archive_cols = conn.schema(:tasks_archive).to_set(&:first)
-            conn.transaction do
-              candidates.each do |row|
-                archive_row = {
-                  original_id:         row[:id],
-                  status:              row[:status],
-                  relationship_id:     row[:relationship_id],
-                  original_created_at: row[:created],
-                  original_updated_at: row[:updated],
-                  archived_at:         Time.now
-                }
-                archive_row[:archive_reason] = 'completed_task_archival' if archive_cols.include?(:archive_reason)
-                conn[:tasks_archive].insert(archive_row)
-              end
-              conn[:tasks].where(id: candidates.select(:id)).delete
-            end
-          end
+          count = archive_table!(source: :tasks, destination: :tasks_archive, cutoff: cutoff,
+                                 batch_size: batch_size, dry_run: false, statuses: %w[completed failed])
 
           log.info "archive_completed_tasks: archived #{count} tasks (cutoff: #{cutoff.iso8601})"
           { archived: count, cutoff: cutoff.iso8601 }
@@ -135,24 +118,40 @@ module Legion
 
         private
 
-        def archive_table!(source:, destination:, cutoff:, batch_size:, dry_run:)
+        def archive_table!(source:, destination:, cutoff:, batch_size:, dry_run:, statuses: nil)
           conn = Legion::Data.connection
-          candidates = conn[source].where { created_at < cutoff }.limit(batch_size)
-          count = candidates.count
-          return count if dry_run || count.zero?
-
-          conn.transaction do
-            candidates.each do |row|
-              archive_row = row.dup
-              archive_row[:original_id] = archive_row.delete(:id)
-              archive_row[:original_created_at] = archive_row.delete(:created_at)
-              archive_row[:original_updated_at] = archive_row.delete(:updated_at)
-              archive_row[:archived_at] = Time.now
-              conn[destination].insert(archive_row)
+          date_column = Policy::DATE_COLUMN_OVERRIDES.fetch(source.to_s, :created_at)
+          candidates = conn[source].where(Sequel[date_column] < cutoff).order(:id).limit(batch_size)
+          candidates = candidates.where(status: statuses) if statuses
+          if source == :tasks
+            references = conn.from(Sequel.as(:tasks, :dependent_tasks))
+                             .where(Sequel.|({ parent_id: Sequel[:tasks][:id] }, { master_id: Sequel[:tasks][:id] }))
+                             .exclude(id: Sequel[:tasks][:id]).exists
+            candidates = candidates.exclude(references)
+          end
+          return candidates.count if dry_run
+
+          mapping = source == :tasks ? TASK_COLUMN_MAP : COLUMN_MAP
+          conn.transaction(savepoint: true, **(conn.database_type == :sqlite ? { mode: :immediate } : {})) do
+            locked = conn.database_type == :sqlite ? candidates : candidates.for_update
+            selected_ids = locked.select_map(:id)
+            selected = conn[source].where(id: selected_ids).order(:id)
+            selected = selected.exclude(references) if source == :tasks
+            selected = selected.for_update unless conn.database_type == :sqlite
+            rows = selected.all
+            rows.each do |row|
+              archive_row = row.to_h { |column, value| [mapping.fetch(column, column), value] }
+              conn[destination].insert(archive_row.merge(archived_at: Time.now))
             end
-            conn[source].where(id: candidates.select(:id)).delete
+            conn[source].where(id: rows.map { |row| row[:id] }).delete
+            rows.size
           end
-          count
+        end
+
+        def restore_row_values(conn:, source:, archived:)
+          mapping = source == :tasks ? TASK_COLUMN_MAP : COLUMN_MAP
+          columns = conn.schema(source).reject { |_, info| info[:generated] }.map(&:first)
+          archived.map { |row| columns.to_h { |column| [column, row[mapping.fetch(column, column)]] } }
         end
 
         def db_ready?(table)

data/lib/legion/data/archiver.rb

@@ -5,6 +5,7 @@ require 'digest'
 require 'fileutils'
 require 'json'
 require 'securerandom'
+require 'stringio'
 require 'tmpdir'
 require 'zlib'
 
@@ -47,15 +48,15 @@ module Legion
           raise
         end
 
-        def upload_batch(data:, table:, year:, month:, batch_n:, backend:)
-          log.info "Archiver storing batch table=#{table} backend=#{backend || :tmpdir} year=#{year} month=#{month} batch=#{batch_n}"
+        def upload_batch(data:, table:, year:, month:, batch_id:, backend:)
+          log.info "Archiver storing batch table=#{table} backend=#{backend || :tmpdir} year=#{year} month=#{month} batch=#{batch_id}"
           case backend
           when :s3
-            upload_s3(data: data, table: table, year: year, month: month, batch_n: batch_n)
+            upload_s3(data: data, table: table, year: year, month: month, batch_id: batch_id)
           when :azure
-            upload_azure(data: data, table: table, year: year, month: month, batch_n: batch_n)
+            upload_azure(data: data, table: table, year: year, month: month, batch_id: batch_id)
           else
-            upload_tmpdir(data: data, table: table, year: year, month: month, batch_n: batch_n)
+            upload_tmpdir(data: data, table: table, year: year, month: month, batch_id: batch_id)
           end
         end
 
@@ -102,7 +103,6 @@ module Legion
             table:           table,
             cutoff:          cutoff,
             batch_size:      batch_size,
-            batch_n:         batches + 1,
             now:             now,
             storage_backend: storage_backend
           ))
@@ -114,18 +114,19 @@ module Legion
           { batches: batches, total_rows: total_rows, paths: paths }
         end
 
-        def archive_batch(conn:, table:, cutoff:, batch_size:, batch_n:, now:, storage_backend:)
+        def archive_batch(conn:, table:, cutoff:, batch_size:, now:, storage_backend:)
           rows = conn[table].where { created_at < cutoff }.limit(batch_size).all
           return if rows.empty?
 
+          batch_id = SecureRandom.uuid
           compressed = gzip_compress(serialize_rows(rows))
           path = upload_batch(
-            data:    compressed,
-            table:   table.to_s,
-            year:    now.year,
-            month:   now.month,
-            batch_n: batch_n,
-            backend: storage_backend
+            data:     compressed,
+            table:    table.to_s,
+            year:     now.year,
+            month:    now.month,
+            batch_id: batch_id,
+            backend:  storage_backend
           )
 
           record_archived_batch(
@@ -134,16 +135,17 @@ module Legion
             rows:       rows,
             compressed: compressed,
             path:       path,
+            batch_id:   batch_id,
             now:        now
           )
 
           { row_count: rows.size, path: path }
         end
 
-        def record_archived_batch(conn:, table:, rows:, compressed:, path:, now:)
+        def record_archived_batch(conn:, table:, rows:, compressed:, path:, batch_id:, now:)
           conn.transaction do
             conn[:archive_manifest].insert(
-              batch_id:     SecureRandom.uuid,
+              batch_id:     batch_id,
               source_table: table.to_s,
               row_count:    rows.size,
               checksum:     Digest::SHA256.hexdigest(compressed),
@@ -167,47 +169,47 @@ module Legion
           output.string
         end
 
-        def upload_s3(data:, table:, year:, month:, batch_n:)
+        def upload_s3(data:, table:, year:, month:, batch_id:)
           raise UploadError, 'S3 backend not available: Legion::Extensions::S3::Runners::Put not defined' unless defined?(Legion::Extensions::S3::Runners::Put)
 
-          key = "legion-archive/#{table}/#{year}/#{month}/batch_#{batch_n}.jsonl.gz"
+          key = "legion-archive/#{table}/#{year}/#{month}/batch_#{batch_id}.jsonl.gz"
           Legion::Extensions::S3::Runners::Put.run(key: key, body: data)
           log.info "Archiver uploaded batch to s3 key=#{key}"
           "s3://#{key}"
         rescue UploadError => e
-          handle_exception(e, level: :error, handled: false, operation: :upload_s3, table: table, year: year, month: month, batch_n: batch_n)
+          handle_exception(e, level: :error, handled: false, operation: :upload_s3, table: table, year: year, month: month, batch_id: batch_id)
           raise
         rescue StandardError => e
-          handle_exception(e, level: :error, handled: true, operation: :upload_s3, table: table, year: year, month: month, batch_n: batch_n)
+          handle_exception(e, level: :error, handled: true, operation: :upload_s3, table: table, year: year, month: month, batch_id: batch_id)
           raise UploadError, "S3 upload failed: #{e.message}"
         end
 
-        def upload_azure(data:, table:, year:, month:, batch_n:)
+        def upload_azure(data:, table:, year:, month:, batch_id:)
           unless defined?(Legion::Extensions::AzureStorage::Runners::Upload)
             raise UploadError, 'Azure backend not available: Legion::Extensions::AzureStorage::Runners::Upload not defined'
           end
 
-          blob_name = "legion-archive/#{table}/#{year}/#{month}/batch_#{batch_n}.jsonl.gz"
+          blob_name = "legion-archive/#{table}/#{year}/#{month}/batch_#{batch_id}.jsonl.gz"
           Legion::Extensions::AzureStorage::Runners::Upload.run(blob_name: blob_name, data: data)
           log.info "Archiver uploaded batch to azure blob=#{blob_name}"
           "azure://#{blob_name}"
         rescue UploadError => e
-          handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month, batch_n: batch_n)
+          handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month, batch_id: batch_id)
           raise
         rescue StandardError => e
-          handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month, batch_n: batch_n)
+          handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month, batch_id: batch_id)
           raise UploadError, "Azure upload failed: #{e.message}"
         end
 
-        def upload_tmpdir(data:, table:, year:, month:, batch_n:)
+        def upload_tmpdir(data:, table:, year:, month:, batch_id:)
           dir = File.join(Dir.tmpdir, 'legion-archive', table.to_s, year.to_s, month.to_s)
           FileUtils.mkdir_p(dir)
-          path = File.join(dir, "batch_#{batch_n}.jsonl.gz")
+          path = File.join(dir, "batch_#{batch_id}.jsonl.gz")
           File.binwrite(path, data)
           log.info "Archiver stored batch locally path=#{path}"
           "file://#{path}"
         rescue StandardError => e
-          handle_exception(e, level: :error, handled: true, operation: :upload_tmpdir, table: table, year: year, month: month, batch_n: batch_n)
+          handle_exception(e, level: :error, handled: true, operation: :upload_tmpdir, table: table, year: year, month: month, batch_id: batch_id)
           raise UploadError, "Tmpdir upload failed: #{e.message}"
         end
       end

data/lib/legion/data/audit_log_hash_chain.rb

@@ -19,6 +19,7 @@ module Legion
 
         def verify(records)
           previous_hash = GENESIS_HASH
+          length = 0
           records.each do |record|
             return invalid(record, :parent_mismatch) unless value_for(record, :previous_hash).to_s == previous_hash
 
@@ -26,9 +27,10 @@ module Legion
             return invalid(record, :hash_mismatch) unless value_for(record, :record_hash).to_s == expected
 
             previous_hash = expected
+            length += 1
           end
 
-          { valid: true, length: records.size }
+          { valid: true, length: length }
         end
 
         def canonical_payload(record)

data/lib/legion/data/audit_record.rb

@@ -2,6 +2,7 @@
 
 require 'legion/logging/helper'
 require 'digest'
+require 'legion/data/database_lock'
 
 module Legion
   module Data
@@ -23,7 +24,8 @@ module Legion
           return { error: 'db unavailable' } unless db_ready?
 
           conn = Legion::Data.connection
-          conn.transaction do
+          Legion::Data::DatabaseLock.with_claim(database: conn, table: :append_claims,
+                                                key: { kind: 'audit', stream_id: chain_id }) do
             parent_hash = latest_chain_hash(conn, chain_id)
             ts          = truncate_to_us(Time.now)
             ch          = compute_chain_hash(parent_hash, content_hash, ts, content_type)
@@ -46,7 +48,7 @@ module Legion
           end
         end
 
-        # Walk all records in the chain ordered by creation time and verify that
+        # Walk all records in append order and verify that
         # each record's stored chain_hash matches a freshly computed one.
         #
         # @param chain_id [String]
@@ -56,7 +58,7 @@ module Legion
 
           records = Legion::Data.connection[:audit_records]
                                 .where(chain_id: chain_id)
-                                .order(:created_at, :id)
+                                .order(:id)
                                 .all
 
           prev_hash = GENESIS_HASH
@@ -88,7 +90,7 @@ module Legion
 
           ds = Legion::Data.connection[:audit_records].where(chain_id: chain_id)
           ds = ds.where { created_at >= since } if since
-          ds.order(:created_at, :id).limit(limit).all.map { |r| deserialize(r) }
+          ds.order(:id).limit(limit).all.map { |r| deserialize(r) }
         end
 
         # Return records filtered by content_type across all chains.
@@ -136,11 +138,12 @@ module Legion
         end
 
         def latest_chain_hash(conn, chain_id)
-          last = conn[:audit_records]
-                 .select(:chain_hash)
-                 .where(chain_id: chain_id)
-                 .order(Sequel.desc(:created_at), Sequel.desc(:id))
-                 .first
+          records = conn[:audit_records]
+                    .select(:chain_hash)
+                    .where(chain_id: chain_id)
+                    .order(Sequel.desc(:id))
+          records = records.for_update unless conn.database_type == :sqlite
+          last = records.first
           last ? last[:chain_hash] : GENESIS_HASH
         end