legion-data 1.10.9 → 1.10.17

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 8d929f0ed083e64145e6022ee40af61b677b852513e071ec9a986267837953a2
4
- data.tar.gz: ed00e7cb0c7670378212cc1f8df39c893f0efbaf7a71961188a411a1956dcab9
3
+ metadata.gz: cc2433eff09e371c9b406ba8790fe20da105a2920fb5490947ae590158eee403
4
+ data.tar.gz: 6d4ec2038079bbf0f01be66b2093313af264651e69ff7d38bf0a89a1f8656b43
5
5
  SHA512:
6
- metadata.gz: fdf3cf290669b970b523c3de8e5d6b42b9b14d877bd7260155c2067a708745cd59120605059dc1d5af85fd94f0a63d0c1e4e39ae4b7870409357572085b1d5f2
7
- data.tar.gz: 3b1c0bb52ba3bfcc0249f6a6a4c0185b213f7e6cd8201989f5cd0279be1d9e1df1e364c0479d48920397f7e693ff5a63f557048acdaaefecdd32651e212922c7
6
+ metadata.gz: fdf39d3c2f11766069af4432713e353e87196478585d1c3fe782f1db0aa298f4c0520b579dadb8a343e2b74afdfba485619c29d23188340b45eab641ea726a73
7
+ data.tar.gz: 4a92b1cd8ab39a28fb9b144286602a851a99d64f468678cc704da1fdb79001cfd60354778da7ea41e45c6f300241ae025fe537743c507763b3d2c0967dd8522b
data/CHANGELOG.md CHANGED
@@ -1,5 +1,47 @@
1
1
  # Legion::Data Changelog
2
2
 
3
+ ## [1.10.17] - 2026-10-29
4
+
5
+ ### Fixed
6
+ - Cache the default encryption key under `nil`, separately from every tenant ID, including the literal `__default__`. Key selection no longer depends on which scope was requested first (#74).
7
+ - Scope database spec cleanup to initialized resources so early setup errors are preserved and original state, including nil connection snapshots, is restored (#83).
8
+
9
+ ## [1.10.16] - 2026-10-29
10
+
11
+ ### Fixed
12
+ - Generate each archive batch UUID before upload and use it in S3, Azure, and local object names and the manifest. Separate runs no longer overwrite earlier archives in the same month. Direct `upload_batch` callers now supply `batch_id:` instead of `batch_n:` (#73).
13
+ - Load the archive compressor's StringIO dependency explicitly.
14
+
15
+ ## [1.10.15] - 2026-10-29
16
+
17
+ ### Fixed
18
+ - Isolate query-log write and close I/O failures from database operations. Full disks no longer turn committed writes into apparent failures or replace the original SQL exception; logging failures remain reported through Legion::Logging (#72).
19
+
20
+ ## [1.10.14] - 2026-10-29
21
+
22
+ ### Fixed
23
+ - Propagate fatal connection configuration, shared migration, model load, and local migration failures. Close failed setup connections, clear readiness, and allow setup to retry rather than continuing with partially initialized databases (#71).
24
+
25
+ ## [1.10.13] - 2026-10-29
26
+
27
+ ### Fixed
28
+ - Build PostgreSQL replica pools before connecting, decode replica URLs into valid Sequel server options, and preserve inherited credentials and session timeouts. Clear replica routing on shutdown and reconfiguration (#70).
29
+
30
+ ## [1.10.12] - 2026-10-29
31
+
32
+ ### Fixed
33
+ - Keep PostgreSQL tenant setup, scoped queries, and restoration on one checked-out connection. Preserve nested tenant scopes and clean up after exceptions without leaking tenant state to another borrower (#69).
34
+
35
+ ## [1.10.11] - 2026-10-29
36
+
37
+ ### Fixed
38
+ - Apply PostgreSQL statement and lock timeouts before any connection is opened, including initial, preconnected, and replacement sessions (#68).
39
+
40
+ ## [1.10.10] - 2026-10-29
41
+
42
+ ### Fixed
43
+ - Release Ruby threads while SQLite waits for write locks, with bounded per-connection busy handlers. Apply local SQLite tuning to every pooled connection and honor explicit `data.timeout` values (#67).
44
+
3
45
  ## [1.10.9] - 2026-07-31
4
46
  ### Fixed
5
47
  - **Postgres session timeouts**: Apply `statement_timeout` (default 5000ms) and `lock_timeout` (default 2000ms) via `after_connect` hook on every new Postgres connection. Prevents unbounded `SELECT` or lock-wait from hanging the process indefinitely — the root cause of ~1h50m connection wedges.
data/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  Persistent database storage for the [LegionIO](https://github.com/LegionIO/LegionIO) async job engine and AI coding assistant platform. Provides database connectivity via the [Sequel ORM](https://sequel.jeremyevans.net/), automatic schema migrations (97 numbered migrations), Sequel models for the full LegionIO control plane, and a parallel local SQLite database for on-node agentic cognitive state.
4
4
 
5
- **Version**: 1.8.0 | **Ruby**: >= 3.4 | **License**: Apache-2.0
5
+ **Version**: 1.10.17 | **Ruby**: >= 3.4 | **License**: Apache-2.0
6
6
 
7
7
  ---
8
8
 
@@ -23,6 +23,8 @@ Core responsibilities:
23
23
 
24
24
  The schema is portable by default across SQLite, MySQL, and PostgreSQL. PostgreSQL-only behavior is isolated to features that need PostgreSQL, such as Apollo vector columns.
25
25
 
26
+ `Legion::Data.setup` raises on fatal connection configuration, migration, or model-loading failures rather than announcing successful startup. Failed setup closes its database connections and clears readiness; fix the underlying error before retrying. Local migrations registered after startup also propagate failures and leave the local database disconnected until setup succeeds.
27
+
26
28
  ## Supported Databases
27
29
 
28
30
  | Database | Adapter | Gem | Default |
@@ -239,6 +241,8 @@ Legion::Data::Rls.with_tenant(tenant_id) do
239
241
  end
240
242
  ```
241
243
 
244
+ The scope holds one pooled connection until the block finishes and its prior tenant state has been restored, including when the block raises. Nested scopes restore the outer tenant. Run scoped database work in the calling thread; newly spawned threads do not inherit its tenant scope.
245
+
242
246
  ### Permission Checks
243
247
 
244
248
  ```ruby
@@ -266,6 +270,8 @@ All settings live under the `data` key. The adapter controls which options apply
266
270
  }
267
271
  ```
268
272
 
273
+ SQLite lock waits use a Ruby busy handler that releases the GVL between retries, so other threads can finish transactions while a writer waits. `data.timeout` sets the wait limit in milliseconds for both databases; when unset, shared SQLite uses 5,000 ms and local SQLite uses 30,000 ms. Local connection-specific tuning is applied to every pooled connection. `PRAGMA busy_timeout` does not report the Ruby handler's deadline.
274
+
269
275
  ### MySQL
270
276
 
271
277
  ```json
@@ -300,6 +306,8 @@ All settings live under the `data` key. The adapter controls which options apply
300
306
  }
301
307
  ```
302
308
 
309
+ `data.statement_timeout` (default 5,000 ms) and `data.lock_timeout` (default 2,000 ms) are applied before every PostgreSQL connection enters the pool, including startup and replacement connections. Set a value to `null` to leave that server setting unchanged, or `0` to explicitly disable the corresponding PostgreSQL timeout.
310
+
303
311
  PostgreSQL with `pgvector` is required for Apollo models:
304
312
 
305
313
  ```sql
@@ -390,6 +398,10 @@ Two independent caching tiers, both disabled by default:
390
398
  Legion::Data.reload_static_cache
391
399
  ```
392
400
 
401
+ ### Query File Logging
402
+
403
+ Enable `data.query_log` (or `data.local.query_log`) to write SQL to a dedicated file. Write and close I/O failures, including a full disk, are reported through Legion::Logging without changing the database operation's result or replacing its exception. Opening the configured log file must still succeed during setup.
404
+
393
405
  ### Read Replicas (PostgreSQL)
394
406
 
395
407
  ```json
@@ -401,6 +413,16 @@ Legion::Data.reload_static_cache
401
413
  }
402
414
  ```
403
415
 
416
+ Replica URLs initialize a sharded pool at startup. URL-specific options override primary connection options; omitted credentials and session timeouts are inherited. Duplicate URLs are registered once as `read_0`, `read_1`, and so on. Reads remain on the primary unless explicitly routed, for example `dataset.server(Legion::Data::Connection.read_server)`. With no replicas configured, `read_server` returns `:default`.
417
+
418
+ ### Archive Objects
419
+
420
+ Each archive batch receives a UUID before upload. S3, Azure, and local object paths use `legion-archive/<table>/<year>/<month>/batch_<uuid>.jsonl.gz`; the same UUID is stored in `archive_manifest.batch_id`. Repeated runs do not reuse numbered object names. Existing manifest paths remain valid; no stored objects are renamed. Direct `Archiver.upload_batch` callers must supply `batch_id:` instead of `batch_n:`.
421
+
422
+ ### Encryption Key Cache
423
+
424
+ The default scope (`tenant_id: nil`) and tenant IDs have separate cache entries, including a tenant literally named `__default__`. Restart processes or clear existing key-provider caches when upgrading. This fix does not rewrite existing ciphertext: if that tenant name was used with the old cache, assess records encrypted with the wrong scope's key and re-encrypt them with the intended key. The reader does not fall back to another scope's key.
425
+
404
426
  ---
405
427
 
406
428
  ## Common Fields Standard
@@ -5,6 +5,7 @@ require 'digest'
5
5
  require 'fileutils'
6
6
  require 'json'
7
7
  require 'securerandom'
8
+ require 'stringio'
8
9
  require 'tmpdir'
9
10
  require 'zlib'
10
11
 
@@ -47,15 +48,15 @@ module Legion
47
48
  raise
48
49
  end
49
50
 
50
- def upload_batch(data:, table:, year:, month:, batch_n:, backend:)
51
- log.info "Archiver storing batch table=#{table} backend=#{backend || :tmpdir} year=#{year} month=#{month} batch=#{batch_n}"
51
+ def upload_batch(data:, table:, year:, month:, batch_id:, backend:)
52
+ log.info "Archiver storing batch table=#{table} backend=#{backend || :tmpdir} year=#{year} month=#{month} batch=#{batch_id}"
52
53
  case backend
53
54
  when :s3
54
- upload_s3(data: data, table: table, year: year, month: month, batch_n: batch_n)
55
+ upload_s3(data: data, table: table, year: year, month: month, batch_id: batch_id)
55
56
  when :azure
56
- upload_azure(data: data, table: table, year: year, month: month, batch_n: batch_n)
57
+ upload_azure(data: data, table: table, year: year, month: month, batch_id: batch_id)
57
58
  else
58
- upload_tmpdir(data: data, table: table, year: year, month: month, batch_n: batch_n)
59
+ upload_tmpdir(data: data, table: table, year: year, month: month, batch_id: batch_id)
59
60
  end
60
61
  end
61
62
 
@@ -102,7 +103,6 @@ module Legion
102
103
  table: table,
103
104
  cutoff: cutoff,
104
105
  batch_size: batch_size,
105
- batch_n: batches + 1,
106
106
  now: now,
107
107
  storage_backend: storage_backend
108
108
  ))
@@ -114,18 +114,19 @@ module Legion
114
114
  { batches: batches, total_rows: total_rows, paths: paths }
115
115
  end
116
116
 
117
- def archive_batch(conn:, table:, cutoff:, batch_size:, batch_n:, now:, storage_backend:)
117
+ def archive_batch(conn:, table:, cutoff:, batch_size:, now:, storage_backend:)
118
118
  rows = conn[table].where { created_at < cutoff }.limit(batch_size).all
119
119
  return if rows.empty?
120
120
 
121
+ batch_id = SecureRandom.uuid
121
122
  compressed = gzip_compress(serialize_rows(rows))
122
123
  path = upload_batch(
123
- data: compressed,
124
- table: table.to_s,
125
- year: now.year,
126
- month: now.month,
127
- batch_n: batch_n,
128
- backend: storage_backend
124
+ data: compressed,
125
+ table: table.to_s,
126
+ year: now.year,
127
+ month: now.month,
128
+ batch_id: batch_id,
129
+ backend: storage_backend
129
130
  )
130
131
 
131
132
  record_archived_batch(
@@ -134,16 +135,17 @@ module Legion
134
135
  rows: rows,
135
136
  compressed: compressed,
136
137
  path: path,
138
+ batch_id: batch_id,
137
139
  now: now
138
140
  )
139
141
 
140
142
  { row_count: rows.size, path: path }
141
143
  end
142
144
 
143
- def record_archived_batch(conn:, table:, rows:, compressed:, path:, now:)
145
+ def record_archived_batch(conn:, table:, rows:, compressed:, path:, batch_id:, now:)
144
146
  conn.transaction do
145
147
  conn[:archive_manifest].insert(
146
- batch_id: SecureRandom.uuid,
148
+ batch_id: batch_id,
147
149
  source_table: table.to_s,
148
150
  row_count: rows.size,
149
151
  checksum: Digest::SHA256.hexdigest(compressed),
@@ -167,47 +169,47 @@ module Legion
167
169
  output.string
168
170
  end
169
171
 
170
- def upload_s3(data:, table:, year:, month:, batch_n:)
172
+ def upload_s3(data:, table:, year:, month:, batch_id:)
171
173
  raise UploadError, 'S3 backend not available: Legion::Extensions::S3::Runners::Put not defined' unless defined?(Legion::Extensions::S3::Runners::Put)
172
174
 
173
- key = "legion-archive/#{table}/#{year}/#{month}/batch_#{batch_n}.jsonl.gz"
175
+ key = "legion-archive/#{table}/#{year}/#{month}/batch_#{batch_id}.jsonl.gz"
174
176
  Legion::Extensions::S3::Runners::Put.run(key: key, body: data)
175
177
  log.info "Archiver uploaded batch to s3 key=#{key}"
176
178
  "s3://#{key}"
177
179
  rescue UploadError => e
178
- handle_exception(e, level: :error, handled: false, operation: :upload_s3, table: table, year: year, month: month, batch_n: batch_n)
180
+ handle_exception(e, level: :error, handled: false, operation: :upload_s3, table: table, year: year, month: month, batch_id: batch_id)
179
181
  raise
180
182
  rescue StandardError => e
181
- handle_exception(e, level: :error, handled: true, operation: :upload_s3, table: table, year: year, month: month, batch_n: batch_n)
183
+ handle_exception(e, level: :error, handled: true, operation: :upload_s3, table: table, year: year, month: month, batch_id: batch_id)
182
184
  raise UploadError, "S3 upload failed: #{e.message}"
183
185
  end
184
186
 
185
- def upload_azure(data:, table:, year:, month:, batch_n:)
187
+ def upload_azure(data:, table:, year:, month:, batch_id:)
186
188
  unless defined?(Legion::Extensions::AzureStorage::Runners::Upload)
187
189
  raise UploadError, 'Azure backend not available: Legion::Extensions::AzureStorage::Runners::Upload not defined'
188
190
  end
189
191
 
190
- blob_name = "legion-archive/#{table}/#{year}/#{month}/batch_#{batch_n}.jsonl.gz"
192
+ blob_name = "legion-archive/#{table}/#{year}/#{month}/batch_#{batch_id}.jsonl.gz"
191
193
  Legion::Extensions::AzureStorage::Runners::Upload.run(blob_name: blob_name, data: data)
192
194
  log.info "Archiver uploaded batch to azure blob=#{blob_name}"
193
195
  "azure://#{blob_name}"
194
196
  rescue UploadError => e
195
- handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month, batch_n: batch_n)
197
+ handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month, batch_id: batch_id)
196
198
  raise
197
199
  rescue StandardError => e
198
- handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month, batch_n: batch_n)
200
+ handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month, batch_id: batch_id)
199
201
  raise UploadError, "Azure upload failed: #{e.message}"
200
202
  end
201
203
 
202
- def upload_tmpdir(data:, table:, year:, month:, batch_n:)
204
+ def upload_tmpdir(data:, table:, year:, month:, batch_id:)
203
205
  dir = File.join(Dir.tmpdir, 'legion-archive', table.to_s, year.to_s, month.to_s)
204
206
  FileUtils.mkdir_p(dir)
205
- path = File.join(dir, "batch_#{batch_n}.jsonl.gz")
207
+ path = File.join(dir, "batch_#{batch_id}.jsonl.gz")
206
208
  File.binwrite(path, data)
207
209
  log.info "Archiver stored batch locally path=#{path}"
208
210
  "file://#{path}"
209
211
  rescue StandardError => e
210
- handle_exception(e, level: :error, handled: true, operation: :upload_tmpdir, table: table, year: year, month: month, batch_n: batch_n)
212
+ handle_exception(e, level: :error, handled: true, operation: :upload_tmpdir, table: table, year: year, month: month, batch_id: batch_id)
211
213
  raise UploadError, "Tmpdir upload failed: #{e.message}"
212
214
  end
213
215
  end
@@ -5,6 +5,7 @@ require 'legion/data/auth_failure_handler'
5
5
 
6
6
  require 'fileutils'
7
7
  require 'sequel'
8
+ require 'uri'
8
9
 
9
10
  module Legion
10
11
  module Data
@@ -142,6 +143,9 @@ module Legion
142
143
  @closed = true
143
144
  @file.close unless @file.closed?
144
145
  end
146
+ rescue IOError, SystemCallError => e
147
+ handle_exception(e, level: :warn, handled: true, operation: :query_file_close, path: @path)
148
+ nil
145
149
  end
146
150
 
147
151
  private
@@ -152,9 +156,7 @@ module Legion
152
156
 
153
157
  @file.puts "[#{Time.now.strftime('%Y-%m-%d %H:%M:%S.%L')}] #{level} #{message}"
154
158
  end
155
- rescue IOError => e
156
- return nil if @closed || @file.closed?
157
-
159
+ rescue IOError, SystemCallError => e
158
160
  handle_exception(e, level: :warn, handled: true, operation: :query_file_write, path: @path)
159
161
  nil
160
162
  end
@@ -170,10 +172,14 @@ module Legion
170
172
  end
171
173
 
172
174
  def setup
175
+ Legion::Settings[:data][:connected] = false
176
+ @sequel&.disconnect
177
+ @sequel = nil
173
178
  @adapter = Legion::Settings[:data][:adapter]&.to_sym || :sqlite
174
179
  opts = sequel_opts
175
180
  log.info("Legion::Data::Connection setup adapter=#{adapter}")
176
181
  @fallback_active = false
182
+ @replica_servers = []
177
183
  @sequel = if adapter == :sqlite
178
184
  ::Sequel.connect(opts.merge(adapter: :sqlite, database: sqlite_path))
179
185
  else
@@ -183,7 +189,7 @@ module Legion
183
189
  rescue UnresolvedCredentialError => e
184
190
  handle_exception(e, level: :fatal, handled: false, operation: :data_connect)
185
191
  raise
186
- rescue StandardError => e
192
+ rescue ::Sequel::DatabaseConnectionError => e
187
193
  raise unless dev_fallback?
188
194
 
189
195
  log.error("Legion::Data FALLING BACK TO SQLITE — #{attempted_adapter} network DB connection failed: #{e.message}")
@@ -196,10 +202,15 @@ module Legion
196
202
  ::Sequel.connect(sqlite_opts.merge(adapter: :sqlite, database: sqlite_path))
197
203
  end
198
204
  end
199
- Legion::Settings[:data][:connected] = true
205
+ @replica_servers = @sequel.opts.fetch(:servers, {}).keys
200
206
  log_connection_info
201
207
  configure_extensions
202
- connect_with_replicas
208
+ Legion::Settings[:data][:connected] = true
209
+ rescue StandardError, LoadError => e
210
+ shutdown
211
+ @sequel = nil
212
+ handle_exception(e, level: :fatal, handled: false, operation: :connection_setup)
213
+ raise
203
214
  end
204
215
 
205
216
  # Returns connection metadata for health checks and diagnostics.
@@ -283,6 +294,7 @@ module Legion
283
294
  @query_file_logger&.close
284
295
  @query_file_logger = nil
285
296
  @fallback_active = false
297
+ @replica_servers = []
286
298
  Legion::Settings[:data][:connected] = false
287
299
  log.info 'Legion::Data connection closed'
288
300
  end
@@ -328,28 +340,6 @@ module Legion
328
340
  handle_exception(e, level: :warn, handled: true, operation: :expire_all_pooled_connections)
329
341
  end
330
342
 
331
- def connect_with_replicas
332
- return unless adapter == :postgres
333
-
334
- replica_url = Legion::Settings[:data][:read_replica_url]
335
- replica_list = Array(Legion::Settings[:data][:replicas]).dup
336
-
337
- replica_list.prepend(replica_url) if replica_url && !replica_url.empty?
338
- replica_list.uniq!
339
- replica_list.compact!
340
-
341
- return if replica_list.empty?
342
-
343
- @sequel.extension(:server_block)
344
-
345
- replica_list.each_with_index do |url, idx|
346
- @sequel.add_servers("read_#{idx}": url)
347
- end
348
-
349
- @replica_servers = replica_list.each_with_index.map { |_, idx| :"read_#{idx}" }
350
- log.debug "Registered #{@replica_servers.size} read replica(s)"
351
- end
352
-
353
343
  def read_server
354
344
  return :default if @replica_servers.nil? || @replica_servers.empty?
355
345
 
@@ -386,6 +376,20 @@ module Legion
386
376
  creds
387
377
  end
388
378
 
379
+ def configure_sqlite_busy_handler(connection:, timeout:)
380
+ timeout_seconds = timeout.to_f / 1000
381
+ deadline = nil
382
+ connection.busy_handler do |attempt|
383
+ now = ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
384
+ deadline = now + timeout_seconds if attempt.zero?
385
+ remaining = deadline - now
386
+ next false unless remaining.positive?
387
+
388
+ sleep([0.001, remaining].min)
389
+ true
390
+ end
391
+ end
392
+
389
393
  def creds_builder(final_creds = {})
390
394
  final_creds.merge! Legion::Data::Settings.creds(adapter)
391
395
  final_creds.merge! Legion::Settings[:data][:creds] if Legion::Settings[:data][:creds].is_a? Hash
@@ -472,9 +476,25 @@ module Legion
472
476
  connection_opts = opts.merge(adapter: adapter, **creds_builder)
473
477
  validate_no_unresolved_uris!(connection_opts)
474
478
  connection_opts[:preconnect] = false if adapter != :sqlite && dev_fallback?
479
+ if adapter == :postgres
480
+ servers = postgres_replica_servers
481
+ connection_opts[:servers] = servers unless servers.empty?
482
+ end
475
483
  connection_opts
476
484
  end
477
485
 
486
+ def postgres_replica_servers
487
+ data = Legion::Settings[:data]
488
+ urls = [data[:read_replica_url], *Array(data[:replicas])].compact.reject(&:empty?).uniq
489
+ urls.each_with_index.to_h do |url, index|
490
+ uri = URI.parse(url)
491
+ raise ArgumentError, 'Read replicas require PostgreSQL URLs' unless %w[postgres postgresql].include?(uri.scheme)
492
+
493
+ options = ::Sequel::Database.send(:options_from_uri, uri).compact
494
+ [:"read_#{index}", options]
495
+ end
496
+ end
497
+
478
498
  def validate_no_unresolved_uris!(connection_opts)
479
499
  %i[user username password].each do |key|
480
500
  value = connection_opts[key]
@@ -518,10 +538,18 @@ module Legion
518
538
  val = data.key?(key) && !data[key].nil? ? data[key] : defaults[key]
519
539
  opts[key] = val unless val.nil?
520
540
  end
541
+ opts[:after_connect] = proc { |connection| configure_sqlite_busy_handler(connection: connection, timeout: opts[:timeout]) } if adapter == :sqlite
542
+ opts[:connect_sqls] = postgres_connect_sqls(data) if adapter == :postgres
521
543
 
522
544
  opts
523
545
  end
524
546
 
547
+ def postgres_connect_sqls(data)
548
+ %i[statement_timeout lock_timeout].filter_map do |setting|
549
+ "SET #{setting} = '#{data[setting].to_i}ms'" if data[setting]
550
+ end
551
+ end
552
+
525
553
  def tuning_stats(data)
526
554
  tuning = {}
527
555
 
@@ -655,7 +683,7 @@ module Legion
655
683
  if adapter == :postgres
656
684
  Sequel.extension(:pg_array)
657
685
  @sequel.extension(:pg_array)
658
- install_postgres_session_timeouts
686
+ @sequel.extension(:server_block) unless @replica_servers.empty?
659
687
  end
660
688
 
661
689
  if data[:connection_validation] != false
@@ -670,28 +698,15 @@ module Legion
670
698
 
671
699
  install_auth_failure_hook
672
700
  rescue StandardError => e
673
- handle_exception(e, level: :warn, handled: true, operation: :configure_extensions, adapter: adapter)
674
- end
675
-
676
- def install_postgres_session_timeouts
677
- data = Legion::Settings[:data]
678
- stmt_timeout = data[:statement_timeout]
679
- lck_timeout = data[:lock_timeout]
680
-
681
- @sequel.pool.after_connect = proc do |conn|
682
- conn.exec("SET statement_timeout = '#{stmt_timeout.to_i}ms'") if stmt_timeout
683
- conn.exec("SET lock_timeout = '#{lck_timeout.to_i}ms'") if lck_timeout
684
- end
685
-
686
- log.info("Postgres session timeouts: statement_timeout=#{stmt_timeout}ms, lock_timeout=#{lck_timeout}ms")
687
- rescue StandardError => e
688
- handle_exception(e, level: :warn, handled: true, operation: :install_postgres_session_timeouts)
701
+ handle_exception(e, level: :error, handled: false, operation: :configure_extensions, adapter: adapter)
702
+ raise
689
703
  end
690
704
 
691
705
  def install_auth_failure_hook
692
706
  Legion::Data::AuthFailureHandler.install(@sequel)
693
707
  rescue StandardError => e
694
- handle_exception(e, level: :warn, handled: true, operation: :install_auth_failure_hook)
708
+ handle_exception(e, level: :error, handled: false, operation: :install_auth_failure_hook)
709
+ raise
695
710
  end
696
711
 
697
712
  def build_data_logger
@@ -15,8 +15,7 @@ module Legion
15
15
  end
16
16
 
17
17
  def key_for(tenant_id: nil)
18
- cache_key = tenant_id || '__default__'
19
- @key_cache[cache_key] ||= derive_key(tenant_id)
18
+ @key_cache[tenant_id] ||= derive_key(tenant_id)
20
19
  end
21
20
 
22
21
  def clear_cache!
@@ -32,6 +32,11 @@ module Legion
32
32
  val = data.key?(key) && !data[key].nil? ? data[key] : sqlite_defaults[key]
33
33
  opts[key] = val unless val.nil?
34
34
  end
35
+ opts[:timeout] = data[:timeout] || 30_000
36
+ opts[:after_connect] = proc do |connection|
37
+ Legion::Data::Connection.configure_sqlite_busy_handler(connection: connection, timeout: opts[:timeout])
38
+ connection.execute_batch('PRAGMA synchronous=NORMAL; PRAGMA cache_size=-20000; PRAGMA mmap_size=268435456')
39
+ end
35
40
 
36
41
  if local_settings[:query_log]
37
42
  log_path = File.join(Legion::Data::Connection::QUERY_LOG_DIR, 'data-local-query.log')
@@ -46,14 +51,11 @@ module Legion
46
51
 
47
52
  @connection = ::Sequel.connect(opts)
48
53
  @connection.run('PRAGMA journal_mode=WAL')
49
- @connection.run('PRAGMA busy_timeout=30000')
50
- @connection.run('PRAGMA synchronous=NORMAL')
51
- @connection.run('PRAGMA cache_size=-20000')
52
- @connection.run('PRAGMA mmap_size=268435456')
53
- @connected = true
54
54
  run_migrations
55
- log.info "Legion::Data::Local connected to #{db_file} (WAL mode, 30s busy_timeout, 20MB cache, 256MB mmap)"
56
- rescue StandardError => e
55
+ @connected = true
56
+ log.info "Legion::Data::Local connected to #{db_file} (WAL mode, #{opts[:timeout]}ms busy timeout, 20MB cache, 256MB mmap)"
57
+ rescue StandardError, LoadError => e
58
+ shutdown
57
59
  handle_exception(e, level: :error, handled: false, operation: :local_setup, database: db_file)
58
60
  raise
59
61
  end
@@ -74,6 +76,10 @@ module Legion
74
76
  @registered_migrations ||= {}
75
77
  @registered_migrations[name] = path
76
78
  run_single_migration(name, path) if connected?
79
+ rescue StandardError, LoadError => e
80
+ shutdown
81
+ handle_exception(e, level: :error, handled: false, operation: :register_migrations, name: name, path: path)
82
+ raise
77
83
  end
78
84
 
79
85
  def registered_migrations
@@ -142,8 +148,9 @@ module Legion
142
148
 
143
149
  table = :"schema_migrations_#{name}"
144
150
  ::Sequel::TimestampMigrator.new(@connection, path, table: table).run
145
- rescue StandardError => e
146
- handle_exception(e, level: :warn, handled: true, operation: :local_migration, name: name, path: path)
151
+ rescue StandardError, LoadError => e
152
+ handle_exception(e, level: :error, handled: false, operation: :local_migration, name: name, path: path)
153
+ raise
147
154
  end
148
155
 
149
156
  def local_settings
@@ -47,11 +47,17 @@ module Legion
47
47
  end
48
48
 
49
49
  def with_tenant(tenant_id)
50
- previous = current_tenant
51
- assign_tenant(tenant_id)
52
- yield
53
- ensure
54
- previous ? assign_tenant(previous) : reset_tenant
50
+ return yield unless rls_enabled?
51
+
52
+ Legion::Data.connection.synchronize do
53
+ previous = current_tenant
54
+ begin
55
+ assign_tenant(tenant_id)
56
+ yield
57
+ ensure
58
+ previous ? assign_tenant(previous) : reset_tenant
59
+ end
60
+ end
55
61
  end
56
62
  end
57
63
  end
@@ -2,6 +2,6 @@
2
2
 
3
3
  module Legion
4
4
  module Data
5
- VERSION = '1.10.9'
5
+ VERSION = '1.10.17'
6
6
  end
7
7
  end
data/lib/legion/data.rb CHANGED
@@ -48,6 +48,11 @@ module Legion
48
48
  setup_cache
49
49
  setup_local
50
50
  log.info 'Legion::Data setup complete'
51
+ rescue StandardError, LoadError => e
52
+ shutdown
53
+ Legion::Data::Connection.sequel = nil
54
+ handle_exception(e, level: :fatal, handled: false, operation: :setup)
55
+ raise
51
56
  end
52
57
 
53
58
  def setup_local
@@ -62,10 +67,19 @@ module Legion
62
67
 
63
68
  def setup_global
64
69
  connection_setup
70
+ Legion::Settings[:data][:connected] = false
71
+ Legion::Settings[:data][:migrations][:ran] = false
72
+ Legion::Settings[:data][:models][:loaded] = false
65
73
  migrate
66
74
  load_models
67
- rescue StandardError => e
68
- handle_exception(e, level: :fatal, operation: :setup_global)
75
+ Legion::Settings[:data][:connected] = true
76
+ rescue StandardError, LoadError => e
77
+ Legion::Data::Connection.shutdown
78
+ Legion::Data::Connection.sequel = nil
79
+ Legion::Settings[:data][:migrations][:ran] = false
80
+ Legion::Settings[:data][:models][:loaded] = false
81
+ handle_exception(e, level: :fatal, handled: false, operation: :setup_global)
82
+ raise
69
83
  end
70
84
 
71
85
  def setup_cache
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: legion-data
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.10.9
4
+ version: 1.10.17
5
5
  platform: ruby
6
6
  authors:
7
7
  - Esity