legion-data 1.10.9 → 1.10.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +42 -0
- data/README.md +23 -1
- data/lib/legion/data/archiver.rb +28 -26
- data/lib/legion/data/connection.rb +61 -46
- data/lib/legion/data/encryption/key_provider.rb +1 -2
- data/lib/legion/data/local.rb +16 -9
- data/lib/legion/data/rls.rb +11 -5
- data/lib/legion/data/version.rb +1 -1
- data/lib/legion/data.rb +16 -2
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: cc2433eff09e371c9b406ba8790fe20da105a2920fb5490947ae590158eee403
|
|
4
|
+
data.tar.gz: 6d4ec2038079bbf0f01be66b2093313af264651e69ff7d38bf0a89a1f8656b43
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: fdf39d3c2f11766069af4432713e353e87196478585d1c3fe782f1db0aa298f4c0520b579dadb8a343e2b74afdfba485619c29d23188340b45eab641ea726a73
|
|
7
|
+
data.tar.gz: 4a92b1cd8ab39a28fb9b144286602a851a99d64f468678cc704da1fdb79001cfd60354778da7ea41e45c6f300241ae025fe537743c507763b3d2c0967dd8522b
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,47 @@
|
|
|
1
1
|
# Legion::Data Changelog
|
|
2
2
|
|
|
3
|
+
## [1.10.17] - 2026-10-29
|
|
4
|
+
|
|
5
|
+
### Fixed
|
|
6
|
+
- Cache the default encryption key under `nil`, separately from every tenant ID, including the literal `__default__`. Key selection no longer depends on which scope was requested first (#74).
|
|
7
|
+
- Scope database spec cleanup to initialized resources so early setup errors are preserved and original state, including nil connection snapshots, is restored (#83).
|
|
8
|
+
|
|
9
|
+
## [1.10.16] - 2026-10-29
|
|
10
|
+
|
|
11
|
+
### Fixed
|
|
12
|
+
- Generate each archive batch UUID before upload and use it in S3, Azure, and local object names and the manifest. Separate runs no longer overwrite earlier archives in the same month. Direct `upload_batch` callers now supply `batch_id:` instead of `batch_n:` (#73).
|
|
13
|
+
- Load the archive compressor's StringIO dependency explicitly.
|
|
14
|
+
|
|
15
|
+
## [1.10.15] - 2026-10-29
|
|
16
|
+
|
|
17
|
+
### Fixed
|
|
18
|
+
- Isolate query-log write and close I/O failures from database operations. Full disks no longer turn committed writes into apparent failures or replace the original SQL exception; logging failures remain reported through Legion::Logging (#72).
|
|
19
|
+
|
|
20
|
+
## [1.10.14] - 2026-10-29
|
|
21
|
+
|
|
22
|
+
### Fixed
|
|
23
|
+
- Propagate fatal connection configuration, shared migration, model load, and local migration failures. Close failed setup connections, clear readiness, and allow setup to retry rather than continuing with partially initialized databases (#71).
|
|
24
|
+
|
|
25
|
+
## [1.10.13] - 2026-10-29
|
|
26
|
+
|
|
27
|
+
### Fixed
|
|
28
|
+
- Build PostgreSQL replica pools before connecting, decode replica URLs into valid Sequel server options, and preserve inherited credentials and session timeouts. Clear replica routing on shutdown and reconfiguration (#70).
|
|
29
|
+
|
|
30
|
+
## [1.10.12] - 2026-10-29
|
|
31
|
+
|
|
32
|
+
### Fixed
|
|
33
|
+
- Keep PostgreSQL tenant setup, scoped queries, and restoration on one checked-out connection. Preserve nested tenant scopes and clean up after exceptions without leaking tenant state to another borrower (#69).
|
|
34
|
+
|
|
35
|
+
## [1.10.11] - 2026-10-29
|
|
36
|
+
|
|
37
|
+
### Fixed
|
|
38
|
+
- Apply PostgreSQL statement and lock timeouts before any connection is opened, including initial, preconnected, and replacement sessions (#68).
|
|
39
|
+
|
|
40
|
+
## [1.10.10] - 2026-10-29
|
|
41
|
+
|
|
42
|
+
### Fixed
|
|
43
|
+
- Release Ruby threads while SQLite waits for write locks, with bounded per-connection busy handlers. Apply local SQLite tuning to every pooled connection and honor explicit `data.timeout` values (#67).
|
|
44
|
+
|
|
3
45
|
## [1.10.9] - 2026-07-31
|
|
4
46
|
### Fixed
|
|
5
47
|
- **Postgres session timeouts**: Apply `statement_timeout` (default 5000ms) and `lock_timeout` (default 2000ms) via `after_connect` hook on every new Postgres connection. Prevents unbounded `SELECT` or lock-wait from hanging the process indefinitely — the root cause of ~1h50m connection wedges.
|
data/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Persistent database storage for the [LegionIO](https://github.com/LegionIO/LegionIO) async job engine and AI coding assistant platform. Provides database connectivity via the [Sequel ORM](https://sequel.jeremyevans.net/), automatic schema migrations (97 numbered migrations), Sequel models for the full LegionIO control plane, and a parallel local SQLite database for on-node agentic cognitive state.
|
|
4
4
|
|
|
5
|
-
**Version**: 1.
|
|
5
|
+
**Version**: 1.10.17 | **Ruby**: >= 3.4 | **License**: Apache-2.0
|
|
6
6
|
|
|
7
7
|
---
|
|
8
8
|
|
|
@@ -23,6 +23,8 @@ Core responsibilities:
|
|
|
23
23
|
|
|
24
24
|
The schema is portable by default across SQLite, MySQL, and PostgreSQL. PostgreSQL-only behavior is isolated to features that need PostgreSQL, such as Apollo vector columns.
|
|
25
25
|
|
|
26
|
+
`Legion::Data.setup` raises on fatal connection configuration, migration, or model-loading failures rather than announcing successful startup. Failed setup closes its database connections and clears readiness; fix the underlying error before retrying. Local migrations registered after startup also propagate failures and leave the local database disconnected until setup succeeds.
|
|
27
|
+
|
|
26
28
|
## Supported Databases
|
|
27
29
|
|
|
28
30
|
| Database | Adapter | Gem | Default |
|
|
@@ -239,6 +241,8 @@ Legion::Data::Rls.with_tenant(tenant_id) do
|
|
|
239
241
|
end
|
|
240
242
|
```
|
|
241
243
|
|
|
244
|
+
The scope holds one pooled connection until the block finishes and its prior tenant state has been restored, including when the block raises. Nested scopes restore the outer tenant. Run scoped database work in the calling thread; newly spawned threads do not inherit its tenant scope.
|
|
245
|
+
|
|
242
246
|
### Permission Checks
|
|
243
247
|
|
|
244
248
|
```ruby
|
|
@@ -266,6 +270,8 @@ All settings live under the `data` key. The adapter controls which options apply
|
|
|
266
270
|
}
|
|
267
271
|
```
|
|
268
272
|
|
|
273
|
+
SQLite lock waits use a Ruby busy handler that releases the GVL between retries, so other threads can finish transactions while a writer waits. `data.timeout` sets the wait limit in milliseconds for both databases; when unset, shared SQLite uses 5,000 ms and local SQLite uses 30,000 ms. Local connection-specific tuning is applied to every pooled connection. `PRAGMA busy_timeout` does not report the Ruby handler's deadline.
|
|
274
|
+
|
|
269
275
|
### MySQL
|
|
270
276
|
|
|
271
277
|
```json
|
|
@@ -300,6 +306,8 @@ All settings live under the `data` key. The adapter controls which options apply
|
|
|
300
306
|
}
|
|
301
307
|
```
|
|
302
308
|
|
|
309
|
+
`data.statement_timeout` (default 5,000 ms) and `data.lock_timeout` (default 2,000 ms) are applied before every PostgreSQL connection enters the pool, including startup and replacement connections. Set a value to `null` to leave that server setting unchanged, or `0` to explicitly disable the corresponding PostgreSQL timeout.
|
|
310
|
+
|
|
303
311
|
PostgreSQL with `pgvector` is required for Apollo models:
|
|
304
312
|
|
|
305
313
|
```sql
|
|
@@ -390,6 +398,10 @@ Two independent caching tiers, both disabled by default:
|
|
|
390
398
|
Legion::Data.reload_static_cache
|
|
391
399
|
```
|
|
392
400
|
|
|
401
|
+
### Query File Logging
|
|
402
|
+
|
|
403
|
+
Enable `data.query_log` (or `data.local.query_log`) to write SQL to a dedicated file. Write and close I/O failures, including a full disk, are reported through Legion::Logging without changing the database operation's result or replacing its exception. Opening the configured log file must still succeed during setup.
|
|
404
|
+
|
|
393
405
|
### Read Replicas (PostgreSQL)
|
|
394
406
|
|
|
395
407
|
```json
|
|
@@ -401,6 +413,16 @@ Legion::Data.reload_static_cache
|
|
|
401
413
|
}
|
|
402
414
|
```
|
|
403
415
|
|
|
416
|
+
Replica URLs initialize a sharded pool at startup. URL-specific options override primary connection options; omitted credentials and session timeouts are inherited. Duplicate URLs are registered once as `read_0`, `read_1`, and so on. Reads remain on the primary unless explicitly routed, for example `dataset.server(Legion::Data::Connection.read_server)`. With no replicas configured, `read_server` returns `:default`.
|
|
417
|
+
|
|
418
|
+
### Archive Objects
|
|
419
|
+
|
|
420
|
+
Each archive batch receives a UUID before upload. S3, Azure, and local object paths use `legion-archive/<table>/<year>/<month>/batch_<uuid>.jsonl.gz`; the same UUID is stored in `archive_manifest.batch_id`. Repeated runs do not reuse numbered object names. Existing manifest paths remain valid; no stored objects are renamed. Direct `Archiver.upload_batch` callers must supply `batch_id:` instead of `batch_n:`.
|
|
421
|
+
|
|
422
|
+
### Encryption Key Cache
|
|
423
|
+
|
|
424
|
+
The default scope (`tenant_id: nil`) and tenant IDs have separate cache entries, including a tenant literally named `__default__`. Restart processes or clear existing key-provider caches when upgrading. This fix does not rewrite existing ciphertext: if that tenant name was used with the old cache, assess records encrypted with the wrong scope's key and re-encrypt them with the intended key. The reader does not fall back to another scope's key.
|
|
425
|
+
|
|
404
426
|
---
|
|
405
427
|
|
|
406
428
|
## Common Fields Standard
|
data/lib/legion/data/archiver.rb
CHANGED
|
@@ -5,6 +5,7 @@ require 'digest'
|
|
|
5
5
|
require 'fileutils'
|
|
6
6
|
require 'json'
|
|
7
7
|
require 'securerandom'
|
|
8
|
+
require 'stringio'
|
|
8
9
|
require 'tmpdir'
|
|
9
10
|
require 'zlib'
|
|
10
11
|
|
|
@@ -47,15 +48,15 @@ module Legion
|
|
|
47
48
|
raise
|
|
48
49
|
end
|
|
49
50
|
|
|
50
|
-
def upload_batch(data:, table:, year:, month:,
|
|
51
|
-
log.info "Archiver storing batch table=#{table} backend=#{backend || :tmpdir} year=#{year} month=#{month} batch=#{
|
|
51
|
+
def upload_batch(data:, table:, year:, month:, batch_id:, backend:)
|
|
52
|
+
log.info "Archiver storing batch table=#{table} backend=#{backend || :tmpdir} year=#{year} month=#{month} batch=#{batch_id}"
|
|
52
53
|
case backend
|
|
53
54
|
when :s3
|
|
54
|
-
upload_s3(data: data, table: table, year: year, month: month,
|
|
55
|
+
upload_s3(data: data, table: table, year: year, month: month, batch_id: batch_id)
|
|
55
56
|
when :azure
|
|
56
|
-
upload_azure(data: data, table: table, year: year, month: month,
|
|
57
|
+
upload_azure(data: data, table: table, year: year, month: month, batch_id: batch_id)
|
|
57
58
|
else
|
|
58
|
-
upload_tmpdir(data: data, table: table, year: year, month: month,
|
|
59
|
+
upload_tmpdir(data: data, table: table, year: year, month: month, batch_id: batch_id)
|
|
59
60
|
end
|
|
60
61
|
end
|
|
61
62
|
|
|
@@ -102,7 +103,6 @@ module Legion
|
|
|
102
103
|
table: table,
|
|
103
104
|
cutoff: cutoff,
|
|
104
105
|
batch_size: batch_size,
|
|
105
|
-
batch_n: batches + 1,
|
|
106
106
|
now: now,
|
|
107
107
|
storage_backend: storage_backend
|
|
108
108
|
))
|
|
@@ -114,18 +114,19 @@ module Legion
|
|
|
114
114
|
{ batches: batches, total_rows: total_rows, paths: paths }
|
|
115
115
|
end
|
|
116
116
|
|
|
117
|
-
def archive_batch(conn:, table:, cutoff:, batch_size:,
|
|
117
|
+
def archive_batch(conn:, table:, cutoff:, batch_size:, now:, storage_backend:)
|
|
118
118
|
rows = conn[table].where { created_at < cutoff }.limit(batch_size).all
|
|
119
119
|
return if rows.empty?
|
|
120
120
|
|
|
121
|
+
batch_id = SecureRandom.uuid
|
|
121
122
|
compressed = gzip_compress(serialize_rows(rows))
|
|
122
123
|
path = upload_batch(
|
|
123
|
-
data:
|
|
124
|
-
table:
|
|
125
|
-
year:
|
|
126
|
-
month:
|
|
127
|
-
|
|
128
|
-
backend:
|
|
124
|
+
data: compressed,
|
|
125
|
+
table: table.to_s,
|
|
126
|
+
year: now.year,
|
|
127
|
+
month: now.month,
|
|
128
|
+
batch_id: batch_id,
|
|
129
|
+
backend: storage_backend
|
|
129
130
|
)
|
|
130
131
|
|
|
131
132
|
record_archived_batch(
|
|
@@ -134,16 +135,17 @@ module Legion
|
|
|
134
135
|
rows: rows,
|
|
135
136
|
compressed: compressed,
|
|
136
137
|
path: path,
|
|
138
|
+
batch_id: batch_id,
|
|
137
139
|
now: now
|
|
138
140
|
)
|
|
139
141
|
|
|
140
142
|
{ row_count: rows.size, path: path }
|
|
141
143
|
end
|
|
142
144
|
|
|
143
|
-
def record_archived_batch(conn:, table:, rows:, compressed:, path:, now:)
|
|
145
|
+
def record_archived_batch(conn:, table:, rows:, compressed:, path:, batch_id:, now:)
|
|
144
146
|
conn.transaction do
|
|
145
147
|
conn[:archive_manifest].insert(
|
|
146
|
-
batch_id:
|
|
148
|
+
batch_id: batch_id,
|
|
147
149
|
source_table: table.to_s,
|
|
148
150
|
row_count: rows.size,
|
|
149
151
|
checksum: Digest::SHA256.hexdigest(compressed),
|
|
@@ -167,47 +169,47 @@ module Legion
|
|
|
167
169
|
output.string
|
|
168
170
|
end
|
|
169
171
|
|
|
170
|
-
def upload_s3(data:, table:, year:, month:,
|
|
172
|
+
def upload_s3(data:, table:, year:, month:, batch_id:)
|
|
171
173
|
raise UploadError, 'S3 backend not available: Legion::Extensions::S3::Runners::Put not defined' unless defined?(Legion::Extensions::S3::Runners::Put)
|
|
172
174
|
|
|
173
|
-
key = "legion-archive/#{table}/#{year}/#{month}/batch_#{
|
|
175
|
+
key = "legion-archive/#{table}/#{year}/#{month}/batch_#{batch_id}.jsonl.gz"
|
|
174
176
|
Legion::Extensions::S3::Runners::Put.run(key: key, body: data)
|
|
175
177
|
log.info "Archiver uploaded batch to s3 key=#{key}"
|
|
176
178
|
"s3://#{key}"
|
|
177
179
|
rescue UploadError => e
|
|
178
|
-
handle_exception(e, level: :error, handled: false, operation: :upload_s3, table: table, year: year, month: month,
|
|
180
|
+
handle_exception(e, level: :error, handled: false, operation: :upload_s3, table: table, year: year, month: month, batch_id: batch_id)
|
|
179
181
|
raise
|
|
180
182
|
rescue StandardError => e
|
|
181
|
-
handle_exception(e, level: :error, handled: true, operation: :upload_s3, table: table, year: year, month: month,
|
|
183
|
+
handle_exception(e, level: :error, handled: true, operation: :upload_s3, table: table, year: year, month: month, batch_id: batch_id)
|
|
182
184
|
raise UploadError, "S3 upload failed: #{e.message}"
|
|
183
185
|
end
|
|
184
186
|
|
|
185
|
-
def upload_azure(data:, table:, year:, month:,
|
|
187
|
+
def upload_azure(data:, table:, year:, month:, batch_id:)
|
|
186
188
|
unless defined?(Legion::Extensions::AzureStorage::Runners::Upload)
|
|
187
189
|
raise UploadError, 'Azure backend not available: Legion::Extensions::AzureStorage::Runners::Upload not defined'
|
|
188
190
|
end
|
|
189
191
|
|
|
190
|
-
blob_name = "legion-archive/#{table}/#{year}/#{month}/batch_#{
|
|
192
|
+
blob_name = "legion-archive/#{table}/#{year}/#{month}/batch_#{batch_id}.jsonl.gz"
|
|
191
193
|
Legion::Extensions::AzureStorage::Runners::Upload.run(blob_name: blob_name, data: data)
|
|
192
194
|
log.info "Archiver uploaded batch to azure blob=#{blob_name}"
|
|
193
195
|
"azure://#{blob_name}"
|
|
194
196
|
rescue UploadError => e
|
|
195
|
-
handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month,
|
|
197
|
+
handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month, batch_id: batch_id)
|
|
196
198
|
raise
|
|
197
199
|
rescue StandardError => e
|
|
198
|
-
handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month,
|
|
200
|
+
handle_exception(e, level: :error, handled: false, operation: :upload_azure, table: table, year: year, month: month, batch_id: batch_id)
|
|
199
201
|
raise UploadError, "Azure upload failed: #{e.message}"
|
|
200
202
|
end
|
|
201
203
|
|
|
202
|
-
def upload_tmpdir(data:, table:, year:, month:,
|
|
204
|
+
def upload_tmpdir(data:, table:, year:, month:, batch_id:)
|
|
203
205
|
dir = File.join(Dir.tmpdir, 'legion-archive', table.to_s, year.to_s, month.to_s)
|
|
204
206
|
FileUtils.mkdir_p(dir)
|
|
205
|
-
path = File.join(dir, "batch_#{
|
|
207
|
+
path = File.join(dir, "batch_#{batch_id}.jsonl.gz")
|
|
206
208
|
File.binwrite(path, data)
|
|
207
209
|
log.info "Archiver stored batch locally path=#{path}"
|
|
208
210
|
"file://#{path}"
|
|
209
211
|
rescue StandardError => e
|
|
210
|
-
handle_exception(e, level: :error, handled: true, operation: :upload_tmpdir, table: table, year: year, month: month,
|
|
212
|
+
handle_exception(e, level: :error, handled: true, operation: :upload_tmpdir, table: table, year: year, month: month, batch_id: batch_id)
|
|
211
213
|
raise UploadError, "Tmpdir upload failed: #{e.message}"
|
|
212
214
|
end
|
|
213
215
|
end
|
|
@@ -5,6 +5,7 @@ require 'legion/data/auth_failure_handler'
|
|
|
5
5
|
|
|
6
6
|
require 'fileutils'
|
|
7
7
|
require 'sequel'
|
|
8
|
+
require 'uri'
|
|
8
9
|
|
|
9
10
|
module Legion
|
|
10
11
|
module Data
|
|
@@ -142,6 +143,9 @@ module Legion
|
|
|
142
143
|
@closed = true
|
|
143
144
|
@file.close unless @file.closed?
|
|
144
145
|
end
|
|
146
|
+
rescue IOError, SystemCallError => e
|
|
147
|
+
handle_exception(e, level: :warn, handled: true, operation: :query_file_close, path: @path)
|
|
148
|
+
nil
|
|
145
149
|
end
|
|
146
150
|
|
|
147
151
|
private
|
|
@@ -152,9 +156,7 @@ module Legion
|
|
|
152
156
|
|
|
153
157
|
@file.puts "[#{Time.now.strftime('%Y-%m-%d %H:%M:%S.%L')}] #{level} #{message}"
|
|
154
158
|
end
|
|
155
|
-
rescue IOError => e
|
|
156
|
-
return nil if @closed || @file.closed?
|
|
157
|
-
|
|
159
|
+
rescue IOError, SystemCallError => e
|
|
158
160
|
handle_exception(e, level: :warn, handled: true, operation: :query_file_write, path: @path)
|
|
159
161
|
nil
|
|
160
162
|
end
|
|
@@ -170,10 +172,14 @@ module Legion
|
|
|
170
172
|
end
|
|
171
173
|
|
|
172
174
|
def setup
|
|
175
|
+
Legion::Settings[:data][:connected] = false
|
|
176
|
+
@sequel&.disconnect
|
|
177
|
+
@sequel = nil
|
|
173
178
|
@adapter = Legion::Settings[:data][:adapter]&.to_sym || :sqlite
|
|
174
179
|
opts = sequel_opts
|
|
175
180
|
log.info("Legion::Data::Connection setup adapter=#{adapter}")
|
|
176
181
|
@fallback_active = false
|
|
182
|
+
@replica_servers = []
|
|
177
183
|
@sequel = if adapter == :sqlite
|
|
178
184
|
::Sequel.connect(opts.merge(adapter: :sqlite, database: sqlite_path))
|
|
179
185
|
else
|
|
@@ -183,7 +189,7 @@ module Legion
|
|
|
183
189
|
rescue UnresolvedCredentialError => e
|
|
184
190
|
handle_exception(e, level: :fatal, handled: false, operation: :data_connect)
|
|
185
191
|
raise
|
|
186
|
-
rescue
|
|
192
|
+
rescue ::Sequel::DatabaseConnectionError => e
|
|
187
193
|
raise unless dev_fallback?
|
|
188
194
|
|
|
189
195
|
log.error("Legion::Data FALLING BACK TO SQLITE — #{attempted_adapter} network DB connection failed: #{e.message}")
|
|
@@ -196,10 +202,15 @@ module Legion
|
|
|
196
202
|
::Sequel.connect(sqlite_opts.merge(adapter: :sqlite, database: sqlite_path))
|
|
197
203
|
end
|
|
198
204
|
end
|
|
199
|
-
|
|
205
|
+
@replica_servers = @sequel.opts.fetch(:servers, {}).keys
|
|
200
206
|
log_connection_info
|
|
201
207
|
configure_extensions
|
|
202
|
-
|
|
208
|
+
Legion::Settings[:data][:connected] = true
|
|
209
|
+
rescue StandardError, LoadError => e
|
|
210
|
+
shutdown
|
|
211
|
+
@sequel = nil
|
|
212
|
+
handle_exception(e, level: :fatal, handled: false, operation: :connection_setup)
|
|
213
|
+
raise
|
|
203
214
|
end
|
|
204
215
|
|
|
205
216
|
# Returns connection metadata for health checks and diagnostics.
|
|
@@ -283,6 +294,7 @@ module Legion
|
|
|
283
294
|
@query_file_logger&.close
|
|
284
295
|
@query_file_logger = nil
|
|
285
296
|
@fallback_active = false
|
|
297
|
+
@replica_servers = []
|
|
286
298
|
Legion::Settings[:data][:connected] = false
|
|
287
299
|
log.info 'Legion::Data connection closed'
|
|
288
300
|
end
|
|
@@ -328,28 +340,6 @@ module Legion
|
|
|
328
340
|
handle_exception(e, level: :warn, handled: true, operation: :expire_all_pooled_connections)
|
|
329
341
|
end
|
|
330
342
|
|
|
331
|
-
def connect_with_replicas
|
|
332
|
-
return unless adapter == :postgres
|
|
333
|
-
|
|
334
|
-
replica_url = Legion::Settings[:data][:read_replica_url]
|
|
335
|
-
replica_list = Array(Legion::Settings[:data][:replicas]).dup
|
|
336
|
-
|
|
337
|
-
replica_list.prepend(replica_url) if replica_url && !replica_url.empty?
|
|
338
|
-
replica_list.uniq!
|
|
339
|
-
replica_list.compact!
|
|
340
|
-
|
|
341
|
-
return if replica_list.empty?
|
|
342
|
-
|
|
343
|
-
@sequel.extension(:server_block)
|
|
344
|
-
|
|
345
|
-
replica_list.each_with_index do |url, idx|
|
|
346
|
-
@sequel.add_servers("read_#{idx}": url)
|
|
347
|
-
end
|
|
348
|
-
|
|
349
|
-
@replica_servers = replica_list.each_with_index.map { |_, idx| :"read_#{idx}" }
|
|
350
|
-
log.debug "Registered #{@replica_servers.size} read replica(s)"
|
|
351
|
-
end
|
|
352
|
-
|
|
353
343
|
def read_server
|
|
354
344
|
return :default if @replica_servers.nil? || @replica_servers.empty?
|
|
355
345
|
|
|
@@ -386,6 +376,20 @@ module Legion
|
|
|
386
376
|
creds
|
|
387
377
|
end
|
|
388
378
|
|
|
379
|
+
def configure_sqlite_busy_handler(connection:, timeout:)
|
|
380
|
+
timeout_seconds = timeout.to_f / 1000
|
|
381
|
+
deadline = nil
|
|
382
|
+
connection.busy_handler do |attempt|
|
|
383
|
+
now = ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
|
|
384
|
+
deadline = now + timeout_seconds if attempt.zero?
|
|
385
|
+
remaining = deadline - now
|
|
386
|
+
next false unless remaining.positive?
|
|
387
|
+
|
|
388
|
+
sleep([0.001, remaining].min)
|
|
389
|
+
true
|
|
390
|
+
end
|
|
391
|
+
end
|
|
392
|
+
|
|
389
393
|
def creds_builder(final_creds = {})
|
|
390
394
|
final_creds.merge! Legion::Data::Settings.creds(adapter)
|
|
391
395
|
final_creds.merge! Legion::Settings[:data][:creds] if Legion::Settings[:data][:creds].is_a? Hash
|
|
@@ -472,9 +476,25 @@ module Legion
|
|
|
472
476
|
connection_opts = opts.merge(adapter: adapter, **creds_builder)
|
|
473
477
|
validate_no_unresolved_uris!(connection_opts)
|
|
474
478
|
connection_opts[:preconnect] = false if adapter != :sqlite && dev_fallback?
|
|
479
|
+
if adapter == :postgres
|
|
480
|
+
servers = postgres_replica_servers
|
|
481
|
+
connection_opts[:servers] = servers unless servers.empty?
|
|
482
|
+
end
|
|
475
483
|
connection_opts
|
|
476
484
|
end
|
|
477
485
|
|
|
486
|
+
def postgres_replica_servers
|
|
487
|
+
data = Legion::Settings[:data]
|
|
488
|
+
urls = [data[:read_replica_url], *Array(data[:replicas])].compact.reject(&:empty?).uniq
|
|
489
|
+
urls.each_with_index.to_h do |url, index|
|
|
490
|
+
uri = URI.parse(url)
|
|
491
|
+
raise ArgumentError, 'Read replicas require PostgreSQL URLs' unless %w[postgres postgresql].include?(uri.scheme)
|
|
492
|
+
|
|
493
|
+
options = ::Sequel::Database.send(:options_from_uri, uri).compact
|
|
494
|
+
[:"read_#{index}", options]
|
|
495
|
+
end
|
|
496
|
+
end
|
|
497
|
+
|
|
478
498
|
def validate_no_unresolved_uris!(connection_opts)
|
|
479
499
|
%i[user username password].each do |key|
|
|
480
500
|
value = connection_opts[key]
|
|
@@ -518,10 +538,18 @@ module Legion
|
|
|
518
538
|
val = data.key?(key) && !data[key].nil? ? data[key] : defaults[key]
|
|
519
539
|
opts[key] = val unless val.nil?
|
|
520
540
|
end
|
|
541
|
+
opts[:after_connect] = proc { |connection| configure_sqlite_busy_handler(connection: connection, timeout: opts[:timeout]) } if adapter == :sqlite
|
|
542
|
+
opts[:connect_sqls] = postgres_connect_sqls(data) if adapter == :postgres
|
|
521
543
|
|
|
522
544
|
opts
|
|
523
545
|
end
|
|
524
546
|
|
|
547
|
+
def postgres_connect_sqls(data)
|
|
548
|
+
%i[statement_timeout lock_timeout].filter_map do |setting|
|
|
549
|
+
"SET #{setting} = '#{data[setting].to_i}ms'" if data[setting]
|
|
550
|
+
end
|
|
551
|
+
end
|
|
552
|
+
|
|
525
553
|
def tuning_stats(data)
|
|
526
554
|
tuning = {}
|
|
527
555
|
|
|
@@ -655,7 +683,7 @@ module Legion
|
|
|
655
683
|
if adapter == :postgres
|
|
656
684
|
Sequel.extension(:pg_array)
|
|
657
685
|
@sequel.extension(:pg_array)
|
|
658
|
-
|
|
686
|
+
@sequel.extension(:server_block) unless @replica_servers.empty?
|
|
659
687
|
end
|
|
660
688
|
|
|
661
689
|
if data[:connection_validation] != false
|
|
@@ -670,28 +698,15 @@ module Legion
|
|
|
670
698
|
|
|
671
699
|
install_auth_failure_hook
|
|
672
700
|
rescue StandardError => e
|
|
673
|
-
handle_exception(e, level: :
|
|
674
|
-
|
|
675
|
-
|
|
676
|
-
def install_postgres_session_timeouts
|
|
677
|
-
data = Legion::Settings[:data]
|
|
678
|
-
stmt_timeout = data[:statement_timeout]
|
|
679
|
-
lck_timeout = data[:lock_timeout]
|
|
680
|
-
|
|
681
|
-
@sequel.pool.after_connect = proc do |conn|
|
|
682
|
-
conn.exec("SET statement_timeout = '#{stmt_timeout.to_i}ms'") if stmt_timeout
|
|
683
|
-
conn.exec("SET lock_timeout = '#{lck_timeout.to_i}ms'") if lck_timeout
|
|
684
|
-
end
|
|
685
|
-
|
|
686
|
-
log.info("Postgres session timeouts: statement_timeout=#{stmt_timeout}ms, lock_timeout=#{lck_timeout}ms")
|
|
687
|
-
rescue StandardError => e
|
|
688
|
-
handle_exception(e, level: :warn, handled: true, operation: :install_postgres_session_timeouts)
|
|
701
|
+
handle_exception(e, level: :error, handled: false, operation: :configure_extensions, adapter: adapter)
|
|
702
|
+
raise
|
|
689
703
|
end
|
|
690
704
|
|
|
691
705
|
def install_auth_failure_hook
|
|
692
706
|
Legion::Data::AuthFailureHandler.install(@sequel)
|
|
693
707
|
rescue StandardError => e
|
|
694
|
-
handle_exception(e, level: :
|
|
708
|
+
handle_exception(e, level: :error, handled: false, operation: :install_auth_failure_hook)
|
|
709
|
+
raise
|
|
695
710
|
end
|
|
696
711
|
|
|
697
712
|
def build_data_logger
|
data/lib/legion/data/local.rb
CHANGED
|
@@ -32,6 +32,11 @@ module Legion
|
|
|
32
32
|
val = data.key?(key) && !data[key].nil? ? data[key] : sqlite_defaults[key]
|
|
33
33
|
opts[key] = val unless val.nil?
|
|
34
34
|
end
|
|
35
|
+
opts[:timeout] = data[:timeout] || 30_000
|
|
36
|
+
opts[:after_connect] = proc do |connection|
|
|
37
|
+
Legion::Data::Connection.configure_sqlite_busy_handler(connection: connection, timeout: opts[:timeout])
|
|
38
|
+
connection.execute_batch('PRAGMA synchronous=NORMAL; PRAGMA cache_size=-20000; PRAGMA mmap_size=268435456')
|
|
39
|
+
end
|
|
35
40
|
|
|
36
41
|
if local_settings[:query_log]
|
|
37
42
|
log_path = File.join(Legion::Data::Connection::QUERY_LOG_DIR, 'data-local-query.log')
|
|
@@ -46,14 +51,11 @@ module Legion
|
|
|
46
51
|
|
|
47
52
|
@connection = ::Sequel.connect(opts)
|
|
48
53
|
@connection.run('PRAGMA journal_mode=WAL')
|
|
49
|
-
@connection.run('PRAGMA busy_timeout=30000')
|
|
50
|
-
@connection.run('PRAGMA synchronous=NORMAL')
|
|
51
|
-
@connection.run('PRAGMA cache_size=-20000')
|
|
52
|
-
@connection.run('PRAGMA mmap_size=268435456')
|
|
53
|
-
@connected = true
|
|
54
54
|
run_migrations
|
|
55
|
-
|
|
56
|
-
|
|
55
|
+
@connected = true
|
|
56
|
+
log.info "Legion::Data::Local connected to #{db_file} (WAL mode, #{opts[:timeout]}ms busy timeout, 20MB cache, 256MB mmap)"
|
|
57
|
+
rescue StandardError, LoadError => e
|
|
58
|
+
shutdown
|
|
57
59
|
handle_exception(e, level: :error, handled: false, operation: :local_setup, database: db_file)
|
|
58
60
|
raise
|
|
59
61
|
end
|
|
@@ -74,6 +76,10 @@ module Legion
|
|
|
74
76
|
@registered_migrations ||= {}
|
|
75
77
|
@registered_migrations[name] = path
|
|
76
78
|
run_single_migration(name, path) if connected?
|
|
79
|
+
rescue StandardError, LoadError => e
|
|
80
|
+
shutdown
|
|
81
|
+
handle_exception(e, level: :error, handled: false, operation: :register_migrations, name: name, path: path)
|
|
82
|
+
raise
|
|
77
83
|
end
|
|
78
84
|
|
|
79
85
|
def registered_migrations
|
|
@@ -142,8 +148,9 @@ module Legion
|
|
|
142
148
|
|
|
143
149
|
table = :"schema_migrations_#{name}"
|
|
144
150
|
::Sequel::TimestampMigrator.new(@connection, path, table: table).run
|
|
145
|
-
rescue StandardError => e
|
|
146
|
-
handle_exception(e, level: :
|
|
151
|
+
rescue StandardError, LoadError => e
|
|
152
|
+
handle_exception(e, level: :error, handled: false, operation: :local_migration, name: name, path: path)
|
|
153
|
+
raise
|
|
147
154
|
end
|
|
148
155
|
|
|
149
156
|
def local_settings
|
data/lib/legion/data/rls.rb
CHANGED
|
@@ -47,11 +47,17 @@ module Legion
|
|
|
47
47
|
end
|
|
48
48
|
|
|
49
49
|
def with_tenant(tenant_id)
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
50
|
+
return yield unless rls_enabled?
|
|
51
|
+
|
|
52
|
+
Legion::Data.connection.synchronize do
|
|
53
|
+
previous = current_tenant
|
|
54
|
+
begin
|
|
55
|
+
assign_tenant(tenant_id)
|
|
56
|
+
yield
|
|
57
|
+
ensure
|
|
58
|
+
previous ? assign_tenant(previous) : reset_tenant
|
|
59
|
+
end
|
|
60
|
+
end
|
|
55
61
|
end
|
|
56
62
|
end
|
|
57
63
|
end
|
data/lib/legion/data/version.rb
CHANGED
data/lib/legion/data.rb
CHANGED
|
@@ -48,6 +48,11 @@ module Legion
|
|
|
48
48
|
setup_cache
|
|
49
49
|
setup_local
|
|
50
50
|
log.info 'Legion::Data setup complete'
|
|
51
|
+
rescue StandardError, LoadError => e
|
|
52
|
+
shutdown
|
|
53
|
+
Legion::Data::Connection.sequel = nil
|
|
54
|
+
handle_exception(e, level: :fatal, handled: false, operation: :setup)
|
|
55
|
+
raise
|
|
51
56
|
end
|
|
52
57
|
|
|
53
58
|
def setup_local
|
|
@@ -62,10 +67,19 @@ module Legion
|
|
|
62
67
|
|
|
63
68
|
def setup_global
|
|
64
69
|
connection_setup
|
|
70
|
+
Legion::Settings[:data][:connected] = false
|
|
71
|
+
Legion::Settings[:data][:migrations][:ran] = false
|
|
72
|
+
Legion::Settings[:data][:models][:loaded] = false
|
|
65
73
|
migrate
|
|
66
74
|
load_models
|
|
67
|
-
|
|
68
|
-
|
|
75
|
+
Legion::Settings[:data][:connected] = true
|
|
76
|
+
rescue StandardError, LoadError => e
|
|
77
|
+
Legion::Data::Connection.shutdown
|
|
78
|
+
Legion::Data::Connection.sequel = nil
|
|
79
|
+
Legion::Settings[:data][:migrations][:ran] = false
|
|
80
|
+
Legion::Settings[:data][:models][:loaded] = false
|
|
81
|
+
handle_exception(e, level: :fatal, handled: false, operation: :setup_global)
|
|
82
|
+
raise
|
|
69
83
|
end
|
|
70
84
|
|
|
71
85
|
def setup_cache
|