extralite 2.5 → 2.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.gitignore +1 -0
- data/CHANGELOG.md +34 -13
- data/Gemfile +4 -0
- data/Gemfile-bundle +1 -1
- data/LICENSE +1 -1
- data/README.md +1059 -247
- data/Rakefile +18 -0
- data/TODO.md +0 -7
- data/examples/kv_store.rb +49 -0
- data/examples/multi_fiber.rb +16 -0
- data/examples/on_progress.rb +9 -0
- data/examples/pubsub_store_polyphony.rb +194 -0
- data/examples/pubsub_store_threads.rb +204 -0
- data/ext/extralite/changeset.c +463 -0
- data/ext/extralite/common.c +177 -91
- data/ext/extralite/database.c +745 -276
- data/ext/extralite/extconf-bundle.rb +10 -4
- data/ext/extralite/extconf.rb +34 -34
- data/ext/extralite/extralite.h +104 -47
- data/ext/extralite/extralite_ext.c +6 -0
- data/ext/extralite/iterator.c +14 -86
- data/ext/extralite/query.c +171 -264
- data/extralite-bundle.gemspec +1 -1
- data/extralite.gemspec +1 -1
- data/gemspec.rb +10 -11
- data/lib/extralite/version.rb +1 -1
- data/lib/extralite.rb +69 -10
- data/lib/sequel/adapters/extralite.rb +1 -1
- data/test/helper.rb +9 -1
- data/test/perf_argv_transform.rb +74 -0
- data/test/perf_ary.rb +14 -12
- data/test/perf_hash.rb +17 -15
- data/test/perf_hash_prepared.rb +58 -0
- data/test/perf_hash_transform.rb +66 -0
- data/test/perf_polyphony.rb +74 -0
- data/test/test_changeset.rb +161 -0
- data/test/test_database.rb +720 -104
- data/test/test_extralite.rb +2 -2
- data/test/test_iterator.rb +28 -13
- data/test/test_query.rb +352 -110
- data/test/test_sequel.rb +4 -4
- metadata +24 -16
- data/Gemfile.lock +0 -37
- data/test/perf_prepared.rb +0 -64
data/README.md
CHANGED
@@ -1,247 +1,1069 @@
|
|
1
|
-
|
2
|
-
|
3
|
-
|
4
|
-
|
5
|
-
|
6
|
-
|
1
|
+
<h1 align="center">
|
2
|
+
<br>
|
3
|
+
Extralite
|
4
|
+
</h1>
|
5
|
+
|
6
|
+
<h4 align="center">Ruby on SQLite</h4>
|
7
|
+
|
8
|
+
<p align="center">
|
9
|
+
<a href="http://rubygems.org/gems/extralite">
|
10
|
+
<img src="https://badge.fury.io/rb/extralite.svg" alt="Ruby gem">
|
11
|
+
</a>
|
12
|
+
<a href="https://github.com/digital-fabric/extralite/actions">
|
13
|
+
<img src="https://github.com/digital-fabric/extralite/actions/workflows/test.yml/badge.svg" alt="Tests">
|
14
|
+
</a>
|
15
|
+
<a href="https://github.com/digital-fabric/extralite/blob/master/LICENSE">
|
16
|
+
<img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License">
|
17
|
+
</a>
|
18
|
+
</p>
|
19
|
+
|
20
|
+
<p align="center">
|
21
|
+
<a href="https://www.rubydoc.info/gems/extralite">API reference</a>
|
22
|
+
</p>
|
7
23
|
|
8
24
|
## What is Extralite?
|
9
25
|
|
10
|
-
Extralite is a
|
11
|
-
|
12
|
-
|
26
|
+
Extralite is a fast and innovative SQLite wrapper for Ruby with a rich set of
|
27
|
+
features. It provides multiple ways of retrieving data from SQLite databases,
|
28
|
+
makes it possible to use SQLite databases in multi-threaded and multi-fibered
|
29
|
+
Ruby apps, and includes a comprehensive set of tools for managing SQLite
|
30
|
+
databases.
|
13
31
|
|
14
32
|
Extralite comes in two flavors: the `extralite` gem which uses the
|
15
33
|
system-installed sqlite3 library, and the `extralite-bundle` gem which bundles
|
16
34
|
the latest version of SQLite
|
17
|
-
([3.
|
35
|
+
([3.45.0](https://sqlite.org/releaselog/3_45_0.html)), offering access to the
|
18
36
|
latest features and enhancements.
|
19
37
|
|
20
38
|
## Features
|
21
39
|
|
22
|
-
-
|
23
|
-
[sqlite3](https://github.com/sparklemotion/sqlite3-ruby) gem.
|
24
|
-
-
|
25
|
-
|
26
|
-
-
|
27
|
-
|
28
|
-
-
|
29
|
-
|
30
|
-
|
31
|
-
-
|
32
|
-
|
33
|
-
|
34
|
-
-
|
35
|
-
|
36
|
-
-
|
37
|
-
|
38
|
-
-
|
39
|
-
|
40
|
-
-
|
41
|
-
|
42
|
-
##
|
43
|
-
|
44
|
-
|
40
|
+
- Best-in-class [performance](#performance) (up to 14X the performance of the
|
41
|
+
[sqlite3](https://github.com/sparklemotion/sqlite3-ruby) gem).
|
42
|
+
- Support for [concurrency](#concurrency) out of the box for multi-threaded
|
43
|
+
and multi-fibered apps.
|
44
|
+
- A variety of ways to [retrieve data](#query-modes) - hashes, arrays, single
|
45
|
+
columns, single rows, [transforms](#value-transforms).
|
46
|
+
- Support for [external iteration](#iterating-over-records-in-a-prepared-query),
|
47
|
+
allowing iterating through single records or batches of records.
|
48
|
+
- [Prepared queries](#prepared-queries).
|
49
|
+
- [Parameter binding](#parameter-binding).
|
50
|
+
- [Batch execution](#batch-execution-of-queries) of queries.
|
51
|
+
- [transactions and savepoints](#transactions-and-savepoints).
|
52
|
+
- Advanced features: load [SQLite extensions](#loading-extensions), create
|
53
|
+
[backups](#creating-backups), retrieve [status
|
54
|
+
information](#retrieving-status-information), work with
|
55
|
+
[changesets](#working-with-changesets), interrogate [database
|
56
|
+
limits](#working-with-database-limits), [trace](#tracing-sql-statements)
|
57
|
+
queries.
|
58
|
+
- [Sequel](#usage-with-sequel) adapter.
|
59
|
+
|
60
|
+
## Table of Content
|
61
|
+
|
62
|
+
- [Installing Extralite](#installing-extralite)
|
63
|
+
- [Getting Started](#getting-started)
|
64
|
+
- [Query Modes](#query-modes)
|
65
|
+
- [Parameter binding](#parameter-binding)
|
66
|
+
- [Value Transforms](#value-transforms)
|
67
|
+
- [Data Types](#data-types)
|
68
|
+
- [Prepared Queries](#prepared-queries)
|
69
|
+
- [Batch Execution of Queries](#batch-execution-of-queries)
|
70
|
+
- [Transactions and Savepoints](#transactions-and-savepoints)
|
71
|
+
- [Database Information](#database-information)
|
72
|
+
- [Error Handling](#error-handling)
|
73
|
+
- [Concurrency](#concurrency)
|
74
|
+
- [Advanced Usage](#advanced-usage)
|
75
|
+
- [Usage with Sequel](#usage-with-sequel)
|
76
|
+
- [Performance](#performance)
|
77
|
+
- [License](#license)
|
78
|
+
- [Contributing](#contributing)
|
79
|
+
|
80
|
+
## Installing Extralite
|
81
|
+
|
82
|
+
Using bundler:
|
45
83
|
|
46
84
|
```ruby
|
47
85
|
gem 'extralite'
|
48
86
|
```
|
49
87
|
|
50
|
-
|
88
|
+
Or manually:
|
51
89
|
|
52
|
-
|
90
|
+
```bash
|
91
|
+
$ gem install extralite
|
92
|
+
```
|
53
93
|
|
54
|
-
|
94
|
+
__Note__: Extralite supports Ruby 3.0 and newer.
|
55
95
|
|
56
|
-
|
57
|
-
system-installed version of SQLite3, or would like to use the latest version of
|
58
|
-
SQLite3, you can install the `extralite-bundle` gem, which integrates the
|
59
|
-
SQLite3 source code.
|
96
|
+
### Installing the Extralite-SQLite Bundle
|
60
97
|
|
61
|
-
|
62
|
-
|
63
|
-
|
98
|
+
If you don't have the sqlite3 lib installed on your system, do not want to use
|
99
|
+
the system-installed version of SQLite, or would like to use the latest version
|
100
|
+
of SQLite, you can install the `extralite-bundle` gem, which integrates the
|
101
|
+
SQLite source code.
|
64
102
|
|
65
103
|
Usage of the `extralite-bundle` gem is identical to the usage of the normal
|
66
104
|
`extralite` gem, using `require 'extralite'` to load the gem.
|
67
105
|
|
68
|
-
##
|
106
|
+
## Getting Started
|
107
|
+
|
108
|
+
The following example shows how to open an SQLite database and run some queries:
|
109
|
+
|
110
|
+
```ruby
|
111
|
+
db = Extralite::Database.new('mydb.sqlite')
|
112
|
+
db.execute('create table foo (x, y, z)')
|
113
|
+
db.execute('insert into foo values (?, ?, ?)', 1, 2, 3)
|
114
|
+
db.execute('insert into foo values (?, ?, ?)', 4, 5, 6)
|
115
|
+
db.query('select * from foo') #=> [{x: 1, y: 2, z: 3}, {x: 4, y: 5, z: 6}]
|
116
|
+
```
|
117
|
+
|
118
|
+
The `#execute` method is used to make changes to the database, such as creating
|
119
|
+
tables, or inserting records. It returns the number of records changed by the
|
120
|
+
query.
|
121
|
+
|
122
|
+
The `#query` method is used to read data from the database. It returns an array
|
123
|
+
containing the resulting records, represented as hashes mapping column names (as
|
124
|
+
symbols) to individual values.
|
125
|
+
|
126
|
+
You can also iterate on records by providing a block to `#query`:
|
127
|
+
|
128
|
+
```ruby
|
129
|
+
db.query 'select * from foo' do |r|
|
130
|
+
p record: r
|
131
|
+
end
|
132
|
+
```
|
133
|
+
|
134
|
+
## Query Modes
|
135
|
+
|
136
|
+
Extralite allows you to retrieve data from SQLite database in the form that most
|
137
|
+
a particular context. For some use cases you'll want to work with rows as
|
138
|
+
hashes. In other cases, you'll want to work with rows as arrays, or even as
|
139
|
+
single values, if you're just reading one column.
|
140
|
+
|
141
|
+
For that purpose, Extralite offers three different ways, or modes, of retrieving
|
142
|
+
records:
|
143
|
+
|
144
|
+
- `:hash`: read rows as hashes (this is the default mode).
|
145
|
+
- `:ary`: read rows as arrays.
|
146
|
+
- `:argv`: similar to the `:ary`, except that for queries with a single column,
|
147
|
+
the single column value is returned.
|
148
|
+
|
149
|
+
Extralite provides separate methods for the different modes:
|
150
|
+
|
151
|
+
```ruby
|
152
|
+
# alias #query_hash
|
153
|
+
db.query('select 1') #=> [{ "1" => 1 }]
|
154
|
+
|
155
|
+
db.query_ary('select 1') #=> [[1]]
|
156
|
+
|
157
|
+
db.query_argv('select 1') #=> [1]
|
158
|
+
```
|
159
|
+
|
160
|
+
Notice how all the return values above are arrays. This is because the different
|
161
|
+
`#query_xxx` methods are designed to return multiple rows. If you want to just
|
162
|
+
get back a single row, use one of the `query_single_xxx` methods:
|
163
|
+
|
164
|
+
```ruby
|
165
|
+
# alias #query_single_hash
|
166
|
+
db.query('select 1') #=> { "1" => 1 }
|
167
|
+
|
168
|
+
db.query_single_ary('select 1') #=> [1]
|
169
|
+
|
170
|
+
db.query_single_argv('select 1') #=> 1
|
171
|
+
```
|
172
|
+
|
173
|
+
## Parameter binding
|
174
|
+
|
175
|
+
As shown in the above example, the `#execute` and `#query_xxx` methods accept
|
176
|
+
parameters that can be bound to the query, which means that their values will be
|
177
|
+
used for each corresponding place-holder (expressed using `?`) in the SQL
|
178
|
+
statement:
|
179
|
+
|
180
|
+
```ruby
|
181
|
+
db.query('select x from my_table where y = ? and z = ?', 'foo', 'bar')
|
182
|
+
```
|
183
|
+
|
184
|
+
You can also express place holders by specifying their index (starting from 1)
|
185
|
+
using `?IDX`:
|
186
|
+
|
187
|
+
```ruby
|
188
|
+
# use the same value for both place holders:
|
189
|
+
db.query('select x from my_table where y = ?1 and z = ?1', 42)
|
190
|
+
```
|
191
|
+
|
192
|
+
Another possibility is to use named parameters, which can be done by expressing
|
193
|
+
place holders as `:KEY`, `@KEY` or `$KEY`:
|
194
|
+
|
195
|
+
```ruby
|
196
|
+
db.query('select x from my_table where y = $y and z = $z', y: 'foo', z: 'bar')
|
197
|
+
```
|
198
|
+
|
199
|
+
Extralite supports specifying named parameters using `Struct` or `Data` objects:
|
200
|
+
|
201
|
+
```ruby
|
202
|
+
MyStruct = Struct.new(:x, :z)
|
203
|
+
params = MyStruct.new(42, 6)
|
204
|
+
db.execute('update foo set x = $x where z = $z', params)
|
205
|
+
|
206
|
+
MyData = Data.define(:x, :z)
|
207
|
+
params = MyData.new(43, 3)
|
208
|
+
db.execute('update foo set x = $x where z = $z', params)
|
209
|
+
```
|
210
|
+
|
211
|
+
Parameter binding is especially useful for preventing [SQL-injection
|
212
|
+
attacks](https://en.wikipedia.org/wiki/SQL_injection), but is also useful when
|
213
|
+
combined with [prepared queries](#prepared-queries) when repeatedly running the
|
214
|
+
same query over and over.
|
215
|
+
|
216
|
+
## Data Types
|
217
|
+
|
218
|
+
Extralite supports the following data types for either bound parameters or row
|
219
|
+
values:
|
220
|
+
|
221
|
+
- `Integer`
|
222
|
+
- `Float`
|
223
|
+
- `Boolean` (see below)
|
224
|
+
- `String` (see below)
|
225
|
+
- nil
|
226
|
+
|
227
|
+
### Boolean values
|
228
|
+
|
229
|
+
SQLite does not have a boolean data type. Extralite will automatically translate
|
230
|
+
bound parameter values of `true` or `false` to the integer values `1` and `0`,
|
231
|
+
respectively. Note that boolean values stored in the database will be fetched as
|
232
|
+
integers.
|
233
|
+
|
234
|
+
### String values
|
235
|
+
|
236
|
+
String parameter values are translated by Extralite to either `TEXT` or `BLOB`
|
237
|
+
values according to the string encoding used. Strings with an `ASCII-8BIT` are
|
238
|
+
treated as blobs. Otherwise they are treated as text values.
|
239
|
+
|
240
|
+
Likewise, when fetching records, Extralite will convert a `BLOB` column value to
|
241
|
+
a string with `ASCII-8BIT` encoding, and a `TEXT` column value to a string with
|
242
|
+
`UTF-8` encoding.
|
243
|
+
|
244
|
+
```ruby
|
245
|
+
# The following calls will insert blob values into the database
|
246
|
+
sql = 'insert into foo values (?)'
|
247
|
+
db.execute(sql, File.binread('/path/to/file'))
|
248
|
+
db.execute(sql, Extralite::Blob.new('Hello, 世界!'))
|
249
|
+
db.execute(sql, 'Hello, 世界!'.force_encoding(Encoding::ASCII_8BIT))
|
250
|
+
```
|
251
|
+
|
252
|
+
## Value Transforms
|
253
|
+
|
254
|
+
Extralite allows you to transform rows to any value your application may need by
|
255
|
+
providing a transform proc that takes the raw row values and returns the
|
256
|
+
transformed data. The transform proc is passed each resulting row either as a
|
257
|
+
hash or as a list of values.
|
258
|
+
|
259
|
+
Transforms are useful when you need to transform rows into ORM model instances,
|
260
|
+
or when you need to do some other transformation on the values retrieved from
|
261
|
+
the database.
|
262
|
+
|
263
|
+
To transform results, pass a transform proc as the first parameter to one of the
|
264
|
+
`#query_xxx` methods:
|
265
|
+
|
266
|
+
```ruby
|
267
|
+
transform = ->(h) { MyModel.new(h) }
|
268
|
+
db.query(transform, 'select * from foo')
|
269
|
+
#=> rows as instances of MyModel
|
270
|
+
```
|
271
|
+
|
272
|
+
When using the `argv` mode, the different column values are passed as individual
|
273
|
+
values to the transform proc:
|
274
|
+
|
275
|
+
```ruby
|
276
|
+
transform = ->(a, b, c) { { a:a, b: b, c: JSON.parse(c) } }
|
277
|
+
db.query_argv(transform, 'select a, b, c from foo')
|
278
|
+
#=> transformed rows
|
279
|
+
```
|
280
|
+
|
281
|
+
Value transforms can also be done with [prepared
|
282
|
+
queries](#value-transforms-in-prepared-queries).
|
283
|
+
|
284
|
+
## Prepared Queries
|
285
|
+
|
286
|
+
Prepared queries (also known as prepared statements) allow you to maximize
|
287
|
+
performance and reduce memory usage when running the same query repeatedly. They
|
288
|
+
also allow you to create parameterized queries that can be repeatedly executed
|
289
|
+
with different parameters:
|
290
|
+
|
291
|
+
```ruby
|
292
|
+
query = db.prepare('select * from foo where x = ?')
|
293
|
+
|
294
|
+
# bind parameters and get results as an array of hashes
|
295
|
+
query.bind(1).to_a
|
296
|
+
#=> [{ x: 1, y: 2, z: 3 }]
|
297
|
+
```
|
298
|
+
|
299
|
+
### Binding Values to Prepared Queries
|
300
|
+
|
301
|
+
To bind parameter values to the query, use the `#bind` method. The parameters
|
302
|
+
will remain bound to the query until `#bind` is called again.
|
303
|
+
|
304
|
+
```ruby
|
305
|
+
query.bind(1)
|
306
|
+
|
307
|
+
# run the query any number of times
|
308
|
+
3.times { query.to_a }
|
309
|
+
|
310
|
+
# bind other parameter values
|
311
|
+
query.bind(4)
|
312
|
+
```
|
313
|
+
|
314
|
+
You can also bind parameters when creating the prepared query by passing
|
315
|
+
additional parameters to the `Database#prepare` method:
|
316
|
+
|
317
|
+
```ruby
|
318
|
+
query = db.prepare('select * from foo where x = ?', 1)
|
319
|
+
```
|
320
|
+
|
321
|
+
### Fetching Records from a Prepared Query
|
322
|
+
|
323
|
+
Just like the `Database` interface, prepared queries support getting data using
|
324
|
+
three different modes: as a hash, an array or as individual column values. To
|
325
|
+
set the mode, you can use one of the `#prepare_xxx` methods:
|
326
|
+
|
327
|
+
```ruby
|
328
|
+
# hash mode
|
329
|
+
db.prepare('select * from foo').to_a
|
330
|
+
#=> [{ x: 1, y: 2, z: 3}]
|
331
|
+
|
332
|
+
# argv mode
|
333
|
+
db.prepare_argv('select x from foo').to_a
|
334
|
+
#=> [1]
|
335
|
+
|
336
|
+
# ary mode
|
337
|
+
db.prepare_ary('select * from foo').to_a
|
338
|
+
#=> [[1, 2, 3]]
|
339
|
+
```
|
340
|
+
|
341
|
+
You can also set the query mode by getting or setting `#mode`:
|
342
|
+
|
343
|
+
```ruby
|
344
|
+
q = db.prepare('select * from foo')
|
345
|
+
q.to_a #=> [{ x: 1, y: 2, z: 3}]
|
346
|
+
|
347
|
+
q.mode #=> :hash
|
348
|
+
q.mode = :ary
|
349
|
+
q.to_a "=> [[1, 2, 3]]
|
350
|
+
```
|
351
|
+
|
352
|
+
### Fetching Single Records or Batches of Records
|
353
|
+
|
354
|
+
Prepared queries let you iterate over records one by one, or by batches. For
|
355
|
+
this, use the `#next` method:
|
69
356
|
|
70
357
|
```ruby
|
71
|
-
|
358
|
+
query = db.prepare('select * from foo')
|
359
|
+
|
360
|
+
query.next
|
361
|
+
#=> { x: 1, y: 2, z: 3 }
|
362
|
+
query.next
|
363
|
+
#=> { x: 4, y: 5, z: 6 }
|
364
|
+
|
365
|
+
# Fetch the next 10 records
|
366
|
+
query.reset # go back tpo the beginning
|
367
|
+
query.next(10)
|
368
|
+
#=> [{ x: 1, y: 2, z: 3 }, { x: 4, y: 5, z: 6 }]
|
369
|
+
|
370
|
+
# Fetch the next row as an array
|
371
|
+
query = db.prepare_ary('select * from foo')
|
372
|
+
query.next
|
373
|
+
#=> [1, 2, 3]
|
374
|
+
|
375
|
+
# Fetch the next row as a single column
|
376
|
+
db.prepare_argv('select z from foo').next
|
377
|
+
#=> 3
|
378
|
+
```
|
379
|
+
|
380
|
+
To detect the end of the result, you can use `#eof?`. To go back to the
|
381
|
+
beginning of the result set, use `#reset`. The following example shows how to
|
382
|
+
read the query results in batches of 10:
|
383
|
+
|
384
|
+
```ruby
|
385
|
+
query.reset
|
386
|
+
while !query.eof?
|
387
|
+
records = query.next(10)
|
388
|
+
process_records(records)
|
389
|
+
end
|
390
|
+
```
|
391
|
+
|
392
|
+
### Iterating over Records in a Prepared Query
|
393
|
+
|
394
|
+
In addition to the `#next` method, you can also iterate over query results by
|
395
|
+
using the familiar `#each` method:
|
396
|
+
|
397
|
+
```ruby
|
398
|
+
# iterate over records as hashes
|
399
|
+
query = db.prepare('select * from foo')
|
400
|
+
query.each { |r| ... }
|
401
|
+
|
402
|
+
# iterate over records as arrays
|
403
|
+
query = db.prepare_ary('select * from foo')
|
404
|
+
query.each { |r| ... }
|
72
405
|
|
73
|
-
#
|
74
|
-
|
406
|
+
# iterate over records as single values
|
407
|
+
query = db.prepare_argv('select a, b, c from foo')
|
408
|
+
query.each { |a, b, c| ... }
|
409
|
+
```
|
75
410
|
|
76
|
-
|
77
|
-
db = Extralite::Database.new('/tmp/my.db')
|
411
|
+
### Prepared Query as an Enumerable
|
78
412
|
|
79
|
-
|
80
|
-
|
81
|
-
|
82
|
-
db.query_hash('select 1 as foo') #=> [{ :foo => 1 }]
|
83
|
-
# or iterate over results
|
84
|
-
db.query('select 1 as foo') { |r| p r }
|
85
|
-
# { :foo => 1 }
|
413
|
+
You can also use a prepared query as an enumerable, allowing you to chain
|
414
|
+
enumerable method calls while iterating over the query result set. This is done
|
415
|
+
by calling `#each` without a block:
|
86
416
|
|
87
|
-
|
88
|
-
|
89
|
-
|
90
|
-
db.query_ary('select 1, 2, 3') { |r| p r }
|
91
|
-
# [1, 2, 3]
|
417
|
+
```ruby
|
418
|
+
iterator = query.each
|
419
|
+
#=> Returns an Extralite::Iterator instance
|
92
420
|
|
93
|
-
|
94
|
-
|
421
|
+
iterator.map { |r| r[:x] *100 + r[:y] * 10 + r[:z] }
|
422
|
+
#=> [123, 345]
|
423
|
+
```
|
95
424
|
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
100
|
-
# 42
|
425
|
+
The Iterator class includes the
|
426
|
+
[`Enumerable`](https://rubyapi.org/3.3/o/enumerable) module, with all its
|
427
|
+
methods, such as `#map`, `#select`, `#inject`, `#lazy` etc. You can also
|
428
|
+
instantiate an iterator explicitly:
|
101
429
|
|
102
|
-
|
103
|
-
|
430
|
+
```ruby
|
431
|
+
# You need to pass the query to iterate over:
|
432
|
+
iterator = Extralite::Iterator(query)
|
433
|
+
iterator.each { |r| ... }
|
434
|
+
```
|
104
435
|
|
105
|
-
|
106
|
-
db.query_hash('select ? as foo, ? as bar', 1, 2) #=> [{ :foo => 1, :bar => 2 }]
|
107
|
-
db.query_hash('select ? as foo, ? as bar', [1, 2]) #=> [{ :foo => 1, :bar => 2 }]
|
436
|
+
### Value Transforms in Prepared Queries
|
108
437
|
|
109
|
-
|
110
|
-
|
438
|
+
Prepared queries can automatically transform their result sets by setting a
|
439
|
+
transform block. The transform block receives values according to the query mode
|
440
|
+
(hash, array or argv). To set a transform you can pass a block to one of the
|
441
|
+
`Database#prepare_xxx` methods, or use `Query#transform`:
|
111
442
|
|
112
|
-
|
113
|
-
db.
|
114
|
-
|
115
|
-
db.query('select * from foo where bar = :bar', ':bar' => 42)
|
443
|
+
```ruby
|
444
|
+
q = db.prepare('select * from items where id = ?') { |h| Item.new(h) }
|
445
|
+
q.bind(42).next #=> Item instance
|
116
446
|
|
117
|
-
#
|
118
|
-
|
119
|
-
|
120
|
-
|
121
|
-
db.query_single_column('select :bar', SomeData.new(foo: 41, bar: 42)) #=> [42]
|
447
|
+
# An equivalent
|
448
|
+
q = db.prepare('select * from items where id = ?')
|
449
|
+
q.transform { |h| Item.new(h) }
|
450
|
+
```
|
122
451
|
|
123
|
-
|
124
|
-
db.execute('insert into foo values (?)', File.binread('/path/to/file'))
|
125
|
-
db.execute('insert into foo values (?)', Extralite::Blob.new('Hello, 世界!'))
|
126
|
-
db.execute('insert into foo values (?)', 'Hello, 世界!'.force_encoding(Encoding::ASCII_8BIT))
|
452
|
+
The same can be done for queries in `argv` or `ary` mode:
|
127
453
|
|
128
|
-
|
129
|
-
db.
|
130
|
-
|
454
|
+
```ruby
|
455
|
+
db.prepare_argv('select * from foo') { |a, b, c| a + b + c }
|
456
|
+
|
457
|
+
db.prepare_ary('select * from foo') { |a| a.map(&:to_s).join }
|
458
|
+
```
|
131
459
|
|
132
|
-
|
460
|
+
## Batch Execution of Queries
|
461
|
+
|
462
|
+
Extralite provides methods for batch execution of queries, with multiple sets of
|
463
|
+
parameters. The `#batch_execute` method lets you insert or update a large number
|
464
|
+
of records with a single call:
|
465
|
+
|
466
|
+
```ruby
|
467
|
+
values = [
|
468
|
+
[1, 11, 111],
|
469
|
+
[2, 22, 222],
|
470
|
+
[3, 33, 333]
|
471
|
+
]
|
472
|
+
# insert the above records in one fell swoop, and returns the total number of
|
473
|
+
# changes:
|
474
|
+
db.batch_execute('insert into foo values (?, ?, ?)', values)
|
475
|
+
#=> 3
|
476
|
+
```
|
477
|
+
|
478
|
+
Parameters to the query can also be provided by any object that is an
|
479
|
+
`Enumerable` or has an `#each` method, or any *callable* object that responds to
|
480
|
+
`#call`:
|
481
|
+
|
482
|
+
```ruby
|
483
|
+
# Take parameter values from a Range
|
133
484
|
db.batch_execute('insert into foo values (?)', 1..10)
|
485
|
+
#=> 10
|
134
486
|
|
135
|
-
#
|
136
|
-
|
137
|
-
|
487
|
+
# Insert (chomped) lines from a file
|
488
|
+
File.open('foo.txt') do |f|
|
489
|
+
source = f.each_line.map(&:chomp)
|
490
|
+
db.batch_execute('insert into foo values (?)', source)
|
491
|
+
end
|
138
492
|
|
139
|
-
#
|
140
|
-
|
141
|
-
|
493
|
+
# Insert items from a queue
|
494
|
+
parameters = proc do
|
495
|
+
item = queue.shift
|
496
|
+
# when we're done, we return nil
|
497
|
+
(item == :done) ? nil : item
|
498
|
+
end
|
499
|
+
db.batch_execute('insert into foo values (?)', parameters)
|
500
|
+
#=> number of rows inserted
|
501
|
+
```
|
142
502
|
|
143
|
-
|
144
|
-
|
145
|
-
query.next_ary #=> next row in result_set (as array)
|
146
|
-
query.next_single_column #=> next row in result_set (as single value)
|
503
|
+
Like its cousin `#execute`, the `#batch_execute` returns the total number of
|
504
|
+
changes to the database (rows inserted, deleted or udpated).
|
147
505
|
|
148
|
-
|
149
|
-
query.next_hash(10) #=> next 10 rows in result_set (as hash)
|
150
|
-
query.next_ary(10) #=> next 10 rows in result_set (as array)
|
151
|
-
query.next_single_column(10) #=> next 10 rows in result_set (as single value)
|
506
|
+
### Batch Execution of Queries that Return Rows
|
152
507
|
|
153
|
-
|
154
|
-
|
155
|
-
|
156
|
-
|
508
|
+
Extralite also provides a `#batch_query` method that like `#batch_execute` takes
|
509
|
+
a parameter source and returns an array containing the result sets for all query
|
510
|
+
invocations. If a block is given, the result sets are passed to the block
|
511
|
+
instead.
|
157
512
|
|
158
|
-
|
159
|
-
|
160
|
-
query.each_ary { |r| ... } #=> iterate over all rows as arrays
|
161
|
-
query.each_single_column { |r| ... } #=> iterate over all rows as single columns
|
513
|
+
The `#batch_query` method is especially useful for batch queries with a
|
514
|
+
`RETURNING` clause:
|
162
515
|
|
163
|
-
|
164
|
-
|
165
|
-
|
166
|
-
|
516
|
+
```ruby
|
517
|
+
updates = [
|
518
|
+
{ id: 3, price: 42 },
|
519
|
+
{ id: 5, price: 43 }
|
520
|
+
]
|
521
|
+
sql = 'update foo set price = $price where id = $id returning id, quantity'
|
522
|
+
db.batch_query(sql, updates)
|
523
|
+
#=> [[{ id: 3, quantity: 4 }], [{ id: 5, quantity: 5 }]]
|
524
|
+
|
525
|
+
# The same with a block (returns the total number of changes)
|
526
|
+
db.batch_query(sql, updates) do |rows|
|
527
|
+
p rows
|
528
|
+
#=> [{ id: 3, quantity: 4 }]
|
529
|
+
#=> [{ id: 5, quantity: 5 }]
|
530
|
+
end
|
531
|
+
#=> 2
|
532
|
+
```
|
167
533
|
|
168
|
-
|
169
|
-
|
534
|
+
The `#batch_query` method, like other row fetching methods, changes the row
|
535
|
+
representation according to the query mode.
|
170
536
|
|
171
|
-
|
172
|
-
rowid = db.last_insert_rowid
|
537
|
+
### Batch Execution of Prepared Queries
|
173
538
|
|
174
|
-
|
175
|
-
|
539
|
+
Batch execution can also be done using prepared queries, using the same methods
|
540
|
+
`#batch_execute` and `#batch_query`:
|
176
541
|
|
177
|
-
|
178
|
-
db.
|
542
|
+
```ruby
|
543
|
+
query = db.prepare 'update foo set x = ? where z = ? returning *'
|
179
544
|
|
180
|
-
|
181
|
-
|
545
|
+
query.batch_execute([[42, 3], [43, 6]])
|
546
|
+
#=> 2
|
182
547
|
|
183
|
-
|
184
|
-
|
548
|
+
query.batch_query([[42, 3], [43, 6]])
|
549
|
+
#=> [{ x: 42, y: 2, z: 3 }, { x: 43, y: 5, z: 6 }]
|
550
|
+
```
|
185
551
|
|
186
|
-
|
187
|
-
db.pragma(:journal_mode) #=> 'delete'
|
188
|
-
db.pragma(journal_mode: 'wal')
|
189
|
-
db.pragma(:journal_mode) #=> 'wal'
|
552
|
+
## Transactions and Savepoints
|
190
553
|
|
191
|
-
|
192
|
-
|
554
|
+
All reads and writes to SQLite databases occur within a
|
555
|
+
[transaction](https://www.sqlite.org/lang_transaction.html). If no explicit
|
556
|
+
transaction has started, the submitted SQL statements passed to `#execute` or
|
557
|
+
`#query` will all run within an implicit transaction:
|
558
|
+
|
559
|
+
```ruby
|
560
|
+
# The following two SQL statements will run in a single implicit transaction:
|
561
|
+
db.execute('insert into foo values (42); insert into bar values (43)')
|
562
|
+
|
563
|
+
# Otherwise, each call to #execute runs in a separate transaction:
|
564
|
+
db.execute('insert into foo values (42)')
|
565
|
+
db.execute('insert into bar values (43)')
|
566
|
+
```
|
567
|
+
|
568
|
+
### Explicit Transactions
|
193
569
|
|
194
|
-
|
570
|
+
While you can issue `BEGIN` and `COMMIT` SQL statements yourself to start and
|
571
|
+
commit explicit transactions, Extralite provides a convenient `#transaction`
|
572
|
+
method that manages starting, commiting and rolling back of transactions
|
573
|
+
automatically:
|
574
|
+
|
575
|
+
```ruby
|
195
576
|
db.transaction do
|
196
|
-
db.execute('insert into foo values (
|
197
|
-
|
198
|
-
db.execute('insert into
|
577
|
+
db.execute('insert into foo values (42)')
|
578
|
+
raise 'Something bad happened' if something_bad_happened
|
579
|
+
db.execute('insert into bar values (43)')
|
199
580
|
end
|
581
|
+
```
|
200
582
|
|
201
|
-
|
202
|
-
|
203
|
-
|
583
|
+
If no exception is raised in the transaction block, the changes are commited. If
|
584
|
+
an exception is raised, the changes are rolled back and the exception is
|
585
|
+
propagated to the application code. You can prevent the exception from being
|
586
|
+
propagated by calling `#rollback!`:
|
587
|
+
|
588
|
+
```ruby
|
589
|
+
db.transaction do
|
590
|
+
db.execute('insert into foo values (42)')
|
591
|
+
rollback! if something_bad_happened
|
592
|
+
db.execute('insert into bar values (43)')
|
593
|
+
end
|
204
594
|
```
|
205
595
|
|
206
|
-
|
596
|
+
### Transaction Modes
|
597
|
+
|
598
|
+
By default, `#transaction` starts an `IMMEDIATE` transaction. To start a
|
599
|
+
`DEFERRED` or `EXCLUSIVE` transaction, pass the desired mode to `#transaction`:
|
207
600
|
|
208
|
-
|
601
|
+
```ruby
|
602
|
+
# Start a DEFERRED transaction
|
603
|
+
db.transaction(:deferred) do
|
604
|
+
...
|
605
|
+
end
|
606
|
+
|
607
|
+
# Start a EXCLUSIVE transaction
|
608
|
+
db.transaction(:exclusive) do
|
609
|
+
...
|
610
|
+
end
|
611
|
+
```
|
209
612
|
|
210
|
-
|
211
|
-
the
|
613
|
+
Note that running an `IMMEDIATE` or `EXCLUSIVE` transaction blocks the database
|
614
|
+
for writing (and also reading in certain cases) for the duration of the
|
615
|
+
transaction. This can cause queries to the same database on a different
|
616
|
+
connection to fail with a `BusyError` exception. This can be mitigated by
|
617
|
+
setting a [busy timeout](#dealing-with-a-busy-database).
|
618
|
+
|
619
|
+
### Savepoints
|
620
|
+
|
621
|
+
In addition to transactions, SQLite also supports the use of
|
622
|
+
[savepoints](https://www.sqlite.org/lang_savepoint.html), which can be used for
|
623
|
+
more fine-grained control of changes within a transaction, and to be able to
|
624
|
+
rollback specific changes without abandoning the entire transaction:
|
212
625
|
|
213
626
|
```ruby
|
214
|
-
|
215
|
-
|
216
|
-
|
627
|
+
db.transaction do
|
628
|
+
db.execute 'insert into foo values (1)'
|
629
|
+
|
630
|
+
db.savepoint :my_savepoint
|
631
|
+
db.execute 'insert into foo values (2)'
|
632
|
+
|
633
|
+
# the following cancels the last insert
|
634
|
+
db.rollback_to :my_savepoint
|
635
|
+
db.execute 'insert into foo values (3)'
|
636
|
+
|
637
|
+
db.release :my_savepoint
|
217
638
|
end
|
639
|
+
```
|
218
640
|
|
219
|
-
|
220
|
-
|
221
|
-
|
222
|
-
|
223
|
-
|
641
|
+
## Database Information
|
642
|
+
|
643
|
+
### Getting the list of tables
|
644
|
+
|
645
|
+
To get the list of tables in a database, use the `#tables` method:
|
646
|
+
|
647
|
+
```ruby
|
648
|
+
db.tables
|
649
|
+
#=> [...]
|
650
|
+
```
|
651
|
+
|
652
|
+
To get the list of tables in an attached database, you can pass the database name to `#tables`:
|
653
|
+
|
654
|
+
```ruby
|
655
|
+
db.execute "attach database 'foo.db' as foo"
|
656
|
+
db.tables('foo')
|
657
|
+
#=> [...]
|
658
|
+
```
|
659
|
+
|
660
|
+
### Getting the last insert row id
|
661
|
+
|
662
|
+
```ruby
|
663
|
+
db.execute 'insert into foo values (?)', 42
|
664
|
+
db.last_insert_rowid
|
665
|
+
#=> 1
|
666
|
+
```
|
667
|
+
|
668
|
+
### Getting the columns names for a given query
|
669
|
+
|
670
|
+
```ruby
|
671
|
+
db.columns('select a, b, c from foo')
|
672
|
+
#=> [:a, :b, :c]
|
673
|
+
|
674
|
+
# Columns a prepared query:
|
675
|
+
query = db.prepare('select x, y from foo')
|
676
|
+
query.columns
|
677
|
+
#=> [:x, :y]
|
678
|
+
```
|
679
|
+
|
680
|
+
### Pragmas
|
681
|
+
|
682
|
+
You can get or set pragma values using `#pragma`:
|
683
|
+
|
684
|
+
```ruby
|
685
|
+
# get a pragma value:
|
686
|
+
db.pragma(:journal_mode)
|
687
|
+
#=> 'delete'
|
688
|
+
|
689
|
+
# set a pragma value:
|
690
|
+
db.pragma(journal_mode: 'wal')
|
691
|
+
db.pragma(:journal_mode)
|
692
|
+
#=> 'wal'
|
693
|
+
```
|
694
|
+
|
695
|
+
You can also pass pragmas when opening the database:
|
696
|
+
|
697
|
+
```ruby
|
698
|
+
db = Extralite::Database.new('path/to/db', pragma: { foreign_keys: true })
|
699
|
+
```
|
700
|
+
|
701
|
+
## Error Handling
|
702
|
+
|
703
|
+
Extralite defines various exception classes that are raised when an error is
|
704
|
+
encountered while interacting with the underlying SQLite library:
|
705
|
+
|
706
|
+
- `Extralite::SQLError` - raised when SQLite encounters an invalid SQL query.
|
707
|
+
- `Extralite::BusyError` - raised when the underlying database is locked for use
|
708
|
+
by another database connection.
|
709
|
+
- `Extralite::InterruptError` - raised when a query has been interrupted.
|
710
|
+
- `Extralite::ParameterError` - raised when an invalid parameter value has been
|
711
|
+
specified.
|
712
|
+
- `Extralite::Error` - raised on all other errors.
|
713
|
+
|
714
|
+
In addition to the above exceptions, further information about the last error
|
715
|
+
that occurred is provided by the following methods:
|
716
|
+
|
717
|
+
- `#errcode` - the [error code](https://www.sqlite.org/rescode.html) returned by
|
718
|
+
the underlying SQLite library.
|
719
|
+
- `#errmsg` - the error message for the last error. For most errors, the error
|
720
|
+
message is copied into the exception message.
|
721
|
+
- `#error_offset` - for SQL errors, the offset into the SQL string where the
|
722
|
+
error was encountered.
|
723
|
+
|
724
|
+
## Concurrency
|
725
|
+
|
726
|
+
Extralite provides a comprehensive set of tools for dealing with concurrency
|
727
|
+
issues, and for making sure that running queries on SQLite databases does not
|
728
|
+
cause the app to freeze.
|
729
|
+
|
730
|
+
**Note**: In order to allow concurrent access your the database, it is highly
|
731
|
+
recommended that you set your database to use [WAL journaling
|
732
|
+
mode](https://www.sqlite.org/wal.html) for *all* database connections.
|
733
|
+
Otherwise, you risking running into performance problems and having queries fail
|
734
|
+
with `BusyError` exceptions. You can easily open your database in WAL journaling
|
735
|
+
mode by passing a `wal: true` option:
|
736
|
+
|
737
|
+
```ruby
|
738
|
+
# This will set PRAGMA journal_mode=1 and PRAGMA synchronous=1
|
739
|
+
db = Extralite::Database.new('path/to/db', wal: true)
|
740
|
+
```
|
741
|
+
|
742
|
+
### The Ruby GVL
|
743
|
+
|
744
|
+
In order to support multi-threading, Extralite releases the [Ruby
|
745
|
+
GVL](https://www.speedshop.co/2020/05/11/the-ruby-gvl-and-scaling.html)
|
746
|
+
periodically while running queries. This allows other threads to run while the
|
747
|
+
underlying SQLite library is busy preparing queries, fetching records and
|
748
|
+
backing up databases. By default, the GVL is when preparing the query, and once
|
749
|
+
for every 1000 iterated records. The GVL release threshold can be set separately
|
750
|
+
for each database:
|
751
|
+
|
752
|
+
```ruby
|
753
|
+
# release the GVL on preparing the query, and every 10 records
|
754
|
+
db.gvl_release_threshold = 10
|
755
|
+
|
756
|
+
# release the GVL only when preparing the query
|
757
|
+
db.gvl_release_threshold = 0
|
758
|
+
|
759
|
+
# never release the GVL (for single-threaded apps only)
|
760
|
+
db.gvl_release_threshold = -1
|
761
|
+
|
762
|
+
db.gvl_release_threshold = nil # use default value (currently 1000)
|
763
|
+
```
|
764
|
+
|
765
|
+
For most applications, there's no need to tune the GVL threshold value, as it
|
766
|
+
provides [excellent](#performance) performance characteristics for both
|
767
|
+
single-threaded and multi-threaded applications.
|
768
|
+
|
769
|
+
In a heavily multi-threaded application, releasing the GVL more often (lower
|
770
|
+
threshold value) will lead to less latency (for threads not running a query),
|
771
|
+
but will also hurt the throughput (for the thread running the query). Releasing
|
772
|
+
the GVL less often (higher threshold value) will lead to better throughput for
|
773
|
+
queries, while increasing latency for threads not running a query. The following
|
774
|
+
diagram demonstrates the relationship between the GVL release threshold value,
|
775
|
+
latency and throughput:
|
776
|
+
|
777
|
+
```
|
778
|
+
less latency & throughput <<< GVL release threshold >>> more latency & throughput
|
779
|
+
```
|
780
|
+
|
781
|
+
### Dealing with a Busy Database
|
782
|
+
|
783
|
+
When multiple threads or processes access the same database, the database may be
|
784
|
+
locked for writing by one process, which will block other processes wishing to
|
785
|
+
write to the database. When attempting to write to a locked database, a
|
786
|
+
`Extralite::BusyError` will be raised:
|
787
|
+
|
788
|
+
```ruby
|
789
|
+
ready = nil
|
790
|
+
locker = Thread.new do
|
791
|
+
db1 = Extralite::Database.new('my.db')
|
792
|
+
# Lock the database for 3 seconds
|
793
|
+
db1.transaction do
|
794
|
+
ready = true
|
795
|
+
sleep(3)
|
796
|
+
end
|
797
|
+
end
|
798
|
+
|
799
|
+
db2 = Extralite::Database.new('my.db')
|
800
|
+
# wait for writer1 to enter a transaction
|
801
|
+
sleep(0) while !ready
|
802
|
+
# This will raise a Extralite::BusyError
|
803
|
+
db2.transaction { }
|
804
|
+
# Extralite::BusyError!
|
805
|
+
```
|
806
|
+
|
807
|
+
You can mitigate this by setting a busy timeout. This will cause SQLite to wait
|
808
|
+
for the database to become unlocked up to the specified timeout period:
|
809
|
+
|
810
|
+
```ruby
|
811
|
+
# Wait for up to 5 seconds before giving up
|
812
|
+
db2.busy_timeout = 5
|
813
|
+
# Now it'll work!
|
814
|
+
db2.transaction { }
|
815
|
+
```
|
816
|
+
|
817
|
+
As stated above, setting the database to use WAL journaling mode greatly reduces
|
818
|
+
contention between different process/threads accessing the same database. For
|
819
|
+
most use cases, setting the busy timeout solves the problem of failing to run
|
820
|
+
queries because of a busy database, as normally transactions are short-lived.
|
821
|
+
|
822
|
+
However, in some cases, such as when running a multi-fibered app or when
|
823
|
+
implementing your own timeout mechanisms, you'll want to set a [progress
|
824
|
+
handler](#the-progress-handler).
|
825
|
+
|
826
|
+
### Interrupting a Query
|
827
|
+
|
828
|
+
To interrupt an ongoing query, use the `#interrupt` method. Normally this is
|
829
|
+
done from a separate thread. Here's a way to implement a timeout using
|
830
|
+
`#interrupt`:
|
831
|
+
|
832
|
+
```ruby
|
833
|
+
def run_query_with_timeout(sql, timeout)
|
834
|
+
timeout_thread = Thread.new do
|
835
|
+
t0 = Time.now
|
836
|
+
sleep(timeout)
|
837
|
+
@db.interrupt
|
838
|
+
end
|
839
|
+
result = @db.query(sql)
|
224
840
|
timeout_thread.kill
|
225
|
-
|
841
|
+
result
|
226
842
|
end
|
843
|
+
|
844
|
+
run_query_with_timeout('select 1 as foo', 5)
|
845
|
+
#=> [{ foo: 1 }]
|
846
|
+
|
847
|
+
# A timeout will cause a Extralite::InterruptError to be raised
|
848
|
+
run_query_with_timeout(slow_sql, 5)
|
849
|
+
#=> Extralite::InterruptError!
|
227
850
|
```
|
228
851
|
|
229
|
-
|
852
|
+
You can also call `#interrupt` from within the [progress
|
853
|
+
handler](#the-progress-handler).
|
854
|
+
|
855
|
+
### The Progress Handler
|
856
|
+
|
857
|
+
Extralite also supports setting up a progress handler, which is a piece of code
|
858
|
+
that will be called while a query is in progress, or while the database is busy.
|
859
|
+
This is useful especially when you want to implement a general purpose timeout
|
860
|
+
mechanism that deals with both a busy database and with slow queries.
|
861
|
+
|
862
|
+
The progress handler can also be used for performing any kind of operation while
|
863
|
+
a query is in progress. Here are some use cases:
|
230
864
|
|
231
|
-
|
232
|
-
|
233
|
-
the
|
234
|
-
|
235
|
-
|
865
|
+
- Interrupting queries that take too long to run.
|
866
|
+
- Interrupting queries on an exceptional condition, such as a received signal.
|
867
|
+
- Updating the UI while a query is running.
|
868
|
+
- Switching between fibers in multi-fibered apps.
|
869
|
+
- Switching between threads in multi-threaded apps.
|
870
|
+
- Instrumenting the performance of queries.
|
871
|
+
|
872
|
+
Setting the progress handler requires that Extralite hold the GVL while running
|
873
|
+
all queries. Therefore, it should be used with care. In a multi-threaded app,
|
874
|
+
you'll need to call `Thread.pass` from the progress handler in order for other
|
875
|
+
threads to be able to run while the query is in progress. The progress handler
|
876
|
+
is set per-database using `#on_progress`. This method takes a single parameter
|
877
|
+
that specifies the approximate number of SQLite VM instructions between
|
878
|
+
successive calls to the progress handler:
|
879
|
+
|
880
|
+
```ruby
|
881
|
+
db.on_progress do
|
882
|
+
check_for_timeout
|
883
|
+
# Allow other threads to run
|
884
|
+
Thread.pass
|
885
|
+
end
|
886
|
+
```
|
887
|
+
|
888
|
+
The progress handler can be used to interrupt queries in progress. This can be
|
889
|
+
done by either calling `#interrupt`, or by raising an exception. As discussed
|
890
|
+
above, calling `#interrupt` causes the query to raise a
|
891
|
+
`Extralite::InterruptError` exception:
|
892
|
+
|
893
|
+
```ruby
|
894
|
+
db.on_progress { db.interrupt }
|
895
|
+
db.query('select 1')
|
896
|
+
#=> Extralite::InterruptError!
|
897
|
+
```
|
898
|
+
|
899
|
+
You can also interrupt queries in progress by raising an exception. The query
|
900
|
+
will be stopped, and the exception will propagate to the call site:
|
901
|
+
|
902
|
+
```ruby
|
903
|
+
db.on_progress do
|
904
|
+
raise 'BOOM!'
|
905
|
+
end
|
906
|
+
|
907
|
+
db.query('select 1')
|
908
|
+
#=> BOOM!
|
909
|
+
```
|
910
|
+
|
911
|
+
Here's how a timeout might be implemented using the progress handler:
|
912
|
+
|
913
|
+
```ruby
|
914
|
+
def setup_progress_handler
|
915
|
+
@db.on_progress do
|
916
|
+
raise TimeoutError if Time.now - @t0 >= @timeout
|
917
|
+
Thread.pass
|
918
|
+
end
|
919
|
+
end
|
920
|
+
|
921
|
+
# In this example, we just return nil on timeout
|
922
|
+
def run_query_with_timeout(sql, timeout)
|
923
|
+
@t0 = Time.now
|
924
|
+
@db.query(sql)
|
925
|
+
rescue TimeoutError
|
926
|
+
nil
|
927
|
+
end
|
928
|
+
|
929
|
+
run_query_with_timeout('select 1 as foo', 5)
|
930
|
+
#=> [{ foo: 1 }]
|
931
|
+
|
932
|
+
run_query_with_timeout(slow_sql, 5)
|
933
|
+
#=> nil
|
934
|
+
```
|
935
|
+
|
936
|
+
**Note**: you must not issue any query from within the progress handler.
|
937
|
+
|
938
|
+
### Dealing with a Busy Database in the Progress Handler
|
939
|
+
|
940
|
+
As mentioned above, the progress handler is also called when the database is
|
941
|
+
busy, regardless of the progress period given to `#on_progress`. You can detect
|
942
|
+
if the database is busy by checking the first argument passed to the progress
|
943
|
+
handler, which will be true when busy:
|
236
944
|
|
237
945
|
```ruby
|
238
|
-
db.
|
239
|
-
|
240
|
-
|
946
|
+
db.on_progress do |busy|
|
947
|
+
if busy
|
948
|
+
foo
|
949
|
+
else
|
950
|
+
bar
|
951
|
+
end
|
952
|
+
end
|
241
953
|
```
|
242
954
|
|
243
|
-
|
244
|
-
|
955
|
+
This allows you to implement separate logic to deal with busy states, for
|
956
|
+
example sleeping for a small period of time, or implementing a different timeout
|
957
|
+
period.
|
958
|
+
|
959
|
+
### Advanced Progress Handler Settings
|
960
|
+
|
961
|
+
You can further tune the behaviour of the progress handler with the following
|
962
|
+
options passed to `#on_progress`:
|
963
|
+
|
964
|
+
- `:mode`: the following modes are supported:
|
965
|
+
- `:none` : the progress handler is disabled.
|
966
|
+
- `:normal`: the progress handler is called on query progress (this is the
|
967
|
+
default mode).
|
968
|
+
- `:once`: the progress handler is called once before running the query.
|
969
|
+
- `:at_least_once`: the progress handler is called once before running the
|
970
|
+
query, and on query progress.
|
971
|
+
- `:period`: controls the approximate number of SQLite VM instructions executed
|
972
|
+
between consecutive calls to the progress handler. Default value: 1000.
|
973
|
+
- `:tick`: controls the granularity of the progress handler. This is the value
|
974
|
+
passed internally to the SQLite library. Default value: 10.
|
975
|
+
|
976
|
+
```ruby
|
977
|
+
db.on_progress(mode: :at_least_once, period: 640, tick: 5) { snooze }
|
978
|
+
```
|
979
|
+
|
980
|
+
### Global Progress Handler Settings
|
981
|
+
|
982
|
+
You can set the global progress handler behaviour by calling
|
983
|
+
`Extralite.on_progress`. You can use this API to set the global progress
|
984
|
+
settings, without needing to set a progress handler individually for each
|
985
|
+
`Database` instance. This method takes the same options as
|
986
|
+
`Database#on_progress`:
|
987
|
+
|
988
|
+
```ruby
|
989
|
+
Extralite.on_progress(mode: :at_least_once, period: 640, tick: 5) { snooze }
|
990
|
+
|
991
|
+
# the new database instance uses the global progress handler settings
|
992
|
+
db = Database.new(':memory:')
|
993
|
+
```
|
994
|
+
|
995
|
+
### Extralite and Fibers
|
996
|
+
|
997
|
+
The progress handler can also be used to switch between fibers in a
|
998
|
+
multi-fibered Ruby app, based on libraries such as
|
999
|
+
[Async](https://github.com/socketry/async) or
|
1000
|
+
[Polyphony](https://github.com/digital-fabric/polyphony). A general solution
|
1001
|
+
(that also works for multi-threaded apps) is to call `sleep(0)` in the progress
|
1002
|
+
handler. This will work for switching between fibers using either Polyphony or
|
1003
|
+
any fiber scheduler gem, such as Async et al:
|
1004
|
+
|
1005
|
+
```ruby
|
1006
|
+
db.on_progress(100) { sleep(0) }
|
1007
|
+
```
|
1008
|
+
|
1009
|
+
For Polyphony-based apps, you can also call `snooze` to allow other fibers to
|
1010
|
+
run while a query is progressing. If your Polyphony app is multi-threaded,
|
1011
|
+
you'll also need to call `Thread.pass` in order to allow other threads to run:
|
1012
|
+
|
1013
|
+
```ruby
|
1014
|
+
db.on_progress(100) do
|
1015
|
+
snooze
|
1016
|
+
Thread.pass
|
1017
|
+
end
|
1018
|
+
```
|
1019
|
+
|
1020
|
+
Note that with Polyphony, once you install the progress handler, you can just
|
1021
|
+
use the regular `#move_on_after` and `#cancel_after` methods to implement
|
1022
|
+
timeouts for queries:
|
1023
|
+
|
1024
|
+
```ruby
|
1025
|
+
db.on_progress(100) { snooze }
|
1026
|
+
|
1027
|
+
cancel_after(3) do
|
1028
|
+
db.query(long_running_query)
|
1029
|
+
end
|
1030
|
+
```
|
1031
|
+
|
1032
|
+
### Thread Safety
|
1033
|
+
|
1034
|
+
A single database instance can be safely used in multiple threads simultaneously
|
1035
|
+
as long as the following conditions are met:
|
1036
|
+
|
1037
|
+
- No explicit transactions are used.
|
1038
|
+
- Each thread issues queries by calling `Database#query_xxx`, or uses a separate
|
1039
|
+
`Query` instance.
|
1040
|
+
- The GVL release threshold is not `0` (i.e. the GVL is released periodically
|
1041
|
+
while running queries.)
|
1042
|
+
|
1043
|
+
### Use with Ractors
|
1044
|
+
|
1045
|
+
Extralite databases can safely be used inside ractors. A ractor has the benefit
|
1046
|
+
of using a separate GVL from the maine one, which allows true parallelism for
|
1047
|
+
Ruby apps. So when you use Extralite to access SQLite databases from within a
|
1048
|
+
ractor, you can do so without any considerations for what's happening outside
|
1049
|
+
the ractor when it runs queries.
|
1050
|
+
|
1051
|
+
**Note**: Ractors are considered an experimental feature of Ruby. You may
|
1052
|
+
encounter errors or inconsistent behaviour when using ractors.
|
1053
|
+
|
1054
|
+
## Advanced Usage
|
1055
|
+
|
1056
|
+
### Loading Extensions
|
1057
|
+
|
1058
|
+
Extensions can be loaded by calling `#load_extension`:
|
1059
|
+
|
1060
|
+
```ruby
|
1061
|
+
db.load_extension('/path/to/extension.so')
|
1062
|
+
```
|
1063
|
+
|
1064
|
+
A pretty comprehensive set of extensions can be found here:
|
1065
|
+
|
1066
|
+
https://github.com/nalgeon/sqlean
|
245
1067
|
|
246
1068
|
### Creating Backups
|
247
1069
|
|
@@ -266,6 +1088,51 @@ db.backup('backup.db') do |remaining, total|
|
|
266
1088
|
end
|
267
1089
|
```
|
268
1090
|
|
1091
|
+
### Working with Changesets
|
1092
|
+
|
1093
|
+
__Note__: as the session extension is by default disabled in SQLite
|
1094
|
+
distributions, support for changesets is currently only available withthe
|
1095
|
+
bundled version of Extralite, `extralite-bundle`.
|
1096
|
+
|
1097
|
+
Changesets can be used to track and persist changes to data in a database. They
|
1098
|
+
can also be used to apply the same changes to another database, or to undo them.
|
1099
|
+
To track changes to a database, use the `#track_changes` method:
|
1100
|
+
|
1101
|
+
```ruby
|
1102
|
+
# track changes to the foo and bar tables:
|
1103
|
+
changeset = db.track_changes(:foo, :bar) do
|
1104
|
+
insert_a_bunch_of_records(db)
|
1105
|
+
end
|
1106
|
+
|
1107
|
+
# to track changes to all tables, pass nil:
|
1108
|
+
changeset = db.track_changes(nil) do
|
1109
|
+
insert_a_bunch_of_records(db)
|
1110
|
+
end
|
1111
|
+
```
|
1112
|
+
|
1113
|
+
You can then apply the same changes to another database:
|
1114
|
+
|
1115
|
+
```ruby
|
1116
|
+
changeset.apply(some_other_db)
|
1117
|
+
```
|
1118
|
+
|
1119
|
+
To undo the changes, obtain an inverted changeset and apply it to the database:
|
1120
|
+
|
1121
|
+
```ruby
|
1122
|
+
changeset.invert.apply(db)
|
1123
|
+
```
|
1124
|
+
|
1125
|
+
You can also save and load the changeset:
|
1126
|
+
|
1127
|
+
```ruby
|
1128
|
+
# save the changeset
|
1129
|
+
IO.write('my.changes', changeset.to_blob)
|
1130
|
+
|
1131
|
+
# load the changeset
|
1132
|
+
changeset = Extralite::Changeset.new
|
1133
|
+
changeset.load(IO.read('my.changes'))
|
1134
|
+
```
|
1135
|
+
|
269
1136
|
### Retrieving Status Information
|
270
1137
|
|
271
1138
|
Extralite provides methods for retrieving status information about the sqlite
|
@@ -305,16 +1172,6 @@ value = db.limit(Extralite::SQLITE_LIMIT_ATTACHED)
|
|
305
1172
|
db.limit(Extralite::SQLITE_LIMIT_ATTACHED, new_value)
|
306
1173
|
```
|
307
1174
|
|
308
|
-
### Setting the Busy Timeout
|
309
|
-
|
310
|
-
When accessing a database concurrently it can be handy to set a busy timeout, in
|
311
|
-
order to not have to deal with rescuing `Extralite::BusyError` exceptions. The
|
312
|
-
timeout is given in seconds:
|
313
|
-
|
314
|
-
```ruby
|
315
|
-
db.busy_timeout = 5
|
316
|
-
```
|
317
|
-
|
318
1175
|
### Tracing SQL Statements
|
319
1176
|
|
320
1177
|
To trace all SQL statements executed on the database, pass a block to
|
@@ -342,58 +1199,6 @@ p articles.to_a
|
|
342
1199
|
|
343
1200
|
(Make sure you include `extralite` as a dependency in your `Gemfile`.)
|
344
1201
|
|
345
|
-
## Concurrency
|
346
|
-
|
347
|
-
### The Ruby GVL
|
348
|
-
|
349
|
-
Extralite releases the [Ruby
|
350
|
-
GVL](https://www.speedshop.co/2020/05/11/the-ruby-gvl-and-scaling.html) while
|
351
|
-
making calls to the sqlite3 library that might block, such as when backing up a
|
352
|
-
database, or when preparing a query. This allows other threads to run while the
|
353
|
-
underlying sqlite3 library is busy preparing queries, fetching records and
|
354
|
-
backing up databases.
|
355
|
-
|
356
|
-
Extralite also releases the GVL periodically when iterating over records. By
|
357
|
-
default, the GVL is released every 1000 records iterated. The GVL release
|
358
|
-
threshold can be set separately for each database:
|
359
|
-
|
360
|
-
```ruby
|
361
|
-
db.gvl_release_threshold = 10 # release GVL every 10 records
|
362
|
-
|
363
|
-
db.gvl_release_threshold = nil # use default value (currently 1000)
|
364
|
-
```
|
365
|
-
|
366
|
-
For most applications, there's no need to tune the GVL threshold value, as it
|
367
|
-
provides [excellent](#performance) performance characteristics for both
|
368
|
-
single-threaded and multi-threaded applications.
|
369
|
-
|
370
|
-
In a heavily multi-threaded application, releasing the GVL more often (lower
|
371
|
-
threshold value) will lead to less latency (for threads not running a query),
|
372
|
-
but will also hurt the throughput (for the thread running the query). Releasing
|
373
|
-
the GVL less often (higher threshold value) will lead to better throughput for
|
374
|
-
queries, while increasing latency for threads not running a query. The following
|
375
|
-
diagram demonstrates the relationship between the GVL release threshold value,
|
376
|
-
latency and throughput:
|
377
|
-
|
378
|
-
```
|
379
|
-
less latency & throughput <<< GVL release threshold >>> more latency & throughput
|
380
|
-
```
|
381
|
-
|
382
|
-
### Thread Safety
|
383
|
-
|
384
|
-
A single database instance can be safely used in multiple threads simultaneously
|
385
|
-
as long as the following conditions are met:
|
386
|
-
|
387
|
-
- No explicit transactions are used.
|
388
|
-
- Each thread issues queries by calling `Database#query_xxx`, or uses a separate
|
389
|
-
`Query` instance.
|
390
|
-
- The GVL release threshold is not `0` (i.e. the GVL is released periodically
|
391
|
-
while running queries.)
|
392
|
-
|
393
|
-
### Use with Ractors
|
394
|
-
|
395
|
-
Extralite databases can be used inside ractors
|
396
|
-
|
397
1202
|
## Performance
|
398
1203
|
|
399
1204
|
A benchmark script is included, creating a table of various row counts, then
|
@@ -403,37 +1208,44 @@ large number of rows.
|
|
403
1208
|
|
404
1209
|
### Rows as Hashes
|
405
1210
|
|
406
|
-
|
1211
|
+
[Benchmark source
|
1212
|
+
code](https://github.com/digital-fabric/extralite/blob/main/test/perf_hash.rb)
|
407
1213
|
|
408
|
-
|Row count|sqlite3 1.
|
1214
|
+
|Row count|sqlite3 1.7.0|Extralite 2.5|Advantage|
|
409
1215
|
|-:|-:|-:|-:|
|
410
|
-
|10|
|
411
|
-
|1K|
|
412
|
-
|100K|
|
1216
|
+
|10|184.9K rows/s|473.2K rows/s|__2.56x__|
|
1217
|
+
|1K|290.5K rows/s|2320.7K rows/s|__7.98x__|
|
1218
|
+
|100K|143.0K rows/s|2061.3K rows/s|__14.41x__|
|
413
1219
|
|
414
1220
|
### Rows as Arrays
|
415
1221
|
|
416
|
-
[Benchmark source
|
1222
|
+
[Benchmark source
|
1223
|
+
code](https://github.com/digital-fabric/extralite/blob/main/test/perf_ary.rb)
|
417
1224
|
|
418
|
-
|Row count|sqlite3 1.
|
1225
|
+
|Row count|sqlite3 1.7.0|Extralite 2.5|Advantage|
|
419
1226
|
|-:|-:|-:|-:|
|
420
|
-
|10|
|
421
|
-
|1K|
|
422
|
-
|100K|
|
1227
|
+
|10|276.9K rows/s|472.3K rows/s|__1.71x__|
|
1228
|
+
|1K|615.6K rows/s|2324.3K rows/s|__3.78x__|
|
1229
|
+
|100K|477.4K rows/s|1982.7K rows/s|__4.15x__|
|
423
1230
|
|
424
1231
|
### Prepared Queries (Prepared Statements)
|
425
1232
|
|
426
|
-
[Benchmark source
|
1233
|
+
[Benchmark source
|
1234
|
+
code](https://github.com/digital-fabric/extralite/blob/main/test/perf_hash_prepared.rb)
|
427
1235
|
|
428
|
-
|Row count|sqlite3 1.
|
1236
|
+
|Row count|sqlite3 1.7.0|Extralite 2.5|Advantage|
|
429
1237
|
|-:|-:|-:|-:|
|
430
|
-
|10|
|
431
|
-
|1K|
|
432
|
-
|100K|
|
1238
|
+
|10|228.5K rows/s|707.9K rows/s|__3.10x__|
|
1239
|
+
|1K|296.5K rows/s|2396.2K rows/s|__8.08x__|
|
1240
|
+
|100K|145.9K rows/s|2107.3K rows/s|__14.45x__|
|
1241
|
+
|
1242
|
+
As those benchmarks show, Extralite is capabale of reading up to 2.4M rows per
|
1243
|
+
second, and can be more than 14 times faster than the `sqlite3` gem.
|
433
1244
|
|
434
|
-
|
435
|
-
|
436
|
-
|
1245
|
+
Note that the benchmarks above were performed on synthetic data, in a
|
1246
|
+
single-threaded environment, with the GVL release threshold set to -1, which
|
1247
|
+
means that both Extralite and the `sqlite3` gem hold the GVL for the duration of
|
1248
|
+
the query.
|
437
1249
|
|
438
1250
|
## License
|
439
1251
|
|