extralite 2.5 → 2.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (45) hide show
  1. checksums.yaml +4 -4
  2. data/.gitignore +1 -0
  3. data/CHANGELOG.md +34 -13
  4. data/Gemfile +4 -0
  5. data/Gemfile-bundle +1 -1
  6. data/LICENSE +1 -1
  7. data/README.md +1059 -247
  8. data/Rakefile +18 -0
  9. data/TODO.md +0 -7
  10. data/examples/kv_store.rb +49 -0
  11. data/examples/multi_fiber.rb +16 -0
  12. data/examples/on_progress.rb +9 -0
  13. data/examples/pubsub_store_polyphony.rb +194 -0
  14. data/examples/pubsub_store_threads.rb +204 -0
  15. data/ext/extralite/changeset.c +463 -0
  16. data/ext/extralite/common.c +177 -91
  17. data/ext/extralite/database.c +745 -276
  18. data/ext/extralite/extconf-bundle.rb +10 -4
  19. data/ext/extralite/extconf.rb +34 -34
  20. data/ext/extralite/extralite.h +104 -47
  21. data/ext/extralite/extralite_ext.c +6 -0
  22. data/ext/extralite/iterator.c +14 -86
  23. data/ext/extralite/query.c +171 -264
  24. data/extralite-bundle.gemspec +1 -1
  25. data/extralite.gemspec +1 -1
  26. data/gemspec.rb +10 -11
  27. data/lib/extralite/version.rb +1 -1
  28. data/lib/extralite.rb +69 -10
  29. data/lib/sequel/adapters/extralite.rb +1 -1
  30. data/test/helper.rb +9 -1
  31. data/test/perf_argv_transform.rb +74 -0
  32. data/test/perf_ary.rb +14 -12
  33. data/test/perf_hash.rb +17 -15
  34. data/test/perf_hash_prepared.rb +58 -0
  35. data/test/perf_hash_transform.rb +66 -0
  36. data/test/perf_polyphony.rb +74 -0
  37. data/test/test_changeset.rb +161 -0
  38. data/test/test_database.rb +720 -104
  39. data/test/test_extralite.rb +2 -2
  40. data/test/test_iterator.rb +28 -13
  41. data/test/test_query.rb +352 -110
  42. data/test/test_sequel.rb +4 -4
  43. metadata +24 -16
  44. data/Gemfile.lock +0 -37
  45. data/test/perf_prepared.rb +0 -64
data/README.md CHANGED
@@ -1,247 +1,1069 @@
1
- # Extralite - a Super Fast Ruby Gem for Working with SQLite3 Databases
2
-
3
- * Source code: https://github.com/digital-fabric/extralite
4
- * Documentation: http://www.rubydoc.info/gems/extralite
5
-
6
- [![Ruby gem](https://badge.fury.io/rb/extralite.svg)](https://rubygems.org/gems/extralite) [![Tests](https://github.com/digital-fabric/extralite/workflows/Tests/badge.svg)](https://github.com/digital-fabric/extralite/actions?query=workflow%3ATests) [![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/digital-fabric/extralite/blob/master/LICENSE)
1
+ <h1 align="center">
2
+ <br>
3
+ Extralite
4
+ </h1>
5
+
6
+ <h4 align="center">Ruby on SQLite</h4>
7
+
8
+ <p align="center">
9
+ <a href="http://rubygems.org/gems/extralite">
10
+ <img src="https://badge.fury.io/rb/extralite.svg" alt="Ruby gem">
11
+ </a>
12
+ <a href="https://github.com/digital-fabric/extralite/actions">
13
+ <img src="https://github.com/digital-fabric/extralite/actions/workflows/test.yml/badge.svg" alt="Tests">
14
+ </a>
15
+ <a href="https://github.com/digital-fabric/extralite/blob/master/LICENSE">
16
+ <img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License">
17
+ </a>
18
+ </p>
19
+
20
+ <p align="center">
21
+ <a href="https://www.rubydoc.info/gems/extralite">API reference</a>
22
+ </p>
7
23
 
8
24
  ## What is Extralite?
9
25
 
10
- Extralite is a super fast, extra-lightweight (about 1300 lines of C-code)
11
- SQLite3 wrapper for Ruby. It provides a minimal set of methods for interacting
12
- with an SQLite3 database, as well as prepared queries (prepared statements).
26
+ Extralite is a fast and innovative SQLite wrapper for Ruby with a rich set of
27
+ features. It provides multiple ways of retrieving data from SQLite databases,
28
+ makes it possible to use SQLite databases in multi-threaded and multi-fibered
29
+ Ruby apps, and includes a comprehensive set of tools for managing SQLite
30
+ databases.
13
31
 
14
32
  Extralite comes in two flavors: the `extralite` gem which uses the
15
33
  system-installed sqlite3 library, and the `extralite-bundle` gem which bundles
16
34
  the latest version of SQLite
17
- ([3.44.2](https://sqlite.org/releaselog/3_44_2.html)), offering access to the
35
+ ([3.45.0](https://sqlite.org/releaselog/3_45_0.html)), offering access to the
18
36
  latest features and enhancements.
19
37
 
20
38
  ## Features
21
39
 
22
- - Super fast - [up to 11x faster](#performance) than the
23
- [sqlite3](https://github.com/sparklemotion/sqlite3-ruby) gem.
24
- - A variety of methods for different data access patterns: rows as hashes, rows
25
- as arrays, single row, single column, single value.
26
- - Prepared statements.
27
- - Parameter binding.
28
- - External iteration - get single records or batches of records.
29
- - Use system-installed sqlite3, or the [bundled latest version of
30
- SQLite3](#installing-the-extralite-sqlite3-bundle).
31
- - Improved [concurrency](#concurrency) for multithreaded apps: the Ruby GVL is
32
- released peridically while preparing SQL statements and while iterating over
33
- results.
34
- - Automatically execute SQL strings containing multiple semicolon-separated
35
- queries (handy for creating/modifying schemas).
36
- - Execute the same query with multiple parameter lists (useful for inserting
37
- records).
38
- - Load extensions (loading of extensions is autmatically enabled. You can find
39
- some useful extensions here: https://github.com/nalgeon/sqlean.)
40
- - Includes a [Sequel adapter](#usage-with-sequel).
41
-
42
- ## Installation
43
-
44
- To use Extralite in your Ruby app, add the following to your `Gemfile`:
40
+ - Best-in-class [performance](#performance) (up to 14X the performance of the
41
+ [sqlite3](https://github.com/sparklemotion/sqlite3-ruby) gem).
42
+ - Support for [concurrency](#concurrency) out of the box for multi-threaded
43
+ and multi-fibered apps.
44
+ - A variety of ways to [retrieve data](#query-modes) - hashes, arrays, single
45
+ columns, single rows, [transforms](#value-transforms).
46
+ - Support for [external iteration](#iterating-over-records-in-a-prepared-query),
47
+ allowing iterating through single records or batches of records.
48
+ - [Prepared queries](#prepared-queries).
49
+ - [Parameter binding](#parameter-binding).
50
+ - [Batch execution](#batch-execution-of-queries) of queries.
51
+ - [transactions and savepoints](#transactions-and-savepoints).
52
+ - Advanced features: load [SQLite extensions](#loading-extensions), create
53
+ [backups](#creating-backups), retrieve [status
54
+ information](#retrieving-status-information), work with
55
+ [changesets](#working-with-changesets), interrogate [database
56
+ limits](#working-with-database-limits), [trace](#tracing-sql-statements)
57
+ queries.
58
+ - [Sequel](#usage-with-sequel) adapter.
59
+
60
+ ## Table of Content
61
+
62
+ - [Installing Extralite](#installing-extralite)
63
+ - [Getting Started](#getting-started)
64
+ - [Query Modes](#query-modes)
65
+ - [Parameter binding](#parameter-binding)
66
+ - [Value Transforms](#value-transforms)
67
+ - [Data Types](#data-types)
68
+ - [Prepared Queries](#prepared-queries)
69
+ - [Batch Execution of Queries](#batch-execution-of-queries)
70
+ - [Transactions and Savepoints](#transactions-and-savepoints)
71
+ - [Database Information](#database-information)
72
+ - [Error Handling](#error-handling)
73
+ - [Concurrency](#concurrency)
74
+ - [Advanced Usage](#advanced-usage)
75
+ - [Usage with Sequel](#usage-with-sequel)
76
+ - [Performance](#performance)
77
+ - [License](#license)
78
+ - [Contributing](#contributing)
79
+
80
+ ## Installing Extralite
81
+
82
+ Using bundler:
45
83
 
46
84
  ```ruby
47
85
  gem 'extralite'
48
86
  ```
49
87
 
50
- You can also run `gem install extralite` if you just want to check it out.
88
+ Or manually:
51
89
 
52
- __Note__: Extralite supports Ruby 3.0 and higher.
90
+ ```bash
91
+ $ gem install extralite
92
+ ```
53
93
 
54
- ### Installing the Extralite-SQLite3 Bundle
94
+ __Note__: Extralite supports Ruby 3.0 and newer.
55
95
 
56
- If you don't have sqlite3 installed on your system, do not want to use the
57
- system-installed version of SQLite3, or would like to use the latest version of
58
- SQLite3, you can install the `extralite-bundle` gem, which integrates the
59
- SQLite3 source code.
96
+ ### Installing the Extralite-SQLite Bundle
60
97
 
61
- > **Important note**: The `extralite-bundle` gem will take a while to install
62
- > (on my modest machine it takes about a minute), due to the size of the sqlite3
63
- > code.
98
+ If you don't have the sqlite3 lib installed on your system, do not want to use
99
+ the system-installed version of SQLite, or would like to use the latest version
100
+ of SQLite, you can install the `extralite-bundle` gem, which integrates the
101
+ SQLite source code.
64
102
 
65
103
  Usage of the `extralite-bundle` gem is identical to the usage of the normal
66
104
  `extralite` gem, using `require 'extralite'` to load the gem.
67
105
 
68
- ## Synopsis
106
+ ## Getting Started
107
+
108
+ The following example shows how to open an SQLite database and run some queries:
109
+
110
+ ```ruby
111
+ db = Extralite::Database.new('mydb.sqlite')
112
+ db.execute('create table foo (x, y, z)')
113
+ db.execute('insert into foo values (?, ?, ?)', 1, 2, 3)
114
+ db.execute('insert into foo values (?, ?, ?)', 4, 5, 6)
115
+ db.query('select * from foo') #=> [{x: 1, y: 2, z: 3}, {x: 4, y: 5, z: 6}]
116
+ ```
117
+
118
+ The `#execute` method is used to make changes to the database, such as creating
119
+ tables, or inserting records. It returns the number of records changed by the
120
+ query.
121
+
122
+ The `#query` method is used to read data from the database. It returns an array
123
+ containing the resulting records, represented as hashes mapping column names (as
124
+ symbols) to individual values.
125
+
126
+ You can also iterate on records by providing a block to `#query`:
127
+
128
+ ```ruby
129
+ db.query 'select * from foo' do |r|
130
+ p record: r
131
+ end
132
+ ```
133
+
134
+ ## Query Modes
135
+
136
+ Extralite allows you to retrieve data from SQLite database in the form that most
137
+ a particular context. For some use cases you'll want to work with rows as
138
+ hashes. In other cases, you'll want to work with rows as arrays, or even as
139
+ single values, if you're just reading one column.
140
+
141
+ For that purpose, Extralite offers three different ways, or modes, of retrieving
142
+ records:
143
+
144
+ - `:hash`: read rows as hashes (this is the default mode).
145
+ - `:ary`: read rows as arrays.
146
+ - `:argv`: similar to the `:ary`, except that for queries with a single column,
147
+ the single column value is returned.
148
+
149
+ Extralite provides separate methods for the different modes:
150
+
151
+ ```ruby
152
+ # alias #query_hash
153
+ db.query('select 1') #=> [{ "1" => 1 }]
154
+
155
+ db.query_ary('select 1') #=> [[1]]
156
+
157
+ db.query_argv('select 1') #=> [1]
158
+ ```
159
+
160
+ Notice how all the return values above are arrays. This is because the different
161
+ `#query_xxx` methods are designed to return multiple rows. If you want to just
162
+ get back a single row, use one of the `query_single_xxx` methods:
163
+
164
+ ```ruby
165
+ # alias #query_single_hash
166
+ db.query('select 1') #=> { "1" => 1 }
167
+
168
+ db.query_single_ary('select 1') #=> [1]
169
+
170
+ db.query_single_argv('select 1') #=> 1
171
+ ```
172
+
173
+ ## Parameter binding
174
+
175
+ As shown in the above example, the `#execute` and `#query_xxx` methods accept
176
+ parameters that can be bound to the query, which means that their values will be
177
+ used for each corresponding place-holder (expressed using `?`) in the SQL
178
+ statement:
179
+
180
+ ```ruby
181
+ db.query('select x from my_table where y = ? and z = ?', 'foo', 'bar')
182
+ ```
183
+
184
+ You can also express place holders by specifying their index (starting from 1)
185
+ using `?IDX`:
186
+
187
+ ```ruby
188
+ # use the same value for both place holders:
189
+ db.query('select x from my_table where y = ?1 and z = ?1', 42)
190
+ ```
191
+
192
+ Another possibility is to use named parameters, which can be done by expressing
193
+ place holders as `:KEY`, `@KEY` or `$KEY`:
194
+
195
+ ```ruby
196
+ db.query('select x from my_table where y = $y and z = $z', y: 'foo', z: 'bar')
197
+ ```
198
+
199
+ Extralite supports specifying named parameters using `Struct` or `Data` objects:
200
+
201
+ ```ruby
202
+ MyStruct = Struct.new(:x, :z)
203
+ params = MyStruct.new(42, 6)
204
+ db.execute('update foo set x = $x where z = $z', params)
205
+
206
+ MyData = Data.define(:x, :z)
207
+ params = MyData.new(43, 3)
208
+ db.execute('update foo set x = $x where z = $z', params)
209
+ ```
210
+
211
+ Parameter binding is especially useful for preventing [SQL-injection
212
+ attacks](https://en.wikipedia.org/wiki/SQL_injection), but is also useful when
213
+ combined with [prepared queries](#prepared-queries) when repeatedly running the
214
+ same query over and over.
215
+
216
+ ## Data Types
217
+
218
+ Extralite supports the following data types for either bound parameters or row
219
+ values:
220
+
221
+ - `Integer`
222
+ - `Float`
223
+ - `Boolean` (see below)
224
+ - `String` (see below)
225
+ - nil
226
+
227
+ ### Boolean values
228
+
229
+ SQLite does not have a boolean data type. Extralite will automatically translate
230
+ bound parameter values of `true` or `false` to the integer values `1` and `0`,
231
+ respectively. Note that boolean values stored in the database will be fetched as
232
+ integers.
233
+
234
+ ### String values
235
+
236
+ String parameter values are translated by Extralite to either `TEXT` or `BLOB`
237
+ values according to the string encoding used. Strings with an `ASCII-8BIT` are
238
+ treated as blobs. Otherwise they are treated as text values.
239
+
240
+ Likewise, when fetching records, Extralite will convert a `BLOB` column value to
241
+ a string with `ASCII-8BIT` encoding, and a `TEXT` column value to a string with
242
+ `UTF-8` encoding.
243
+
244
+ ```ruby
245
+ # The following calls will insert blob values into the database
246
+ sql = 'insert into foo values (?)'
247
+ db.execute(sql, File.binread('/path/to/file'))
248
+ db.execute(sql, Extralite::Blob.new('Hello, 世界!'))
249
+ db.execute(sql, 'Hello, 世界!'.force_encoding(Encoding::ASCII_8BIT))
250
+ ```
251
+
252
+ ## Value Transforms
253
+
254
+ Extralite allows you to transform rows to any value your application may need by
255
+ providing a transform proc that takes the raw row values and returns the
256
+ transformed data. The transform proc is passed each resulting row either as a
257
+ hash or as a list of values.
258
+
259
+ Transforms are useful when you need to transform rows into ORM model instances,
260
+ or when you need to do some other transformation on the values retrieved from
261
+ the database.
262
+
263
+ To transform results, pass a transform proc as the first parameter to one of the
264
+ `#query_xxx` methods:
265
+
266
+ ```ruby
267
+ transform = ->(h) { MyModel.new(h) }
268
+ db.query(transform, 'select * from foo')
269
+ #=> rows as instances of MyModel
270
+ ```
271
+
272
+ When using the `argv` mode, the different column values are passed as individual
273
+ values to the transform proc:
274
+
275
+ ```ruby
276
+ transform = ->(a, b, c) { { a:a, b: b, c: JSON.parse(c) } }
277
+ db.query_argv(transform, 'select a, b, c from foo')
278
+ #=> transformed rows
279
+ ```
280
+
281
+ Value transforms can also be done with [prepared
282
+ queries](#value-transforms-in-prepared-queries).
283
+
284
+ ## Prepared Queries
285
+
286
+ Prepared queries (also known as prepared statements) allow you to maximize
287
+ performance and reduce memory usage when running the same query repeatedly. They
288
+ also allow you to create parameterized queries that can be repeatedly executed
289
+ with different parameters:
290
+
291
+ ```ruby
292
+ query = db.prepare('select * from foo where x = ?')
293
+
294
+ # bind parameters and get results as an array of hashes
295
+ query.bind(1).to_a
296
+ #=> [{ x: 1, y: 2, z: 3 }]
297
+ ```
298
+
299
+ ### Binding Values to Prepared Queries
300
+
301
+ To bind parameter values to the query, use the `#bind` method. The parameters
302
+ will remain bound to the query until `#bind` is called again.
303
+
304
+ ```ruby
305
+ query.bind(1)
306
+
307
+ # run the query any number of times
308
+ 3.times { query.to_a }
309
+
310
+ # bind other parameter values
311
+ query.bind(4)
312
+ ```
313
+
314
+ You can also bind parameters when creating the prepared query by passing
315
+ additional parameters to the `Database#prepare` method:
316
+
317
+ ```ruby
318
+ query = db.prepare('select * from foo where x = ?', 1)
319
+ ```
320
+
321
+ ### Fetching Records from a Prepared Query
322
+
323
+ Just like the `Database` interface, prepared queries support getting data using
324
+ three different modes: as a hash, an array or as individual column values. To
325
+ set the mode, you can use one of the `#prepare_xxx` methods:
326
+
327
+ ```ruby
328
+ # hash mode
329
+ db.prepare('select * from foo').to_a
330
+ #=> [{ x: 1, y: 2, z: 3}]
331
+
332
+ # argv mode
333
+ db.prepare_argv('select x from foo').to_a
334
+ #=> [1]
335
+
336
+ # ary mode
337
+ db.prepare_ary('select * from foo').to_a
338
+ #=> [[1, 2, 3]]
339
+ ```
340
+
341
+ You can also set the query mode by getting or setting `#mode`:
342
+
343
+ ```ruby
344
+ q = db.prepare('select * from foo')
345
+ q.to_a #=> [{ x: 1, y: 2, z: 3}]
346
+
347
+ q.mode #=> :hash
348
+ q.mode = :ary
349
+ q.to_a "=> [[1, 2, 3]]
350
+ ```
351
+
352
+ ### Fetching Single Records or Batches of Records
353
+
354
+ Prepared queries let you iterate over records one by one, or by batches. For
355
+ this, use the `#next` method:
69
356
 
70
357
  ```ruby
71
- require 'extralite'
358
+ query = db.prepare('select * from foo')
359
+
360
+ query.next
361
+ #=> { x: 1, y: 2, z: 3 }
362
+ query.next
363
+ #=> { x: 4, y: 5, z: 6 }
364
+
365
+ # Fetch the next 10 records
366
+ query.reset # go back tpo the beginning
367
+ query.next(10)
368
+ #=> [{ x: 1, y: 2, z: 3 }, { x: 4, y: 5, z: 6 }]
369
+
370
+ # Fetch the next row as an array
371
+ query = db.prepare_ary('select * from foo')
372
+ query.next
373
+ #=> [1, 2, 3]
374
+
375
+ # Fetch the next row as a single column
376
+ db.prepare_argv('select z from foo').next
377
+ #=> 3
378
+ ```
379
+
380
+ To detect the end of the result, you can use `#eof?`. To go back to the
381
+ beginning of the result set, use `#reset`. The following example shows how to
382
+ read the query results in batches of 10:
383
+
384
+ ```ruby
385
+ query.reset
386
+ while !query.eof?
387
+ records = query.next(10)
388
+ process_records(records)
389
+ end
390
+ ```
391
+
392
+ ### Iterating over Records in a Prepared Query
393
+
394
+ In addition to the `#next` method, you can also iterate over query results by
395
+ using the familiar `#each` method:
396
+
397
+ ```ruby
398
+ # iterate over records as hashes
399
+ query = db.prepare('select * from foo')
400
+ query.each { |r| ... }
401
+
402
+ # iterate over records as arrays
403
+ query = db.prepare_ary('select * from foo')
404
+ query.each { |r| ... }
72
405
 
73
- # get sqlite3 version
74
- Extralite.sqlite3_version #=> "3.35.2"
406
+ # iterate over records as single values
407
+ query = db.prepare_argv('select a, b, c from foo')
408
+ query.each { |a, b, c| ... }
409
+ ```
75
410
 
76
- # open a database
77
- db = Extralite::Database.new('/tmp/my.db')
411
+ ### Prepared Query as an Enumerable
78
412
 
79
- # get query results as array of hashes
80
- db.query('select 1 as foo') #=> [{ :foo => 1 }]
81
- # or:
82
- db.query_hash('select 1 as foo') #=> [{ :foo => 1 }]
83
- # or iterate over results
84
- db.query('select 1 as foo') { |r| p r }
85
- # { :foo => 1 }
413
+ You can also use a prepared query as an enumerable, allowing you to chain
414
+ enumerable method calls while iterating over the query result set. This is done
415
+ by calling `#each` without a block:
86
416
 
87
- # get query results as array of arrays
88
- db.query_ary('select 1, 2, 3') #=> [[1, 2, 3]]
89
- # or iterate over results
90
- db.query_ary('select 1, 2, 3') { |r| p r }
91
- # [1, 2, 3]
417
+ ```ruby
418
+ iterator = query.each
419
+ #=> Returns an Extralite::Iterator instance
92
420
 
93
- # get a single row as a hash
94
- db.query_single_row("select 1 as foo") #=> { :foo => 1 }
421
+ iterator.map { |r| r[:x] *100 + r[:y] * 10 + r[:z] }
422
+ #=> [123, 345]
423
+ ```
95
424
 
96
- # get single column query results as array of values
97
- db.query_single_column('select 42') #=> [42]
98
- # or iterate over results
99
- db.query_single_column('select 42') { |v| p v }
100
- # 42
425
+ The Iterator class includes the
426
+ [`Enumerable`](https://rubyapi.org/3.3/o/enumerable) module, with all its
427
+ methods, such as `#map`, `#select`, `#inject`, `#lazy` etc. You can also
428
+ instantiate an iterator explicitly:
101
429
 
102
- # get single value from first row of results
103
- db.query_single_value("select 'foo'") #=> "foo"
430
+ ```ruby
431
+ # You need to pass the query to iterate over:
432
+ iterator = Extralite::Iterator(query)
433
+ iterator.each { |r| ... }
434
+ ```
104
435
 
105
- # parameter binding (works for all query_xxx methods)
106
- db.query_hash('select ? as foo, ? as bar', 1, 2) #=> [{ :foo => 1, :bar => 2 }]
107
- db.query_hash('select ? as foo, ? as bar', [1, 2]) #=> [{ :foo => 1, :bar => 2 }]
436
+ ### Value Transforms in Prepared Queries
108
437
 
109
- # explicit parameter indexes
110
- db.query_hash('select ?2 as foo, ?1 as bar, ?1 * ?2 as baz', 6, 7) #=> [{ :foo => 7, :bar => 6, :baz => 42 }
438
+ Prepared queries can automatically transform their result sets by setting a
439
+ transform block. The transform block receives values according to the query mode
440
+ (hash, array or argv). To set a transform you can pass a block to one of the
441
+ `Database#prepare_xxx` methods, or use `Query#transform`:
111
442
 
112
- # named parameters
113
- db.query('select * from foo where bar = :bar', bar: 42)
114
- db.query('select * from foo where bar = :bar', 'bar' => 42)
115
- db.query('select * from foo where bar = :bar', ':bar' => 42)
443
+ ```ruby
444
+ q = db.prepare('select * from items where id = ?') { |h| Item.new(h) }
445
+ q.bind(42).next #=> Item instance
116
446
 
117
- # parameter binding of named parameters from Struct and Data
118
- SomeStruct = Struct.new(:foo, :bar)
119
- db.query_single_column('select :bar', SomeStruct.new(41, 42)) #=> [42]
120
- SomeData = Data.define(:foo, :bar)
121
- db.query_single_column('select :bar', SomeData.new(foo: 41, bar: 42)) #=> [42]
447
+ # An equivalent
448
+ q = db.prepare('select * from items where id = ?')
449
+ q.transform { |h| Item.new(h) }
450
+ ```
122
451
 
123
- # parameter binding for binary data (BLOBs)
124
- db.execute('insert into foo values (?)', File.binread('/path/to/file'))
125
- db.execute('insert into foo values (?)', Extralite::Blob.new('Hello, 世界!'))
126
- db.execute('insert into foo values (?)', 'Hello, 世界!'.force_encoding(Encoding::ASCII_8BIT))
452
+ The same can be done for queries in `argv` or `ary` mode:
127
453
 
128
- # insert multiple rows
129
- db.batch_execute('insert into foo values (?)', ['bar', 'baz'])
130
- db.batch_execute('insert into foo values (?, ?)', [[1, 2], [3, 4]])
454
+ ```ruby
455
+ db.prepare_argv('select * from foo') { |a, b, c| a + b + c }
456
+
457
+ db.prepare_ary('select * from foo') { |a| a.map(&:to_s).join }
458
+ ```
131
459
 
132
- # batch execute from enumerable
460
+ ## Batch Execution of Queries
461
+
462
+ Extralite provides methods for batch execution of queries, with multiple sets of
463
+ parameters. The `#batch_execute` method lets you insert or update a large number
464
+ of records with a single call:
465
+
466
+ ```ruby
467
+ values = [
468
+ [1, 11, 111],
469
+ [2, 22, 222],
470
+ [3, 33, 333]
471
+ ]
472
+ # insert the above records in one fell swoop, and returns the total number of
473
+ # changes:
474
+ db.batch_execute('insert into foo values (?, ?, ?)', values)
475
+ #=> 3
476
+ ```
477
+
478
+ Parameters to the query can also be provided by any object that is an
479
+ `Enumerable` or has an `#each` method, or any *callable* object that responds to
480
+ `#call`:
481
+
482
+ ```ruby
483
+ # Take parameter values from a Range
133
484
  db.batch_execute('insert into foo values (?)', 1..10)
485
+ #=> 10
134
486
 
135
- # batch execute from block
136
- source = [[1, 2], [2, 3], [3, 4]]
137
- db.batch_execute('insert into foo values (?, ?)') { source.shift }
487
+ # Insert (chomped) lines from a file
488
+ File.open('foo.txt') do |f|
489
+ source = f.each_line.map(&:chomp)
490
+ db.batch_execute('insert into foo values (?)', source)
491
+ end
138
492
 
139
- # prepared queries
140
- query = db.prepare('select ? as foo, ? as bar') #=> Extralite::Query
141
- query.bind(1, 2) #=> [{ :foo => 1, :bar => 2 }]
493
+ # Insert items from a queue
494
+ parameters = proc do
495
+ item = queue.shift
496
+ # when we're done, we return nil
497
+ (item == :done) ? nil : item
498
+ end
499
+ db.batch_execute('insert into foo values (?)', parameters)
500
+ #=> number of rows inserted
501
+ ```
142
502
 
143
- query.next #=> next row in result_set (as hash)
144
- query.next_hash #=> next row in result_set (as hash)
145
- query.next_ary #=> next row in result_set (as array)
146
- query.next_single_column #=> next row in result_set (as single value)
503
+ Like its cousin `#execute`, the `#batch_execute` returns the total number of
504
+ changes to the database (rows inserted, deleted or udpated).
147
505
 
148
- query.next(10) #=> next 10 rows in result_set (as hash)
149
- query.next_hash(10) #=> next 10 rows in result_set (as hash)
150
- query.next_ary(10) #=> next 10 rows in result_set (as array)
151
- query.next_single_column(10) #=> next 10 rows in result_set (as single value)
506
+ ### Batch Execution of Queries that Return Rows
152
507
 
153
- query.to_a #=> all rows as array of hashes
154
- query.to_a_hash #=> all rows as array of hashes
155
- query.to_a_ary #=> all rows as array of arrays
156
- query.to_a_single_column #=> all rows as array of single values
508
+ Extralite also provides a `#batch_query` method that like `#batch_execute` takes
509
+ a parameter source and returns an array containing the result sets for all query
510
+ invocations. If a block is given, the result sets are passed to the block
511
+ instead.
157
512
 
158
- query.each { |r| ... } #=> iterate over all rows as hashes
159
- query.each_hash { |r| ... } #=> iterate over all rows as hashes
160
- query.each_ary { |r| ... } #=> iterate over all rows as arrays
161
- query.each_single_column { |r| ... } #=> iterate over all rows as single columns
513
+ The `#batch_query` method is especially useful for batch queries with a
514
+ `RETURNING` clause:
162
515
 
163
- iterator = query.each #=> create enumerable iterator
164
- iterator.next #=> next row
165
- iterator.each { |r| ... } #=> iterate over all rows
166
- values = iterator.map { |r| r[:foo] * 10 } #=> map all rows
516
+ ```ruby
517
+ updates = [
518
+ { id: 3, price: 42 },
519
+ { id: 5, price: 43 }
520
+ ]
521
+ sql = 'update foo set price = $price where id = $id returning id, quantity'
522
+ db.batch_query(sql, updates)
523
+ #=> [[{ id: 3, quantity: 4 }], [{ id: 5, quantity: 5 }]]
524
+
525
+ # The same with a block (returns the total number of changes)
526
+ db.batch_query(sql, updates) do |rows|
527
+ p rows
528
+ #=> [{ id: 3, quantity: 4 }]
529
+ #=> [{ id: 5, quantity: 5 }]
530
+ end
531
+ #=> 2
532
+ ```
167
533
 
168
- iterator = query.each_ary #=> create enumerable iterator with rows as arrays
169
- iterator = query.each_single_column #=> create enumerable iterator with single values
534
+ The `#batch_query` method, like other row fetching methods, changes the row
535
+ representation according to the query mode.
170
536
 
171
- # get last insert rowid
172
- rowid = db.last_insert_rowid
537
+ ### Batch Execution of Prepared Queries
173
538
 
174
- # get number of rows changed in last query
175
- number_of_rows_affected = db.changes
539
+ Batch execution can also be done using prepared queries, using the same methods
540
+ `#batch_execute` and `#batch_query`:
176
541
 
177
- # get column names for the given sql
178
- db.columns('select a, b, c from foo') => [:a, :b, :c]
542
+ ```ruby
543
+ query = db.prepare 'update foo set x = ? where z = ? returning *'
179
544
 
180
- # get db filename
181
- db.filename #=> "/tmp/my.db"
545
+ query.batch_execute([[42, 3], [43, 6]])
546
+ #=> 2
182
547
 
183
- # get list of tables
184
- db.tables #=> ['foo', 'bar']
548
+ query.batch_query([[42, 3], [43, 6]])
549
+ #=> [{ x: 42, y: 2, z: 3 }, { x: 43, y: 5, z: 6 }]
550
+ ```
185
551
 
186
- # get and set pragmas
187
- db.pragma(:journal_mode) #=> 'delete'
188
- db.pragma(journal_mode: 'wal')
189
- db.pragma(:journal_mode) #=> 'wal'
552
+ ## Transactions and Savepoints
190
553
 
191
- # load an extension
192
- db.load_extension('/path/to/extension.so')
554
+ All reads and writes to SQLite databases occur within a
555
+ [transaction](https://www.sqlite.org/lang_transaction.html). If no explicit
556
+ transaction has started, the submitted SQL statements passed to `#execute` or
557
+ `#query` will all run within an implicit transaction:
558
+
559
+ ```ruby
560
+ # The following two SQL statements will run in a single implicit transaction:
561
+ db.execute('insert into foo values (42); insert into bar values (43)')
562
+
563
+ # Otherwise, each call to #execute runs in a separate transaction:
564
+ db.execute('insert into foo values (42)')
565
+ db.execute('insert into bar values (43)')
566
+ ```
567
+
568
+ ### Explicit Transactions
193
569
 
194
- # run queries in a transaction
570
+ While you can issue `BEGIN` and `COMMIT` SQL statements yourself to start and
571
+ commit explicit transactions, Extralite provides a convenient `#transaction`
572
+ method that manages starting, commiting and rolling back of transactions
573
+ automatically:
574
+
575
+ ```ruby
195
576
  db.transaction do
196
- db.execute('insert into foo values (?)', 1)
197
- db.execute('insert into foo values (?)', 2)
198
- db.execute('insert into foo values (?)', 3)
577
+ db.execute('insert into foo values (42)')
578
+ raise 'Something bad happened' if something_bad_happened
579
+ db.execute('insert into bar values (43)')
199
580
  end
581
+ ```
200
582
 
201
- # close database
202
- db.close
203
- db.closed? #=> true
583
+ If no exception is raised in the transaction block, the changes are commited. If
584
+ an exception is raised, the changes are rolled back and the exception is
585
+ propagated to the application code. You can prevent the exception from being
586
+ propagated by calling `#rollback!`:
587
+
588
+ ```ruby
589
+ db.transaction do
590
+ db.execute('insert into foo values (42)')
591
+ rollback! if something_bad_happened
592
+ db.execute('insert into bar values (43)')
593
+ end
204
594
  ```
205
595
 
206
- ## More Features
596
+ ### Transaction Modes
597
+
598
+ By default, `#transaction` starts an `IMMEDIATE` transaction. To start a
599
+ `DEFERRED` or `EXCLUSIVE` transaction, pass the desired mode to `#transaction`:
207
600
 
208
- ### Interrupting Long-running Queries
601
+ ```ruby
602
+ # Start a DEFERRED transaction
603
+ db.transaction(:deferred) do
604
+ ...
605
+ end
606
+
607
+ # Start a EXCLUSIVE transaction
608
+ db.transaction(:exclusive) do
609
+ ...
610
+ end
611
+ ```
209
612
 
210
- When running long-running queries, you can use `Database#interrupt` to interrupt
211
- the query:
613
+ Note that running an `IMMEDIATE` or `EXCLUSIVE` transaction blocks the database
614
+ for writing (and also reading in certain cases) for the duration of the
615
+ transaction. This can cause queries to the same database on a different
616
+ connection to fail with a `BusyError` exception. This can be mitigated by
617
+ setting a [busy timeout](#dealing-with-a-busy-database).
618
+
619
+ ### Savepoints
620
+
621
+ In addition to transactions, SQLite also supports the use of
622
+ [savepoints](https://www.sqlite.org/lang_savepoint.html), which can be used for
623
+ more fine-grained control of changes within a transaction, and to be able to
624
+ rollback specific changes without abandoning the entire transaction:
212
625
 
213
626
  ```ruby
214
- timeout_thread = Thread.new do
215
- sleep 10
216
- db.interrupt
627
+ db.transaction do
628
+ db.execute 'insert into foo values (1)'
629
+
630
+ db.savepoint :my_savepoint
631
+ db.execute 'insert into foo values (2)'
632
+
633
+ # the following cancels the last insert
634
+ db.rollback_to :my_savepoint
635
+ db.execute 'insert into foo values (3)'
636
+
637
+ db.release :my_savepoint
217
638
  end
639
+ ```
218
640
 
219
- result = begin
220
- db.query(super_slow_sql)
221
- rescue Extralite::InterruptError
222
- nil
223
- ensure
641
+ ## Database Information
642
+
643
+ ### Getting the list of tables
644
+
645
+ To get the list of tables in a database, use the `#tables` method:
646
+
647
+ ```ruby
648
+ db.tables
649
+ #=> [...]
650
+ ```
651
+
652
+ To get the list of tables in an attached database, you can pass the database name to `#tables`:
653
+
654
+ ```ruby
655
+ db.execute "attach database 'foo.db' as foo"
656
+ db.tables('foo')
657
+ #=> [...]
658
+ ```
659
+
660
+ ### Getting the last insert row id
661
+
662
+ ```ruby
663
+ db.execute 'insert into foo values (?)', 42
664
+ db.last_insert_rowid
665
+ #=> 1
666
+ ```
667
+
668
+ ### Getting the columns names for a given query
669
+
670
+ ```ruby
671
+ db.columns('select a, b, c from foo')
672
+ #=> [:a, :b, :c]
673
+
674
+ # Columns a prepared query:
675
+ query = db.prepare('select x, y from foo')
676
+ query.columns
677
+ #=> [:x, :y]
678
+ ```
679
+
680
+ ### Pragmas
681
+
682
+ You can get or set pragma values using `#pragma`:
683
+
684
+ ```ruby
685
+ # get a pragma value:
686
+ db.pragma(:journal_mode)
687
+ #=> 'delete'
688
+
689
+ # set a pragma value:
690
+ db.pragma(journal_mode: 'wal')
691
+ db.pragma(:journal_mode)
692
+ #=> 'wal'
693
+ ```
694
+
695
+ You can also pass pragmas when opening the database:
696
+
697
+ ```ruby
698
+ db = Extralite::Database.new('path/to/db', pragma: { foreign_keys: true })
699
+ ```
700
+
701
+ ## Error Handling
702
+
703
+ Extralite defines various exception classes that are raised when an error is
704
+ encountered while interacting with the underlying SQLite library:
705
+
706
+ - `Extralite::SQLError` - raised when SQLite encounters an invalid SQL query.
707
+ - `Extralite::BusyError` - raised when the underlying database is locked for use
708
+ by another database connection.
709
+ - `Extralite::InterruptError` - raised when a query has been interrupted.
710
+ - `Extralite::ParameterError` - raised when an invalid parameter value has been
711
+ specified.
712
+ - `Extralite::Error` - raised on all other errors.
713
+
714
+ In addition to the above exceptions, further information about the last error
715
+ that occurred is provided by the following methods:
716
+
717
+ - `#errcode` - the [error code](https://www.sqlite.org/rescode.html) returned by
718
+ the underlying SQLite library.
719
+ - `#errmsg` - the error message for the last error. For most errors, the error
720
+ message is copied into the exception message.
721
+ - `#error_offset` - for SQL errors, the offset into the SQL string where the
722
+ error was encountered.
723
+
724
+ ## Concurrency
725
+
726
+ Extralite provides a comprehensive set of tools for dealing with concurrency
727
+ issues, and for making sure that running queries on SQLite databases does not
728
+ cause the app to freeze.
729
+
730
+ **Note**: In order to allow concurrent access your the database, it is highly
731
+ recommended that you set your database to use [WAL journaling
732
+ mode](https://www.sqlite.org/wal.html) for *all* database connections.
733
+ Otherwise, you risking running into performance problems and having queries fail
734
+ with `BusyError` exceptions. You can easily open your database in WAL journaling
735
+ mode by passing a `wal: true` option:
736
+
737
+ ```ruby
738
+ # This will set PRAGMA journal_mode=1 and PRAGMA synchronous=1
739
+ db = Extralite::Database.new('path/to/db', wal: true)
740
+ ```
741
+
742
+ ### The Ruby GVL
743
+
744
+ In order to support multi-threading, Extralite releases the [Ruby
745
+ GVL](https://www.speedshop.co/2020/05/11/the-ruby-gvl-and-scaling.html)
746
+ periodically while running queries. This allows other threads to run while the
747
+ underlying SQLite library is busy preparing queries, fetching records and
748
+ backing up databases. By default, the GVL is when preparing the query, and once
749
+ for every 1000 iterated records. The GVL release threshold can be set separately
750
+ for each database:
751
+
752
+ ```ruby
753
+ # release the GVL on preparing the query, and every 10 records
754
+ db.gvl_release_threshold = 10
755
+
756
+ # release the GVL only when preparing the query
757
+ db.gvl_release_threshold = 0
758
+
759
+ # never release the GVL (for single-threaded apps only)
760
+ db.gvl_release_threshold = -1
761
+
762
+ db.gvl_release_threshold = nil # use default value (currently 1000)
763
+ ```
764
+
765
+ For most applications, there's no need to tune the GVL threshold value, as it
766
+ provides [excellent](#performance) performance characteristics for both
767
+ single-threaded and multi-threaded applications.
768
+
769
+ In a heavily multi-threaded application, releasing the GVL more often (lower
770
+ threshold value) will lead to less latency (for threads not running a query),
771
+ but will also hurt the throughput (for the thread running the query). Releasing
772
+ the GVL less often (higher threshold value) will lead to better throughput for
773
+ queries, while increasing latency for threads not running a query. The following
774
+ diagram demonstrates the relationship between the GVL release threshold value,
775
+ latency and throughput:
776
+
777
+ ```
778
+ less latency & throughput <<< GVL release threshold >>> more latency & throughput
779
+ ```
780
+
781
+ ### Dealing with a Busy Database
782
+
783
+ When multiple threads or processes access the same database, the database may be
784
+ locked for writing by one process, which will block other processes wishing to
785
+ write to the database. When attempting to write to a locked database, a
786
+ `Extralite::BusyError` will be raised:
787
+
788
+ ```ruby
789
+ ready = nil
790
+ locker = Thread.new do
791
+ db1 = Extralite::Database.new('my.db')
792
+ # Lock the database for 3 seconds
793
+ db1.transaction do
794
+ ready = true
795
+ sleep(3)
796
+ end
797
+ end
798
+
799
+ db2 = Extralite::Database.new('my.db')
800
+ # wait for writer1 to enter a transaction
801
+ sleep(0) while !ready
802
+ # This will raise a Extralite::BusyError
803
+ db2.transaction { }
804
+ # Extralite::BusyError!
805
+ ```
806
+
807
+ You can mitigate this by setting a busy timeout. This will cause SQLite to wait
808
+ for the database to become unlocked up to the specified timeout period:
809
+
810
+ ```ruby
811
+ # Wait for up to 5 seconds before giving up
812
+ db2.busy_timeout = 5
813
+ # Now it'll work!
814
+ db2.transaction { }
815
+ ```
816
+
817
+ As stated above, setting the database to use WAL journaling mode greatly reduces
818
+ contention between different process/threads accessing the same database. For
819
+ most use cases, setting the busy timeout solves the problem of failing to run
820
+ queries because of a busy database, as normally transactions are short-lived.
821
+
822
+ However, in some cases, such as when running a multi-fibered app or when
823
+ implementing your own timeout mechanisms, you'll want to set a [progress
824
+ handler](#the-progress-handler).
825
+
826
+ ### Interrupting a Query
827
+
828
+ To interrupt an ongoing query, use the `#interrupt` method. Normally this is
829
+ done from a separate thread. Here's a way to implement a timeout using
830
+ `#interrupt`:
831
+
832
+ ```ruby
833
+ def run_query_with_timeout(sql, timeout)
834
+ timeout_thread = Thread.new do
835
+ t0 = Time.now
836
+ sleep(timeout)
837
+ @db.interrupt
838
+ end
839
+ result = @db.query(sql)
224
840
  timeout_thread.kill
225
- timeout_thread.join
841
+ result
226
842
  end
843
+
844
+ run_query_with_timeout('select 1 as foo', 5)
845
+ #=> [{ foo: 1 }]
846
+
847
+ # A timeout will cause a Extralite::InterruptError to be raised
848
+ run_query_with_timeout(slow_sql, 5)
849
+ #=> Extralite::InterruptError!
227
850
  ```
228
851
 
229
- ### Running transactions
852
+ You can also call `#interrupt` from within the [progress
853
+ handler](#the-progress-handler).
854
+
855
+ ### The Progress Handler
856
+
857
+ Extralite also supports setting up a progress handler, which is a piece of code
858
+ that will be called while a query is in progress, or while the database is busy.
859
+ This is useful especially when you want to implement a general purpose timeout
860
+ mechanism that deals with both a busy database and with slow queries.
861
+
862
+ The progress handler can also be used for performing any kind of operation while
863
+ a query is in progress. Here are some use cases:
230
864
 
231
- In order to run multiple queries in a single transaction, use
232
- `Database#transaction`, passing a block that runs the queries. You can specify
233
- the [transaction
234
- mode](https://www.sqlite.org/lang_transaction.html#deferred_immediate_and_exclusive_transactions).
235
- The default mode is `:immediate`:
865
+ - Interrupting queries that take too long to run.
866
+ - Interrupting queries on an exceptional condition, such as a received signal.
867
+ - Updating the UI while a query is running.
868
+ - Switching between fibers in multi-fibered apps.
869
+ - Switching between threads in multi-threaded apps.
870
+ - Instrumenting the performance of queries.
871
+
872
+ Setting the progress handler requires that Extralite hold the GVL while running
873
+ all queries. Therefore, it should be used with care. In a multi-threaded app,
874
+ you'll need to call `Thread.pass` from the progress handler in order for other
875
+ threads to be able to run while the query is in progress. The progress handler
876
+ is set per-database using `#on_progress`. This method takes a single parameter
877
+ that specifies the approximate number of SQLite VM instructions between
878
+ successive calls to the progress handler:
879
+
880
+ ```ruby
881
+ db.on_progress do
882
+ check_for_timeout
883
+ # Allow other threads to run
884
+ Thread.pass
885
+ end
886
+ ```
887
+
888
+ The progress handler can be used to interrupt queries in progress. This can be
889
+ done by either calling `#interrupt`, or by raising an exception. As discussed
890
+ above, calling `#interrupt` causes the query to raise a
891
+ `Extralite::InterruptError` exception:
892
+
893
+ ```ruby
894
+ db.on_progress { db.interrupt }
895
+ db.query('select 1')
896
+ #=> Extralite::InterruptError!
897
+ ```
898
+
899
+ You can also interrupt queries in progress by raising an exception. The query
900
+ will be stopped, and the exception will propagate to the call site:
901
+
902
+ ```ruby
903
+ db.on_progress do
904
+ raise 'BOOM!'
905
+ end
906
+
907
+ db.query('select 1')
908
+ #=> BOOM!
909
+ ```
910
+
911
+ Here's how a timeout might be implemented using the progress handler:
912
+
913
+ ```ruby
914
+ def setup_progress_handler
915
+ @db.on_progress do
916
+ raise TimeoutError if Time.now - @t0 >= @timeout
917
+ Thread.pass
918
+ end
919
+ end
920
+
921
+ # In this example, we just return nil on timeout
922
+ def run_query_with_timeout(sql, timeout)
923
+ @t0 = Time.now
924
+ @db.query(sql)
925
+ rescue TimeoutError
926
+ nil
927
+ end
928
+
929
+ run_query_with_timeout('select 1 as foo', 5)
930
+ #=> [{ foo: 1 }]
931
+
932
+ run_query_with_timeout(slow_sql, 5)
933
+ #=> nil
934
+ ```
935
+
936
+ **Note**: you must not issue any query from within the progress handler.
937
+
938
+ ### Dealing with a Busy Database in the Progress Handler
939
+
940
+ As mentioned above, the progress handler is also called when the database is
941
+ busy, regardless of the progress period given to `#on_progress`. You can detect
942
+ if the database is busy by checking the first argument passed to the progress
943
+ handler, which will be true when busy:
236
944
 
237
945
  ```ruby
238
- db.transaction { ... } # Run an immediate transaction
239
- db.transaction(:deferred) { ... } # Run a deferred transaction
240
- db.transaction(:exclusive) { ... } # Run an exclusive transaction
946
+ db.on_progress do |busy|
947
+ if busy
948
+ foo
949
+ else
950
+ bar
951
+ end
952
+ end
241
953
  ```
242
954
 
243
- If an exception is raised in the given block, the transaction will be rolled
244
- back. Otherwise, it is committed.
955
+ This allows you to implement separate logic to deal with busy states, for
956
+ example sleeping for a small period of time, or implementing a different timeout
957
+ period.
958
+
959
+ ### Advanced Progress Handler Settings
960
+
961
+ You can further tune the behaviour of the progress handler with the following
962
+ options passed to `#on_progress`:
963
+
964
+ - `:mode`: the following modes are supported:
965
+ - `:none` : the progress handler is disabled.
966
+ - `:normal`: the progress handler is called on query progress (this is the
967
+ default mode).
968
+ - `:once`: the progress handler is called once before running the query.
969
+ - `:at_least_once`: the progress handler is called once before running the
970
+ query, and on query progress.
971
+ - `:period`: controls the approximate number of SQLite VM instructions executed
972
+ between consecutive calls to the progress handler. Default value: 1000.
973
+ - `:tick`: controls the granularity of the progress handler. This is the value
974
+ passed internally to the SQLite library. Default value: 10.
975
+
976
+ ```ruby
977
+ db.on_progress(mode: :at_least_once, period: 640, tick: 5) { snooze }
978
+ ```
979
+
980
+ ### Global Progress Handler Settings
981
+
982
+ You can set the global progress handler behaviour by calling
983
+ `Extralite.on_progress`. You can use this API to set the global progress
984
+ settings, without needing to set a progress handler individually for each
985
+ `Database` instance. This method takes the same options as
986
+ `Database#on_progress`:
987
+
988
+ ```ruby
989
+ Extralite.on_progress(mode: :at_least_once, period: 640, tick: 5) { snooze }
990
+
991
+ # the new database instance uses the global progress handler settings
992
+ db = Database.new(':memory:')
993
+ ```
994
+
995
+ ### Extralite and Fibers
996
+
997
+ The progress handler can also be used to switch between fibers in a
998
+ multi-fibered Ruby app, based on libraries such as
999
+ [Async](https://github.com/socketry/async) or
1000
+ [Polyphony](https://github.com/digital-fabric/polyphony). A general solution
1001
+ (that also works for multi-threaded apps) is to call `sleep(0)` in the progress
1002
+ handler. This will work for switching between fibers using either Polyphony or
1003
+ any fiber scheduler gem, such as Async et al:
1004
+
1005
+ ```ruby
1006
+ db.on_progress(100) { sleep(0) }
1007
+ ```
1008
+
1009
+ For Polyphony-based apps, you can also call `snooze` to allow other fibers to
1010
+ run while a query is progressing. If your Polyphony app is multi-threaded,
1011
+ you'll also need to call `Thread.pass` in order to allow other threads to run:
1012
+
1013
+ ```ruby
1014
+ db.on_progress(100) do
1015
+ snooze
1016
+ Thread.pass
1017
+ end
1018
+ ```
1019
+
1020
+ Note that with Polyphony, once you install the progress handler, you can just
1021
+ use the regular `#move_on_after` and `#cancel_after` methods to implement
1022
+ timeouts for queries:
1023
+
1024
+ ```ruby
1025
+ db.on_progress(100) { snooze }
1026
+
1027
+ cancel_after(3) do
1028
+ db.query(long_running_query)
1029
+ end
1030
+ ```
1031
+
1032
+ ### Thread Safety
1033
+
1034
+ A single database instance can be safely used in multiple threads simultaneously
1035
+ as long as the following conditions are met:
1036
+
1037
+ - No explicit transactions are used.
1038
+ - Each thread issues queries by calling `Database#query_xxx`, or uses a separate
1039
+ `Query` instance.
1040
+ - The GVL release threshold is not `0` (i.e. the GVL is released periodically
1041
+ while running queries.)
1042
+
1043
+ ### Use with Ractors
1044
+
1045
+ Extralite databases can safely be used inside ractors. A ractor has the benefit
1046
+ of using a separate GVL from the maine one, which allows true parallelism for
1047
+ Ruby apps. So when you use Extralite to access SQLite databases from within a
1048
+ ractor, you can do so without any considerations for what's happening outside
1049
+ the ractor when it runs queries.
1050
+
1051
+ **Note**: Ractors are considered an experimental feature of Ruby. You may
1052
+ encounter errors or inconsistent behaviour when using ractors.
1053
+
1054
+ ## Advanced Usage
1055
+
1056
+ ### Loading Extensions
1057
+
1058
+ Extensions can be loaded by calling `#load_extension`:
1059
+
1060
+ ```ruby
1061
+ db.load_extension('/path/to/extension.so')
1062
+ ```
1063
+
1064
+ A pretty comprehensive set of extensions can be found here:
1065
+
1066
+ https://github.com/nalgeon/sqlean
245
1067
 
246
1068
  ### Creating Backups
247
1069
 
@@ -266,6 +1088,51 @@ db.backup('backup.db') do |remaining, total|
266
1088
  end
267
1089
  ```
268
1090
 
1091
+ ### Working with Changesets
1092
+
1093
+ __Note__: as the session extension is by default disabled in SQLite
1094
+ distributions, support for changesets is currently only available withthe
1095
+ bundled version of Extralite, `extralite-bundle`.
1096
+
1097
+ Changesets can be used to track and persist changes to data in a database. They
1098
+ can also be used to apply the same changes to another database, or to undo them.
1099
+ To track changes to a database, use the `#track_changes` method:
1100
+
1101
+ ```ruby
1102
+ # track changes to the foo and bar tables:
1103
+ changeset = db.track_changes(:foo, :bar) do
1104
+ insert_a_bunch_of_records(db)
1105
+ end
1106
+
1107
+ # to track changes to all tables, pass nil:
1108
+ changeset = db.track_changes(nil) do
1109
+ insert_a_bunch_of_records(db)
1110
+ end
1111
+ ```
1112
+
1113
+ You can then apply the same changes to another database:
1114
+
1115
+ ```ruby
1116
+ changeset.apply(some_other_db)
1117
+ ```
1118
+
1119
+ To undo the changes, obtain an inverted changeset and apply it to the database:
1120
+
1121
+ ```ruby
1122
+ changeset.invert.apply(db)
1123
+ ```
1124
+
1125
+ You can also save and load the changeset:
1126
+
1127
+ ```ruby
1128
+ # save the changeset
1129
+ IO.write('my.changes', changeset.to_blob)
1130
+
1131
+ # load the changeset
1132
+ changeset = Extralite::Changeset.new
1133
+ changeset.load(IO.read('my.changes'))
1134
+ ```
1135
+
269
1136
  ### Retrieving Status Information
270
1137
 
271
1138
  Extralite provides methods for retrieving status information about the sqlite
@@ -305,16 +1172,6 @@ value = db.limit(Extralite::SQLITE_LIMIT_ATTACHED)
305
1172
  db.limit(Extralite::SQLITE_LIMIT_ATTACHED, new_value)
306
1173
  ```
307
1174
 
308
- ### Setting the Busy Timeout
309
-
310
- When accessing a database concurrently it can be handy to set a busy timeout, in
311
- order to not have to deal with rescuing `Extralite::BusyError` exceptions. The
312
- timeout is given in seconds:
313
-
314
- ```ruby
315
- db.busy_timeout = 5
316
- ```
317
-
318
1175
  ### Tracing SQL Statements
319
1176
 
320
1177
  To trace all SQL statements executed on the database, pass a block to
@@ -342,58 +1199,6 @@ p articles.to_a
342
1199
 
343
1200
  (Make sure you include `extralite` as a dependency in your `Gemfile`.)
344
1201
 
345
- ## Concurrency
346
-
347
- ### The Ruby GVL
348
-
349
- Extralite releases the [Ruby
350
- GVL](https://www.speedshop.co/2020/05/11/the-ruby-gvl-and-scaling.html) while
351
- making calls to the sqlite3 library that might block, such as when backing up a
352
- database, or when preparing a query. This allows other threads to run while the
353
- underlying sqlite3 library is busy preparing queries, fetching records and
354
- backing up databases.
355
-
356
- Extralite also releases the GVL periodically when iterating over records. By
357
- default, the GVL is released every 1000 records iterated. The GVL release
358
- threshold can be set separately for each database:
359
-
360
- ```ruby
361
- db.gvl_release_threshold = 10 # release GVL every 10 records
362
-
363
- db.gvl_release_threshold = nil # use default value (currently 1000)
364
- ```
365
-
366
- For most applications, there's no need to tune the GVL threshold value, as it
367
- provides [excellent](#performance) performance characteristics for both
368
- single-threaded and multi-threaded applications.
369
-
370
- In a heavily multi-threaded application, releasing the GVL more often (lower
371
- threshold value) will lead to less latency (for threads not running a query),
372
- but will also hurt the throughput (for the thread running the query). Releasing
373
- the GVL less often (higher threshold value) will lead to better throughput for
374
- queries, while increasing latency for threads not running a query. The following
375
- diagram demonstrates the relationship between the GVL release threshold value,
376
- latency and throughput:
377
-
378
- ```
379
- less latency & throughput <<< GVL release threshold >>> more latency & throughput
380
- ```
381
-
382
- ### Thread Safety
383
-
384
- A single database instance can be safely used in multiple threads simultaneously
385
- as long as the following conditions are met:
386
-
387
- - No explicit transactions are used.
388
- - Each thread issues queries by calling `Database#query_xxx`, or uses a separate
389
- `Query` instance.
390
- - The GVL release threshold is not `0` (i.e. the GVL is released periodically
391
- while running queries.)
392
-
393
- ### Use with Ractors
394
-
395
- Extralite databases can be used inside ractors
396
-
397
1202
  ## Performance
398
1203
 
399
1204
  A benchmark script is included, creating a table of various row counts, then
@@ -403,37 +1208,44 @@ large number of rows.
403
1208
 
404
1209
  ### Rows as Hashes
405
1210
 
406
- [Benchmark source code](https://github.com/digital-fabric/extralite/blob/main/test/perf_hash.rb)
1211
+ [Benchmark source
1212
+ code](https://github.com/digital-fabric/extralite/blob/main/test/perf_hash.rb)
407
1213
 
408
- |Row count|sqlite3 1.6.0|Extralite 1.21|Advantage|
1214
+ |Row count|sqlite3 1.7.0|Extralite 2.5|Advantage|
409
1215
  |-:|-:|-:|-:|
410
- |10|63.7K rows/s|94.0K rows/s|__1.48x__|
411
- |1K|299.2K rows/s|1.983M rows/s|__6.63x__|
412
- |100K|185.4K rows/s|2.033M rows/s|__10.97x__|
1216
+ |10|184.9K rows/s|473.2K rows/s|__2.56x__|
1217
+ |1K|290.5K rows/s|2320.7K rows/s|__7.98x__|
1218
+ |100K|143.0K rows/s|2061.3K rows/s|__14.41x__|
413
1219
 
414
1220
  ### Rows as Arrays
415
1221
 
416
- [Benchmark source code](https://github.com/digital-fabric/extralite/blob/main/test/perf_ary.rb)
1222
+ [Benchmark source
1223
+ code](https://github.com/digital-fabric/extralite/blob/main/test/perf_ary.rb)
417
1224
 
418
- |Row count|sqlite3 1.6.0|Extralite 1.21|Advantage|
1225
+ |Row count|sqlite3 1.7.0|Extralite 2.5|Advantage|
419
1226
  |-:|-:|-:|-:|
420
- |10|71.2K rows/s|92.1K rows/s|__1.29x__|
421
- |1K|502.1K rows/s|2.065M rows/s|__4.11x__|
422
- |100K|455.7K rows/s|2.511M rows/s|__5.51x__|
1227
+ |10|276.9K rows/s|472.3K rows/s|__1.71x__|
1228
+ |1K|615.6K rows/s|2324.3K rows/s|__3.78x__|
1229
+ |100K|477.4K rows/s|1982.7K rows/s|__4.15x__|
423
1230
 
424
1231
  ### Prepared Queries (Prepared Statements)
425
1232
 
426
- [Benchmark source code](https://github.com/digital-fabric/extralite/blob/main/test/perf_prepared.rb)
1233
+ [Benchmark source
1234
+ code](https://github.com/digital-fabric/extralite/blob/main/test/perf_hash_prepared.rb)
427
1235
 
428
- |Row count|sqlite3 1.6.0|Extralite 1.21|Advantage|
1236
+ |Row count|sqlite3 1.7.0|Extralite 2.5|Advantage|
429
1237
  |-:|-:|-:|-:|
430
- |10|232.2K rows/s|741.6K rows/s|__3.19x__|
431
- |1K|299.8K rows/s|2386.0M rows/s|__7.96x__|
432
- |100K|183.1K rows/s|1.893M rows/s|__10.34x__|
1238
+ |10|228.5K rows/s|707.9K rows/s|__3.10x__|
1239
+ |1K|296.5K rows/s|2396.2K rows/s|__8.08x__|
1240
+ |100K|145.9K rows/s|2107.3K rows/s|__14.45x__|
1241
+
1242
+ As those benchmarks show, Extralite is capabale of reading up to 2.4M rows per
1243
+ second, and can be more than 14 times faster than the `sqlite3` gem.
433
1244
 
434
- As those benchmarks show, Extralite is capabale of reading up to 2.5M
435
- rows/second when fetching rows as arrays, and up to 2M rows/second when fetching
436
- rows as hashes.
1245
+ Note that the benchmarks above were performed on synthetic data, in a
1246
+ single-threaded environment, with the GVL release threshold set to -1, which
1247
+ means that both Extralite and the `sqlite3` gem hold the GVL for the duration of
1248
+ the query.
437
1249
 
438
1250
  ## License
439
1251