extralite-bundle 2.5 → 2.7

Sign up to get free protection for your applications and to get access to all the features.
Files changed (45) hide show
  1. checksums.yaml +4 -4
  2. data/.gitignore +1 -0
  3. data/CHANGELOG.md +34 -13
  4. data/Gemfile +4 -0
  5. data/Gemfile-bundle +1 -1
  6. data/LICENSE +1 -1
  7. data/README.md +1059 -247
  8. data/Rakefile +18 -0
  9. data/TODO.md +0 -7
  10. data/examples/kv_store.rb +49 -0
  11. data/examples/multi_fiber.rb +16 -0
  12. data/examples/on_progress.rb +9 -0
  13. data/examples/pubsub_store_polyphony.rb +194 -0
  14. data/examples/pubsub_store_threads.rb +204 -0
  15. data/ext/extralite/changeset.c +463 -0
  16. data/ext/extralite/common.c +177 -91
  17. data/ext/extralite/database.c +745 -276
  18. data/ext/extralite/extconf-bundle.rb +10 -4
  19. data/ext/extralite/extconf.rb +34 -34
  20. data/ext/extralite/extralite.h +104 -47
  21. data/ext/extralite/extralite_ext.c +6 -0
  22. data/ext/extralite/iterator.c +14 -86
  23. data/ext/extralite/query.c +171 -264
  24. data/extralite-bundle.gemspec +1 -1
  25. data/extralite.gemspec +1 -1
  26. data/gemspec.rb +10 -11
  27. data/lib/extralite/version.rb +1 -1
  28. data/lib/extralite.rb +69 -10
  29. data/lib/sequel/adapters/extralite.rb +1 -1
  30. data/test/helper.rb +9 -1
  31. data/test/perf_argv_transform.rb +74 -0
  32. data/test/perf_ary.rb +14 -12
  33. data/test/perf_hash.rb +17 -15
  34. data/test/perf_hash_prepared.rb +58 -0
  35. data/test/perf_hash_transform.rb +66 -0
  36. data/test/perf_polyphony.rb +74 -0
  37. data/test/test_changeset.rb +161 -0
  38. data/test/test_database.rb +720 -104
  39. data/test/test_extralite.rb +2 -2
  40. data/test/test_iterator.rb +28 -13
  41. data/test/test_query.rb +352 -110
  42. data/test/test_sequel.rb +4 -4
  43. metadata +24 -16
  44. data/Gemfile.lock +0 -37
  45. data/test/perf_prepared.rb +0 -64
data/README.md CHANGED
@@ -1,247 +1,1069 @@
1
- # Extralite - a Super Fast Ruby Gem for Working with SQLite3 Databases
2
-
3
- * Source code: https://github.com/digital-fabric/extralite
4
- * Documentation: http://www.rubydoc.info/gems/extralite
5
-
6
- [![Ruby gem](https://badge.fury.io/rb/extralite.svg)](https://rubygems.org/gems/extralite) [![Tests](https://github.com/digital-fabric/extralite/workflows/Tests/badge.svg)](https://github.com/digital-fabric/extralite/actions?query=workflow%3ATests) [![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/digital-fabric/extralite/blob/master/LICENSE)
1
+ <h1 align="center">
2
+ <br>
3
+ Extralite
4
+ </h1>
5
+
6
+ <h4 align="center">Ruby on SQLite</h4>
7
+
8
+ <p align="center">
9
+ <a href="http://rubygems.org/gems/extralite">
10
+ <img src="https://badge.fury.io/rb/extralite.svg" alt="Ruby gem">
11
+ </a>
12
+ <a href="https://github.com/digital-fabric/extralite/actions">
13
+ <img src="https://github.com/digital-fabric/extralite/actions/workflows/test.yml/badge.svg" alt="Tests">
14
+ </a>
15
+ <a href="https://github.com/digital-fabric/extralite/blob/master/LICENSE">
16
+ <img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License">
17
+ </a>
18
+ </p>
19
+
20
+ <p align="center">
21
+ <a href="https://www.rubydoc.info/gems/extralite">API reference</a>
22
+ </p>
7
23
 
8
24
  ## What is Extralite?
9
25
 
10
- Extralite is a super fast, extra-lightweight (about 1300 lines of C-code)
11
- SQLite3 wrapper for Ruby. It provides a minimal set of methods for interacting
12
- with an SQLite3 database, as well as prepared queries (prepared statements).
26
+ Extralite is a fast and innovative SQLite wrapper for Ruby with a rich set of
27
+ features. It provides multiple ways of retrieving data from SQLite databases,
28
+ makes it possible to use SQLite databases in multi-threaded and multi-fibered
29
+ Ruby apps, and includes a comprehensive set of tools for managing SQLite
30
+ databases.
13
31
 
14
32
  Extralite comes in two flavors: the `extralite` gem which uses the
15
33
  system-installed sqlite3 library, and the `extralite-bundle` gem which bundles
16
34
  the latest version of SQLite
17
- ([3.44.2](https://sqlite.org/releaselog/3_44_2.html)), offering access to the
35
+ ([3.45.0](https://sqlite.org/releaselog/3_45_0.html)), offering access to the
18
36
  latest features and enhancements.
19
37
 
20
38
  ## Features
21
39
 
22
- - Super fast - [up to 11x faster](#performance) than the
23
- [sqlite3](https://github.com/sparklemotion/sqlite3-ruby) gem.
24
- - A variety of methods for different data access patterns: rows as hashes, rows
25
- as arrays, single row, single column, single value.
26
- - Prepared statements.
27
- - Parameter binding.
28
- - External iteration - get single records or batches of records.
29
- - Use system-installed sqlite3, or the [bundled latest version of
30
- SQLite3](#installing-the-extralite-sqlite3-bundle).
31
- - Improved [concurrency](#concurrency) for multithreaded apps: the Ruby GVL is
32
- released peridically while preparing SQL statements and while iterating over
33
- results.
34
- - Automatically execute SQL strings containing multiple semicolon-separated
35
- queries (handy for creating/modifying schemas).
36
- - Execute the same query with multiple parameter lists (useful for inserting
37
- records).
38
- - Load extensions (loading of extensions is autmatically enabled. You can find
39
- some useful extensions here: https://github.com/nalgeon/sqlean.)
40
- - Includes a [Sequel adapter](#usage-with-sequel).
41
-
42
- ## Installation
43
-
44
- To use Extralite in your Ruby app, add the following to your `Gemfile`:
40
+ - Best-in-class [performance](#performance) (up to 14X the performance of the
41
+ [sqlite3](https://github.com/sparklemotion/sqlite3-ruby) gem).
42
+ - Support for [concurrency](#concurrency) out of the box for multi-threaded
43
+ and multi-fibered apps.
44
+ - A variety of ways to [retrieve data](#query-modes) - hashes, arrays, single
45
+ columns, single rows, [transforms](#value-transforms).
46
+ - Support for [external iteration](#iterating-over-records-in-a-prepared-query),
47
+ allowing iterating through single records or batches of records.
48
+ - [Prepared queries](#prepared-queries).
49
+ - [Parameter binding](#parameter-binding).
50
+ - [Batch execution](#batch-execution-of-queries) of queries.
51
+ - [transactions and savepoints](#transactions-and-savepoints).
52
+ - Advanced features: load [SQLite extensions](#loading-extensions), create
53
+ [backups](#creating-backups), retrieve [status
54
+ information](#retrieving-status-information), work with
55
+ [changesets](#working-with-changesets), interrogate [database
56
+ limits](#working-with-database-limits), [trace](#tracing-sql-statements)
57
+ queries.
58
+ - [Sequel](#usage-with-sequel) adapter.
59
+
60
+ ## Table of Content
61
+
62
+ - [Installing Extralite](#installing-extralite)
63
+ - [Getting Started](#getting-started)
64
+ - [Query Modes](#query-modes)
65
+ - [Parameter binding](#parameter-binding)
66
+ - [Value Transforms](#value-transforms)
67
+ - [Data Types](#data-types)
68
+ - [Prepared Queries](#prepared-queries)
69
+ - [Batch Execution of Queries](#batch-execution-of-queries)
70
+ - [Transactions and Savepoints](#transactions-and-savepoints)
71
+ - [Database Information](#database-information)
72
+ - [Error Handling](#error-handling)
73
+ - [Concurrency](#concurrency)
74
+ - [Advanced Usage](#advanced-usage)
75
+ - [Usage with Sequel](#usage-with-sequel)
76
+ - [Performance](#performance)
77
+ - [License](#license)
78
+ - [Contributing](#contributing)
79
+
80
+ ## Installing Extralite
81
+
82
+ Using bundler:
45
83
 
46
84
  ```ruby
47
85
  gem 'extralite'
48
86
  ```
49
87
 
50
- You can also run `gem install extralite` if you just want to check it out.
88
+ Or manually:
51
89
 
52
- __Note__: Extralite supports Ruby 3.0 and higher.
90
+ ```bash
91
+ $ gem install extralite
92
+ ```
53
93
 
54
- ### Installing the Extralite-SQLite3 Bundle
94
+ __Note__: Extralite supports Ruby 3.0 and newer.
55
95
 
56
- If you don't have sqlite3 installed on your system, do not want to use the
57
- system-installed version of SQLite3, or would like to use the latest version of
58
- SQLite3, you can install the `extralite-bundle` gem, which integrates the
59
- SQLite3 source code.
96
+ ### Installing the Extralite-SQLite Bundle
60
97
 
61
- > **Important note**: The `extralite-bundle` gem will take a while to install
62
- > (on my modest machine it takes about a minute), due to the size of the sqlite3
63
- > code.
98
+ If you don't have the sqlite3 lib installed on your system, do not want to use
99
+ the system-installed version of SQLite, or would like to use the latest version
100
+ of SQLite, you can install the `extralite-bundle` gem, which integrates the
101
+ SQLite source code.
64
102
 
65
103
  Usage of the `extralite-bundle` gem is identical to the usage of the normal
66
104
  `extralite` gem, using `require 'extralite'` to load the gem.
67
105
 
68
- ## Synopsis
106
+ ## Getting Started
107
+
108
+ The following example shows how to open an SQLite database and run some queries:
109
+
110
+ ```ruby
111
+ db = Extralite::Database.new('mydb.sqlite')
112
+ db.execute('create table foo (x, y, z)')
113
+ db.execute('insert into foo values (?, ?, ?)', 1, 2, 3)
114
+ db.execute('insert into foo values (?, ?, ?)', 4, 5, 6)
115
+ db.query('select * from foo') #=> [{x: 1, y: 2, z: 3}, {x: 4, y: 5, z: 6}]
116
+ ```
117
+
118
+ The `#execute` method is used to make changes to the database, such as creating
119
+ tables, or inserting records. It returns the number of records changed by the
120
+ query.
121
+
122
+ The `#query` method is used to read data from the database. It returns an array
123
+ containing the resulting records, represented as hashes mapping column names (as
124
+ symbols) to individual values.
125
+
126
+ You can also iterate on records by providing a block to `#query`:
127
+
128
+ ```ruby
129
+ db.query 'select * from foo' do |r|
130
+ p record: r
131
+ end
132
+ ```
133
+
134
+ ## Query Modes
135
+
136
+ Extralite allows you to retrieve data from SQLite database in the form that most
137
+ a particular context. For some use cases you'll want to work with rows as
138
+ hashes. In other cases, you'll want to work with rows as arrays, or even as
139
+ single values, if you're just reading one column.
140
+
141
+ For that purpose, Extralite offers three different ways, or modes, of retrieving
142
+ records:
143
+
144
+ - `:hash`: read rows as hashes (this is the default mode).
145
+ - `:ary`: read rows as arrays.
146
+ - `:argv`: similar to the `:ary`, except that for queries with a single column,
147
+ the single column value is returned.
148
+
149
+ Extralite provides separate methods for the different modes:
150
+
151
+ ```ruby
152
+ # alias #query_hash
153
+ db.query('select 1') #=> [{ "1" => 1 }]
154
+
155
+ db.query_ary('select 1') #=> [[1]]
156
+
157
+ db.query_argv('select 1') #=> [1]
158
+ ```
159
+
160
+ Notice how all the return values above are arrays. This is because the different
161
+ `#query_xxx` methods are designed to return multiple rows. If you want to just
162
+ get back a single row, use one of the `query_single_xxx` methods:
163
+
164
+ ```ruby
165
+ # alias #query_single_hash
166
+ db.query('select 1') #=> { "1" => 1 }
167
+
168
+ db.query_single_ary('select 1') #=> [1]
169
+
170
+ db.query_single_argv('select 1') #=> 1
171
+ ```
172
+
173
+ ## Parameter binding
174
+
175
+ As shown in the above example, the `#execute` and `#query_xxx` methods accept
176
+ parameters that can be bound to the query, which means that their values will be
177
+ used for each corresponding place-holder (expressed using `?`) in the SQL
178
+ statement:
179
+
180
+ ```ruby
181
+ db.query('select x from my_table where y = ? and z = ?', 'foo', 'bar')
182
+ ```
183
+
184
+ You can also express place holders by specifying their index (starting from 1)
185
+ using `?IDX`:
186
+
187
+ ```ruby
188
+ # use the same value for both place holders:
189
+ db.query('select x from my_table where y = ?1 and z = ?1', 42)
190
+ ```
191
+
192
+ Another possibility is to use named parameters, which can be done by expressing
193
+ place holders as `:KEY`, `@KEY` or `$KEY`:
194
+
195
+ ```ruby
196
+ db.query('select x from my_table where y = $y and z = $z', y: 'foo', z: 'bar')
197
+ ```
198
+
199
+ Extralite supports specifying named parameters using `Struct` or `Data` objects:
200
+
201
+ ```ruby
202
+ MyStruct = Struct.new(:x, :z)
203
+ params = MyStruct.new(42, 6)
204
+ db.execute('update foo set x = $x where z = $z', params)
205
+
206
+ MyData = Data.define(:x, :z)
207
+ params = MyData.new(43, 3)
208
+ db.execute('update foo set x = $x where z = $z', params)
209
+ ```
210
+
211
+ Parameter binding is especially useful for preventing [SQL-injection
212
+ attacks](https://en.wikipedia.org/wiki/SQL_injection), but is also useful when
213
+ combined with [prepared queries](#prepared-queries) when repeatedly running the
214
+ same query over and over.
215
+
216
+ ## Data Types
217
+
218
+ Extralite supports the following data types for either bound parameters or row
219
+ values:
220
+
221
+ - `Integer`
222
+ - `Float`
223
+ - `Boolean` (see below)
224
+ - `String` (see below)
225
+ - nil
226
+
227
+ ### Boolean values
228
+
229
+ SQLite does not have a boolean data type. Extralite will automatically translate
230
+ bound parameter values of `true` or `false` to the integer values `1` and `0`,
231
+ respectively. Note that boolean values stored in the database will be fetched as
232
+ integers.
233
+
234
+ ### String values
235
+
236
+ String parameter values are translated by Extralite to either `TEXT` or `BLOB`
237
+ values according to the string encoding used. Strings with an `ASCII-8BIT` are
238
+ treated as blobs. Otherwise they are treated as text values.
239
+
240
+ Likewise, when fetching records, Extralite will convert a `BLOB` column value to
241
+ a string with `ASCII-8BIT` encoding, and a `TEXT` column value to a string with
242
+ `UTF-8` encoding.
243
+
244
+ ```ruby
245
+ # The following calls will insert blob values into the database
246
+ sql = 'insert into foo values (?)'
247
+ db.execute(sql, File.binread('/path/to/file'))
248
+ db.execute(sql, Extralite::Blob.new('Hello, 世界!'))
249
+ db.execute(sql, 'Hello, 世界!'.force_encoding(Encoding::ASCII_8BIT))
250
+ ```
251
+
252
+ ## Value Transforms
253
+
254
+ Extralite allows you to transform rows to any value your application may need by
255
+ providing a transform proc that takes the raw row values and returns the
256
+ transformed data. The transform proc is passed each resulting row either as a
257
+ hash or as a list of values.
258
+
259
+ Transforms are useful when you need to transform rows into ORM model instances,
260
+ or when you need to do some other transformation on the values retrieved from
261
+ the database.
262
+
263
+ To transform results, pass a transform proc as the first parameter to one of the
264
+ `#query_xxx` methods:
265
+
266
+ ```ruby
267
+ transform = ->(h) { MyModel.new(h) }
268
+ db.query(transform, 'select * from foo')
269
+ #=> rows as instances of MyModel
270
+ ```
271
+
272
+ When using the `argv` mode, the different column values are passed as individual
273
+ values to the transform proc:
274
+
275
+ ```ruby
276
+ transform = ->(a, b, c) { { a:a, b: b, c: JSON.parse(c) } }
277
+ db.query_argv(transform, 'select a, b, c from foo')
278
+ #=> transformed rows
279
+ ```
280
+
281
+ Value transforms can also be done with [prepared
282
+ queries](#value-transforms-in-prepared-queries).
283
+
284
+ ## Prepared Queries
285
+
286
+ Prepared queries (also known as prepared statements) allow you to maximize
287
+ performance and reduce memory usage when running the same query repeatedly. They
288
+ also allow you to create parameterized queries that can be repeatedly executed
289
+ with different parameters:
290
+
291
+ ```ruby
292
+ query = db.prepare('select * from foo where x = ?')
293
+
294
+ # bind parameters and get results as an array of hashes
295
+ query.bind(1).to_a
296
+ #=> [{ x: 1, y: 2, z: 3 }]
297
+ ```
298
+
299
+ ### Binding Values to Prepared Queries
300
+
301
+ To bind parameter values to the query, use the `#bind` method. The parameters
302
+ will remain bound to the query until `#bind` is called again.
303
+
304
+ ```ruby
305
+ query.bind(1)
306
+
307
+ # run the query any number of times
308
+ 3.times { query.to_a }
309
+
310
+ # bind other parameter values
311
+ query.bind(4)
312
+ ```
313
+
314
+ You can also bind parameters when creating the prepared query by passing
315
+ additional parameters to the `Database#prepare` method:
316
+
317
+ ```ruby
318
+ query = db.prepare('select * from foo where x = ?', 1)
319
+ ```
320
+
321
+ ### Fetching Records from a Prepared Query
322
+
323
+ Just like the `Database` interface, prepared queries support getting data using
324
+ three different modes: as a hash, an array or as individual column values. To
325
+ set the mode, you can use one of the `#prepare_xxx` methods:
326
+
327
+ ```ruby
328
+ # hash mode
329
+ db.prepare('select * from foo').to_a
330
+ #=> [{ x: 1, y: 2, z: 3}]
331
+
332
+ # argv mode
333
+ db.prepare_argv('select x from foo').to_a
334
+ #=> [1]
335
+
336
+ # ary mode
337
+ db.prepare_ary('select * from foo').to_a
338
+ #=> [[1, 2, 3]]
339
+ ```
340
+
341
+ You can also set the query mode by getting or setting `#mode`:
342
+
343
+ ```ruby
344
+ q = db.prepare('select * from foo')
345
+ q.to_a #=> [{ x: 1, y: 2, z: 3}]
346
+
347
+ q.mode #=> :hash
348
+ q.mode = :ary
349
+ q.to_a "=> [[1, 2, 3]]
350
+ ```
351
+
352
+ ### Fetching Single Records or Batches of Records
353
+
354
+ Prepared queries let you iterate over records one by one, or by batches. For
355
+ this, use the `#next` method:
69
356
 
70
357
  ```ruby
71
- require 'extralite'
358
+ query = db.prepare('select * from foo')
359
+
360
+ query.next
361
+ #=> { x: 1, y: 2, z: 3 }
362
+ query.next
363
+ #=> { x: 4, y: 5, z: 6 }
364
+
365
+ # Fetch the next 10 records
366
+ query.reset # go back tpo the beginning
367
+ query.next(10)
368
+ #=> [{ x: 1, y: 2, z: 3 }, { x: 4, y: 5, z: 6 }]
369
+
370
+ # Fetch the next row as an array
371
+ query = db.prepare_ary('select * from foo')
372
+ query.next
373
+ #=> [1, 2, 3]
374
+
375
+ # Fetch the next row as a single column
376
+ db.prepare_argv('select z from foo').next
377
+ #=> 3
378
+ ```
379
+
380
+ To detect the end of the result, you can use `#eof?`. To go back to the
381
+ beginning of the result set, use `#reset`. The following example shows how to
382
+ read the query results in batches of 10:
383
+
384
+ ```ruby
385
+ query.reset
386
+ while !query.eof?
387
+ records = query.next(10)
388
+ process_records(records)
389
+ end
390
+ ```
391
+
392
+ ### Iterating over Records in a Prepared Query
393
+
394
+ In addition to the `#next` method, you can also iterate over query results by
395
+ using the familiar `#each` method:
396
+
397
+ ```ruby
398
+ # iterate over records as hashes
399
+ query = db.prepare('select * from foo')
400
+ query.each { |r| ... }
401
+
402
+ # iterate over records as arrays
403
+ query = db.prepare_ary('select * from foo')
404
+ query.each { |r| ... }
72
405
 
73
- # get sqlite3 version
74
- Extralite.sqlite3_version #=> "3.35.2"
406
+ # iterate over records as single values
407
+ query = db.prepare_argv('select a, b, c from foo')
408
+ query.each { |a, b, c| ... }
409
+ ```
75
410
 
76
- # open a database
77
- db = Extralite::Database.new('/tmp/my.db')
411
+ ### Prepared Query as an Enumerable
78
412
 
79
- # get query results as array of hashes
80
- db.query('select 1 as foo') #=> [{ :foo => 1 }]
81
- # or:
82
- db.query_hash('select 1 as foo') #=> [{ :foo => 1 }]
83
- # or iterate over results
84
- db.query('select 1 as foo') { |r| p r }
85
- # { :foo => 1 }
413
+ You can also use a prepared query as an enumerable, allowing you to chain
414
+ enumerable method calls while iterating over the query result set. This is done
415
+ by calling `#each` without a block:
86
416
 
87
- # get query results as array of arrays
88
- db.query_ary('select 1, 2, 3') #=> [[1, 2, 3]]
89
- # or iterate over results
90
- db.query_ary('select 1, 2, 3') { |r| p r }
91
- # [1, 2, 3]
417
+ ```ruby
418
+ iterator = query.each
419
+ #=> Returns an Extralite::Iterator instance
92
420
 
93
- # get a single row as a hash
94
- db.query_single_row("select 1 as foo") #=> { :foo => 1 }
421
+ iterator.map { |r| r[:x] *100 + r[:y] * 10 + r[:z] }
422
+ #=> [123, 345]
423
+ ```
95
424
 
96
- # get single column query results as array of values
97
- db.query_single_column('select 42') #=> [42]
98
- # or iterate over results
99
- db.query_single_column('select 42') { |v| p v }
100
- # 42
425
+ The Iterator class includes the
426
+ [`Enumerable`](https://rubyapi.org/3.3/o/enumerable) module, with all its
427
+ methods, such as `#map`, `#select`, `#inject`, `#lazy` etc. You can also
428
+ instantiate an iterator explicitly:
101
429
 
102
- # get single value from first row of results
103
- db.query_single_value("select 'foo'") #=> "foo"
430
+ ```ruby
431
+ # You need to pass the query to iterate over:
432
+ iterator = Extralite::Iterator(query)
433
+ iterator.each { |r| ... }
434
+ ```
104
435
 
105
- # parameter binding (works for all query_xxx methods)
106
- db.query_hash('select ? as foo, ? as bar', 1, 2) #=> [{ :foo => 1, :bar => 2 }]
107
- db.query_hash('select ? as foo, ? as bar', [1, 2]) #=> [{ :foo => 1, :bar => 2 }]
436
+ ### Value Transforms in Prepared Queries
108
437
 
109
- # explicit parameter indexes
110
- db.query_hash('select ?2 as foo, ?1 as bar, ?1 * ?2 as baz', 6, 7) #=> [{ :foo => 7, :bar => 6, :baz => 42 }
438
+ Prepared queries can automatically transform their result sets by setting a
439
+ transform block. The transform block receives values according to the query mode
440
+ (hash, array or argv). To set a transform you can pass a block to one of the
441
+ `Database#prepare_xxx` methods, or use `Query#transform`:
111
442
 
112
- # named parameters
113
- db.query('select * from foo where bar = :bar', bar: 42)
114
- db.query('select * from foo where bar = :bar', 'bar' => 42)
115
- db.query('select * from foo where bar = :bar', ':bar' => 42)
443
+ ```ruby
444
+ q = db.prepare('select * from items where id = ?') { |h| Item.new(h) }
445
+ q.bind(42).next #=> Item instance
116
446
 
117
- # parameter binding of named parameters from Struct and Data
118
- SomeStruct = Struct.new(:foo, :bar)
119
- db.query_single_column('select :bar', SomeStruct.new(41, 42)) #=> [42]
120
- SomeData = Data.define(:foo, :bar)
121
- db.query_single_column('select :bar', SomeData.new(foo: 41, bar: 42)) #=> [42]
447
+ # An equivalent
448
+ q = db.prepare('select * from items where id = ?')
449
+ q.transform { |h| Item.new(h) }
450
+ ```
122
451
 
123
- # parameter binding for binary data (BLOBs)
124
- db.execute('insert into foo values (?)', File.binread('/path/to/file'))
125
- db.execute('insert into foo values (?)', Extralite::Blob.new('Hello, 世界!'))
126
- db.execute('insert into foo values (?)', 'Hello, 世界!'.force_encoding(Encoding::ASCII_8BIT))
452
+ The same can be done for queries in `argv` or `ary` mode:
127
453
 
128
- # insert multiple rows
129
- db.batch_execute('insert into foo values (?)', ['bar', 'baz'])
130
- db.batch_execute('insert into foo values (?, ?)', [[1, 2], [3, 4]])
454
+ ```ruby
455
+ db.prepare_argv('select * from foo') { |a, b, c| a + b + c }
456
+
457
+ db.prepare_ary('select * from foo') { |a| a.map(&:to_s).join }
458
+ ```
131
459
 
132
- # batch execute from enumerable
460
+ ## Batch Execution of Queries
461
+
462
+ Extralite provides methods for batch execution of queries, with multiple sets of
463
+ parameters. The `#batch_execute` method lets you insert or update a large number
464
+ of records with a single call:
465
+
466
+ ```ruby
467
+ values = [
468
+ [1, 11, 111],
469
+ [2, 22, 222],
470
+ [3, 33, 333]
471
+ ]
472
+ # insert the above records in one fell swoop, and returns the total number of
473
+ # changes:
474
+ db.batch_execute('insert into foo values (?, ?, ?)', values)
475
+ #=> 3
476
+ ```
477
+
478
+ Parameters to the query can also be provided by any object that is an
479
+ `Enumerable` or has an `#each` method, or any *callable* object that responds to
480
+ `#call`:
481
+
482
+ ```ruby
483
+ # Take parameter values from a Range
133
484
  db.batch_execute('insert into foo values (?)', 1..10)
485
+ #=> 10
134
486
 
135
- # batch execute from block
136
- source = [[1, 2], [2, 3], [3, 4]]
137
- db.batch_execute('insert into foo values (?, ?)') { source.shift }
487
+ # Insert (chomped) lines from a file
488
+ File.open('foo.txt') do |f|
489
+ source = f.each_line.map(&:chomp)
490
+ db.batch_execute('insert into foo values (?)', source)
491
+ end
138
492
 
139
- # prepared queries
140
- query = db.prepare('select ? as foo, ? as bar') #=> Extralite::Query
141
- query.bind(1, 2) #=> [{ :foo => 1, :bar => 2 }]
493
+ # Insert items from a queue
494
+ parameters = proc do
495
+ item = queue.shift
496
+ # when we're done, we return nil
497
+ (item == :done) ? nil : item
498
+ end
499
+ db.batch_execute('insert into foo values (?)', parameters)
500
+ #=> number of rows inserted
501
+ ```
142
502
 
143
- query.next #=> next row in result_set (as hash)
144
- query.next_hash #=> next row in result_set (as hash)
145
- query.next_ary #=> next row in result_set (as array)
146
- query.next_single_column #=> next row in result_set (as single value)
503
+ Like its cousin `#execute`, the `#batch_execute` returns the total number of
504
+ changes to the database (rows inserted, deleted or udpated).
147
505
 
148
- query.next(10) #=> next 10 rows in result_set (as hash)
149
- query.next_hash(10) #=> next 10 rows in result_set (as hash)
150
- query.next_ary(10) #=> next 10 rows in result_set (as array)
151
- query.next_single_column(10) #=> next 10 rows in result_set (as single value)
506
+ ### Batch Execution of Queries that Return Rows
152
507
 
153
- query.to_a #=> all rows as array of hashes
154
- query.to_a_hash #=> all rows as array of hashes
155
- query.to_a_ary #=> all rows as array of arrays
156
- query.to_a_single_column #=> all rows as array of single values
508
+ Extralite also provides a `#batch_query` method that like `#batch_execute` takes
509
+ a parameter source and returns an array containing the result sets for all query
510
+ invocations. If a block is given, the result sets are passed to the block
511
+ instead.
157
512
 
158
- query.each { |r| ... } #=> iterate over all rows as hashes
159
- query.each_hash { |r| ... } #=> iterate over all rows as hashes
160
- query.each_ary { |r| ... } #=> iterate over all rows as arrays
161
- query.each_single_column { |r| ... } #=> iterate over all rows as single columns
513
+ The `#batch_query` method is especially useful for batch queries with a
514
+ `RETURNING` clause:
162
515
 
163
- iterator = query.each #=> create enumerable iterator
164
- iterator.next #=> next row
165
- iterator.each { |r| ... } #=> iterate over all rows
166
- values = iterator.map { |r| r[:foo] * 10 } #=> map all rows
516
+ ```ruby
517
+ updates = [
518
+ { id: 3, price: 42 },
519
+ { id: 5, price: 43 }
520
+ ]
521
+ sql = 'update foo set price = $price where id = $id returning id, quantity'
522
+ db.batch_query(sql, updates)
523
+ #=> [[{ id: 3, quantity: 4 }], [{ id: 5, quantity: 5 }]]
524
+
525
+ # The same with a block (returns the total number of changes)
526
+ db.batch_query(sql, updates) do |rows|
527
+ p rows
528
+ #=> [{ id: 3, quantity: 4 }]
529
+ #=> [{ id: 5, quantity: 5 }]
530
+ end
531
+ #=> 2
532
+ ```
167
533
 
168
- iterator = query.each_ary #=> create enumerable iterator with rows as arrays
169
- iterator = query.each_single_column #=> create enumerable iterator with single values
534
+ The `#batch_query` method, like other row fetching methods, changes the row
535
+ representation according to the query mode.
170
536
 
171
- # get last insert rowid
172
- rowid = db.last_insert_rowid
537
+ ### Batch Execution of Prepared Queries
173
538
 
174
- # get number of rows changed in last query
175
- number_of_rows_affected = db.changes
539
+ Batch execution can also be done using prepared queries, using the same methods
540
+ `#batch_execute` and `#batch_query`:
176
541
 
177
- # get column names for the given sql
178
- db.columns('select a, b, c from foo') => [:a, :b, :c]
542
+ ```ruby
543
+ query = db.prepare 'update foo set x = ? where z = ? returning *'
179
544
 
180
- # get db filename
181
- db.filename #=> "/tmp/my.db"
545
+ query.batch_execute([[42, 3], [43, 6]])
546
+ #=> 2
182
547
 
183
- # get list of tables
184
- db.tables #=> ['foo', 'bar']
548
+ query.batch_query([[42, 3], [43, 6]])
549
+ #=> [{ x: 42, y: 2, z: 3 }, { x: 43, y: 5, z: 6 }]
550
+ ```
185
551
 
186
- # get and set pragmas
187
- db.pragma(:journal_mode) #=> 'delete'
188
- db.pragma(journal_mode: 'wal')
189
- db.pragma(:journal_mode) #=> 'wal'
552
+ ## Transactions and Savepoints
190
553
 
191
- # load an extension
192
- db.load_extension('/path/to/extension.so')
554
+ All reads and writes to SQLite databases occur within a
555
+ [transaction](https://www.sqlite.org/lang_transaction.html). If no explicit
556
+ transaction has started, the submitted SQL statements passed to `#execute` or
557
+ `#query` will all run within an implicit transaction:
558
+
559
+ ```ruby
560
+ # The following two SQL statements will run in a single implicit transaction:
561
+ db.execute('insert into foo values (42); insert into bar values (43)')
562
+
563
+ # Otherwise, each call to #execute runs in a separate transaction:
564
+ db.execute('insert into foo values (42)')
565
+ db.execute('insert into bar values (43)')
566
+ ```
567
+
568
+ ### Explicit Transactions
193
569
 
194
- # run queries in a transaction
570
+ While you can issue `BEGIN` and `COMMIT` SQL statements yourself to start and
571
+ commit explicit transactions, Extralite provides a convenient `#transaction`
572
+ method that manages starting, commiting and rolling back of transactions
573
+ automatically:
574
+
575
+ ```ruby
195
576
  db.transaction do
196
- db.execute('insert into foo values (?)', 1)
197
- db.execute('insert into foo values (?)', 2)
198
- db.execute('insert into foo values (?)', 3)
577
+ db.execute('insert into foo values (42)')
578
+ raise 'Something bad happened' if something_bad_happened
579
+ db.execute('insert into bar values (43)')
199
580
  end
581
+ ```
200
582
 
201
- # close database
202
- db.close
203
- db.closed? #=> true
583
+ If no exception is raised in the transaction block, the changes are commited. If
584
+ an exception is raised, the changes are rolled back and the exception is
585
+ propagated to the application code. You can prevent the exception from being
586
+ propagated by calling `#rollback!`:
587
+
588
+ ```ruby
589
+ db.transaction do
590
+ db.execute('insert into foo values (42)')
591
+ rollback! if something_bad_happened
592
+ db.execute('insert into bar values (43)')
593
+ end
204
594
  ```
205
595
 
206
- ## More Features
596
+ ### Transaction Modes
597
+
598
+ By default, `#transaction` starts an `IMMEDIATE` transaction. To start a
599
+ `DEFERRED` or `EXCLUSIVE` transaction, pass the desired mode to `#transaction`:
207
600
 
208
- ### Interrupting Long-running Queries
601
+ ```ruby
602
+ # Start a DEFERRED transaction
603
+ db.transaction(:deferred) do
604
+ ...
605
+ end
606
+
607
+ # Start a EXCLUSIVE transaction
608
+ db.transaction(:exclusive) do
609
+ ...
610
+ end
611
+ ```
209
612
 
210
- When running long-running queries, you can use `Database#interrupt` to interrupt
211
- the query:
613
+ Note that running an `IMMEDIATE` or `EXCLUSIVE` transaction blocks the database
614
+ for writing (and also reading in certain cases) for the duration of the
615
+ transaction. This can cause queries to the same database on a different
616
+ connection to fail with a `BusyError` exception. This can be mitigated by
617
+ setting a [busy timeout](#dealing-with-a-busy-database).
618
+
619
+ ### Savepoints
620
+
621
+ In addition to transactions, SQLite also supports the use of
622
+ [savepoints](https://www.sqlite.org/lang_savepoint.html), which can be used for
623
+ more fine-grained control of changes within a transaction, and to be able to
624
+ rollback specific changes without abandoning the entire transaction:
212
625
 
213
626
  ```ruby
214
- timeout_thread = Thread.new do
215
- sleep 10
216
- db.interrupt
627
+ db.transaction do
628
+ db.execute 'insert into foo values (1)'
629
+
630
+ db.savepoint :my_savepoint
631
+ db.execute 'insert into foo values (2)'
632
+
633
+ # the following cancels the last insert
634
+ db.rollback_to :my_savepoint
635
+ db.execute 'insert into foo values (3)'
636
+
637
+ db.release :my_savepoint
217
638
  end
639
+ ```
218
640
 
219
- result = begin
220
- db.query(super_slow_sql)
221
- rescue Extralite::InterruptError
222
- nil
223
- ensure
641
+ ## Database Information
642
+
643
+ ### Getting the list of tables
644
+
645
+ To get the list of tables in a database, use the `#tables` method:
646
+
647
+ ```ruby
648
+ db.tables
649
+ #=> [...]
650
+ ```
651
+
652
+ To get the list of tables in an attached database, you can pass the database name to `#tables`:
653
+
654
+ ```ruby
655
+ db.execute "attach database 'foo.db' as foo"
656
+ db.tables('foo')
657
+ #=> [...]
658
+ ```
659
+
660
+ ### Getting the last insert row id
661
+
662
+ ```ruby
663
+ db.execute 'insert into foo values (?)', 42
664
+ db.last_insert_rowid
665
+ #=> 1
666
+ ```
667
+
668
+ ### Getting the columns names for a given query
669
+
670
+ ```ruby
671
+ db.columns('select a, b, c from foo')
672
+ #=> [:a, :b, :c]
673
+
674
+ # Columns a prepared query:
675
+ query = db.prepare('select x, y from foo')
676
+ query.columns
677
+ #=> [:x, :y]
678
+ ```
679
+
680
+ ### Pragmas
681
+
682
+ You can get or set pragma values using `#pragma`:
683
+
684
+ ```ruby
685
+ # get a pragma value:
686
+ db.pragma(:journal_mode)
687
+ #=> 'delete'
688
+
689
+ # set a pragma value:
690
+ db.pragma(journal_mode: 'wal')
691
+ db.pragma(:journal_mode)
692
+ #=> 'wal'
693
+ ```
694
+
695
+ You can also pass pragmas when opening the database:
696
+
697
+ ```ruby
698
+ db = Extralite::Database.new('path/to/db', pragma: { foreign_keys: true })
699
+ ```
700
+
701
+ ## Error Handling
702
+
703
+ Extralite defines various exception classes that are raised when an error is
704
+ encountered while interacting with the underlying SQLite library:
705
+
706
+ - `Extralite::SQLError` - raised when SQLite encounters an invalid SQL query.
707
+ - `Extralite::BusyError` - raised when the underlying database is locked for use
708
+ by another database connection.
709
+ - `Extralite::InterruptError` - raised when a query has been interrupted.
710
+ - `Extralite::ParameterError` - raised when an invalid parameter value has been
711
+ specified.
712
+ - `Extralite::Error` - raised on all other errors.
713
+
714
+ In addition to the above exceptions, further information about the last error
715
+ that occurred is provided by the following methods:
716
+
717
+ - `#errcode` - the [error code](https://www.sqlite.org/rescode.html) returned by
718
+ the underlying SQLite library.
719
+ - `#errmsg` - the error message for the last error. For most errors, the error
720
+ message is copied into the exception message.
721
+ - `#error_offset` - for SQL errors, the offset into the SQL string where the
722
+ error was encountered.
723
+
724
+ ## Concurrency
725
+
726
+ Extralite provides a comprehensive set of tools for dealing with concurrency
727
+ issues, and for making sure that running queries on SQLite databases does not
728
+ cause the app to freeze.
729
+
730
+ **Note**: In order to allow concurrent access your the database, it is highly
731
+ recommended that you set your database to use [WAL journaling
732
+ mode](https://www.sqlite.org/wal.html) for *all* database connections.
733
+ Otherwise, you risking running into performance problems and having queries fail
734
+ with `BusyError` exceptions. You can easily open your database in WAL journaling
735
+ mode by passing a `wal: true` option:
736
+
737
+ ```ruby
738
+ # This will set PRAGMA journal_mode=1 and PRAGMA synchronous=1
739
+ db = Extralite::Database.new('path/to/db', wal: true)
740
+ ```
741
+
742
+ ### The Ruby GVL
743
+
744
+ In order to support multi-threading, Extralite releases the [Ruby
745
+ GVL](https://www.speedshop.co/2020/05/11/the-ruby-gvl-and-scaling.html)
746
+ periodically while running queries. This allows other threads to run while the
747
+ underlying SQLite library is busy preparing queries, fetching records and
748
+ backing up databases. By default, the GVL is when preparing the query, and once
749
+ for every 1000 iterated records. The GVL release threshold can be set separately
750
+ for each database:
751
+
752
+ ```ruby
753
+ # release the GVL on preparing the query, and every 10 records
754
+ db.gvl_release_threshold = 10
755
+
756
+ # release the GVL only when preparing the query
757
+ db.gvl_release_threshold = 0
758
+
759
+ # never release the GVL (for single-threaded apps only)
760
+ db.gvl_release_threshold = -1
761
+
762
+ db.gvl_release_threshold = nil # use default value (currently 1000)
763
+ ```
764
+
765
+ For most applications, there's no need to tune the GVL threshold value, as it
766
+ provides [excellent](#performance) performance characteristics for both
767
+ single-threaded and multi-threaded applications.
768
+
769
+ In a heavily multi-threaded application, releasing the GVL more often (lower
770
+ threshold value) will lead to less latency (for threads not running a query),
771
+ but will also hurt the throughput (for the thread running the query). Releasing
772
+ the GVL less often (higher threshold value) will lead to better throughput for
773
+ queries, while increasing latency for threads not running a query. The following
774
+ diagram demonstrates the relationship between the GVL release threshold value,
775
+ latency and throughput:
776
+
777
+ ```
778
+ less latency & throughput <<< GVL release threshold >>> more latency & throughput
779
+ ```
780
+
781
+ ### Dealing with a Busy Database
782
+
783
+ When multiple threads or processes access the same database, the database may be
784
+ locked for writing by one process, which will block other processes wishing to
785
+ write to the database. When attempting to write to a locked database, a
786
+ `Extralite::BusyError` will be raised:
787
+
788
+ ```ruby
789
+ ready = nil
790
+ locker = Thread.new do
791
+ db1 = Extralite::Database.new('my.db')
792
+ # Lock the database for 3 seconds
793
+ db1.transaction do
794
+ ready = true
795
+ sleep(3)
796
+ end
797
+ end
798
+
799
+ db2 = Extralite::Database.new('my.db')
800
+ # wait for writer1 to enter a transaction
801
+ sleep(0) while !ready
802
+ # This will raise a Extralite::BusyError
803
+ db2.transaction { }
804
+ # Extralite::BusyError!
805
+ ```
806
+
807
+ You can mitigate this by setting a busy timeout. This will cause SQLite to wait
808
+ for the database to become unlocked up to the specified timeout period:
809
+
810
+ ```ruby
811
+ # Wait for up to 5 seconds before giving up
812
+ db2.busy_timeout = 5
813
+ # Now it'll work!
814
+ db2.transaction { }
815
+ ```
816
+
817
+ As stated above, setting the database to use WAL journaling mode greatly reduces
818
+ contention between different process/threads accessing the same database. For
819
+ most use cases, setting the busy timeout solves the problem of failing to run
820
+ queries because of a busy database, as normally transactions are short-lived.
821
+
822
+ However, in some cases, such as when running a multi-fibered app or when
823
+ implementing your own timeout mechanisms, you'll want to set a [progress
824
+ handler](#the-progress-handler).
825
+
826
+ ### Interrupting a Query
827
+
828
+ To interrupt an ongoing query, use the `#interrupt` method. Normally this is
829
+ done from a separate thread. Here's a way to implement a timeout using
830
+ `#interrupt`:
831
+
832
+ ```ruby
833
+ def run_query_with_timeout(sql, timeout)
834
+ timeout_thread = Thread.new do
835
+ t0 = Time.now
836
+ sleep(timeout)
837
+ @db.interrupt
838
+ end
839
+ result = @db.query(sql)
224
840
  timeout_thread.kill
225
- timeout_thread.join
841
+ result
226
842
  end
843
+
844
+ run_query_with_timeout('select 1 as foo', 5)
845
+ #=> [{ foo: 1 }]
846
+
847
+ # A timeout will cause a Extralite::InterruptError to be raised
848
+ run_query_with_timeout(slow_sql, 5)
849
+ #=> Extralite::InterruptError!
227
850
  ```
228
851
 
229
- ### Running transactions
852
+ You can also call `#interrupt` from within the [progress
853
+ handler](#the-progress-handler).
854
+
855
+ ### The Progress Handler
856
+
857
+ Extralite also supports setting up a progress handler, which is a piece of code
858
+ that will be called while a query is in progress, or while the database is busy.
859
+ This is useful especially when you want to implement a general purpose timeout
860
+ mechanism that deals with both a busy database and with slow queries.
861
+
862
+ The progress handler can also be used for performing any kind of operation while
863
+ a query is in progress. Here are some use cases:
230
864
 
231
- In order to run multiple queries in a single transaction, use
232
- `Database#transaction`, passing a block that runs the queries. You can specify
233
- the [transaction
234
- mode](https://www.sqlite.org/lang_transaction.html#deferred_immediate_and_exclusive_transactions).
235
- The default mode is `:immediate`:
865
+ - Interrupting queries that take too long to run.
866
+ - Interrupting queries on an exceptional condition, such as a received signal.
867
+ - Updating the UI while a query is running.
868
+ - Switching between fibers in multi-fibered apps.
869
+ - Switching between threads in multi-threaded apps.
870
+ - Instrumenting the performance of queries.
871
+
872
+ Setting the progress handler requires that Extralite hold the GVL while running
873
+ all queries. Therefore, it should be used with care. In a multi-threaded app,
874
+ you'll need to call `Thread.pass` from the progress handler in order for other
875
+ threads to be able to run while the query is in progress. The progress handler
876
+ is set per-database using `#on_progress`. This method takes a single parameter
877
+ that specifies the approximate number of SQLite VM instructions between
878
+ successive calls to the progress handler:
879
+
880
+ ```ruby
881
+ db.on_progress do
882
+ check_for_timeout
883
+ # Allow other threads to run
884
+ Thread.pass
885
+ end
886
+ ```
887
+
888
+ The progress handler can be used to interrupt queries in progress. This can be
889
+ done by either calling `#interrupt`, or by raising an exception. As discussed
890
+ above, calling `#interrupt` causes the query to raise a
891
+ `Extralite::InterruptError` exception:
892
+
893
+ ```ruby
894
+ db.on_progress { db.interrupt }
895
+ db.query('select 1')
896
+ #=> Extralite::InterruptError!
897
+ ```
898
+
899
+ You can also interrupt queries in progress by raising an exception. The query
900
+ will be stopped, and the exception will propagate to the call site:
901
+
902
+ ```ruby
903
+ db.on_progress do
904
+ raise 'BOOM!'
905
+ end
906
+
907
+ db.query('select 1')
908
+ #=> BOOM!
909
+ ```
910
+
911
+ Here's how a timeout might be implemented using the progress handler:
912
+
913
+ ```ruby
914
+ def setup_progress_handler
915
+ @db.on_progress do
916
+ raise TimeoutError if Time.now - @t0 >= @timeout
917
+ Thread.pass
918
+ end
919
+ end
920
+
921
+ # In this example, we just return nil on timeout
922
+ def run_query_with_timeout(sql, timeout)
923
+ @t0 = Time.now
924
+ @db.query(sql)
925
+ rescue TimeoutError
926
+ nil
927
+ end
928
+
929
+ run_query_with_timeout('select 1 as foo', 5)
930
+ #=> [{ foo: 1 }]
931
+
932
+ run_query_with_timeout(slow_sql, 5)
933
+ #=> nil
934
+ ```
935
+
936
+ **Note**: you must not issue any query from within the progress handler.
937
+
938
+ ### Dealing with a Busy Database in the Progress Handler
939
+
940
+ As mentioned above, the progress handler is also called when the database is
941
+ busy, regardless of the progress period given to `#on_progress`. You can detect
942
+ if the database is busy by checking the first argument passed to the progress
943
+ handler, which will be true when busy:
236
944
 
237
945
  ```ruby
238
- db.transaction { ... } # Run an immediate transaction
239
- db.transaction(:deferred) { ... } # Run a deferred transaction
240
- db.transaction(:exclusive) { ... } # Run an exclusive transaction
946
+ db.on_progress do |busy|
947
+ if busy
948
+ foo
949
+ else
950
+ bar
951
+ end
952
+ end
241
953
  ```
242
954
 
243
- If an exception is raised in the given block, the transaction will be rolled
244
- back. Otherwise, it is committed.
955
+ This allows you to implement separate logic to deal with busy states, for
956
+ example sleeping for a small period of time, or implementing a different timeout
957
+ period.
958
+
959
+ ### Advanced Progress Handler Settings
960
+
961
+ You can further tune the behaviour of the progress handler with the following
962
+ options passed to `#on_progress`:
963
+
964
+ - `:mode`: the following modes are supported:
965
+ - `:none` : the progress handler is disabled.
966
+ - `:normal`: the progress handler is called on query progress (this is the
967
+ default mode).
968
+ - `:once`: the progress handler is called once before running the query.
969
+ - `:at_least_once`: the progress handler is called once before running the
970
+ query, and on query progress.
971
+ - `:period`: controls the approximate number of SQLite VM instructions executed
972
+ between consecutive calls to the progress handler. Default value: 1000.
973
+ - `:tick`: controls the granularity of the progress handler. This is the value
974
+ passed internally to the SQLite library. Default value: 10.
975
+
976
+ ```ruby
977
+ db.on_progress(mode: :at_least_once, period: 640, tick: 5) { snooze }
978
+ ```
979
+
980
+ ### Global Progress Handler Settings
981
+
982
+ You can set the global progress handler behaviour by calling
983
+ `Extralite.on_progress`. You can use this API to set the global progress
984
+ settings, without needing to set a progress handler individually for each
985
+ `Database` instance. This method takes the same options as
986
+ `Database#on_progress`:
987
+
988
+ ```ruby
989
+ Extralite.on_progress(mode: :at_least_once, period: 640, tick: 5) { snooze }
990
+
991
+ # the new database instance uses the global progress handler settings
992
+ db = Database.new(':memory:')
993
+ ```
994
+
995
+ ### Extralite and Fibers
996
+
997
+ The progress handler can also be used to switch between fibers in a
998
+ multi-fibered Ruby app, based on libraries such as
999
+ [Async](https://github.com/socketry/async) or
1000
+ [Polyphony](https://github.com/digital-fabric/polyphony). A general solution
1001
+ (that also works for multi-threaded apps) is to call `sleep(0)` in the progress
1002
+ handler. This will work for switching between fibers using either Polyphony or
1003
+ any fiber scheduler gem, such as Async et al:
1004
+
1005
+ ```ruby
1006
+ db.on_progress(100) { sleep(0) }
1007
+ ```
1008
+
1009
+ For Polyphony-based apps, you can also call `snooze` to allow other fibers to
1010
+ run while a query is progressing. If your Polyphony app is multi-threaded,
1011
+ you'll also need to call `Thread.pass` in order to allow other threads to run:
1012
+
1013
+ ```ruby
1014
+ db.on_progress(100) do
1015
+ snooze
1016
+ Thread.pass
1017
+ end
1018
+ ```
1019
+
1020
+ Note that with Polyphony, once you install the progress handler, you can just
1021
+ use the regular `#move_on_after` and `#cancel_after` methods to implement
1022
+ timeouts for queries:
1023
+
1024
+ ```ruby
1025
+ db.on_progress(100) { snooze }
1026
+
1027
+ cancel_after(3) do
1028
+ db.query(long_running_query)
1029
+ end
1030
+ ```
1031
+
1032
+ ### Thread Safety
1033
+
1034
+ A single database instance can be safely used in multiple threads simultaneously
1035
+ as long as the following conditions are met:
1036
+
1037
+ - No explicit transactions are used.
1038
+ - Each thread issues queries by calling `Database#query_xxx`, or uses a separate
1039
+ `Query` instance.
1040
+ - The GVL release threshold is not `0` (i.e. the GVL is released periodically
1041
+ while running queries.)
1042
+
1043
+ ### Use with Ractors
1044
+
1045
+ Extralite databases can safely be used inside ractors. A ractor has the benefit
1046
+ of using a separate GVL from the maine one, which allows true parallelism for
1047
+ Ruby apps. So when you use Extralite to access SQLite databases from within a
1048
+ ractor, you can do so without any considerations for what's happening outside
1049
+ the ractor when it runs queries.
1050
+
1051
+ **Note**: Ractors are considered an experimental feature of Ruby. You may
1052
+ encounter errors or inconsistent behaviour when using ractors.
1053
+
1054
+ ## Advanced Usage
1055
+
1056
+ ### Loading Extensions
1057
+
1058
+ Extensions can be loaded by calling `#load_extension`:
1059
+
1060
+ ```ruby
1061
+ db.load_extension('/path/to/extension.so')
1062
+ ```
1063
+
1064
+ A pretty comprehensive set of extensions can be found here:
1065
+
1066
+ https://github.com/nalgeon/sqlean
245
1067
 
246
1068
  ### Creating Backups
247
1069
 
@@ -266,6 +1088,51 @@ db.backup('backup.db') do |remaining, total|
266
1088
  end
267
1089
  ```
268
1090
 
1091
+ ### Working with Changesets
1092
+
1093
+ __Note__: as the session extension is by default disabled in SQLite
1094
+ distributions, support for changesets is currently only available withthe
1095
+ bundled version of Extralite, `extralite-bundle`.
1096
+
1097
+ Changesets can be used to track and persist changes to data in a database. They
1098
+ can also be used to apply the same changes to another database, or to undo them.
1099
+ To track changes to a database, use the `#track_changes` method:
1100
+
1101
+ ```ruby
1102
+ # track changes to the foo and bar tables:
1103
+ changeset = db.track_changes(:foo, :bar) do
1104
+ insert_a_bunch_of_records(db)
1105
+ end
1106
+
1107
+ # to track changes to all tables, pass nil:
1108
+ changeset = db.track_changes(nil) do
1109
+ insert_a_bunch_of_records(db)
1110
+ end
1111
+ ```
1112
+
1113
+ You can then apply the same changes to another database:
1114
+
1115
+ ```ruby
1116
+ changeset.apply(some_other_db)
1117
+ ```
1118
+
1119
+ To undo the changes, obtain an inverted changeset and apply it to the database:
1120
+
1121
+ ```ruby
1122
+ changeset.invert.apply(db)
1123
+ ```
1124
+
1125
+ You can also save and load the changeset:
1126
+
1127
+ ```ruby
1128
+ # save the changeset
1129
+ IO.write('my.changes', changeset.to_blob)
1130
+
1131
+ # load the changeset
1132
+ changeset = Extralite::Changeset.new
1133
+ changeset.load(IO.read('my.changes'))
1134
+ ```
1135
+
269
1136
  ### Retrieving Status Information
270
1137
 
271
1138
  Extralite provides methods for retrieving status information about the sqlite
@@ -305,16 +1172,6 @@ value = db.limit(Extralite::SQLITE_LIMIT_ATTACHED)
305
1172
  db.limit(Extralite::SQLITE_LIMIT_ATTACHED, new_value)
306
1173
  ```
307
1174
 
308
- ### Setting the Busy Timeout
309
-
310
- When accessing a database concurrently it can be handy to set a busy timeout, in
311
- order to not have to deal with rescuing `Extralite::BusyError` exceptions. The
312
- timeout is given in seconds:
313
-
314
- ```ruby
315
- db.busy_timeout = 5
316
- ```
317
-
318
1175
  ### Tracing SQL Statements
319
1176
 
320
1177
  To trace all SQL statements executed on the database, pass a block to
@@ -342,58 +1199,6 @@ p articles.to_a
342
1199
 
343
1200
  (Make sure you include `extralite` as a dependency in your `Gemfile`.)
344
1201
 
345
- ## Concurrency
346
-
347
- ### The Ruby GVL
348
-
349
- Extralite releases the [Ruby
350
- GVL](https://www.speedshop.co/2020/05/11/the-ruby-gvl-and-scaling.html) while
351
- making calls to the sqlite3 library that might block, such as when backing up a
352
- database, or when preparing a query. This allows other threads to run while the
353
- underlying sqlite3 library is busy preparing queries, fetching records and
354
- backing up databases.
355
-
356
- Extralite also releases the GVL periodically when iterating over records. By
357
- default, the GVL is released every 1000 records iterated. The GVL release
358
- threshold can be set separately for each database:
359
-
360
- ```ruby
361
- db.gvl_release_threshold = 10 # release GVL every 10 records
362
-
363
- db.gvl_release_threshold = nil # use default value (currently 1000)
364
- ```
365
-
366
- For most applications, there's no need to tune the GVL threshold value, as it
367
- provides [excellent](#performance) performance characteristics for both
368
- single-threaded and multi-threaded applications.
369
-
370
- In a heavily multi-threaded application, releasing the GVL more often (lower
371
- threshold value) will lead to less latency (for threads not running a query),
372
- but will also hurt the throughput (for the thread running the query). Releasing
373
- the GVL less often (higher threshold value) will lead to better throughput for
374
- queries, while increasing latency for threads not running a query. The following
375
- diagram demonstrates the relationship between the GVL release threshold value,
376
- latency and throughput:
377
-
378
- ```
379
- less latency & throughput <<< GVL release threshold >>> more latency & throughput
380
- ```
381
-
382
- ### Thread Safety
383
-
384
- A single database instance can be safely used in multiple threads simultaneously
385
- as long as the following conditions are met:
386
-
387
- - No explicit transactions are used.
388
- - Each thread issues queries by calling `Database#query_xxx`, or uses a separate
389
- `Query` instance.
390
- - The GVL release threshold is not `0` (i.e. the GVL is released periodically
391
- while running queries.)
392
-
393
- ### Use with Ractors
394
-
395
- Extralite databases can be used inside ractors
396
-
397
1202
  ## Performance
398
1203
 
399
1204
  A benchmark script is included, creating a table of various row counts, then
@@ -403,37 +1208,44 @@ large number of rows.
403
1208
 
404
1209
  ### Rows as Hashes
405
1210
 
406
- [Benchmark source code](https://github.com/digital-fabric/extralite/blob/main/test/perf_hash.rb)
1211
+ [Benchmark source
1212
+ code](https://github.com/digital-fabric/extralite/blob/main/test/perf_hash.rb)
407
1213
 
408
- |Row count|sqlite3 1.6.0|Extralite 1.21|Advantage|
1214
+ |Row count|sqlite3 1.7.0|Extralite 2.5|Advantage|
409
1215
  |-:|-:|-:|-:|
410
- |10|63.7K rows/s|94.0K rows/s|__1.48x__|
411
- |1K|299.2K rows/s|1.983M rows/s|__6.63x__|
412
- |100K|185.4K rows/s|2.033M rows/s|__10.97x__|
1216
+ |10|184.9K rows/s|473.2K rows/s|__2.56x__|
1217
+ |1K|290.5K rows/s|2320.7K rows/s|__7.98x__|
1218
+ |100K|143.0K rows/s|2061.3K rows/s|__14.41x__|
413
1219
 
414
1220
  ### Rows as Arrays
415
1221
 
416
- [Benchmark source code](https://github.com/digital-fabric/extralite/blob/main/test/perf_ary.rb)
1222
+ [Benchmark source
1223
+ code](https://github.com/digital-fabric/extralite/blob/main/test/perf_ary.rb)
417
1224
 
418
- |Row count|sqlite3 1.6.0|Extralite 1.21|Advantage|
1225
+ |Row count|sqlite3 1.7.0|Extralite 2.5|Advantage|
419
1226
  |-:|-:|-:|-:|
420
- |10|71.2K rows/s|92.1K rows/s|__1.29x__|
421
- |1K|502.1K rows/s|2.065M rows/s|__4.11x__|
422
- |100K|455.7K rows/s|2.511M rows/s|__5.51x__|
1227
+ |10|276.9K rows/s|472.3K rows/s|__1.71x__|
1228
+ |1K|615.6K rows/s|2324.3K rows/s|__3.78x__|
1229
+ |100K|477.4K rows/s|1982.7K rows/s|__4.15x__|
423
1230
 
424
1231
  ### Prepared Queries (Prepared Statements)
425
1232
 
426
- [Benchmark source code](https://github.com/digital-fabric/extralite/blob/main/test/perf_prepared.rb)
1233
+ [Benchmark source
1234
+ code](https://github.com/digital-fabric/extralite/blob/main/test/perf_hash_prepared.rb)
427
1235
 
428
- |Row count|sqlite3 1.6.0|Extralite 1.21|Advantage|
1236
+ |Row count|sqlite3 1.7.0|Extralite 2.5|Advantage|
429
1237
  |-:|-:|-:|-:|
430
- |10|232.2K rows/s|741.6K rows/s|__3.19x__|
431
- |1K|299.8K rows/s|2386.0M rows/s|__7.96x__|
432
- |100K|183.1K rows/s|1.893M rows/s|__10.34x__|
1238
+ |10|228.5K rows/s|707.9K rows/s|__3.10x__|
1239
+ |1K|296.5K rows/s|2396.2K rows/s|__8.08x__|
1240
+ |100K|145.9K rows/s|2107.3K rows/s|__14.45x__|
1241
+
1242
+ As those benchmarks show, Extralite is capabale of reading up to 2.4M rows per
1243
+ second, and can be more than 14 times faster than the `sqlite3` gem.
433
1244
 
434
- As those benchmarks show, Extralite is capabale of reading up to 2.5M
435
- rows/second when fetching rows as arrays, and up to 2M rows/second when fetching
436
- rows as hashes.
1245
+ Note that the benchmarks above were performed on synthetic data, in a
1246
+ single-threaded environment, with the GVL release threshold set to -1, which
1247
+ means that both Extralite and the `sqlite3` gem hold the GVL for the duration of
1248
+ the query.
437
1249
 
438
1250
  ## License
439
1251