rails_cursor_pagination 0.1.3 → 0.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +14 -1
- data/README.md +49 -8
- data/lib/rails_cursor_pagination.rb +17 -13
- data/lib/rails_cursor_pagination/paginator.rb +68 -50
- data/lib/rails_cursor_pagination/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 92ff9fc0d88af01bbcee5b6bc749790494b476fe425492f799a1ebd0e4a35cf7
|
4
|
+
data.tar.gz: 421e730132cc775c717e10bfb05309a98419d6096166f24db631390cfb4792cb
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 653cbdf05a63e16adab8f991dcf9fa90b0799424174c8b8838795f2f8cccdb89cd823e7a5708e485a8113e2af3ea6362c8a2696bfdac783ac30b8108795df841
|
7
|
+
data.tar.gz: 6307cfb673d4d18ed4dff27efdc5de2f042cfeb09332d23cb7813ff7bf79ed761b219b7486330ff924dcbe624325b41a75a71955312dcd30992a84299ba5193c
|
data/CHANGELOG.md
CHANGED
@@ -14,6 +14,19 @@ These are the latest changes on the project's `master` branch that have not yet
|
|
14
14
|
Follow the same format as previous releases by categorizing your feature into "Added", "Changed", "Deprecated", "Removed", "Fixed", or "Security".
|
15
15
|
--->
|
16
16
|
|
17
|
+
## [0.2.0] - 2021-04-19
|
18
|
+
|
19
|
+
### Changed
|
20
|
+
- **Breaking change:** The way records are retrieved from a given cursor has been changed to no longer use `CONCAT` but instead simply use a compound `WHERE` clause in case of a custom order and having both the custom field as well as the `id` field in the `ORDER BY` query. This is a breaking change since it now changes the internal order of how records with the same value of the `order_by` field are returned.
|
21
|
+
- Remove `ORDER BY` clause from `COUNT` queries
|
22
|
+
|
23
|
+
### Fixed
|
24
|
+
- Only trigger one SQL query to load the records from the database and use it to determine if there was a previous / is a next page
|
25
|
+
- Memoize the `Paginator#page` method which is invoked multiple times to prevent it from mapping over the `records` again and again and rebuilding all cursors
|
26
|
+
|
27
|
+
### Added
|
28
|
+
- Description about `order_by` on arbitrary SQL to README.md
|
29
|
+
|
17
30
|
## [0.1.3] - 2021-03-17
|
18
31
|
|
19
32
|
### Changed
|
@@ -21,7 +34,7 @@ These are the latest changes on the project's `master` branch that have not yet
|
|
21
34
|
- Reference changelog file in the gemspec instead of the general releases Github tab
|
22
35
|
|
23
36
|
### Removed
|
24
|
-
- Remove bulk from release: The previous gem releases contained files like the content of the `bin` folder or the Gemfile used for testing. Since this is not useful for gem users, adjust the gemspec file accordingly.
|
37
|
+
- Remove bulk from release: The previous gem releases contained files like the content of the `bin` folder or the Gemfile used for testing. Since this is not useful for gem users, adjust the gemspec file accordingly.
|
25
38
|
|
26
39
|
## [0.1.2] - 2021-02-04
|
27
40
|
|
data/README.md
CHANGED
@@ -1,5 +1,9 @@
|
|
1
1
|
# RailsCursorPagination
|
2
2
|
|
3
|
+
[![Gem Version](https://badge.fury.io/rb/rails_cursor_pagination.svg)](https://badge.fury.io/rb/rails_cursor_pagination)
|
4
|
+
[![License](http://img.shields.io/badge/license-MIT-brightgreen.svg)](https://tldrlegal.com/license/mit-license)
|
5
|
+
[![Tests](https://github.com/xing/rails_cursor_pagination/actions/workflows/test.yml/badge.svg?branch=master)](https://github.com/xing/rails_cursor_pagination/actions/workflows/test.yml?query=branch%3Amaster)
|
6
|
+
|
3
7
|
This library allows to paginate through an `ActiveRecord` relation using cursor pagination.
|
4
8
|
It also supports ordering by any column on the relation in either ascending or descending order.
|
5
9
|
|
@@ -159,9 +163,44 @@ Of course, this can both be combined with `first`, `last`, `before`, and `after`
|
|
159
163
|
|
160
164
|
**Important:**
|
161
165
|
If your app regularly orders by another column, you might want to add a database index for this.
|
162
|
-
Say that your order column is `author` then
|
166
|
+
Say that your order column is `author` then you'll want to add a compound index on `(author, id)`.
|
167
|
+
If your table is called `posts` you can use a query like this in MySQL or Postgres:
|
168
|
+
```sql
|
169
|
+
CREATE INDEX index_posts_on_author_and_id ON posts (author, id);
|
170
|
+
```
|
171
|
+
Or you can just do it via an `ActiveRecord::Migration`:
|
172
|
+
```ruby
|
173
|
+
class AddAuthorAndIdIndexToPosts < ActiveRecord::Migration
|
174
|
+
def change
|
175
|
+
add_index :posts, %i[author id]
|
176
|
+
end
|
177
|
+
end
|
178
|
+
```
|
163
179
|
|
164
180
|
Please take a look at the _"How does it work?"_ to find out more why this is necessary.
|
181
|
+
|
182
|
+
#### Order by more complex logic
|
183
|
+
|
184
|
+
Sometimes you might not only want to oder by a column ascending or descending, but need more complex logic.
|
185
|
+
Imagine you would also store the post's `category` on the `posts` table (as a plain string for simplicity's sake).
|
186
|
+
And the category could be `pinned`, `announcement`, or `general`.
|
187
|
+
Then you might want to show all `pinned` posts first, followed by the `announcement` ones and lastly show the `general` posts.
|
188
|
+
|
189
|
+
In MySQL you could e.g. use a `FIELD(category, 'pinned', 'announcement', 'general')` query in the `ORDER BY` clause to achieve this.
|
190
|
+
However, you cannot add an index to such a statement.
|
191
|
+
And therefore, the performance of this is – especially when using cursor pagination where we not only have an `ORDER BY` clause but also need it twice in the `WHERE` clauses – is pretty dismal.
|
192
|
+
|
193
|
+
For this reason, the gem currently only supports ordering by natural columns of the relation.
|
194
|
+
You **cannot** pass a generic SQL query to the `order_by` parameter.
|
195
|
+
|
196
|
+
Implementing support for arbitrary SQL queries would also be fairly complex to handle in this gem.
|
197
|
+
We would have to ensure that SQL injection attacks aren't possible by passing malicious code to the `oder_by` parameter.
|
198
|
+
And we would need to return the data produced by the statement so that it can be encoded in the cursor.
|
199
|
+
This is, for now, out of scope of the functionality of this gem.
|
200
|
+
|
201
|
+
What is recommended if you _do_ need to order by more complex logic is to have a separate column that you only use for ordering.
|
202
|
+
You can use `ActiveRecord` hooks to automatically update this column whenever you change your data.
|
203
|
+
Or, for example in MySQL, you can also use a [generated column](https://dev.mysql.com/doc/refman/5.7/en/create-table-generated-columns.html) that is automatically being updated by the database based on some stored logic.
|
165
204
|
|
166
205
|
### Configuration options
|
167
206
|
|
@@ -310,11 +349,11 @@ This will issue the following SQL query:
|
|
310
349
|
```sql
|
311
350
|
SELECT *
|
312
351
|
FROM "posts"
|
313
|
-
ORDER BY
|
352
|
+
ORDER BY "posts"."author" ASC, "posts"."id" ASC
|
314
353
|
LIMIT 2
|
315
354
|
```
|
316
355
|
|
317
|
-
As you can see, this will now order by
|
356
|
+
As you can see, this will now order by the author first, and if two records have the same author it will order them by ID.
|
318
357
|
Ordering only the author is not enough since we cannot know if the custom column only has unique values.
|
319
358
|
And we need to guarantee the correct order of ambiguous records independent of the direction of ordering.
|
320
359
|
This unique order is the basis of being able to paginate forward and backward repeatedly and getting the correct records.
|
@@ -340,17 +379,19 @@ We get this SQL query:
|
|
340
379
|
```sql
|
341
380
|
SELECT *
|
342
381
|
FROM "posts"
|
343
|
-
WHERE
|
344
|
-
ORDER BY
|
382
|
+
WHERE (author > 'Jane' OR (author = 'Jane') AND ("posts"."id" > 4))
|
383
|
+
ORDER BY "posts"."author" ASC, "posts"."id" ASC
|
345
384
|
LIMIT 2
|
346
385
|
```
|
347
386
|
|
348
|
-
You can see how the cursor is being
|
387
|
+
You can see how the cursor is being used by the WHERE clause to uniquely identify the row and properly filter based on this.
|
388
|
+
We only want to get records that either have a name that is alphabetically _after_ `"Jane"` or another `"Jane"` record with an ID that is higher than `4`.
|
349
389
|
We will get the records #5 and #2 as response.
|
350
390
|
|
351
|
-
|
391
|
+
When using a custom `order_by`, this affects both filtering as well as ordering.
|
352
392
|
Therefore, it is recommended to add an index for columns that are frequently used for ordering.
|
353
|
-
In our test case we would want to add
|
393
|
+
In our test case we would want to add a compound index for the `(author, id)` column combination.
|
394
|
+
Databases like MySQL and Postgres are able to then use the leftmost part of the index, in our case `author`, by its own _or_ can use it combined with the `id` index.
|
354
395
|
|
355
396
|
## Development
|
356
397
|
|
@@ -96,11 +96,11 @@
|
|
96
96
|
#
|
97
97
|
# SELECT *
|
98
98
|
# FROM "posts"
|
99
|
-
# ORDER BY
|
99
|
+
# ORDER BY "posts"."author" ASC, "posts"."id" ASC
|
100
100
|
# LIMIT 2
|
101
101
|
#
|
102
|
-
# As you can see, this will now order by
|
103
|
-
#
|
102
|
+
# As you can see, this will now order by the author first, and if two records
|
103
|
+
# have the same author it will order them by ID. Ordering only the author is not
|
104
104
|
# enough since we cannot know if the custom column only has unique values.
|
105
105
|
# And we need to guarantee the correct order of ambiguous records independent
|
106
106
|
# of the direction of ordering. This unique order is the basis of being able
|
@@ -128,18 +128,22 @@
|
|
128
128
|
#
|
129
129
|
# SELECT *
|
130
130
|
# FROM "posts"
|
131
|
-
# WHERE
|
132
|
-
# ORDER BY
|
131
|
+
# WHERE (author > 'Jane' OR (author = 'Jane') AND ("posts"."id" > 4))
|
132
|
+
# ORDER BY "posts"."author" ASC, "posts"."id" ASC
|
133
133
|
# LIMIT 2
|
134
134
|
#
|
135
|
-
# You can see how the cursor is being
|
136
|
-
#
|
137
|
-
#
|
138
|
-
#
|
139
|
-
#
|
140
|
-
#
|
141
|
-
#
|
142
|
-
#
|
135
|
+
# You can see how the cursor is being used by the WHERE clause to uniquely
|
136
|
+
# identify the row and properly filter based on this. We only want to get
|
137
|
+
# records that either have a name that is alphabetically after `"Jane"` or
|
138
|
+
# another `"Jane"` record with an ID that is higher than `4`. We will get the
|
139
|
+
# records #5 and #2 as response.
|
140
|
+
#
|
141
|
+
# When using a custom `order_by`, this affects both filtering as well as
|
142
|
+
# ordering. Therefore, it is recommended to add an index for columns that are
|
143
|
+
# frequently used for ordering. In our test case we would want to add a compound
|
144
|
+
# index for the `(author, id)` column combination. Databases like MySQL and
|
145
|
+
# Postgres are able to then use the leftmost part of the index, in our case
|
146
|
+
# `author`, by its own _or_ can use it combined with the `id` index.
|
143
147
|
#
|
144
148
|
module RailsCursorPagination
|
145
149
|
class Error < StandardError; end
|
@@ -34,8 +34,12 @@ module RailsCursorPagination
|
|
34
34
|
# Cursor to paginate upto (excluding). Can be combined with `last`.
|
35
35
|
# @param order_by [Symbol, String, nil]
|
36
36
|
# Column to order by. If none is provided, will default to ID column.
|
37
|
-
# NOTE: this will cause
|
38
|
-
# to add
|
37
|
+
# NOTE: this will cause the query to filter on both the given column as
|
38
|
+
# well as the ID column. So you might want to add a compound index to your
|
39
|
+
# database similar to:
|
40
|
+
# ```sql
|
41
|
+
# CREATE INDEX <index_name> ON <table_name> (<order_by_field>, id)
|
42
|
+
# ```
|
39
43
|
# @param order [Symbol, nil]
|
40
44
|
# Ordering to apply, either `:asc` or `:desc`. Defaults to `:asc`.
|
41
45
|
#
|
@@ -143,11 +147,13 @@ module RailsCursorPagination
|
|
143
147
|
#
|
144
148
|
# @return [Array<Hash>] List of hashes, each with a `cursor` and `data`
|
145
149
|
def page
|
146
|
-
|
147
|
-
|
148
|
-
|
149
|
-
|
150
|
-
|
150
|
+
memoize :page do
|
151
|
+
records.map do |item|
|
152
|
+
{
|
153
|
+
cursor: cursor_for_record(item),
|
154
|
+
data: item
|
155
|
+
}
|
156
|
+
end
|
151
157
|
end
|
152
158
|
end
|
153
159
|
|
@@ -155,7 +161,7 @@ module RailsCursorPagination
|
|
155
161
|
#
|
156
162
|
# @return [Integer]
|
157
163
|
def total
|
158
|
-
memoize(:total) { @relation.size }
|
164
|
+
memoize(:total) { @relation.reorder('').size }
|
159
165
|
end
|
160
166
|
|
161
167
|
# Check if the pagination direction is forward
|
@@ -182,11 +188,12 @@ module RailsCursorPagination
|
|
182
188
|
# When paginating forward, we can only have a previous page if we were
|
183
189
|
# provided with a cursor and there were records discarded after applying
|
184
190
|
# this filter. These records would have to be on previous pages.
|
185
|
-
@cursor.present? &&
|
191
|
+
@cursor.present? &&
|
192
|
+
filtered_and_sorted_relation.reorder('').size < total
|
186
193
|
else
|
187
194
|
# When paginating backwards, if we managed to load one more record than
|
188
195
|
# requested, this record will be available on the previous page.
|
189
|
-
@page_size
|
196
|
+
records_plus_one.size > @page_size
|
190
197
|
end
|
191
198
|
end
|
192
199
|
|
@@ -197,12 +204,12 @@ module RailsCursorPagination
|
|
197
204
|
if paginate_forward?
|
198
205
|
# When paginating forward, if we managed to load one more record than
|
199
206
|
# requested, this record will be available on the next page.
|
200
|
-
@page_size
|
207
|
+
records_plus_one.size > @page_size
|
201
208
|
else
|
202
209
|
# When paginating backward, if applying our cursor reduced the number
|
203
210
|
# records returned, we know that the missing records will be on
|
204
211
|
# subsequent pages.
|
205
|
-
filtered_and_sorted_relation.size < total
|
212
|
+
filtered_and_sorted_relation.reorder('').size < total
|
206
213
|
end
|
207
214
|
end
|
208
215
|
|
@@ -210,7 +217,7 @@ module RailsCursorPagination
|
|
210
217
|
#
|
211
218
|
# @return [Array<ActiveRecord>]
|
212
219
|
def records
|
213
|
-
records =
|
220
|
+
records = records_plus_one.first(@page_size)
|
214
221
|
|
215
222
|
paginate_forward? ? records : records.reverse
|
216
223
|
end
|
@@ -218,11 +225,13 @@ module RailsCursorPagination
|
|
218
225
|
# Apply limit to filtered and sorted relation that contains one item more
|
219
226
|
# than the user-requested page size. This is useful for determining if there
|
220
227
|
# is an additional page available without having to do a separate DB query.
|
228
|
+
# Then, fetch the records from the database to prevent multiple queries to
|
229
|
+
# load the records and count them.
|
221
230
|
#
|
222
231
|
# @return [ActiveRecord::Relation]
|
223
|
-
def
|
224
|
-
memoize :
|
225
|
-
filtered_and_sorted_relation.limit(@page_size + 1)
|
232
|
+
def records_plus_one
|
233
|
+
memoize :records_plus_one do
|
234
|
+
filtered_and_sorted_relation.limit(@page_size + 1).load
|
226
235
|
end
|
227
236
|
end
|
228
237
|
|
@@ -374,38 +383,6 @@ module RailsCursorPagination
|
|
374
383
|
decoded_cursor.first
|
375
384
|
end
|
376
385
|
|
377
|
-
# The SQL identifier of the column we need to consider for both ordering and
|
378
|
-
# filtering.
|
379
|
-
#
|
380
|
-
# In case we have a custom field order, this is a concatenation
|
381
|
-
# of the custom order field and the ID column joined by a dash. This is to
|
382
|
-
# ensure uniqueness of records even if they might have duplicates in the
|
383
|
-
# custom order field. If we don't have a custom order, it just returns a
|
384
|
-
# reference to the table's ID column.
|
385
|
-
#
|
386
|
-
# This uses the fully qualified and escaped reference to the ID column to
|
387
|
-
# prevent ambiguity in case of a query that uses JOINs and therefore might
|
388
|
-
# have multiple ID columns.
|
389
|
-
#
|
390
|
-
# @return [String]
|
391
|
-
def sql_column
|
392
|
-
memoize :sql_column do
|
393
|
-
escaped_table_name = @relation.quoted_table_name
|
394
|
-
escaped_id_column = @relation.connection.quote_column_name(:id)
|
395
|
-
|
396
|
-
id_column = "#{escaped_table_name}.#{escaped_id_column}"
|
397
|
-
|
398
|
-
sql =
|
399
|
-
if custom_order_field?
|
400
|
-
"CONCAT(#{@order_field}, '-', #{id_column})"
|
401
|
-
else
|
402
|
-
id_column
|
403
|
-
end
|
404
|
-
|
405
|
-
Arel.sql(sql)
|
406
|
-
end
|
407
|
-
end
|
408
|
-
|
409
386
|
# Ensure that the relation has the ID column and any potential `order_by`
|
410
387
|
# column selected. These are required to generate the record's cursor and
|
411
388
|
# therefore it's crucial that they are part of the selected fields.
|
@@ -432,19 +409,60 @@ module RailsCursorPagination
|
|
432
409
|
#
|
433
410
|
# @return [ActiveRecord::Relation]
|
434
411
|
def sorted_relation
|
412
|
+
unless custom_order_field?
|
413
|
+
return relation_with_cursor_fields.reorder id: pagination_sorting.upcase
|
414
|
+
end
|
415
|
+
|
435
416
|
relation_with_cursor_fields
|
436
|
-
.reorder(
|
417
|
+
.reorder(@order_field => pagination_sorting.upcase,
|
418
|
+
id: pagination_sorting.upcase)
|
419
|
+
end
|
420
|
+
|
421
|
+
# Return a properly escaped reference to the ID column prefixed with the
|
422
|
+
# table name. This prefixing is important in case of another model having
|
423
|
+
# been joined to the passed relation.
|
424
|
+
#
|
425
|
+
# @return [String (frozen)]
|
426
|
+
def id_column
|
427
|
+
escaped_table_name = @relation.quoted_table_name
|
428
|
+
escaped_id_column = @relation.connection.quote_column_name(:id)
|
429
|
+
|
430
|
+
"#{escaped_table_name}.#{escaped_id_column}".freeze
|
437
431
|
end
|
438
432
|
|
439
433
|
# Applies the filtering based on the provided cursor and order column to the
|
440
434
|
# sorted relation.
|
441
435
|
#
|
436
|
+
# In case a custom `order_by` field is provided, we have to filter based on
|
437
|
+
# this field and the ID column to ensure reproducible results.
|
438
|
+
#
|
439
|
+
# To better understand this, let's consider our example with the `posts`
|
440
|
+
# table. Say that we're paginating forward and add `order_by: :author` to
|
441
|
+
# the call, and if the cursor that is passed encodes `['Jane', 4]`. In this
|
442
|
+
# case we will have to select all posts that either have an author whose
|
443
|
+
# name is alphanumerically greater than 'Jane', or if the author is 'Jane'
|
444
|
+
# we have to ensure that the post's ID is greater than `4`.
|
445
|
+
#
|
446
|
+
# So our SQL WHERE clause needs to be something like:
|
447
|
+
# WHERE author > 'Jane' OR author = 'Jane' AND id > 4
|
448
|
+
#
|
442
449
|
# @return [ActiveRecord::Relation]
|
443
450
|
def filtered_and_sorted_relation
|
444
451
|
memoize :filtered_and_sorted_relation do
|
445
452
|
next sorted_relation if @cursor.blank?
|
446
453
|
|
447
|
-
|
454
|
+
unless custom_order_field?
|
455
|
+
next sorted_relation.where "#{id_column} #{filter_operator} ?",
|
456
|
+
decoded_cursor_id
|
457
|
+
end
|
458
|
+
|
459
|
+
sorted_relation
|
460
|
+
.where("#{@order_field} #{filter_operator} ?", decoded_cursor_field)
|
461
|
+
.or(
|
462
|
+
sorted_relation
|
463
|
+
.where("#{@order_field} = ?", decoded_cursor_field)
|
464
|
+
.where("#{id_column} #{filter_operator} ?", decoded_cursor_id)
|
465
|
+
)
|
448
466
|
end
|
449
467
|
end
|
450
468
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: rails_cursor_pagination
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Nicolas Fricke
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-04-19 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activerecord
|