rails_cursor_pagination 0.1.3 → 0.2.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: ff75c17ca149149eba59773520bc6690c7f0c5a86066d3783f68314e03a58e8e
4
- data.tar.gz: bb0a908d6a7c204da69851ac3bb83dad64d829dc9f0ac955d33c5164231fb0e7
3
+ metadata.gz: 92ff9fc0d88af01bbcee5b6bc749790494b476fe425492f799a1ebd0e4a35cf7
4
+ data.tar.gz: 421e730132cc775c717e10bfb05309a98419d6096166f24db631390cfb4792cb
5
5
  SHA512:
6
- metadata.gz: 5bfc9913cb0232569b0ca322e0b50dbf6a591f79ef7c97baf1dd3fd1850e5875163fa9468d91532a576b4e3bd36126a39ec835f531b6a4d29b308acaebcbaf2b
7
- data.tar.gz: 4f4624b26ce65bd8a70f18602931b5d5a2209ce9b912babed5bba534bfffeef06da0fbd287461a4aa5cc58bc18e6217c8ba7232e6fc71ba4ba3f369125514fcb
6
+ metadata.gz: 653cbdf05a63e16adab8f991dcf9fa90b0799424174c8b8838795f2f8cccdb89cd823e7a5708e485a8113e2af3ea6362c8a2696bfdac783ac30b8108795df841
7
+ data.tar.gz: 6307cfb673d4d18ed4dff27efdc5de2f042cfeb09332d23cb7813ff7bf79ed761b219b7486330ff924dcbe624325b41a75a71955312dcd30992a84299ba5193c
data/CHANGELOG.md CHANGED
@@ -14,6 +14,19 @@ These are the latest changes on the project's `master` branch that have not yet
14
14
  Follow the same format as previous releases by categorizing your feature into "Added", "Changed", "Deprecated", "Removed", "Fixed", or "Security".
15
15
  --->
16
16
 
17
+ ## [0.2.0] - 2021-04-19
18
+
19
+ ### Changed
20
+ - **Breaking change:** The way records are retrieved from a given cursor has been changed to no longer use `CONCAT` but instead simply use a compound `WHERE` clause in case of a custom order and having both the custom field as well as the `id` field in the `ORDER BY` query. This is a breaking change since it now changes the internal order of how records with the same value of the `order_by` field are returned.
21
+ - Remove `ORDER BY` clause from `COUNT` queries
22
+
23
+ ### Fixed
24
+ - Only trigger one SQL query to load the records from the database and use it to determine if there was a previous / is a next page
25
+ - Memoize the `Paginator#page` method which is invoked multiple times to prevent it from mapping over the `records` again and again and rebuilding all cursors
26
+
27
+ ### Added
28
+ - Description about `order_by` on arbitrary SQL to README.md
29
+
17
30
  ## [0.1.3] - 2021-03-17
18
31
 
19
32
  ### Changed
@@ -21,7 +34,7 @@ These are the latest changes on the project's `master` branch that have not yet
21
34
  - Reference changelog file in the gemspec instead of the general releases Github tab
22
35
 
23
36
  ### Removed
24
- - Remove bulk from release: The previous gem releases contained files like the content of the `bin` folder or the Gemfile used for testing. Since this is not useful for gem users, adjust the gemspec file accordingly.
37
+ - Remove bulk from release: The previous gem releases contained files like the content of the `bin` folder or the Gemfile used for testing. Since this is not useful for gem users, adjust the gemspec file accordingly.
25
38
 
26
39
  ## [0.1.2] - 2021-02-04
27
40
 
data/README.md CHANGED
@@ -1,5 +1,9 @@
1
1
  # RailsCursorPagination
2
2
 
3
+ [![Gem Version](https://badge.fury.io/rb/rails_cursor_pagination.svg)](https://badge.fury.io/rb/rails_cursor_pagination)
4
+ [![License](http://img.shields.io/badge/license-MIT-brightgreen.svg)](https://tldrlegal.com/license/mit-license)
5
+ [![Tests](https://github.com/xing/rails_cursor_pagination/actions/workflows/test.yml/badge.svg?branch=master)](https://github.com/xing/rails_cursor_pagination/actions/workflows/test.yml?query=branch%3Amaster)
6
+
3
7
  This library allows to paginate through an `ActiveRecord` relation using cursor pagination.
4
8
  It also supports ordering by any column on the relation in either ascending or descending order.
5
9
 
@@ -159,9 +163,44 @@ Of course, this can both be combined with `first`, `last`, `before`, and `after`
159
163
 
160
164
  **Important:**
161
165
  If your app regularly orders by another column, you might want to add a database index for this.
162
- Say that your order column is `author` then your index should be on `CONCAT(author, '-', id)`.
166
+ Say that your order column is `author` then you'll want to add a compound index on `(author, id)`.
167
+ If your table is called `posts` you can use a query like this in MySQL or Postgres:
168
+ ```sql
169
+ CREATE INDEX index_posts_on_author_and_id ON posts (author, id);
170
+ ```
171
+ Or you can just do it via an `ActiveRecord::Migration`:
172
+ ```ruby
173
+ class AddAuthorAndIdIndexToPosts < ActiveRecord::Migration
174
+ def change
175
+ add_index :posts, %i[author id]
176
+ end
177
+ end
178
+ ```
163
179
 
164
180
  Please take a look at the _"How does it work?"_ to find out more why this is necessary.
181
+
182
+ #### Order by more complex logic
183
+
184
+ Sometimes you might not only want to oder by a column ascending or descending, but need more complex logic.
185
+ Imagine you would also store the post's `category` on the `posts` table (as a plain string for simplicity's sake).
186
+ And the category could be `pinned`, `announcement`, or `general`.
187
+ Then you might want to show all `pinned` posts first, followed by the `announcement` ones and lastly show the `general` posts.
188
+
189
+ In MySQL you could e.g. use a `FIELD(category, 'pinned', 'announcement', 'general')` query in the `ORDER BY` clause to achieve this.
190
+ However, you cannot add an index to such a statement.
191
+ And therefore, the performance of this is – especially when using cursor pagination where we not only have an `ORDER BY` clause but also need it twice in the `WHERE` clauses – is pretty dismal.
192
+
193
+ For this reason, the gem currently only supports ordering by natural columns of the relation.
194
+ You **cannot** pass a generic SQL query to the `order_by` parameter.
195
+
196
+ Implementing support for arbitrary SQL queries would also be fairly complex to handle in this gem.
197
+ We would have to ensure that SQL injection attacks aren't possible by passing malicious code to the `oder_by` parameter.
198
+ And we would need to return the data produced by the statement so that it can be encoded in the cursor.
199
+ This is, for now, out of scope of the functionality of this gem.
200
+
201
+ What is recommended if you _do_ need to order by more complex logic is to have a separate column that you only use for ordering.
202
+ You can use `ActiveRecord` hooks to automatically update this column whenever you change your data.
203
+ Or, for example in MySQL, you can also use a [generated column](https://dev.mysql.com/doc/refman/5.7/en/create-table-generated-columns.html) that is automatically being updated by the database based on some stored logic.
165
204
 
166
205
  ### Configuration options
167
206
 
@@ -310,11 +349,11 @@ This will issue the following SQL query:
310
349
  ```sql
311
350
  SELECT *
312
351
  FROM "posts"
313
- ORDER BY CONCAT(author, '-', "posts"."id") ASC
352
+ ORDER BY "posts"."author" ASC, "posts"."id" ASC
314
353
  LIMIT 2
315
354
  ```
316
355
 
317
- As you can see, this will now order by a concatenation of the requested column, a dash `-`, and the ID column.
356
+ As you can see, this will now order by the author first, and if two records have the same author it will order them by ID.
318
357
  Ordering only the author is not enough since we cannot know if the custom column only has unique values.
319
358
  And we need to guarantee the correct order of ambiguous records independent of the direction of ordering.
320
359
  This unique order is the basis of being able to paginate forward and backward repeatedly and getting the correct records.
@@ -340,17 +379,19 @@ We get this SQL query:
340
379
  ```sql
341
380
  SELECT *
342
381
  FROM "posts"
343
- WHERE CONCAT(author, '-', "posts"."id") > 'Jane-4'
344
- ORDER BY CONCAT(author, '-', "posts"."id") ASC
382
+ WHERE (author > 'Jane' OR (author = 'Jane') AND ("posts"."id" > 4))
383
+ ORDER BY "posts"."author" ASC, "posts"."id" ASC
345
384
  LIMIT 2
346
385
  ```
347
386
 
348
- You can see how the cursor is being translated into the WHERE clause to uniquely identify the row and properly filter based on this.
387
+ You can see how the cursor is being used by the WHERE clause to uniquely identify the row and properly filter based on this.
388
+ We only want to get records that either have a name that is alphabetically _after_ `"Jane"` or another `"Jane"` record with an ID that is higher than `4`.
349
389
  We will get the records #5 and #2 as response.
350
390
 
351
- As you can see, when using a custom `order_by`, the concatenation is used for both filtering and ordering.
391
+ When using a custom `order_by`, this affects both filtering as well as ordering.
352
392
  Therefore, it is recommended to add an index for columns that are frequently used for ordering.
353
- In our test case we would want to add an index for `CONCAT(author, '-', id)`.
393
+ In our test case we would want to add a compound index for the `(author, id)` column combination.
394
+ Databases like MySQL and Postgres are able to then use the leftmost part of the index, in our case `author`, by its own _or_ can use it combined with the `id` index.
354
395
 
355
396
  ## Development
356
397
 
@@ -96,11 +96,11 @@
96
96
  #
97
97
  # SELECT *
98
98
  # FROM "posts"
99
- # ORDER BY CONCAT(author, '-', "posts"."id") ASC
99
+ # ORDER BY "posts"."author" ASC, "posts"."id" ASC
100
100
  # LIMIT 2
101
101
  #
102
- # As you can see, this will now order by a concatenation of the requested
103
- # column, a dash `-`, and the ID column. Ordering only the author is not
102
+ # As you can see, this will now order by the author first, and if two records
103
+ # have the same author it will order them by ID. Ordering only the author is not
104
104
  # enough since we cannot know if the custom column only has unique values.
105
105
  # And we need to guarantee the correct order of ambiguous records independent
106
106
  # of the direction of ordering. This unique order is the basis of being able
@@ -128,18 +128,22 @@
128
128
  #
129
129
  # SELECT *
130
130
  # FROM "posts"
131
- # WHERE CONCAT(author, '-', "posts"."id") > 'Jane-4'
132
- # ORDER BY CONCAT(author, '-', "posts"."id") ASC
131
+ # WHERE (author > 'Jane' OR (author = 'Jane') AND ("posts"."id" > 4))
132
+ # ORDER BY "posts"."author" ASC, "posts"."id" ASC
133
133
  # LIMIT 2
134
134
  #
135
- # You can see how the cursor is being translated into the WHERE clause to
136
- # uniquely identify the row and properly filter based on this. We will get
137
- # the records #5 and #2 as response.
138
- #
139
- # As you can see, when using a custom `order_by`, the concatenation is used
140
- # for both filtering and ordering. Therefore, it is recommended to add an
141
- # index for columns that are frequently used for ordering. In our test case
142
- # we would want to add an index for `CONCAT(author, '-', id)`.
135
+ # You can see how the cursor is being used by the WHERE clause to uniquely
136
+ # identify the row and properly filter based on this. We only want to get
137
+ # records that either have a name that is alphabetically after `"Jane"` or
138
+ # another `"Jane"` record with an ID that is higher than `4`. We will get the
139
+ # records #5 and #2 as response.
140
+ #
141
+ # When using a custom `order_by`, this affects both filtering as well as
142
+ # ordering. Therefore, it is recommended to add an index for columns that are
143
+ # frequently used for ordering. In our test case we would want to add a compound
144
+ # index for the `(author, id)` column combination. Databases like MySQL and
145
+ # Postgres are able to then use the leftmost part of the index, in our case
146
+ # `author`, by its own _or_ can use it combined with the `id` index.
143
147
  #
144
148
  module RailsCursorPagination
145
149
  class Error < StandardError; end
@@ -34,8 +34,12 @@ module RailsCursorPagination
34
34
  # Cursor to paginate upto (excluding). Can be combined with `last`.
35
35
  # @param order_by [Symbol, String, nil]
36
36
  # Column to order by. If none is provided, will default to ID column.
37
- # NOTE: this will cause an SQL `CONCAT` query. Therefore, you might want
38
- # to add an index to your database: `CONCAT(<order_by_field>, '-', id)`
37
+ # NOTE: this will cause the query to filter on both the given column as
38
+ # well as the ID column. So you might want to add a compound index to your
39
+ # database similar to:
40
+ # ```sql
41
+ # CREATE INDEX <index_name> ON <table_name> (<order_by_field>, id)
42
+ # ```
39
43
  # @param order [Symbol, nil]
40
44
  # Ordering to apply, either `:asc` or `:desc`. Defaults to `:asc`.
41
45
  #
@@ -143,11 +147,13 @@ module RailsCursorPagination
143
147
  #
144
148
  # @return [Array<Hash>] List of hashes, each with a `cursor` and `data`
145
149
  def page
146
- records.map do |item|
147
- {
148
- cursor: cursor_for_record(item),
149
- data: item
150
- }
150
+ memoize :page do
151
+ records.map do |item|
152
+ {
153
+ cursor: cursor_for_record(item),
154
+ data: item
155
+ }
156
+ end
151
157
  end
152
158
  end
153
159
 
@@ -155,7 +161,7 @@ module RailsCursorPagination
155
161
  #
156
162
  # @return [Integer]
157
163
  def total
158
- memoize(:total) { @relation.size }
164
+ memoize(:total) { @relation.reorder('').size }
159
165
  end
160
166
 
161
167
  # Check if the pagination direction is forward
@@ -182,11 +188,12 @@ module RailsCursorPagination
182
188
  # When paginating forward, we can only have a previous page if we were
183
189
  # provided with a cursor and there were records discarded after applying
184
190
  # this filter. These records would have to be on previous pages.
185
- @cursor.present? && filtered_and_sorted_relation.size < total
191
+ @cursor.present? &&
192
+ filtered_and_sorted_relation.reorder('').size < total
186
193
  else
187
194
  # When paginating backwards, if we managed to load one more record than
188
195
  # requested, this record will be available on the previous page.
189
- @page_size < limited_relation_plus_one.size
196
+ records_plus_one.size > @page_size
190
197
  end
191
198
  end
192
199
 
@@ -197,12 +204,12 @@ module RailsCursorPagination
197
204
  if paginate_forward?
198
205
  # When paginating forward, if we managed to load one more record than
199
206
  # requested, this record will be available on the next page.
200
- @page_size < limited_relation_plus_one.size
207
+ records_plus_one.size > @page_size
201
208
  else
202
209
  # When paginating backward, if applying our cursor reduced the number
203
210
  # records returned, we know that the missing records will be on
204
211
  # subsequent pages.
205
- filtered_and_sorted_relation.size < total
212
+ filtered_and_sorted_relation.reorder('').size < total
206
213
  end
207
214
  end
208
215
 
@@ -210,7 +217,7 @@ module RailsCursorPagination
210
217
  #
211
218
  # @return [Array<ActiveRecord>]
212
219
  def records
213
- records = limited_relation_plus_one.first(@page_size)
220
+ records = records_plus_one.first(@page_size)
214
221
 
215
222
  paginate_forward? ? records : records.reverse
216
223
  end
@@ -218,11 +225,13 @@ module RailsCursorPagination
218
225
  # Apply limit to filtered and sorted relation that contains one item more
219
226
  # than the user-requested page size. This is useful for determining if there
220
227
  # is an additional page available without having to do a separate DB query.
228
+ # Then, fetch the records from the database to prevent multiple queries to
229
+ # load the records and count them.
221
230
  #
222
231
  # @return [ActiveRecord::Relation]
223
- def limited_relation_plus_one
224
- memoize :limited_relation_plus_one do
225
- filtered_and_sorted_relation.limit(@page_size + 1)
232
+ def records_plus_one
233
+ memoize :records_plus_one do
234
+ filtered_and_sorted_relation.limit(@page_size + 1).load
226
235
  end
227
236
  end
228
237
 
@@ -374,38 +383,6 @@ module RailsCursorPagination
374
383
  decoded_cursor.first
375
384
  end
376
385
 
377
- # The SQL identifier of the column we need to consider for both ordering and
378
- # filtering.
379
- #
380
- # In case we have a custom field order, this is a concatenation
381
- # of the custom order field and the ID column joined by a dash. This is to
382
- # ensure uniqueness of records even if they might have duplicates in the
383
- # custom order field. If we don't have a custom order, it just returns a
384
- # reference to the table's ID column.
385
- #
386
- # This uses the fully qualified and escaped reference to the ID column to
387
- # prevent ambiguity in case of a query that uses JOINs and therefore might
388
- # have multiple ID columns.
389
- #
390
- # @return [String]
391
- def sql_column
392
- memoize :sql_column do
393
- escaped_table_name = @relation.quoted_table_name
394
- escaped_id_column = @relation.connection.quote_column_name(:id)
395
-
396
- id_column = "#{escaped_table_name}.#{escaped_id_column}"
397
-
398
- sql =
399
- if custom_order_field?
400
- "CONCAT(#{@order_field}, '-', #{id_column})"
401
- else
402
- id_column
403
- end
404
-
405
- Arel.sql(sql)
406
- end
407
- end
408
-
409
386
  # Ensure that the relation has the ID column and any potential `order_by`
410
387
  # column selected. These are required to generate the record's cursor and
411
388
  # therefore it's crucial that they are part of the selected fields.
@@ -432,19 +409,60 @@ module RailsCursorPagination
432
409
  #
433
410
  # @return [ActiveRecord::Relation]
434
411
  def sorted_relation
412
+ unless custom_order_field?
413
+ return relation_with_cursor_fields.reorder id: pagination_sorting.upcase
414
+ end
415
+
435
416
  relation_with_cursor_fields
436
- .reorder(sql_column => pagination_sorting.upcase)
417
+ .reorder(@order_field => pagination_sorting.upcase,
418
+ id: pagination_sorting.upcase)
419
+ end
420
+
421
+ # Return a properly escaped reference to the ID column prefixed with the
422
+ # table name. This prefixing is important in case of another model having
423
+ # been joined to the passed relation.
424
+ #
425
+ # @return [String (frozen)]
426
+ def id_column
427
+ escaped_table_name = @relation.quoted_table_name
428
+ escaped_id_column = @relation.connection.quote_column_name(:id)
429
+
430
+ "#{escaped_table_name}.#{escaped_id_column}".freeze
437
431
  end
438
432
 
439
433
  # Applies the filtering based on the provided cursor and order column to the
440
434
  # sorted relation.
441
435
  #
436
+ # In case a custom `order_by` field is provided, we have to filter based on
437
+ # this field and the ID column to ensure reproducible results.
438
+ #
439
+ # To better understand this, let's consider our example with the `posts`
440
+ # table. Say that we're paginating forward and add `order_by: :author` to
441
+ # the call, and if the cursor that is passed encodes `['Jane', 4]`. In this
442
+ # case we will have to select all posts that either have an author whose
443
+ # name is alphanumerically greater than 'Jane', or if the author is 'Jane'
444
+ # we have to ensure that the post's ID is greater than `4`.
445
+ #
446
+ # So our SQL WHERE clause needs to be something like:
447
+ # WHERE author > 'Jane' OR author = 'Jane' AND id > 4
448
+ #
442
449
  # @return [ActiveRecord::Relation]
443
450
  def filtered_and_sorted_relation
444
451
  memoize :filtered_and_sorted_relation do
445
452
  next sorted_relation if @cursor.blank?
446
453
 
447
- sorted_relation.where "#{sql_column} #{filter_operator} ?", filter_value
454
+ unless custom_order_field?
455
+ next sorted_relation.where "#{id_column} #{filter_operator} ?",
456
+ decoded_cursor_id
457
+ end
458
+
459
+ sorted_relation
460
+ .where("#{@order_field} #{filter_operator} ?", decoded_cursor_field)
461
+ .or(
462
+ sorted_relation
463
+ .where("#{@order_field} = ?", decoded_cursor_field)
464
+ .where("#{id_column} #{filter_operator} ?", decoded_cursor_id)
465
+ )
448
466
  end
449
467
  end
450
468
 
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module RailsCursorPagination
4
- VERSION = '0.1.3'
4
+ VERSION = '0.2.0'
5
5
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: rails_cursor_pagination
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.3
4
+ version: 0.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Nicolas Fricke
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2021-03-17 00:00:00.000000000 Z
11
+ date: 2021-04-19 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: activerecord