activerecord_cursor_paginate 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/CHANGELOG.md +5 -0
- data/LICENSE.txt +21 -0
- data/README.md +374 -0
- data/lib/activerecord_cursor_paginate/config.rb +11 -0
- data/lib/activerecord_cursor_paginate/cursor.rb +73 -0
- data/lib/activerecord_cursor_paginate/extension.rb +18 -0
- data/lib/activerecord_cursor_paginate/page.rb +85 -0
- data/lib/activerecord_cursor_paginate/paginator.rb +214 -0
- data/lib/activerecord_cursor_paginate/version.rb +5 -0
- data/lib/activerecord_cursor_paginate.rb +32 -0
- metadata +70 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 6a96b7bf3c40469653bf87dcf68cc159b30c389ed734047ef165b12d6ff7ef24
|
4
|
+
data.tar.gz: 2dea9c8b3847b5a72d6a915e9051e8b61891940e9feaa855edc822d49a393750
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: f54a87c366b7cfd08e59530b29fab0e0d5736cc323c4bab81ba0c3873ae5faa1254a2a9c94563e7aa9751debdb8802f28454a5781ddccf6811fe912c6808afca
|
7
|
+
data.tar.gz: c5689804eb34fa24e4e38f80994c9e3f16f45ab2594fe1fe9482397d2260b7d6db3453168a61fcb5f7e614ad97d8dfb6d8a1a42b4e560d780cebda759fd9f26d
|
data/CHANGELOG.md
ADDED
data/LICENSE.txt
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Copyright (c) 2024 fatkodima
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
13
|
+
all copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
21
|
+
THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,374 @@
|
|
1
|
+
# ActiveRecordCursorPaginate
|
2
|
+
|
3
|
+
This library allows to paginate through an `ActiveRecord` relation using cursor pagination.
|
4
|
+
It also supports ordering by any column on the relation in either ascending or descending order.
|
5
|
+
|
6
|
+
Cursor pagination allows to paginate results and gracefully deal with deletions / additions on previous pages.
|
7
|
+
Where a regular limit / offset pagination would jump in results if a record on a previous page gets deleted or added while requesting the next page, cursor pagination just returns the records following the one identified in the request.
|
8
|
+
|
9
|
+
To learn more about cursor pagination, check out the _"How does it work?"_ section below.
|
10
|
+
|
11
|
+
[![Build Status](https://github.com/fatkodima/activerecord_cursor_paginate/actions/workflows/test.yml/badge.svg?branch=master)](https://github.com/fatkodima/activerecord_cursor_paginate/actions/workflows/test.yml)
|
12
|
+
|
13
|
+
## Requirements
|
14
|
+
|
15
|
+
- Ruby 2.7+
|
16
|
+
- Rails (ActiveRecord) 7.0+
|
17
|
+
|
18
|
+
If you need support for older ruby and rails, please open an issue.
|
19
|
+
|
20
|
+
## Installation
|
21
|
+
|
22
|
+
Add this line to your application's Gemfile:
|
23
|
+
|
24
|
+
```ruby
|
25
|
+
gem "activerecord_cursor_paginate"
|
26
|
+
```
|
27
|
+
|
28
|
+
And then run:
|
29
|
+
|
30
|
+
```sh
|
31
|
+
$ bundle install
|
32
|
+
```
|
33
|
+
|
34
|
+
## Usage
|
35
|
+
|
36
|
+
Let's assume we have a `Post` model of which we want to fetch some data and then paginate through it.
|
37
|
+
Therefore, we first apply our scopes, `where` clauses or other functionality as usual:
|
38
|
+
|
39
|
+
```ruby
|
40
|
+
posts = Post.where(author: "Jane")
|
41
|
+
```
|
42
|
+
|
43
|
+
And then we create our paginator to fetch the first response page:
|
44
|
+
|
45
|
+
```ruby
|
46
|
+
paginator = posts.cursor_paginate
|
47
|
+
page = paginator.fetch
|
48
|
+
page.records # => [#<Post:0x00007fd7071b2ea8 @id=1>, #<Post:0x00007fd7071bb738 @id=2>, ..., #<Post:0x00007fd707238260 @id=10>]
|
49
|
+
page.count # => 10
|
50
|
+
page.empty? # => false
|
51
|
+
page.cursors # => ["MQ==", "Mg==", ..., "MTA="]
|
52
|
+
| |
|
53
|
+
| |
|
54
|
+
page.previous_cursor # => "MQ==" |
|
55
|
+
page.next_cursor # => "MTA=" ------------------|
|
56
|
+
page.has_previous? # => false
|
57
|
+
page.has_next? # => true
|
58
|
+
```
|
59
|
+
|
60
|
+
Note that any ordering of the relation at this stage will be ignored by the gem.
|
61
|
+
Take a look at the next section _"Ordering"_ to see how you can have an order different than ascending IDs.
|
62
|
+
|
63
|
+
To then get the next result page, you simply need to pass the last cursor of the returned page item via:
|
64
|
+
|
65
|
+
```ruby
|
66
|
+
paginator = posts.cursor_paginate(after: "MTA=")
|
67
|
+
```
|
68
|
+
|
69
|
+
This will then fetch the next result page.
|
70
|
+
You can also just as easily paginate to previous pages by using `before` instead of `after` and using the first cursor of the current page.
|
71
|
+
|
72
|
+
```ruby
|
73
|
+
paginator = posts.cursor_paginate(before: "MQ==")
|
74
|
+
```
|
75
|
+
|
76
|
+
By default, this will always return up to 10 results. But you can also specify how many records should be returned via `limit` parameter.
|
77
|
+
|
78
|
+
```ruby
|
79
|
+
paginator = posts.cursor_paginate(after: "MTA=", limit: 2)
|
80
|
+
```
|
81
|
+
|
82
|
+
```ruby
|
83
|
+
paginator = posts.cursor_paginate(before: "MQ==", limit: 2)
|
84
|
+
```
|
85
|
+
|
86
|
+
You can also easily iterate over the whole relation:
|
87
|
+
|
88
|
+
```ruby
|
89
|
+
paginator = posts.cursor_paginate
|
90
|
+
|
91
|
+
# Will lazily iterate over the pages.
|
92
|
+
paginator.pages.each do |page|
|
93
|
+
# do something with the page
|
94
|
+
end
|
95
|
+
```
|
96
|
+
|
97
|
+
### Ordering
|
98
|
+
|
99
|
+
As said, this gem ignores any previous ordering added to the passed relation.
|
100
|
+
But you can still paginate through relations with an order different than by ascending IDs.
|
101
|
+
|
102
|
+
You can specify a different column and direction to order the results by via an `order` parameter.
|
103
|
+
|
104
|
+
```ruby
|
105
|
+
# Order records ascending by the `:author` field.
|
106
|
+
paginator = posts.cursor_paginate(order: :author)
|
107
|
+
|
108
|
+
# Order records descending by the `:author` field.
|
109
|
+
paginator = posts.cursor_paginate(order: { author: :desc })
|
110
|
+
|
111
|
+
# Order records ascending by the `:author` and `:title` fields.
|
112
|
+
paginator = posts.cursor_paginate(order: [:author, :title])
|
113
|
+
|
114
|
+
# Order records ascending by the `:author` and descending by the `:title` fields.
|
115
|
+
paginator = posts.cursor_paginate(order: { author: :asc, title: :desc })
|
116
|
+
```
|
117
|
+
|
118
|
+
**Important:**
|
119
|
+
If your app regularly orders by another column, you might want to add a database index for this.
|
120
|
+
Say that your order column is `author` then you'll want to add a compound index on `(author, id)`.
|
121
|
+
If your table is called `posts` you can use a query like this in MySQL or PostgreSQL:
|
122
|
+
|
123
|
+
```sql
|
124
|
+
CREATE INDEX index_posts_on_author_and_id ON posts (author, id)
|
125
|
+
```
|
126
|
+
|
127
|
+
Or you can just do it via an `ActiveRecord::Migration`:
|
128
|
+
|
129
|
+
```ruby
|
130
|
+
class AddAuthorAndIdIndexToPosts < ActiveRecord::Migration[7.1]
|
131
|
+
def change
|
132
|
+
add_index :posts, [:author, :id]
|
133
|
+
end
|
134
|
+
end
|
135
|
+
```
|
136
|
+
|
137
|
+
Take a look at the _"How does it work?"_ to find out more why this is necessary.
|
138
|
+
|
139
|
+
#### Ordering and `JOIN`s
|
140
|
+
|
141
|
+
To order by a column from the `JOIN`ed table, you need to explicitly specify and fully qualify the column name for the `:order` parameter:
|
142
|
+
|
143
|
+
```ruby
|
144
|
+
paginator = User.joins(:projects).cursor_paginate(order: [Arel.sql("projects.id"), :id])
|
145
|
+
page = paginator.fetch
|
146
|
+
```
|
147
|
+
|
148
|
+
**Note**: Make sure to wrap custom SQL expressions by `Arel.sql`.
|
149
|
+
|
150
|
+
#### Order by more complex logic
|
151
|
+
|
152
|
+
Sometimes you might not only want to oder by a column ascending or descending, but need more complex logic.
|
153
|
+
Imagine you would also store the post's `category` on the `posts` table (as a plain string for simplicity's sake).
|
154
|
+
And the category could be `pinned`, `announcement`, or `general`.
|
155
|
+
Then you might want to show all `pinned` posts first, followed by the `announcement` ones and lastly show the `general` posts.
|
156
|
+
|
157
|
+
In MySQL you could e.g. use a `FIELD(category, 'pinned', 'announcement', 'general')` query in the `ORDER BY` clause to achieve this.
|
158
|
+
However, you cannot add an index to such a statement. And therefore, the performance of this is pretty dismal.
|
159
|
+
|
160
|
+
The gem supports ordering by custom SQL expressions, but make sure the performance will not suffer.
|
161
|
+
|
162
|
+
What is recommended if you _do_ need to order by more complex logic is to have a separate column that you only use for ordering.
|
163
|
+
You can use `ActiveRecord` hooks to automatically update this column whenever you change your data.
|
164
|
+
Or, for example in MySQL, you can also use a [generated column](https://dev.mysql.com/doc/refman/5.7/en/create-table-generated-columns.html) that is automatically being updated by the database based on some stored logic.
|
165
|
+
|
166
|
+
For example, if you want paginate `users` by a lowercased `email`, you can use the following:
|
167
|
+
|
168
|
+
```ruby
|
169
|
+
paginator = User.cursor_paginate(order: Arel.sql("lower(email)"))
|
170
|
+
page = paginator.fetch
|
171
|
+
```
|
172
|
+
|
173
|
+
**Note**: Make sure to wrap custom SQL expressions by `Arel.sql`.
|
174
|
+
|
175
|
+
### Configuration
|
176
|
+
|
177
|
+
You can change the default page size to a value that better fits the needs of your application.
|
178
|
+
So if a user doesn't request a given `:limit` value, the default amount of records is being returned.
|
179
|
+
|
180
|
+
To change the default, simply add an initializer to your app that does the following:
|
181
|
+
|
182
|
+
```ruby
|
183
|
+
# config/initializers/activerecord_cursor_paginate.rb
|
184
|
+
ActiveRecordCursorPaginate.configure do |config|
|
185
|
+
config.default_page_size = 50
|
186
|
+
end
|
187
|
+
```
|
188
|
+
|
189
|
+
This would set the default page size to 50.
|
190
|
+
|
191
|
+
You can also set a global `max_page_size` to prevent a client from requesting too large pages.
|
192
|
+
|
193
|
+
```ruby
|
194
|
+
ActiveRecordCursorPaginate.configure do |config|
|
195
|
+
config.max_page_size = 100
|
196
|
+
end
|
197
|
+
```
|
198
|
+
|
199
|
+
## How does it work?
|
200
|
+
|
201
|
+
This library allows to paginate through a passed relation using a cursor
|
202
|
+
for before or after parameters. It also supports ordering by any column
|
203
|
+
on the relation in either ascending or descending order.
|
204
|
+
|
205
|
+
Cursor pagination allows to paginate results and gracefully deal with
|
206
|
+
deletions / additions on previous pages. Where a regular limit / offset
|
207
|
+
pagination would jump in results if a record on a previous page gets deleted
|
208
|
+
or added while requesting the next page, cursor pagination just returns the
|
209
|
+
records following the one identified in the request.
|
210
|
+
|
211
|
+
How this works is that it uses a "cursor", which is an encoded value that
|
212
|
+
uniquely identifies a given row for the requested order. Then, based on
|
213
|
+
this cursor, you can request the "n records AFTER the cursor"
|
214
|
+
(forward-pagination) or the "n records BEFORE the cursor" (backward-pagination).
|
215
|
+
|
216
|
+
As an example, assume we have a table called "posts" with this data:
|
217
|
+
|
218
|
+
| id | author |
|
219
|
+
|----|--------|
|
220
|
+
| 1 | Jane |
|
221
|
+
| 2 | John |
|
222
|
+
| 3 | John |
|
223
|
+
| 4 | Jane |
|
224
|
+
| 5 | Jane |
|
225
|
+
| 6 | John |
|
226
|
+
| 7 | John |
|
227
|
+
|
228
|
+
Now if we make a basic request without any `before`, `after`, custom `order` column,
|
229
|
+
this will just request the first page of this relation.
|
230
|
+
|
231
|
+
```ruby
|
232
|
+
paginator = relation.cursor_paginate
|
233
|
+
page = paginator.fetch
|
234
|
+
```
|
235
|
+
|
236
|
+
Assume that our default page size here is 2 and we would get a query like this:
|
237
|
+
|
238
|
+
```sql
|
239
|
+
SELECT *
|
240
|
+
FROM "posts"
|
241
|
+
ORDER BY "posts"."id" ASC
|
242
|
+
LIMIT 2
|
243
|
+
```
|
244
|
+
|
245
|
+
This will return the first page of results, containing post #1 and #2. Since
|
246
|
+
no custom order is defined, each item in the returned collection will have a
|
247
|
+
cursor that only encodes the record's ID.
|
248
|
+
|
249
|
+
If we want to now request the next page, we can pass in the cursor of record
|
250
|
+
#2 which would be `"Mg=="` (can get via `page.cursor`). So now we can request
|
251
|
+
the next page by calling:
|
252
|
+
|
253
|
+
```ruby
|
254
|
+
paginator = relation.cursor_paginate(limit: 2, after: "Mg==")
|
255
|
+
page = paginator.fetch
|
256
|
+
```
|
257
|
+
|
258
|
+
And this will decode the given cursor and issue a query like:
|
259
|
+
|
260
|
+
```sql
|
261
|
+
SELECT *
|
262
|
+
FROM "posts"
|
263
|
+
WHERE "posts"."id" > 2
|
264
|
+
ORDER BY "posts"."id" ASC
|
265
|
+
LIMIT 2
|
266
|
+
```
|
267
|
+
|
268
|
+
Which would return posts #3 and #4. If we now want to paginate back, we can
|
269
|
+
request the posts that came before the first post, whose cursor would be
|
270
|
+
`"Mw=="` (can get via `page.previous_cursor`):
|
271
|
+
|
272
|
+
```ruby
|
273
|
+
paginator = relation.cursor_paginate(limit: 2, before: "Mw==")
|
274
|
+
page = paginator.fetch
|
275
|
+
```
|
276
|
+
|
277
|
+
Since we now paginate backward, the resulting SQL query needs to be flipped
|
278
|
+
around to get the last two records that have an ID smaller than the given one:
|
279
|
+
|
280
|
+
```sql
|
281
|
+
SELECT *
|
282
|
+
FROM "posts"
|
283
|
+
WHERE "posts"."id" < 3
|
284
|
+
ORDER BY "posts"."id" DESC
|
285
|
+
LIMIT 2
|
286
|
+
```
|
287
|
+
|
288
|
+
This would return posts #2 and #1. Since we still requested them in
|
289
|
+
ascending order, the result will be reversed before it is returned.
|
290
|
+
|
291
|
+
Now, in case that the user wants to order by a column different than the ID,
|
292
|
+
we require this information in our cursor. Therefore, when requesting the
|
293
|
+
first page like this:
|
294
|
+
|
295
|
+
```ruby
|
296
|
+
paginator = relation.cursor_paginate(order: :author)
|
297
|
+
page = paginator.fetch
|
298
|
+
```
|
299
|
+
|
300
|
+
This will issue the following SQL query:
|
301
|
+
|
302
|
+
```sql
|
303
|
+
SELECT *
|
304
|
+
FROM "posts"
|
305
|
+
ORDER BY "posts"."author" ASC, "posts"."id" ASC
|
306
|
+
LIMIT 2
|
307
|
+
```
|
308
|
+
|
309
|
+
As you can see, this will now order by the author first, and if two records
|
310
|
+
have the same author it will order them by ID. Ordering only the author is not
|
311
|
+
enough since we cannot know if the custom column only has unique values.
|
312
|
+
And we need to guarantee the correct order of ambiguous records independent
|
313
|
+
of the direction of ordering. This unique order is the basis of being able
|
314
|
+
to paginate forward and backward repeatedly and getting the correct records.
|
315
|
+
|
316
|
+
The query will then return records #1 and #4. But the cursor for these
|
317
|
+
records will also be different to the previous query where we ordered by ID
|
318
|
+
only. It is important that the cursor encodes all the data we need to
|
319
|
+
uniquely identify a row and filter based upon it. Therefore, we need to
|
320
|
+
encode the same information as we used for the ordering in our SQL query.
|
321
|
+
Hence, the cursor for pagination with a custom column contains a tuple of
|
322
|
+
data, the first record being the custom order column followed by the
|
323
|
+
record's ID.
|
324
|
+
|
325
|
+
Therefore, the cursor of record #4 will encode `['Jane', 4]`, which yields
|
326
|
+
this cursor: `"WyJKYW5lIiw0XQ=="`.
|
327
|
+
|
328
|
+
If we now want to request the next page via:
|
329
|
+
|
330
|
+
```ruby
|
331
|
+
paginator = relation.cursor_paginate(order: :author, limit: 2, after: "WyJKYW5lIiw0XQ==")
|
332
|
+
page = paginator.fetch
|
333
|
+
```
|
334
|
+
|
335
|
+
We get this SQL query:
|
336
|
+
|
337
|
+
```sql
|
338
|
+
SELECT *
|
339
|
+
FROM "posts"
|
340
|
+
WHERE (author > 'Jane' OR (author = 'Jane') AND ("posts"."id" > 4))
|
341
|
+
ORDER BY "posts"."author" ASC, "posts"."id" ASC
|
342
|
+
LIMIT 2
|
343
|
+
```
|
344
|
+
|
345
|
+
You can see how the cursor is being used by the WHERE clause to uniquely
|
346
|
+
identify the row and properly filter based on this. We only want to get
|
347
|
+
records that either have a name that is alphabetically after `"Jane"` or
|
348
|
+
another `"Jane"` record with an ID that is higher than `4`. We will get the
|
349
|
+
records #5 and #2 as response.
|
350
|
+
|
351
|
+
When using a custom `order`, this affects both filtering as well as
|
352
|
+
ordering. Therefore, it is recommended to add an index for columns that are
|
353
|
+
frequently used for ordering. In our test case we would want to add a compound
|
354
|
+
index for the `(author, id)` column combination. Databases like MySQL and
|
355
|
+
PostgreSQL are able to then use the leftmost part of the index, in our case
|
356
|
+
`author`, by its own or can use it combined with the `id` index.
|
357
|
+
|
358
|
+
## Credits
|
359
|
+
|
360
|
+
Thanks to [rails_cursor_pagination gem](https://github.com/xing/rails_cursor_pagination) for the original ideas.
|
361
|
+
|
362
|
+
## Development
|
363
|
+
|
364
|
+
After checking out the repo, run `bundle install` to install dependencies. Then, run `bundle exec rake` to run the linter and tests. This project uses multiple Gemfiles to test against multiple versions of Active Record; you can run the tests against the specific version with `BUNDLE_GEMFILE=gemfiles/activerecord_71.gemfile bundle exec rake test`.
|
365
|
+
|
366
|
+
To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
|
367
|
+
|
368
|
+
## Contributing
|
369
|
+
|
370
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/fatkodima/activerecord_cursor_paginate.
|
371
|
+
|
372
|
+
## License
|
373
|
+
|
374
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
@@ -0,0 +1,73 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "base64"
|
4
|
+
require "json"
|
5
|
+
|
6
|
+
module ActiveRecordCursorPaginate
|
7
|
+
# @private
|
8
|
+
class Cursor
|
9
|
+
class << self
|
10
|
+
def from_record(record, columns:)
|
11
|
+
columns = columns.map { |column| column.to_s.split(".").last }
|
12
|
+
values = columns.map { |column| record[column] }
|
13
|
+
new(columns: columns, values: values)
|
14
|
+
end
|
15
|
+
|
16
|
+
def decode(cursor_string:, columns:)
|
17
|
+
decoded = JSON.parse(Base64.strict_decode64(cursor_string))
|
18
|
+
|
19
|
+
if (columns.size == 1 && decoded.is_a?(Array)) ||
|
20
|
+
(decoded.is_a?(Array) && decoded.size != columns.size)
|
21
|
+
raise InvalidCursorError,
|
22
|
+
"The given cursor `#{cursor_string}` was decoded as `#{decoded}` but could not be parsed"
|
23
|
+
end
|
24
|
+
|
25
|
+
decoded =
|
26
|
+
if decoded.is_a?(Array)
|
27
|
+
decoded.map { |value| deserialize_time_if_needed(value) }
|
28
|
+
else
|
29
|
+
deserialize_time_if_needed(decoded)
|
30
|
+
end
|
31
|
+
|
32
|
+
new(columns: columns, values: decoded)
|
33
|
+
rescue ArgumentError, JSON::ParserError # ArgumentError is raised by strict_decode64
|
34
|
+
raise InvalidCursorError, "The given cursor `#{cursor_string}` could not be decoded"
|
35
|
+
end
|
36
|
+
|
37
|
+
private
|
38
|
+
def deserialize_time_if_needed(value)
|
39
|
+
if value.is_a?(String) && value.start_with?(TIMESTAMP_PREFIX)
|
40
|
+
seconds_with_frac = value.delete_prefix(TIMESTAMP_PREFIX).to_r / (10**6)
|
41
|
+
Time.at(seconds_with_frac).utc
|
42
|
+
else
|
43
|
+
value
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
attr_reader :columns, :values
|
49
|
+
|
50
|
+
def initialize(columns:, values:)
|
51
|
+
@columns = Array(columns)
|
52
|
+
@values = Array(values)
|
53
|
+
|
54
|
+
raise ArgumentError, "Cursor values can not be nil" if @values.any?(nil)
|
55
|
+
raise ArgumentError, ":columns and :values have different sizes" if @columns.size != @values.size
|
56
|
+
end
|
57
|
+
|
58
|
+
def encode
|
59
|
+
serialized_values = values.map do |value|
|
60
|
+
if value.is_a?(Time)
|
61
|
+
TIMESTAMP_PREFIX + value.strftime("%s%6N")
|
62
|
+
else
|
63
|
+
value
|
64
|
+
end
|
65
|
+
end
|
66
|
+
unencoded_cursor = (serialized_values.size == 1 ? serialized_values.first : serialized_values)
|
67
|
+
Base64.strict_encode64(unencoded_cursor.to_json)
|
68
|
+
end
|
69
|
+
|
70
|
+
TIMESTAMP_PREFIX = "0aIX2_" # something random
|
71
|
+
private_constant :TIMESTAMP_PREFIX
|
72
|
+
end
|
73
|
+
end
|
@@ -0,0 +1,18 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module ActiveRecordCursorPaginate
|
4
|
+
module Extension
|
5
|
+
# Convenient method to use on ActiveRecord::Relation to get a paginator.
|
6
|
+
# @return [ActiveRecordCursorPaginate::Paginator]
|
7
|
+
#
|
8
|
+
# @example
|
9
|
+
# paginator = Post.all.cursor_paginate(limit: 2, after: "Mg==")
|
10
|
+
# page = paginator.fetch
|
11
|
+
#
|
12
|
+
def cursor_paginate(after: nil, before: nil, limit: nil, order: nil)
|
13
|
+
relation = (is_a?(ActiveRecord::Relation) ? self : all)
|
14
|
+
Paginator.new(relation, after: after, before: before, limit: limit, order: order)
|
15
|
+
end
|
16
|
+
alias cursor_pagination cursor_paginate
|
17
|
+
end
|
18
|
+
end
|
@@ -0,0 +1,85 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module ActiveRecordCursorPaginate
|
4
|
+
# Represents a batch of records retrieved via a single iteration of
|
5
|
+
# cursor-based pagination.
|
6
|
+
#
|
7
|
+
class Page
|
8
|
+
# Records this page contains.
|
9
|
+
# @return [ActiveRecord::Base]
|
10
|
+
#
|
11
|
+
attr_reader :records
|
12
|
+
|
13
|
+
def initialize(records, order_columns:, has_previous: false, has_next: false)
|
14
|
+
@records = records
|
15
|
+
@order_columns = order_columns
|
16
|
+
@has_previous = has_previous
|
17
|
+
@has_next = has_next
|
18
|
+
end
|
19
|
+
|
20
|
+
# Number of records in this page.
|
21
|
+
# @return [Integer]
|
22
|
+
#
|
23
|
+
def count
|
24
|
+
records.size
|
25
|
+
end
|
26
|
+
|
27
|
+
# Whether this page is empty.
|
28
|
+
# @return [Boolean]
|
29
|
+
#
|
30
|
+
def empty?
|
31
|
+
count == 0
|
32
|
+
end
|
33
|
+
|
34
|
+
# Returns the cursor, which can be used to retrieve the next page.
|
35
|
+
# @return [String]
|
36
|
+
#
|
37
|
+
def next_cursor
|
38
|
+
cursor_for_record(records.last)
|
39
|
+
end
|
40
|
+
alias cursor next_cursor
|
41
|
+
|
42
|
+
# Returns the cursor, which can be used to retrieve the previous page.
|
43
|
+
# @return [String]
|
44
|
+
#
|
45
|
+
def previous_cursor
|
46
|
+
cursor_for_record(records.first)
|
47
|
+
end
|
48
|
+
|
49
|
+
# Whether this page has a previous page.
|
50
|
+
# @return [Boolean]
|
51
|
+
#
|
52
|
+
def has_previous?
|
53
|
+
@has_previous
|
54
|
+
end
|
55
|
+
|
56
|
+
# Whether this page has a next page.
|
57
|
+
# @return [Boolean]
|
58
|
+
#
|
59
|
+
def has_next?
|
60
|
+
@has_next
|
61
|
+
end
|
62
|
+
|
63
|
+
# Returns cursor for a specific record.
|
64
|
+
#
|
65
|
+
# @param record [ActiveRecord::Base]
|
66
|
+
# @return [String]
|
67
|
+
#
|
68
|
+
def cursor_for(record)
|
69
|
+
cursor_for_record(record)
|
70
|
+
end
|
71
|
+
|
72
|
+
# Returns cursors for all the records on this page.
|
73
|
+
# @return [Array<String>]
|
74
|
+
#
|
75
|
+
def cursors
|
76
|
+
records.map { |record| cursor_for_record(record) }
|
77
|
+
end
|
78
|
+
|
79
|
+
private
|
80
|
+
def cursor_for_record(record)
|
81
|
+
cursor = Cursor.from_record(record, columns: @order_columns)
|
82
|
+
cursor.encode
|
83
|
+
end
|
84
|
+
end
|
85
|
+
end
|
@@ -0,0 +1,214 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module ActiveRecordCursorPaginate
|
4
|
+
# Use this Paginator class to effortlessly paginate through ActiveRecord
|
5
|
+
# relations using cursor pagination.
|
6
|
+
#
|
7
|
+
# @example Iterating one page at a time
|
8
|
+
# ActiveRecordCursorPaginate::Paginator
|
9
|
+
# .new(relation, order: :author, limit: 2, after: "WyJKYW5lIiw0XQ==")
|
10
|
+
# .fetch
|
11
|
+
#
|
12
|
+
# @example Iterating over the whole relation
|
13
|
+
# paginator = ActiveRecordCursorPaginate::Paginator
|
14
|
+
# .new(relation, order: :author, limit: 2, after: "WyJKYW5lIiw0XQ==")
|
15
|
+
#
|
16
|
+
# # Will lazily iterate over the pages.
|
17
|
+
# paginator.pages.each do |page|
|
18
|
+
# # do something with the page
|
19
|
+
# end
|
20
|
+
#
|
21
|
+
class Paginator
|
22
|
+
# Create a new instance of the `ActiveRecordCursorPaginate::Paginator`
|
23
|
+
#
|
24
|
+
# @param relation [ActiveRecord::Relation] Relation that will be paginated.
|
25
|
+
# @param before [String, nil] Cursor to paginate upto (excluding).
|
26
|
+
# @param after [String, nil] Cursor to paginate forward from.
|
27
|
+
# @param limit [Integer, nil] Number of records to return in pagination.
|
28
|
+
# @param order [Symbol, String, nil, Array<Symbol, String>, Hash]
|
29
|
+
# Column(s) to order by, optionally with directions (either `:asc` or `:desc`,
|
30
|
+
# defaults to `:asc`). If none is provided, will default to ID column.
|
31
|
+
# NOTE: this will cause the query to filter on both the given column as
|
32
|
+
# well as the ID column. So you might want to add a compound index to your
|
33
|
+
# database similar to:
|
34
|
+
# ```sql
|
35
|
+
# CREATE INDEX <index_name> ON <table_name> (<order_fields>..., id)
|
36
|
+
# ```
|
37
|
+
# @raise [ArgumentError] If any parameter is not valid
|
38
|
+
#
|
39
|
+
def initialize(relation, before: nil, after: nil, limit: nil, order: nil)
|
40
|
+
unless relation.is_a?(ActiveRecord::Relation)
|
41
|
+
raise ArgumentError, "relation is not an ActiveRecord::Relation"
|
42
|
+
end
|
43
|
+
|
44
|
+
if before.present? && after.present?
|
45
|
+
raise ArgumentError, "Only one of :before and :after can be provided"
|
46
|
+
end
|
47
|
+
|
48
|
+
@relation = relation
|
49
|
+
@primary_key = @relation.primary_key
|
50
|
+
@cursor = before || after
|
51
|
+
@is_forward_pagination = before.blank?
|
52
|
+
|
53
|
+
config = ActiveRecordCursorPaginate.config
|
54
|
+
@page_size = limit || config.default_page_size
|
55
|
+
@page_size = [@page_size, config.max_page_size].min if config.max_page_size
|
56
|
+
|
57
|
+
order = normalize_order(order)
|
58
|
+
@columns = order.keys
|
59
|
+
@directions = order.values
|
60
|
+
end
|
61
|
+
|
62
|
+
# Get the paginated result.
|
63
|
+
# @return [ActiveRecordCursorPaginate::Page]
|
64
|
+
#
|
65
|
+
# @note Calling this method advances the paginator.
|
66
|
+
#
|
67
|
+
def fetch
|
68
|
+
relation = @relation
|
69
|
+
|
70
|
+
# Non trivial columns (expressions or joined tables columns).
|
71
|
+
if @columns.any?(/\W/)
|
72
|
+
arel_columns = @columns.map.with_index do |column, i|
|
73
|
+
arel_column(column).as("cursor_column_#{i + 1}")
|
74
|
+
end
|
75
|
+
cursor_column_names = 1.upto(@columns.size).map { |i| "cursor_column_#{i}" }
|
76
|
+
|
77
|
+
relation =
|
78
|
+
if relation.select_values.empty?
|
79
|
+
relation.select(Arel.star, arel_columns)
|
80
|
+
else
|
81
|
+
relation.select(arel_columns)
|
82
|
+
end
|
83
|
+
else
|
84
|
+
cursor_column_names = @columns
|
85
|
+
end
|
86
|
+
|
87
|
+
pagination_directions = @directions.map { |direction| pagination_direction(direction) }
|
88
|
+
relation = relation.reorder(cursor_column_names.zip(pagination_directions).to_h)
|
89
|
+
|
90
|
+
if @cursor
|
91
|
+
decoded_cursor = Cursor.decode(cursor_string: @cursor, columns: @columns)
|
92
|
+
relation = apply_cursor(relation, decoded_cursor)
|
93
|
+
end
|
94
|
+
|
95
|
+
relation = relation.limit(@page_size + 1)
|
96
|
+
records_plus_one = relation.to_a
|
97
|
+
has_additional = records_plus_one.size > @page_size
|
98
|
+
|
99
|
+
records = records_plus_one.take(@page_size)
|
100
|
+
records.reverse! unless @is_forward_pagination
|
101
|
+
|
102
|
+
if @is_forward_pagination
|
103
|
+
has_next_page = has_additional
|
104
|
+
has_previous_page = @cursor.present?
|
105
|
+
else
|
106
|
+
has_next_page = @cursor.present?
|
107
|
+
has_previous_page = has_additional
|
108
|
+
end
|
109
|
+
|
110
|
+
page = Page.new(
|
111
|
+
records,
|
112
|
+
order_columns: cursor_column_names,
|
113
|
+
has_next: has_next_page,
|
114
|
+
has_previous: has_previous_page
|
115
|
+
)
|
116
|
+
|
117
|
+
advance_by_page(page) unless page.empty?
|
118
|
+
|
119
|
+
page
|
120
|
+
end
|
121
|
+
alias page fetch
|
122
|
+
|
123
|
+
# Returns an enumerator that can be used to iterate over the whole relation.
|
124
|
+
# @return [Enumerator]
|
125
|
+
#
|
126
|
+
def pages
|
127
|
+
Enumerator.new do |yielder|
|
128
|
+
loop do
|
129
|
+
page = fetch
|
130
|
+
break if page.empty?
|
131
|
+
|
132
|
+
yielder.yield(page)
|
133
|
+
end
|
134
|
+
end
|
135
|
+
end
|
136
|
+
|
137
|
+
private
|
138
|
+
def normalize_order(order)
|
139
|
+
order ||= {}
|
140
|
+
default_direction = :asc
|
141
|
+
|
142
|
+
result =
|
143
|
+
case order
|
144
|
+
when String, Symbol
|
145
|
+
{ order => default_direction }
|
146
|
+
when Hash
|
147
|
+
order
|
148
|
+
when Array
|
149
|
+
order.to_h { |column| [column, default_direction] }
|
150
|
+
else
|
151
|
+
raise ArgumentError, "Invalid order: #{order.inspect}"
|
152
|
+
end
|
153
|
+
|
154
|
+
result = result.with_indifferent_access
|
155
|
+
result.transform_values! { |direction| direction.downcase.to_sym }
|
156
|
+
Array(@primary_key).each { |column| result[column] ||= default_direction }
|
157
|
+
result
|
158
|
+
end
|
159
|
+
|
160
|
+
def apply_cursor(relation, cursor)
|
161
|
+
operators = @directions.map { |direction| pagination_operator(direction) }
|
162
|
+
cursor_positions = cursor.columns.zip(cursor.values, operators)
|
163
|
+
|
164
|
+
where_clause = nil
|
165
|
+
cursor_positions.reverse_each.with_index do |(column, value, operator), index|
|
166
|
+
where_clause =
|
167
|
+
if index == 0
|
168
|
+
arel_column(column).public_send(operator, value)
|
169
|
+
else
|
170
|
+
arel_column(column).public_send(operator, value).or(
|
171
|
+
arel_column(column).eq(value).and(where_clause)
|
172
|
+
)
|
173
|
+
end
|
174
|
+
end
|
175
|
+
|
176
|
+
relation.where(where_clause)
|
177
|
+
end
|
178
|
+
|
179
|
+
def arel_column(column)
|
180
|
+
if Arel.arel_node?(column)
|
181
|
+
column
|
182
|
+
elsif column.match?(/\A\w+\.\w+\z/)
|
183
|
+
Arel.sql(column)
|
184
|
+
else
|
185
|
+
@relation.arel_table[column]
|
186
|
+
end
|
187
|
+
end
|
188
|
+
|
189
|
+
def pagination_direction(direction)
|
190
|
+
if @is_forward_pagination
|
191
|
+
direction
|
192
|
+
else
|
193
|
+
direction == :asc ? :desc : :asc
|
194
|
+
end
|
195
|
+
end
|
196
|
+
|
197
|
+
def pagination_operator(direction)
|
198
|
+
if @is_forward_pagination
|
199
|
+
direction == :asc ? :gt : :lt
|
200
|
+
else
|
201
|
+
direction == :asc ? :lt : :gt
|
202
|
+
end
|
203
|
+
end
|
204
|
+
|
205
|
+
def advance_by_page(page)
|
206
|
+
@cursor =
|
207
|
+
if @is_forward_pagination
|
208
|
+
page.next_cursor
|
209
|
+
else
|
210
|
+
page.previous_cursor
|
211
|
+
end
|
212
|
+
end
|
213
|
+
end
|
214
|
+
end
|
@@ -0,0 +1,32 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "active_record"
|
4
|
+
|
5
|
+
require_relative "activerecord_cursor_paginate/version"
|
6
|
+
require_relative "activerecord_cursor_paginate/cursor"
|
7
|
+
require_relative "activerecord_cursor_paginate/page"
|
8
|
+
require_relative "activerecord_cursor_paginate/paginator"
|
9
|
+
require_relative "activerecord_cursor_paginate/extension"
|
10
|
+
require_relative "activerecord_cursor_paginate/config"
|
11
|
+
|
12
|
+
module ActiveRecordCursorPaginate
|
13
|
+
class Error < StandardError; end
|
14
|
+
|
15
|
+
# Error that gets raised if a cursor given as `before` or `after` cannot be
|
16
|
+
# properly parsed.
|
17
|
+
class InvalidCursorError < Error; end
|
18
|
+
|
19
|
+
class << self
|
20
|
+
def configure
|
21
|
+
yield config
|
22
|
+
end
|
23
|
+
|
24
|
+
def config
|
25
|
+
@config ||= Config.new
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
ActiveSupport.on_load(:active_record) do
|
31
|
+
extend ActiveRecordCursorPaginate::Extension
|
32
|
+
end
|
metadata
ADDED
@@ -0,0 +1,70 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: activerecord_cursor_paginate
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- fatkodima
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2024-03-08 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: activerecord
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '7.0'
|
20
|
+
type: :runtime
|
21
|
+
prerelease: false
|
22
|
+
version_requirements: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - ">="
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '7.0'
|
27
|
+
description:
|
28
|
+
email:
|
29
|
+
- fatkodima123@gmail.com
|
30
|
+
executables: []
|
31
|
+
extensions: []
|
32
|
+
extra_rdoc_files: []
|
33
|
+
files:
|
34
|
+
- CHANGELOG.md
|
35
|
+
- LICENSE.txt
|
36
|
+
- README.md
|
37
|
+
- lib/activerecord_cursor_paginate.rb
|
38
|
+
- lib/activerecord_cursor_paginate/config.rb
|
39
|
+
- lib/activerecord_cursor_paginate/cursor.rb
|
40
|
+
- lib/activerecord_cursor_paginate/extension.rb
|
41
|
+
- lib/activerecord_cursor_paginate/page.rb
|
42
|
+
- lib/activerecord_cursor_paginate/paginator.rb
|
43
|
+
- lib/activerecord_cursor_paginate/version.rb
|
44
|
+
homepage: https://github.com/fatkodima/activerecord_cursor_paginate
|
45
|
+
licenses:
|
46
|
+
- MIT
|
47
|
+
metadata:
|
48
|
+
homepage_uri: https://github.com/fatkodima/activerecord_cursor_paginate
|
49
|
+
source_code_uri: https://github.com/fatkodima/activerecord_cursor_paginate
|
50
|
+
changelog_uri: https://github.com/fatkodima/activerecord_cursor_paginate/blob/master/CHANGELOG.md
|
51
|
+
post_install_message:
|
52
|
+
rdoc_options: []
|
53
|
+
require_paths:
|
54
|
+
- lib
|
55
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
56
|
+
requirements:
|
57
|
+
- - ">="
|
58
|
+
- !ruby/object:Gem::Version
|
59
|
+
version: '2.7'
|
60
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
61
|
+
requirements:
|
62
|
+
- - ">="
|
63
|
+
- !ruby/object:Gem::Version
|
64
|
+
version: '0'
|
65
|
+
requirements: []
|
66
|
+
rubygems_version: 3.5.4
|
67
|
+
signing_key:
|
68
|
+
specification_version: 4
|
69
|
+
summary: Cursor-based pagination for ActiveRecord.
|
70
|
+
test_files: []
|