spree-typesense 6.0.0 → 6.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +17 -0
- data/README.md +7 -1
- data/app/models/spree/search_provider/typesense/index_layout.rb +14 -0
- data/app/models/spree/search_provider/typesense.rb +119 -8
- data/lib/spree/typesense/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 668e304126b464f435d440829a4334a397cd9cc356254b2ab832bab8b32ccab5
|
|
4
|
+
data.tar.gz: 83b2fb0a898a9ede0ea26536d707bec06acafbdb952a0e90ebced1489d9c8e31
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 704bcf15d7e794935b84a69f43576fd4b9cf8c49188bb404485d9fb8d794cd29514173bf5a4e67ac178e5d6b347d2d192416d44c3b1fd38c1e49e7e788d3131c
|
|
7
|
+
data.tar.gz: 9d75c0e0686c3a4f3eda56fa084cff0f9370a43b5f913f34ea85b00b0a167a728781f4f119a28ba7450d44709d2703dbb3bf3de5d54831f262703a423c05e73e
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,22 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 6.0.1 (2026-11-20)
|
|
4
|
+
|
|
5
|
+
- **Reindex no longer orders the scope it batches** (#21). `find_in_batches`
|
|
6
|
+
ignores a scoped order, so Rails logged "Scoped order is ignored" on every
|
|
7
|
+
reindex, and an app with `error_on_ignored_order` failed every reindex.
|
|
8
|
+
- **Overlapping reindexes** (#22): the one that started last goes live. An
|
|
9
|
+
earlier one that finds the `__building` marker taken when it ends (or
|
|
10
|
+
fails) drops its collection, leaves the live alias and the marker alone,
|
|
11
|
+
and returns 0. It used to go live without the writes made since the newer
|
|
12
|
+
one started, and remove the newer one's marker. If the newer one failed
|
|
13
|
+
instead, the earlier one raises `ReindexOverlapError` rather than go live
|
|
14
|
+
without those writes.
|
|
15
|
+
- **A product saved during a reindex keeps its newer documents** (#23). A
|
|
16
|
+
batch read before the save left out that product's documents rather than
|
|
17
|
+
overwrite the ones the save wrote into the new collection, which used to
|
|
18
|
+
go live with the older ones.
|
|
19
|
+
|
|
3
20
|
## 6.0.0 (2026-11-19)
|
|
4
21
|
|
|
5
22
|
The first release as **spree-typesense** (`gem 'spree-typesense', '~> 6.0'`),
|
data/README.md
CHANGED
|
@@ -90,7 +90,13 @@ and with it the documents of products that were deleted or left the scope.
|
|
|
90
90
|
- Searches keep answering from the previous collection until the swap.
|
|
91
91
|
- While a reindex runs, the collection it is filling is marked by the alias
|
|
92
92
|
`products_<store code>__building`, and a product saved meanwhile is written
|
|
93
|
-
to both, so the change survives the swap.
|
|
93
|
+
to both, so the change survives the swap. A batch the reindex read before
|
|
94
|
+
that save leaves the product out rather than overwrite the newer documents.
|
|
95
|
+
- If a second reindex starts while one is running (a manual rake run during a
|
|
96
|
+
deploy's, say), the second takes the marker and is the one that goes live.
|
|
97
|
+
The first drops its collection when it ends, logs a warning and returns 0.
|
|
98
|
+
If the second fails instead, the first raises `ReindexOverlapError`, as it
|
|
99
|
+
has missed the writes the second received: run the reindex again.
|
|
94
100
|
- If Typesense rejects any document, or the reindex fails, the new collection
|
|
95
101
|
is dropped and the alias left where it was: the store keeps the index it
|
|
96
102
|
had, and the rake task fails naming the rejected documents.
|
|
@@ -71,6 +71,20 @@ module Spree
|
|
|
71
71
|
nil
|
|
72
72
|
end
|
|
73
73
|
|
|
74
|
+
# Removes the alias only while it points at the collection: a newer
|
|
75
|
+
# reindex may have taken the `__building` marker since (#22).
|
|
76
|
+
# Typesense has no compare-and-delete, so this reads, then deletes;
|
|
77
|
+
# a newer reindex taking the marker between the two loses it.
|
|
78
|
+
# @param alias_name [String]
|
|
79
|
+
# @param collection_name [String]
|
|
80
|
+
# @return [Boolean] whether the alias was removed
|
|
81
|
+
def release!(alias_name, collection_name)
|
|
82
|
+
return false unless alias_target(alias_name) == collection_name
|
|
83
|
+
|
|
84
|
+
unpoint!(alias_name)
|
|
85
|
+
true
|
|
86
|
+
end
|
|
87
|
+
|
|
74
88
|
# Drops the collection; one already gone is fine.
|
|
75
89
|
# @param collection_name [String]
|
|
76
90
|
def drop!(collection_name)
|
|
@@ -26,6 +26,20 @@ module Spree
|
|
|
26
26
|
# error, as multi_search does per search.
|
|
27
27
|
class SearchError < ::Typesense::Error; end
|
|
28
28
|
|
|
29
|
+
# Raised by #reindex when a newer reindex took the `__building` marker
|
|
30
|
+
# from it and then stopped without going live: it failed, or someone
|
|
31
|
+
# removed the marker. The collection this reindex built has missed the
|
|
32
|
+
# single-product writes made meanwhile, so it is dropped rather than
|
|
33
|
+
# swapped in (#22).
|
|
34
|
+
class ReindexOverlapError < StandardError
|
|
35
|
+
# @param building [String] the collection this reindex built
|
|
36
|
+
def initialize(building)
|
|
37
|
+
super("a newer reindex took over while this one was building '#{building}', then stopped without " \
|
|
38
|
+
'going live; that collection missed the product writes made meanwhile, so it was dropped and ' \
|
|
39
|
+
'the live index left as it was: run the reindex again')
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
|
|
29
43
|
# The network errors typesense-ruby 5.0's ApiCall retries on and, once
|
|
30
44
|
# its retries are spent, raises as they are.
|
|
31
45
|
NETWORK_ERRORS = [
|
|
@@ -114,7 +128,7 @@ module Spree
|
|
|
114
128
|
end
|
|
115
129
|
|
|
116
130
|
# Mirrors meilisearch.rb#reindex: default scope is store.products (not
|
|
117
|
-
# Spree::Product.where(store_id:)), batch size 500,
|
|
131
|
+
# Spree::Product.where(store_id:)), batch size 500, in primary key order,
|
|
118
132
|
# lazy-preloaded associations. Returns the number of DOCUMENTS indexed
|
|
119
133
|
# (not products — one product can emit N locale x currency documents).
|
|
120
134
|
#
|
|
@@ -135,6 +149,15 @@ module Spree
|
|
|
135
149
|
# names them all, so `rake spree:search:reindex` fails visibly; the new
|
|
136
150
|
# collection is then dropped and the live alias left where it was, so
|
|
137
151
|
# no product goes missing from search. Any other error does the same.
|
|
152
|
+
#
|
|
153
|
+
# Reindexes can overlap (a manual rake run during the deploy hook's).
|
|
154
|
+
# The one that started last holds the marker, so only it receives
|
|
155
|
+
# single-product writes from then on, and only it goes live: an older
|
|
156
|
+
# one that finds the marker taken when it ends, or fails, drops its
|
|
157
|
+
# collection and returns 0, leaving the newer one to swap in (#22). If
|
|
158
|
+
# the newer one has instead stopped without going live, the older one
|
|
159
|
+
# has still missed those writes: it drops its collection and raises
|
|
160
|
+
# ReindexOverlapError.
|
|
138
161
|
def reindex(scope = nil)
|
|
139
162
|
scope ||= store.products
|
|
140
163
|
layout = index_layout
|
|
@@ -144,10 +167,18 @@ module Spree
|
|
|
144
167
|
|
|
145
168
|
begin
|
|
146
169
|
indexed, failures = index_scope(scope, into: building)
|
|
147
|
-
rescue StandardError
|
|
170
|
+
rescue StandardError => e
|
|
171
|
+
return give_way(layout, building, e) if marker_state(layout, building) == :superseded
|
|
172
|
+
|
|
148
173
|
abandon_build(layout, building)
|
|
149
174
|
raise
|
|
150
175
|
end
|
|
176
|
+
case marker_state(layout, building)
|
|
177
|
+
when :superseded then return give_way(layout, building)
|
|
178
|
+
when :lost
|
|
179
|
+
abandon_build(layout, building)
|
|
180
|
+
raise ReindexOverlapError, building
|
|
181
|
+
end
|
|
151
182
|
|
|
152
183
|
Rails.logger.info do
|
|
153
184
|
"[spree_typesense] reindex complete: #{indexed} documents indexed, #{failures.size} failed"
|
|
@@ -365,9 +396,13 @@ module Spree
|
|
|
365
396
|
def index_scope(scope, into:)
|
|
366
397
|
indexed = 0
|
|
367
398
|
failures = []
|
|
368
|
-
|
|
399
|
+
# find_in_batches walks by primary key and ignores a scoped order,
|
|
400
|
+
# which Rails warns about, or rejects under error_on_ignored_order,
|
|
401
|
+
# so the scope reaches it without one (#21).
|
|
402
|
+
scope.unscope(:order).preload_associations_lazily.find_in_batches(batch_size: 500) do |batch|
|
|
369
403
|
read_at = current_time_ms
|
|
370
404
|
documents = batch.flat_map { |product| presenter_class.new(product, store).call }
|
|
405
|
+
documents = without_newer_writes(documents, into: into, read_at: read_at)
|
|
371
406
|
next if documents.empty?
|
|
372
407
|
|
|
373
408
|
accepted = documents.size
|
|
@@ -383,16 +418,92 @@ module Spree
|
|
|
383
418
|
[indexed, failures]
|
|
384
419
|
end
|
|
385
420
|
|
|
386
|
-
#
|
|
387
|
-
#
|
|
388
|
-
#
|
|
421
|
+
# The batch's documents, less those of products a single-product write
|
|
422
|
+
# (#index) has put into `into` since the batch was read (#23). That
|
|
423
|
+
# write read the product later, so its documents are the newer ones:
|
|
424
|
+
# importing the batch's over them would take the older ones live. A
|
|
425
|
+
# write landing between this lookup and the import can still be
|
|
426
|
+
# overwritten, but that window is one request, where the batch's is
|
|
427
|
+
# the time it takes to build its documents.
|
|
428
|
+
def without_newer_writes(documents, into:, read_at:)
|
|
429
|
+
return documents if documents.empty?
|
|
430
|
+
|
|
431
|
+
newer = product_ids_written_since(documents.map { |d| document_product_id(d) }.uniq, into: into, since: read_at)
|
|
432
|
+
return documents if newer.empty?
|
|
433
|
+
|
|
434
|
+
Rails.logger.info do
|
|
435
|
+
"[spree_typesense] left out #{newer.sort.join(', ')}: saved since this batch was read, so #{into} holds newer documents"
|
|
436
|
+
end
|
|
437
|
+
documents.reject { |d| newer.include?(document_product_id(d)) }
|
|
438
|
+
end
|
|
439
|
+
|
|
440
|
+
# Which of the products have a document in `into` stamped `since` or
|
|
441
|
+
# later: one facet search, sent as a POST, as a batch's ids are longer
|
|
442
|
+
# than Typesense's GET limit allows.
|
|
443
|
+
def product_ids_written_since(product_ids, into:, since:)
|
|
444
|
+
quoted = product_ids.map { |id| "`#{id}`" }.join(',')
|
|
445
|
+
response = client.multi_search.perform(
|
|
446
|
+
{ searches: [{ collection: into, q: '*', per_page: 0,
|
|
447
|
+
filter_by: "product_id:=[#{quoted}] && indexed_at:>=#{since.to_i}",
|
|
448
|
+
facet_by: 'product_id', max_facet_values: product_ids.size }] }
|
|
449
|
+
)
|
|
450
|
+
result = Array(response['results']).first || {}
|
|
451
|
+
raise search_error_for(result) if result['error']
|
|
452
|
+
|
|
453
|
+
(Facets.normalize(result['facet_counts'])['product_id'] || {}).keys
|
|
454
|
+
end
|
|
455
|
+
|
|
456
|
+
def document_product_id(document) = sanitize_prefixed_id(document[:product_id] || document['product_id'])
|
|
457
|
+
|
|
458
|
+
# Forgets a reindex's collection: removes the marker while it is still
|
|
459
|
+
# this reindex's, and drops the collection, leaving the live alias as
|
|
460
|
+
# it was. Best effort, so the error that stopped the reindex is the one
|
|
461
|
+
# raised.
|
|
389
462
|
def abandon_build(layout, building)
|
|
390
|
-
layout.
|
|
463
|
+
layout.release!(layout.building_name, building)
|
|
391
464
|
layout.drop!(building)
|
|
392
465
|
rescue StandardError => e
|
|
393
466
|
Rails.logger.error { "[spree_typesense] could not clean up '#{building}': #{e.class}: #{e.message}" }
|
|
394
467
|
end
|
|
395
468
|
|
|
469
|
+
# Whose the marker is as this reindex of `building` ends:
|
|
470
|
+
# - :mine while it still points at `building`
|
|
471
|
+
# - :superseded when a newer reindex holds it, or has gone live (the
|
|
472
|
+
# live alias points at a newer collection) and removed it
|
|
473
|
+
# - :lost when it is gone and nothing newer went live: the newer
|
|
474
|
+
# reindex failed, or someone removed the marker
|
|
475
|
+
# When the marker cannot be read, the reindex goes on as its own.
|
|
476
|
+
def marker_state(layout, building)
|
|
477
|
+
holder = layout.alias_target(layout.building_name)
|
|
478
|
+
return :mine if holder == building
|
|
479
|
+
return :superseded if holder
|
|
480
|
+
|
|
481
|
+
live = layout.alias_target(layout.live_name)
|
|
482
|
+
live && layout.version_pattern.match?(live) && live > building ? :superseded : :lost
|
|
483
|
+
rescue StandardError
|
|
484
|
+
:mine
|
|
485
|
+
end
|
|
486
|
+
|
|
487
|
+
# A newer reindex took the marker while this one was filling
|
|
488
|
+
# `building`, so `building` has missed the single-product writes made
|
|
489
|
+
# since, and the newer one will swap in a collection that has them.
|
|
490
|
+
# Drops `building`, leaves the live alias and the marker alone, and
|
|
491
|
+
# reports nothing indexed. A failure of this reindex is logged: the
|
|
492
|
+
# newer one meets it too, if it is not passing.
|
|
493
|
+
def give_way(layout, building, error = nil)
|
|
494
|
+
Rails.logger.warn do
|
|
495
|
+
"[spree_typesense] a newer reindex took over while this one was building #{building}" \
|
|
496
|
+
"#{" (and this one failed: #{error.class}: #{error.message})" if error}; " \
|
|
497
|
+
"dropping #{building} and leaving #{layout.live_name} to the newer reindex"
|
|
498
|
+
end
|
|
499
|
+
begin
|
|
500
|
+
layout.drop!(building)
|
|
501
|
+
rescue StandardError => e
|
|
502
|
+
Rails.logger.warn { "[spree_typesense] could not drop '#{building}' (#{e.class}: #{e.message}); the next reindex will" }
|
|
503
|
+
end
|
|
504
|
+
0
|
|
505
|
+
end
|
|
506
|
+
|
|
396
507
|
# Points the live alias at the collection just built, then drops what
|
|
397
508
|
# it replaced. Typesense refuses an alias whose name a collection holds,
|
|
398
509
|
# so a real collection holding the live name is dropped first; searches
|
|
@@ -409,7 +520,7 @@ module Spree
|
|
|
409
520
|
layout.drop!(layout.live_name) if legacy
|
|
410
521
|
point_live_alias(layout, building)
|
|
411
522
|
|
|
412
|
-
after_swap("remove the #{layout.building_name} alias") { layout.
|
|
523
|
+
after_swap("remove the #{layout.building_name} alias") { layout.release!(layout.building_name, building) }
|
|
413
524
|
replaced = after_swap('list the collections it replaced') do
|
|
414
525
|
layout.versioned_collections.select { |name| name < building }
|
|
415
526
|
end || []
|