spree-typesense 6.0.0 → 6.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: d4511197f622f06c51312eb55ee9666bcb8b2576e8b99488685788ff1941f901
4
- data.tar.gz: 507bde5c69e8803ea0b19560e654ae8e2e094f2f3cc224e3da91df0f0f0edd77
3
+ metadata.gz: 668e304126b464f435d440829a4334a397cd9cc356254b2ab832bab8b32ccab5
4
+ data.tar.gz: 83b2fb0a898a9ede0ea26536d707bec06acafbdb952a0e90ebced1489d9c8e31
5
5
  SHA512:
6
- metadata.gz: 67cb48607bccdef46db48344acddce96347d69f739f541e9a83acb5638faf25823d9506d4e33b5aeef10c5e5b02ba8c007302406bc1a264dab2572b7e4966cb9
7
- data.tar.gz: f682513ba927b3dca9e18ea0979b9259e86de789aa92912781fde7d9d3e9d644022ed4c5b2a8c82138434fae939f3ce9fe81484b94da2007cb46c1eb427fe47f
6
+ metadata.gz: 704bcf15d7e794935b84a69f43576fd4b9cf8c49188bb404485d9fb8d794cd29514173bf5a4e67ac178e5d6b347d2d192416d44c3b1fd38c1e49e7e788d3131c
7
+ data.tar.gz: 9d75c0e0686c3a4f3eda56fa084cff0f9370a43b5f913f34ea85b00b0a167a728781f4f119a28ba7450d44709d2703dbb3bf3de5d54831f262703a423c05e73e
data/CHANGELOG.md CHANGED
@@ -1,5 +1,22 @@
1
1
  # Changelog
2
2
 
3
+ ## 6.0.1 (2026-11-20)
4
+
5
+ - **Reindex no longer orders the scope it batches** (#21). `find_in_batches`
6
+ ignores a scoped order, so Rails logged "Scoped order is ignored" on every
7
+ reindex, and an app with `error_on_ignored_order` failed every reindex.
8
+ - **Overlapping reindexes** (#22): the one that started last goes live. An
9
+ earlier one that finds the `__building` marker taken when it ends (or
10
+ fails) drops its collection, leaves the live alias and the marker alone,
11
+ and returns 0. It used to go live without the writes made since the newer
12
+ one started, and remove the newer one's marker. If the newer one failed
13
+ instead, the earlier one raises `ReindexOverlapError` rather than go live
14
+ without those writes.
15
+ - **A product saved during a reindex keeps its newer documents** (#23). A
16
+ batch read before the save left out that product's documents rather than
17
+ overwrite the ones the save wrote into the new collection, which used to
18
+ go live with the older ones.
19
+
3
20
  ## 6.0.0 (2026-11-19)
4
21
 
5
22
  The first release as **spree-typesense** (`gem 'spree-typesense', '~> 6.0'`),
data/README.md CHANGED
@@ -90,7 +90,13 @@ and with it the documents of products that were deleted or left the scope.
90
90
  - Searches keep answering from the previous collection until the swap.
91
91
  - While a reindex runs, the collection it is filling is marked by the alias
92
92
  `products_<store code>__building`, and a product saved meanwhile is written
93
- to both, so the change survives the swap.
93
+ to both, so the change survives the swap. A batch the reindex read before
94
+ that save leaves the product out rather than overwrite the newer documents.
95
+ - If a second reindex starts while one is running (a manual rake run during a
96
+ deploy's, say), the second takes the marker and is the one that goes live.
97
+ The first drops its collection when it ends, logs a warning and returns 0.
98
+ If the second fails instead, the first raises `ReindexOverlapError`, as it
99
+ has missed the writes the second received: run the reindex again.
94
100
  - If Typesense rejects any document, or the reindex fails, the new collection
95
101
  is dropped and the alias left where it was: the store keeps the index it
96
102
  had, and the rake task fails naming the rejected documents.
@@ -71,6 +71,20 @@ module Spree
71
71
  nil
72
72
  end
73
73
 
74
+ # Removes the alias only while it points at the collection: a newer
75
+ # reindex may have taken the `__building` marker since (#22).
76
+ # Typesense has no compare-and-delete, so this reads, then deletes;
77
+ # a newer reindex taking the marker between the two loses it.
78
+ # @param alias_name [String]
79
+ # @param collection_name [String]
80
+ # @return [Boolean] whether the alias was removed
81
+ def release!(alias_name, collection_name)
82
+ return false unless alias_target(alias_name) == collection_name
83
+
84
+ unpoint!(alias_name)
85
+ true
86
+ end
87
+
74
88
  # Drops the collection; one already gone is fine.
75
89
  # @param collection_name [String]
76
90
  def drop!(collection_name)
@@ -26,6 +26,20 @@ module Spree
26
26
  # error, as multi_search does per search.
27
27
  class SearchError < ::Typesense::Error; end
28
28
 
29
+ # Raised by #reindex when a newer reindex took the `__building` marker
30
+ # from it and then stopped without going live: it failed, or someone
31
+ # removed the marker. The collection this reindex built has missed the
32
+ # single-product writes made meanwhile, so it is dropped rather than
33
+ # swapped in (#22).
34
+ class ReindexOverlapError < StandardError
35
+ # @param building [String] the collection this reindex built
36
+ def initialize(building)
37
+ super("a newer reindex took over while this one was building '#{building}', then stopped without " \
38
+ 'going live; that collection missed the product writes made meanwhile, so it was dropped and ' \
39
+ 'the live index left as it was: run the reindex again')
40
+ end
41
+ end
42
+
29
43
  # The network errors typesense-ruby 5.0's ApiCall retries on and, once
30
44
  # its retries are spent, raises as they are.
31
45
  NETWORK_ERRORS = [
@@ -114,7 +128,7 @@ module Spree
114
128
  end
115
129
 
116
130
  # Mirrors meilisearch.rb#reindex: default scope is store.products (not
117
- # Spree::Product.where(store_id:)), batch size 500, deterministic order,
131
+ # Spree::Product.where(store_id:)), batch size 500, in primary key order,
118
132
  # lazy-preloaded associations. Returns the number of DOCUMENTS indexed
119
133
  # (not products — one product can emit N locale x currency documents).
120
134
  #
@@ -135,6 +149,15 @@ module Spree
135
149
  # names them all, so `rake spree:search:reindex` fails visibly; the new
136
150
  # collection is then dropped and the live alias left where it was, so
137
151
  # no product goes missing from search. Any other error does the same.
152
+ #
153
+ # Reindexes can overlap (a manual rake run during the deploy hook's).
154
+ # The one that started last holds the marker, so only it receives
155
+ # single-product writes from then on, and only it goes live: an older
156
+ # one that finds the marker taken when it ends, or fails, drops its
157
+ # collection and returns 0, leaving the newer one to swap in (#22). If
158
+ # the newer one has instead stopped without going live, the older one
159
+ # has still missed those writes: it drops its collection and raises
160
+ # ReindexOverlapError.
138
161
  def reindex(scope = nil)
139
162
  scope ||= store.products
140
163
  layout = index_layout
@@ -144,10 +167,18 @@ module Spree
144
167
 
145
168
  begin
146
169
  indexed, failures = index_scope(scope, into: building)
147
- rescue StandardError
170
+ rescue StandardError => e
171
+ return give_way(layout, building, e) if marker_state(layout, building) == :superseded
172
+
148
173
  abandon_build(layout, building)
149
174
  raise
150
175
  end
176
+ case marker_state(layout, building)
177
+ when :superseded then return give_way(layout, building)
178
+ when :lost
179
+ abandon_build(layout, building)
180
+ raise ReindexOverlapError, building
181
+ end
151
182
 
152
183
  Rails.logger.info do
153
184
  "[spree_typesense] reindex complete: #{indexed} documents indexed, #{failures.size} failed"
@@ -365,9 +396,13 @@ module Spree
365
396
  def index_scope(scope, into:)
366
397
  indexed = 0
367
398
  failures = []
368
- scope.reorder(id: :asc).preload_associations_lazily.find_in_batches(batch_size: 500) do |batch|
399
+ # find_in_batches walks by primary key and ignores a scoped order,
400
+ # which Rails warns about, or rejects under error_on_ignored_order,
401
+ # so the scope reaches it without one (#21).
402
+ scope.unscope(:order).preload_associations_lazily.find_in_batches(batch_size: 500) do |batch|
369
403
  read_at = current_time_ms
370
404
  documents = batch.flat_map { |product| presenter_class.new(product, store).call }
405
+ documents = without_newer_writes(documents, into: into, read_at: read_at)
371
406
  next if documents.empty?
372
407
 
373
408
  accepted = documents.size
@@ -383,16 +418,92 @@ module Spree
383
418
  [indexed, failures]
384
419
  end
385
420
 
386
- # Forgets a reindex's collection: removes the marker and drops it,
387
- # leaving the live alias as it was. Best effort, so the error that
388
- # stopped the reindex is the one raised.
421
+ # The batch's documents, less those of products a single-product write
422
+ # (#index) has put into `into` since the batch was read (#23). That
423
+ # write read the product later, so its documents are the newer ones:
424
+ # importing the batch's over them would take the older ones live. A
425
+ # write landing between this lookup and the import can still be
426
+ # overwritten, but that window is one request, where the batch's is
427
+ # the time it takes to build its documents.
428
+ def without_newer_writes(documents, into:, read_at:)
429
+ return documents if documents.empty?
430
+
431
+ newer = product_ids_written_since(documents.map { |d| document_product_id(d) }.uniq, into: into, since: read_at)
432
+ return documents if newer.empty?
433
+
434
+ Rails.logger.info do
435
+ "[spree_typesense] left out #{newer.sort.join(', ')}: saved since this batch was read, so #{into} holds newer documents"
436
+ end
437
+ documents.reject { |d| newer.include?(document_product_id(d)) }
438
+ end
439
+
440
+ # Which of the products have a document in `into` stamped `since` or
441
+ # later: one facet search, sent as a POST, as a batch's ids are longer
442
+ # than Typesense's GET limit allows.
443
+ def product_ids_written_since(product_ids, into:, since:)
444
+ quoted = product_ids.map { |id| "`#{id}`" }.join(',')
445
+ response = client.multi_search.perform(
446
+ { searches: [{ collection: into, q: '*', per_page: 0,
447
+ filter_by: "product_id:=[#{quoted}] && indexed_at:>=#{since.to_i}",
448
+ facet_by: 'product_id', max_facet_values: product_ids.size }] }
449
+ )
450
+ result = Array(response['results']).first || {}
451
+ raise search_error_for(result) if result['error']
452
+
453
+ (Facets.normalize(result['facet_counts'])['product_id'] || {}).keys
454
+ end
455
+
456
+ def document_product_id(document) = sanitize_prefixed_id(document[:product_id] || document['product_id'])
457
+
458
+ # Forgets a reindex's collection: removes the marker while it is still
459
+ # this reindex's, and drops the collection, leaving the live alias as
460
+ # it was. Best effort, so the error that stopped the reindex is the one
461
+ # raised.
389
462
  def abandon_build(layout, building)
390
- layout.unpoint!(layout.building_name)
463
+ layout.release!(layout.building_name, building)
391
464
  layout.drop!(building)
392
465
  rescue StandardError => e
393
466
  Rails.logger.error { "[spree_typesense] could not clean up '#{building}': #{e.class}: #{e.message}" }
394
467
  end
395
468
 
469
+ # Whose the marker is as this reindex of `building` ends:
470
+ # - :mine while it still points at `building`
471
+ # - :superseded when a newer reindex holds it, or has gone live (the
472
+ # live alias points at a newer collection) and removed it
473
+ # - :lost when it is gone and nothing newer went live: the newer
474
+ # reindex failed, or someone removed the marker
475
+ # When the marker cannot be read, the reindex goes on as its own.
476
+ def marker_state(layout, building)
477
+ holder = layout.alias_target(layout.building_name)
478
+ return :mine if holder == building
479
+ return :superseded if holder
480
+
481
+ live = layout.alias_target(layout.live_name)
482
+ live && layout.version_pattern.match?(live) && live > building ? :superseded : :lost
483
+ rescue StandardError
484
+ :mine
485
+ end
486
+
487
+ # A newer reindex took the marker while this one was filling
488
+ # `building`, so `building` has missed the single-product writes made
489
+ # since, and the newer one will swap in a collection that has them.
490
+ # Drops `building`, leaves the live alias and the marker alone, and
491
+ # reports nothing indexed. A failure of this reindex is logged: the
492
+ # newer one meets it too, if it is not passing.
493
+ def give_way(layout, building, error = nil)
494
+ Rails.logger.warn do
495
+ "[spree_typesense] a newer reindex took over while this one was building #{building}" \
496
+ "#{" (and this one failed: #{error.class}: #{error.message})" if error}; " \
497
+ "dropping #{building} and leaving #{layout.live_name} to the newer reindex"
498
+ end
499
+ begin
500
+ layout.drop!(building)
501
+ rescue StandardError => e
502
+ Rails.logger.warn { "[spree_typesense] could not drop '#{building}' (#{e.class}: #{e.message}); the next reindex will" }
503
+ end
504
+ 0
505
+ end
506
+
396
507
  # Points the live alias at the collection just built, then drops what
397
508
  # it replaced. Typesense refuses an alias whose name a collection holds,
398
509
  # so a real collection holding the live name is dropped first; searches
@@ -409,7 +520,7 @@ module Spree
409
520
  layout.drop!(layout.live_name) if legacy
410
521
  point_live_alias(layout, building)
411
522
 
412
- after_swap("remove the #{layout.building_name} alias") { layout.unpoint!(layout.building_name) }
523
+ after_swap("remove the #{layout.building_name} alias") { layout.release!(layout.building_name, building) }
413
524
  replaced = after_swap('list the collections it replaced') do
414
525
  layout.versioned_collections.select { |name| name < building }
415
526
  end || []
@@ -1,6 +1,6 @@
1
1
  module Spree
2
2
  module Typesense
3
3
  # Major version tracks Spree's major version: 6.x supports Spree 6.x.
4
- VERSION = '6.0.0'.freeze
4
+ VERSION = '6.0.1'.freeze
5
5
  end
6
6
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: spree-typesense
3
3
  version: !ruby/object:Gem::Version
4
- version: 6.0.0
4
+ version: 6.0.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Aypex