search-engine-for-typesense 30.1.8.20 → 30.1.8.24

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0ee014706c77ccb6794d31e57efbc2e2efe3f5dd167e18adb2ada205678fdea1
-  data.tar.gz: 11544aa9e90f22be02e65df7e848c4a66ef452f544e9f9ee0ea84ba9dfe31871
+  metadata.gz: 5f32e5d4e9f5b31b0ce518fc880b11df77f7023d759c876bbae7e5cc4ff81202
+  data.tar.gz: 541c01e233fe52c156e456b41a7caa77cde65d7517d9e876da4d026944a368e9
 SHA512:
-  metadata.gz: 40d1c46d229a68f0f04402d3fed7ea62fa68d91d76bb92096e2062667518c68f19cf32d5ed3155140c304a38328714e19fd718f950f21cd6f1f890848c827075
-  data.tar.gz: 6e4e43fd5ca84f0f53ba3cc8a06d1dc2eadd43be935b54f4bdc2fcd7aef38e1c860a6e37c38cd73acdf1c4a8683d769bc4a58e80bd0ab3f356eed4de37e393e7
+  metadata.gz: b2f888f079aa3991467920c1f66ca4538feb0043710c1ccf6ad76a3ecef16a26243db14921f75ea3fea09387331021fb9088c6041798f0999b6d924d10389902
+  data.tar.gz: fe6e8b4b9a36b1b4de3e7952d68e2309cab02a62caa2306d8ecccd445f91fed30db9c065c849f4f9bb382bcd7189fb9b8c3bec239e8de9229fc7964f498ff8d6

data/CHANGELOG.md

@@ -0,0 +1,74 @@
+# Changelog
+
+## 30.1.8.24
+
+- Preserve caller transactions during PostgreSQL streaming by rejecting active transactions before opening a cursor.
+  Apply statement timeouts inside the streaming transaction and restore connection settings on completion or failure.
+- Synchronize both record creation and updates after commit, and propagate failed custom document identity calculations.
+- Propagate parallel import worker failures and timeouts instead of returning successful summaries for incomplete imports.
+- Execute ID-filtered queries through normal search materialization so text filters, selection, and memoization apply.
+- Use native Typesense joins without incomplete client-side key lookups. Missing references and invalid search
+  configurations now raise errors instead of returning empty results. Existing joins require valid server references.
+- Allow upserts to omit server-generated embeddings and preserve boolean coercions that produce false.
+- Correct generated models, documented declarations, and demo collection references and timestamp mappings.
+
+## 30.1.8.23
+
+- Run retained-physical alias rollbacks through the same cutover guard. Rollback accepts explicit destination
+  and expected-current compare-and-swap targets, validates older retained physicals, and makes legacy retries a
+  conservative no-op instead of toggling the alias forward again.
+- Make row-trigger notification payloads constant per source table and collection so PostgreSQL folds repeated
+  notifications from one transaction into one low-latency wakeup. Durable event identity remains in the outbox
+  table, which continues to be the source of truth.
+- Add `replace_search_engine_outbox_trigger_function` for forward migrations that only need to refresh generated
+  function bodies, without dropping triggers or taking avoidable trigger DDL locks on source tables.
+- Renew target-scoped delivery leases before every dependency-ordered collection group. Deliveries whose fenced
+  lease cannot be renewed are never passed to a processor, so slow earlier groups cannot consume the lease budget
+  of later groups in the same claim.
+- Add opt-in `postgres_outbox.clear_cache_after_write` invalidation. Enabled hosts clear Typesense's server-side
+  search cache before acknowledging each processed collection group; a clear failure keeps the group retryable.
+
+No database migration is added by this release. Existing PostgreSQL outbox delivery/slot migrations remain
+compatible.
+
+## 30.1.8.22
+
+- Add `schema.around_rebuild`, an exact-once guard around full physical create/index/alias-swap/retention
+  lifecycles. In-place schema updates remain outside the guard, while guard and indexing failures propagate
+  without leaking interrupted physical collections.
+- Add PostgreSQL delivery-target cutover locks. Full rebuilds can pause new claims, wait for acknowledged
+  processing deliveries, and exclude cooperating direct Typesense writers with one deterministic target key.
+  Locks are session-safe, timeout-bounded, and automatically released when the database session dies.
+- Gate every delivery claim before target row work. Timed-out leases are reclaimed only for the bounded rows
+  actually selected; unselected rows remain `processing`, and stale older operations are settled before newer
+  siblings so delete/recreation ordering cannot reverse.
+- Pin built-in outbox upserts and deletes to one resolved physical collection for the claimed operation, so a
+  delayed stale HTTP request cannot follow a logical alias onto a replacement collection after cutover.
+- Reject standalone `reset_stale_processing!` in delivery mode because reset without same-transaction reclaim
+  can create false quiescence. Single-target event mode is unchanged.
+- Make forced cascade rebuild failures propagate instead of downgrading to live partition imports.
+- Mark `search_engine:index:rebuild` and `search_engine:index:rebuild_partition` as live-maintenance paths. When
+  `schema.around_rebuild` is configured, they now require explicit `ALLOW_LIVE_INDEX_MAINTENANCE=true` after
+  operators pause outbox consumers and direct writers. `search_engine:schema:apply` accepts
+  `FORCE_REBUILD=true` as the explicit guarded full-data rebuild path.
+
+No database migration is added by this release. Existing PostgreSQL outbox delivery/slot migrations remain
+compatible.
+
+## 30.1.8.21
+
+- Bound PostgreSQL outbox claim ranking and harden delivery coalescing with parent-first locking, fenced
+  delivery leases, lock-order-safe stale resets, and per-event processing failure isolation.
+- Add explicit, audited, idempotent delivery-target retirement with a required dry-run/apply decision. Target
+  retirement never infers intent from configuration removal and does not alter drain slots.
+- Add strict ordered Typesense import response parsing and `upsert_bulk(..., on_failure: :return)` for stable
+  row-level outcomes. Malformed, ambiguous, short, and long responses now fail closed.
+- Remove raw Typesense `response` data from bulk upsert result/error metadata because it can contain submitted
+  documents. Callers that used `result[:response]` must migrate to `row_results` and aggregate counters.
+- Preserve safe structured error metadata while recursively redacting secrets and payload-bearing fields.
+- Reject unsupported dispatch configuration/overrides instead of silently falling back to inline execution.
+- Make partial async partition imports retryable and observable. Custom partition run stores must add
+  `record_attempt(run_id:, partition_key:, summary:, error:)` as a non-terminal transition.
+
+No database migration is added by this release. Existing PostgreSQL outbox delivery/slot migrations remain
+compatible.

data/README.md

@@ -26,6 +26,7 @@ SearchEngine.configure do |c|
   c.port = 8108
   c.protocol = "http"
   c.api_key = ENV.fetch("TYPESENSE_API_KEY")
+  c.default_infix = "off"
 end
 ```
 
@@ -33,10 +34,10 @@ end
 class SearchEngine::Product < SearchEngine::Base
   collection :products
 
-  attribute :id, :integer
+  identify_by :id
   attribute :name, :string
 
-  query_by %i[name brand description]
+  query_by :name
 end
 
 SearchEngine::Product.where(name: "milk").select(:id, :name).limit(5).to_a
@@ -58,22 +59,50 @@ SearchEngine.configure do |c|
 end
 ```
 
+SQL sources stream PostgreSQL queries in their own read-only transaction. Start streaming
+outside an existing transaction; an active transaction is rejected before cursor SQL runs.
+The configured statement timeout applies only to the streaming transaction.
+
+Joined filters execute as native Typesense joins and require valid collection references.
+Search configuration and missing-reference errors propagate to callers; they do not
+produce empty results or trigger client-side joins.
+
 ## Usage examples
 
 ```ruby
-# Model
+# Models (the source objects below are the host application's ActiveRecord models)
+class SearchEngine::Brand < SearchEngine::Base
+  collection "brands"
+  identify_by :id
+  attribute :name, :string
+  query_by :name
+end
+
 class SearchEngine::Product < SearchEngine::Base
   collection "products"
 
-  attribute :id, :integer
+  identify_by :id
   attribute :name, :string
+  attribute :price_cents, :integer, sort: true
+  attribute :brand_id, :string, facet: true
+  attribute :category, :string, facet: true
+  belongs_to :brands, collection: "brands", local_key: :brand_id, foreign_key: :id
+  query_by :name
+
+  index do
+    source :active_record, model: ::Product
+    map do |record|
+      { name: record.name, price_cents: record.price_cents,
+        brand_id: record.brand_id.to_s, category: record.category }
+    end
+  end
 end
 
 # Basic query
 SearchEngine::Product
   .where(name: "milk")
   # Explicit query_by always wins over model/global defaults
-  .options(query_by: 'name,brand')
+  .options(query_by: 'name')
   .select(:id, :name)
   .order(price_cents: :asc)
   .limit(5)
@@ -95,7 +124,7 @@ rel = SearchEngine::Product
 params = rel.to_h # compiled Typesense params
 
 # Multi-search
-result_set = SearchEngine.multi_search(common: { query_by: SearchEngine.config.default_query_by }) do |m|
+result_set = SearchEngine.multi_search(common: { query_by: "name" }) do |m|
   m.add :products, SearchEngine::Product.where("name:~rud").per(10)
   m.add :brands,   SearchEngine::Brand.all.per(5)
 end
@@ -117,6 +146,11 @@ SearchEngine::Product.upsert_bulk(records: Product.limit(2))
 # Bulk upsert mapped payloads
 SearchEngine::Product.upsert_bulk(data: [mapped])
 
+# Inspect row-level failures without repeating the import request
+result = SearchEngine::Product.upsert_bulk(data: [mapped], on_failure: :return)
+failed_rows = result[:row_results].reject { |row| row[:success] }
+failed_rows.first # => { index: 0, success: false, status: 404, error: "..." }
+
 # Geo search
 class SearchEngine::Venue < SearchEngine::Base
   collection :venues
@@ -148,6 +182,15 @@ result = SearchEngine::Venue.all.order_geo(:location, from: { lat: 54.69, lng: 2
 result.hits.first.geo_distance_meters # => { "location" => 1234 }
 ```
 
+`upsert_bulk` defaults to `on_failure: :raise`. Use the exact symbol `:return` when the caller needs to
+handle valid Typesense row failures itself. Malformed responses and response/document count mismatches always
+raise in either mode; the gem never guesses which submitted document a missing response row belongs to.
+
+Since `30.1.8.21`, bulk results intentionally do not expose the raw Typesense `response` because import
+responses may contain submitted documents. Use the ordered, frozen `row_results` entries (`index`, `success`,
+`status`, `error`) and aggregate counters instead. This is a safety-related result-shape change for callers
+that previously read `result[:response]`.
+
 ## Documentation
 
 See the [Docs](https://nikita-shkoda.mintlify.app/projects/search-engine-for-typesense/v30.1/index)
@@ -188,6 +231,124 @@ Use a shared `Rails.cache` backend, or provide `c.indexer.partition_run_store`,
 the parent indexing process can see the same run metadata. Size the queue carefully: worker concurrency
 multiplies with any per-partition `max_parallel` setting.
 
+A custom partition run store must implement:
+
+```ruby
+create_run(run_id:, collection:, collection_class_name:, into:, partitions:, ttl_s:)
+mark_started(run_id:, partition_key:, job_id: nil)
+record_attempt(run_id:, partition_key:, summary:, error:)
+mark_succeeded(run_id:, partition_key:, summary:)
+mark_failed(run_id:, partition_key:, error:)
+snapshot(run_id:)
+expire(run_id:)
+```
+
+`record_attempt` is non-terminal: it must persist that partition's partial summary and representative error
+while leaving the partition `running`, without changing sibling partitions. Row-level import failures raise
+from `IndexPartitionJob`, so ActiveJob retries the partition. Replaying successful upserts is expected and
+safe. Only retry exhaustion terminalizes the partition as failed.
+
+Only `:inline` and `:active_job` (or their exact String equivalents) are supported for
+`c.indexer.dispatch` and `c.indexer.partition_execution`. Unknown values such as `:sidekiq` fail configuration
+validation instead of silently running inline. Explicit `:active_job` dispatch also fails if ActiveJob is not
+available.
+
+## Guarded blue/green cutovers
+
+`SearchEngine::Schema.apply!` can run an optional guard around the complete full-rebuild lifecycle: physical
+collection creation, indexing, alias swap, and retention. `SearchEngine::Schema.rollback` uses the same guard
+around target resolution, validation, and its alias swap. In-place schema updates intentionally do not invoke
+the guard. The guard must yield exactly once; not yielding or yielding repeatedly raises `ArgumentError`.
+Guard errors and indexing errors propagate, and an interrupted pre-swap physical collection is still cleaned
+up.
+
+In delivery-target outbox deployments, use the guard to stop new target claims, wait for already-claimed
+deliveries to acknowledge, and prevent cooperating direct writers from crossing the alias swap:
+
+```ruby
+# config/initializers/search_engine.rb
+typesense_target_key = ENV.fetch("TYPESENSE_DELIVERY_TARGET_KEY")
+
+SearchEngine.configure do |c|
+  c.schema.around_rebuild = lambda do |collection:, &rebuild|
+    Rails.logger.info("Guarding Typesense rebuild for #{collection} on #{typesense_target_key}")
+    SearchEngine::PostgresOutbox::Repository.new.with_delivery_target_claims_paused(
+      target_key: typesense_target_key,
+      timeout_s: SearchEngine.config.postgres_outbox.processing_timeout_s + 60,
+      poll_interval_s: 0.1,
+      &rebuild
+    )
+  end
+end
+```
+
+The target key is the exact delivery destination key, not the logical collection name. Claim transactions
+automatically take the matching transaction-scoped shared PostgreSQL advisory lock. The rebuild guard takes
+an exclusive session lock, then waits until that target has no `processing` deliveries. It deliberately does
+not reset timed-out leases: lease age cannot prove that an old worker's external Typesense request has
+stopped. A stuck processing delivery therefore times out the rebuild; recover it operationally before
+retrying. PostgreSQL releases the session lock if the owning connection or process dies.
+Set the guard timeout above `processing_timeout_s` with enough margin for the slowest legitimate external
+write and scheduling delay; a shorter fixed timeout causes avoidable rebuild failures.
+
+Every direct Typesense writer outside the delivery drainer must cooperate with the same exact target key:
+
+```ruby
+SearchEngine::PostgresOutbox::Repository.new.with_delivery_target_writes_allowed(
+  target_key: typesense_target_key,
+  timeout_s: 5
+) do
+  SearchEngine::Product.upsert(record: product)
+end
+```
+
+Claimed delivery requests must resolve the logical alias once and write to that pinned physical collection until
+the delivery is acknowledged. The built-in `EventProcessor` does this for both upserts and deletes. Custom
+collection processors must follow the same rule: a stale worker whose HTTP request completes after lease reclaim
+must only be able to mutate the retired physical collection, never whichever physical collection the logical alias
+points to after a later cutover.
+
+Rollback is conservative and retry-safe. The backward-compatible form rolls back one generation only when the
+alias points at the newest retained physical; once it no longer does, repeating the call returns
+`action: :already_rolled_back` without another swap. When newer orphaned physicals exist, or when an operator
+needs an auditable destination, pass the retained target explicitly. Add `expected_current` for compare-and-swap
+protection:
+
+```ruby
+SearchEngine::Schema.rollback(
+  SearchEngine::Product,
+  to: "products_20250101_000000_001",
+  expected_current: "products_20250102_000000_001"
+)
+```
+
+The equivalent task remains backward compatible and also accepts both safety arguments:
+
+```sh
+rails 'search_engine:schema:rollback[products]'
+rails 'search_engine:schema:rollback[products,products_20250101_000000_001,products_20250102_000000_001]'
+```
+
+An explicit destination must be a retained physical older than the current/expected source. Retrying after the
+alias already reached that destination is a successful no-op. Any other expected-current mismatch raises
+`SearchEngine::Schema::RollbackConflict` without changing the alias.
+
+This protocol only gates delivery-target claims. Legacy single-target event claims have no target identity and
+must be stopped before a rebuild. Likewise, an unguarded direct writer can still cross the swap. In
+delivery-target mode, `reset_stale_processing!` rejects standalone use because reset without same-transaction
+reclaim creates a false-quiescence gap. Ordinary claims directly reclaim only the bounded timed-out rows they
+actually select; unselected stale rows remain `processing`, so a cutover continues to fail closed.
+
+Forced cascade rebuilds use the guarded blue/green path and propagate any rebuild/guard failure. They never
+downgrade to a live partition import. The maintenance tasks `search_engine:index:rebuild` and
+`search_engine:index:rebuild_partition` are intentionally different: they write directly into the resolved
+live collection and do not invoke `schema.around_rebuild`. When a guard is configured, these tasks refuse to
+run unless `ALLOW_LIVE_INDEX_MAINTENANCE=true` is explicitly set. Pause all outbox consumers and direct writers
+before using that override. For a guarded forced blue/green data rebuild, run
+`FORCE_REBUILD=true rails 'search_engine:schema:apply[collection]'`. Without `FORCE_REBUILD`, schema apply may
+complete as an in-place schema update and intentionally skip document reindexing. Hosts with no configured
+guard retain the legacy live-task behavior.
+
 ## PostgreSQL outbox sync
 
 Rails callbacks are convenient for ordinary `create`, `update`, and `destroy` flows, but they do not see
@@ -199,7 +360,9 @@ The flow is:
 
 1. A row-level PostgreSQL trigger writes a durable outbox row in the same transaction as the source table
    change.
-2. The trigger calls `pg_notify` as a low-latency nudge after commit.
+2. The trigger calls `pg_notify` as a low-latency nudge after commit. Its payload is constant for a given
+   source table and collection, so PostgreSQL can fold repeated row-level notifications from one transaction
+   into a single wakeup.
 3. A host-managed listener receives notifications, or falls back to polling, and enqueues
    `SearchEngine::PostgresOutbox::DrainJob`.
 4. The drainer claims pending rows, coalesces older rows for the same collection/document pair, orders
@@ -230,6 +393,7 @@ SearchEngine.configure do |c|
   c.postgres_outbox.drain_target_parallelism = 1
   c.postgres_outbox.drain_job_max_batches = 1
   c.postgres_outbox.drain_job_max_runtime_s = nil
+  c.postgres_outbox.clear_cache_after_write = false
   c.postgres_outbox.poll_interval_s = 5
   c.postgres_outbox.retention_s = 7.days.to_i
 
@@ -246,10 +410,28 @@ SearchEngine.configure do |c|
 end
 ```
 
+When an upgrade changes only generated trigger-function behavior, migrations can call
+`replace_search_engine_outbox_trigger_function` with the same arguments used to create the trigger. It runs
+`CREATE OR REPLACE FUNCTION` without dropping and reattaching the existing trigger, avoiding unnecessary
+table-level trigger DDL locks. The trigger must already exist.
+
 `batch_size` is the global fallback for all collections. Use `batch_sizes` when some collections are much
 lighter or heavier than others. Omitted drain limits use the per-collection values; explicit `limit:`
 arguments still override the map and use one global cap for that drain.
 
+When searches set `use_cache`, enable `clear_cache_after_write` if incremental
+writes must be visible before their outbox rows are acknowledged. The drainer
+clears Typesense's server-side search cache once after each processed
+collection group and before database acknowledgement. If cache clearing fails,
+the group remains retryable and its idempotent document writes are replayed.
+The option defaults to `false` so hosts that do not use server-side search
+caching do not add an unnecessary API call.
+
+In delivery-target mode, a claim can contain several dependency-ordered collection groups. The drainer renews
+all unstarted fenced leases before each group and removes any delivery it no longer owns before calling the
+processor. Hosts must still size each individual collection group so its worst-case Typesense retry sequence
+fits below `processing_timeout_s`; renewal protects later groups, not an unbounded current group.
+
 Generate and edit the migrations:
 
 ```bash
@@ -341,6 +523,36 @@ hosts add the optional drain slot table.
 Processors still receive event objects and return event IDs; the parent event status is refreshed from the
 aggregate delivery states.
 
+### Retiring a delivery target
+
+Removing a target from `delivery_targets` does not infer permission to discard its persisted backlog. Remove
+the target from configuration first, keep process configuration stable, then explicitly dry-run and apply
+retirement:
+
+```ruby
+repository = SearchEngine::PostgresOutbox::Repository.new
+
+preview = repository.retire_delivery_target!(
+  target_key: "mirror_a",
+  dry_run: true,
+  reason: "mirror_a was decommissioned"
+)
+
+result = repository.retire_delivery_target!(
+  target_key: "mirror_a",
+  dry_run: false,
+  reason: "mirror_a was decommissioned",
+  operator: "deploy-2026-07-11"
+)
+```
+
+Both calls reject a target that is still configured. Apply supersedes only that exact target's `pending`,
+`processing`, and `failed` deliveries, clears leases, stores a complete JSON audit record in `last_error`, and
+refreshes parent event status under parent-first locks. It is idempotent; a second apply reports zero rows.
+Drain slots are scheduler bookkeeping and are deliberately not changed. Configuration is rechecked directly
+before each database mutation, but Ruby configuration and PostgreSQL cannot share an atomic lock, so do not
+change delivery-target configuration concurrently with retirement.
+
 Pair triggered source models with `sync_strategy: :postgres_outbox` so Active Record callbacks do not also
 write to Typesense for the same changes:
 

data/app/search_engine/search_engine/index_partition_job.rb

@@ -19,6 +19,7 @@ module SearchEngine
     # Handle transient errors with exponential backoff based on Indexer config.
     rescue_from(SearchEngine::Errors::Timeout) { |error| retry_if_possible(error) }
     rescue_from(SearchEngine::Errors::Connection) { |error| retry_if_possible(error) }
+    rescue_from(SearchEngine::Errors::PartitionImportFailed) { |error| retry_if_possible(error) }
     rescue_from(SearchEngine::Errors::Api) do |error|
       if SearchEngine::Indexer::RetryPolicy.transient_status?(error.status.to_i)
         retry_if_possible(error)
@@ -52,6 +53,7 @@ module SearchEngine
       SearchEngine::Instrumentation.with_context(dispatch_mode: :active_job, job_id: job_id) do
         summary = SearchEngine::Indexer.rebuild_partition!(klass, partition: partition, into: into)
       end
+      raise_for_failed_summary!(summary, run_store, run_id, partition_key)
       duration = (monotonic_ms - started).round(1)
       run_store&.mark_succeeded(run_id: run_id, partition_key: partition_key, summary: summary)
 
@@ -64,8 +66,12 @@ module SearchEngine
       nil
     rescue SearchEngine::IndexingRunStore::StaleRun => error
       safe_payload = payload || error_payload(error)
-      instrument_stale_run(error, payload: safe_payload.merge(metadata: metadata || {}), run_id: run_id,
-                           partition_key: partition_key)
+      instrument_stale_run(
+        error,
+        payload: safe_payload.merge(metadata: metadata || {}),
+        run_id: run_id,
+        partition_key: partition_key
+      )
       nil
     rescue StandardError => error
       safe_payload = payload || error_payload(error)
@@ -75,6 +81,12 @@ module SearchEngine
         raise
       end
 
+      if error.is_a?(SearchEngine::Errors::PartitionImportFailed)
+        mark_failed_for_run(run_store, run_id, partition_key, error, payload: safe_payload) if run_id && run_store
+        instrument_error(error, payload: safe_payload)
+        raise
+      end
+
       if run_id && run_store
         mark_failed_for_run(run_store, run_id, partition_key, error, payload: safe_payload)
         instrument_error(error, payload: safe_payload)
@@ -131,7 +143,8 @@ module SearchEngine
     end
 
     def retryable_error?(error)
-      transient_error?(error) && executions.to_i < retry_policy.attempts
+      (transient_error?(error) || error.is_a?(SearchEngine::Errors::PartitionImportFailed)) &&
+        executions.to_i < retry_policy.attempts
     end
 
     def transient_error?(error)
@@ -142,6 +155,28 @@ module SearchEngine
         SearchEngine::Indexer::RetryPolicy.transient_status?(error.status.to_i)
     end
 
+    def summary_failed?(summary)
+      value = if summary.respond_to?(:failed_total)
+                summary.failed_total
+              elsif summary.is_a?(Hash)
+                summary[:failed_total] || summary['failed_total']
+              end
+      value.to_i.positive?
+    end
+
+    def raise_for_failed_summary!(summary, run_store, run_id, partition_key)
+      return unless summary_failed?(summary)
+
+      error = SearchEngine::Errors::PartitionImportFailed.new(summary)
+      run_store&.record_attempt(
+        run_id: run_id,
+        partition_key: partition_key,
+        summary: summary,
+        error: error
+      )
+      raise error
+    end
+
     def error_payload(error)
       {
         collection: arguments_dig_collection,
@@ -174,8 +209,8 @@ module SearchEngine
 
     def mark_failed_for_run(run_store, run_id, partition_key, error, payload:)
       run_store.mark_failed(run_id: run_id, partition_key: partition_key, error: error)
-    rescue SearchEngine::IndexingRunStore::StaleRun => stale_error
-      instrument_stale_run(stale_error, payload: payload, run_id: run_id, partition_key: partition_key)
+    rescue SearchEngine::IndexingRunStore::StaleRun => error
+      instrument_stale_run(error, payload: payload, run_id: run_id, partition_key: partition_key)
     end
 
     def instrument_event(event, payload)

data/app/search_engine/search_engine/postgres_outbox/drain_job.rb

@@ -104,9 +104,9 @@ module SearchEngine
       end
 
       def drainer_for(target_key)
-        return SearchEngine::PostgresOutbox::Drainer.new if target_key.nil?
+        return SearchEngine::PostgresOutbox::Drainer.new(worker_id: worker_id) if target_key.nil?
 
-        SearchEngine::PostgresOutbox::Drainer.new(target_key: target_key)
+        SearchEngine::PostgresOutbox::Drainer.new(target_key: target_key, worker_id: worker_id)
       end
 
       def enqueue_continuation(limit:, target_key:)
@@ -163,6 +163,7 @@ module SearchEngine
           superseded: 0,
           retryable: 0,
           failed: 0,
+          stale: 0,
           collections: [],
           target_key: target_key,
           drain_slot: drain_slot
@@ -174,7 +175,7 @@ module SearchEngine
       end
 
       def merge_batch_summary!(summary, batch_summary)
-        %i[claimed processed superseded retryable failed].each do |key|
+        %i[claimed processed superseded retryable failed stale].each do |key|
           summary[key] += batch_summary[key].to_i
         end
         summary[:collections] |= Array(batch_summary[:collections])

data/lib/generators/search_engine/install/templates/initializer.rb.tt

@@ -160,6 +160,24 @@ SearchEngine.configure do |c|
   # Retention: how many previous physical collections to keep after swap. Default: 0
   # c.schema.retention.keep_last = 0
 
+  # Optional guard around full physical create/index/alias-swap/retention rebuilds and alias rollbacks.
+  # Default: nil. In-place schema updates do not invoke it. The callable must yield exactly once.
+  # Delivery-target hosts can use Repository#with_delivery_target_claims_paused here; every direct
+  # Typesense writer must also use Repository#with_delivery_target_writes_allowed with the same target key.
+  # c.schema.around_rebuild = lambda do |collection:, &rebuild|
+  #   target_key = ENV.fetch("TYPESENSE_DELIVERY_TARGET_KEY")
+  #   SearchEngine::PostgresOutbox::Repository.new.with_delivery_target_claims_paused(
+  #     target_key: target_key,
+  #     timeout_s: SearchEngine.config.postgres_outbox.processing_timeout_s + 60,
+  #     &rebuild
+  #   )
+  # end
+
+  # When searches use Typesense's server-side cache, opt in to clearing it
+  # after each successfully processed outbox collection group and before the
+  # group is acknowledged. Default: false.
+  # c.postgres_outbox.clear_cache_after_write = false
+
   # --- Indexer -------------------------------------------------------------
   # See: https://nikita-shkoda.mintlify.app/projects/search-engine-for-typesense/v30.1/indexer
 
@@ -175,7 +193,8 @@ SearchEngine.configure do |c|
   # Gzip JSONL payloads during import. Default: false
   # c.indexer.gzip = false
 
-  # Dispatch mode for import jobs (:active_job or :inline). Default depends on ActiveJob presence
+  # Dispatch mode for import jobs (:active_job or :inline). Default depends on ActiveJob presence.
+  # Unsupported values fail configuration validation; :active_job requires ActiveJob.
   # c.indexer.dispatch = :active_job
 
   # Queue name for ActiveJob dispatch. Default: 'search_index'
@@ -196,6 +215,8 @@ SearchEngine.configure do |c|
 
   # Operational metadata for async partition runs. Defaults shown.
   # Use shared Rails.cache or provide a custom store visible to workers and parent.
+  # Custom stores must implement the IndexingRunStore contract, including the
+  # non-terminal `record_attempt(run_id:, partition_key:, summary:, error:)` transition.
   # c.indexer.partition_run_store = nil
   # c.indexer.partition_run_ttl_s = 86_400
 

data/lib/generators/search_engine/model/model_generator.rb

@@ -48,12 +48,14 @@ module SearchEngine
         return [] if raw.nil?
 
         tokens = raw.split(/[\s,]+/).map(&:strip).reject(&:empty?)
-        tokens.map do |pair|
+        tokens.filter_map do |pair|
           name, type = pair.split(':', 2)
           raise Thor::Error, "invalid attribute token: #{pair.inspect} (expected name:type)" unless name
 
           type = (type || 'string').to_s
           normalized = normalize_type(type)
+          next if name.to_s.underscore == 'id'
+
           [name.to_s.underscore, normalized]
         end
       end

data/lib/generators/search_engine/model/templates/model.rb.tt

@@ -7,6 +7,7 @@
 # - https://nikita-shkoda.mintlify.app/projects/search-engine-for-typesense/v30.1/vector-search
 class SearchEngine::<%= class_name %> < SearchEngine::Base
   collection "<%= @collection_name %>"
+  identify_by :id
 <% Array(@attributes).each do |(name, type)| -%>
   attribute :<%= name %>, :<%= type %>
 <% end -%>

data/lib/search_engine/active_record_syncable.rb

@@ -265,8 +265,8 @@ module SearchEngine
       end
 
       if timing == :after_commit
-        ar_klass.after_create_commit :__se_syncable_upsert! if actions.include?(:create)
-        ar_klass.after_update_commit :__se_syncable_upsert! if actions.include?(:update)
+        upsert_actions = actions & %i[create update]
+        ar_klass.after_commit :__se_syncable_upsert!, on: upsert_actions unless upsert_actions.empty?
         ar_klass.after_destroy_commit :__se_syncable_delete! if actions.include?(:destroy)
       else
         ar_klass.after_create :__se_syncable_upsert! if actions.include?(:create)

data/lib/search_engine/async_partition_coordinator.rb

@@ -113,9 +113,12 @@ module SearchEngine
     def timeout_result(store, run_id, snapshot)
       mark_non_terminal_failed(store, run_id, snapshot)
       snapshot = store.snapshot(run_id: run_id) || snapshot
+      current_result = SearchEngine::IndexingRun.aggregate_result(snapshot)
+      return finish_success(snapshot, current_result) if current_result[:status] == :ok
+
       result = failed_result(
         snapshot,
-        SearchEngine::IndexingRun.aggregate_result(snapshot),
+        current_result,
         sample_error: "SearchEngine async partition indexing timed out for run #{run_id}"
       )
       instrument('search_engine.indexing_run.failed', event_payload(snapshot, result: result))
@@ -136,6 +139,9 @@ module SearchEngine
           partition_key: partition_key,
           error: 'partition did not finish before timeout'
         )
+      rescue SearchEngine::IndexingRunStore::StaleRun
+        # The partition completed between the snapshot and terminal transition.
+        # Re-read the run after this pass instead of overwriting terminal success.
       end
     end
     private_class_method :mark_non_terminal_failed