engram 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 412e5e9bcb45b4889a24f5b6739a476b84057c3a62aca2bd767856cd4f725e3e
-  data.tar.gz: 26b9be259f6e937ba91a432bfa12f566c65a721ea49fefb7fa0a057c9fa435af
+  metadata.gz: 8a38dfd456e4e89f37a9d4d8949a5bb77b9736d95621e9172ba1db72fb3a5771
+  data.tar.gz: 0b537ee13bea0313fd770c529024af45e9ace8fb84f7e49621c2dc7b9d6b888c
 SHA512:
-  metadata.gz: fcd14fc54223897ed9d342ee574e279af725507b7653ff7545955f9b15e2815962edb40fdbf1c018fb99813fab84229ad061249fc666f9ff398f7826dede4da0
-  data.tar.gz: d27dfc3039f3c2e4dcd466be5c1481036dc823e6e1760e966eb8bd76dca4ee22ed2e71eae503ab2355a9696a05b000bf014efc6798e7b344ae4f975db48854f3
+  metadata.gz: 768d5d18a9ce04f60db58957127df76fbdc30482df4976d7d04b082af005850ba821ba45a35417c33946668af60331e869cf930e6239409ba10168a453f58665
+  data.tar.gz: 7ec7997ba40947deede93b6e970a549bf6caae32eed79f0e4c3922ebecbf05bc1545b5565afc521c9dab6c0f95b27898b3ae9bd9f3f57d9935a48d18e9211322

data/CHANGELOG.md

@@ -5,6 +5,63 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [0.5.0] - 2026-07-17
+
+### Added
+- Embedding provenance metadata is stored with new memories so applications can detect model
+  and dimension drift during store search result validation.
+- `Memory#rebuild_embeddings` and `Engram::UseCases::RebuildEmbeddings` for scoped,
+  deterministic rebuilding of stale vectors plus provenance metadata.
+- Added a focused rake task `engram:rebuild_embeddings` with batch control and optional
+  forced full-scope rebuild mode for recovery after provider/model changes. The task is
+  packaged with the gem and loaded into host Rails apps by the Railtie, where it depends on
+  `:environment` so app initializers run first. `STALE_ONLY` accepts `false`, `0`, or `no`.
+
+### Changed
+- `MemoryStore` mutations now require an explicit `scope:` and enforce the `(scope, id)` boundary
+  for update, delete, and touch operations. Custom stores must adopt the new scoped signatures;
+  `update` returns the updated record or raises `Engram::Error`, while delete and touch return an
+  affected-row count.
+- **Breaking (pre-1.0):** the `ProcessedTurns` port migrated from the previous check/mark API
+  to `claim`, `complete`, `release`, and `completed?`; custom adapters must implement the new
+  claim lifecycle. A successful claim returns a truthy opaque token.
+- Observation idempotency now uses atomic, scope-aware claims with an in-progress lease and
+  completed state. `InMemoryProcessedTurns` is thread-safe and releases failures immediately.
+  Generic cache release is a no-op until lease expiry because ActiveSupport cache has no atomic
+  compare-and-delete; completion never deletes a possibly newer claim. Completed markers
+  suppress later calls only for their configured `ttl`. A failed completion write leaves the
+  lease to suppress work until expiry, after which already-applied work may replay. `lease_ttl`
+  should cover the longest observation while remaining much shorter than `ttl`, and
+  `Rails::CacheProcessedTurns` uses atomic `unless_exist` writes with separate claim and
+  completion keys. This coordinates concurrent work but does not make multi-decision memory
+  persistence and claim completion a crash-proof transaction. Lease expiry may permit overlap,
+  so claims do not guarantee ownership, fencing, or exactly-once execution.
+- Added API stability and migration posture documentation for pre-1.0 freeze planning, including public surface boundaries and legacy compatibility points.
+- Store search result validation now raises a clear `Engram::Error` when stored embedding
+  metadata or vector dimensions conflict with the active embedder, while legacy records without
+  metadata remain searchable when their vector dimensions match the active embedder.
+- Caller metadata keys named `_engram` are now reserved for Engram-owned embedding provenance;
+  rename any application metadata stored under that key before adding new memories.
+- `RubyLLMEmbedder` now requests explicitly configured `dimensions:` from the provider (on
+  RubyLLM versions that support the option) and validates every returned vector against the
+  configured dimensions, raising a clear `Engram::Error` on mismatch. Previously the option
+  was recorded in metadata but never sent or checked, so a mismatch surfaced later as opaque
+  pgvector insert failures. If you use a model whose native vector size differs from 1536,
+  set `dimensions:` to that model's actual output size.
+
+### Fixed
+- `Observe` now raises `Engram::ObservationInProgressError` when a turn is claimed but not
+  completed, rather than reporting a successful no-op. `ObserveJob` retries this error with
+  backoff that outlasts the default claim lease; direct callers should handle it as retryable.
+- Stale detection in `rebuild_embeddings` now compares against the embedder's declared
+  dimensions instead of the stored record's vector length, so a dimensions-only embedder
+  change marks existing rows stale. Previously the default stale-only rebuild silently
+  skipped every row after such a change. Embedders that do not declare dimensions keep the
+  previous record-length comparison.
+- `LLMConsolidator` now ignores malformed decisions and rejects `update`/`forget` targets
+  that were not shown to the model, preventing invalid model output from partially applying
+  a turn. Cross-scope updates from custom consolidators still raise in `Observe`.
+
 ## [0.4.0] - 2026-06-06
 
 ### Added

data/README.md

@@ -61,6 +61,41 @@ chat.ask("Why am I being rate limited?")
 - Persistence policy that rejects obvious secrets and transient task-progress updates before storage.
 - Idempotent observation, recency/importance-aware ranking, recall touching, and stale-memory pruning.
 
+
+## API stability and migration posture
+
+Engram is intentionally marked pre-1.0; default behavior is to prioritize safety and compatibility, while still allowing occasional focused API additions.
+
+### Current public surface
+
+The following are part of the documented public surface and should remain stable except where
+explicitly version-gated:
+
+- Core types and facade: `Engram::Memory`, `Engram::Record`, `Engram::Decision`, `Engram::PersistencePolicy`, and `Engram.with_memory`.
+- Store and adapter ports: `Engram::Ports::MemoryStore`, `Engram::Ports::Embedder`, `Engram::Ports::Completion`.
+- Rails integration points: `has_memory`, `Memory#observe_later`, and generator outputs under `engram:` rake tasks.
+- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget_stale`, and `rebuild_embeddings`.
+- RubyLLM adapter contract points and evaluator entrypoints (`rake eval`, `rake eval:real`).
+
+### Backward-compatibility commitments (pre-1.0)
+
+- Compatibility adjustments should be additive where possible.
+- Behavioral changes that could break callers should be documented under `CHANGELOG.md` and tested with migration scenarios.
+- Legacy compatibility notes already in effect:
+  - `kind: "semantic"` is normalized to `:fact` for read paths.
+  - Existing rows created without embedding provenance are still readable when dimensions match the active embedder.
+
+### 1.0 pre-freeze checklist
+
+Before the 1.0 release marker, freeze the public API by:
+
+1. Finalizing configuration and initializer keys in docs and examples.
+2. Writing upgrade notes for any remaining behavior-shifting defaults.
+3. Verifying migrations/rebuild flows for legacy rows remain observable and recoverable.
+4. Keeping security and persistence-policy boundaries explicit in host-app guidance.
+
+Use `CHANGELOG.md` as the authoritative source for breaking/compatibility changes while still in pre-1.0.
+
 ## Installation
 
 ```ruby
@@ -112,15 +147,23 @@ current_user.memory.observe_later([
 ```
 
 `observe_later` uses ActiveJob, so configure the queue adapter you already use in
-production (Sidekiq, Solid Queue, GoodJob, etc.). For idempotency across retries and
-processes, use the Rails cache-backed processed-turn store:
+production (Sidekiq, Solid Queue, GoodJob, etc.). To coordinate observation claims across
+retries and processes, use the Rails cache-backed processed-turn store with a shared cache
+whose `write(..., unless_exist: true)` operation is atomic (for example Redis or Solid Cache):
 
 ```ruby
 Engram.configure do |config|
-  config.processed_turns = Engram::Rails::CacheProcessedTurns.new
+  config.processed_turns = Engram::Rails::CacheProcessedTurns.new(lease_ttl: 5.minutes)
 end
 ```
 
+The adapter rejects backends that do not return a boolean result for `unless_exist`, but it
+cannot detect a backend that accepts the option without implementing it atomically. Rails'
+memory cache coordinates threads in one process only, and `NullStore` is not suitable.
+Completed suppression is bounded by `ttl` (24 hours by default). Set `lease_ttl` longer than
+the longest expected observation, but normally much shorter than `ttl`, so crashed work can
+be retried promptly without weakening the completed-turn suppression window.
+
 ## Postgres + pgvector setup
 
 The Rails generator creates an `engram_memories` table with a `vector` extension and a
@@ -153,7 +196,9 @@ If you change embedding models, keep the database column dimension in sync with
 embedding vector length. A model that returns 768-dimensional vectors needs a 768-dimensional
 `vector` column; a 1536-dimensional migration will not be compatible with it. The install
 generator rejects non-positive or non-integer `--dimensions` values so an invalid vector
-size does not land in a migration.
+size does not land in a migration. `RubyLLMEmbedder` requests explicitly configured
+`dimensions:` from the provider (models like `text-embedding-3-*` support shortening) and
+raises a clear error when the model's output does not match the configured dimensions.
 
 For production recall performance, add one approximate vector index after the table has
 representative data. HNSW is the recommended default for read-heavy applications because it
@@ -317,9 +362,28 @@ with the old `semantic` kind value.
 
 ## Tuning and maintenance
 
-Observation is idempotent per turn: observing the same messages twice does nothing the
-second time, so retries do not create duplicate memories or repeat LLM calls. In Rails,
-use a persistent processed-turn store so this also holds across job retries and processes.
+Observation uses a scope-and-turn claim before extraction. While a claim lease is live and
+the turn has not completed, calls for the same scope and turn raise
+`Engram::ObservationInProgressError` instead of reporting success without doing the work; a
+completed marker suppresses later calls (returning no decisions) until its configured `ttl`
+expires. `ObserveJob` retries `ObservationInProgressError` with polynomial backoff that
+outlasts the default lease. Direct `observe` callers should also treat the error as retryable.
+The in-memory adapter releases failures immediately. Generic Rails cache release
+is deliberately a no-op until lease expiry because ActiveSupport cache has no atomic
+compare-and-delete. In Rails, use a shared cache with atomic `unless_exist` writes for
+cross-process coordination. Set `lease_ttl` longer than the longest expected observation but
+much shorter than the completed-marker `ttl`; after a worker crash (or an overlong
+observation), the lease expires so a retry can proceed.
+
+Lease expiry can permit old and new workers to overlap; claims are not fencing tokens or an
+ownership guarantee. Successful work records completion but cannot safely delete a possibly
+newer cache claim. This is idempotency coordination, not crash-proof exactly-once persistence. An observation
+can apply multiple decisions, and the memory writes plus completed marker are not one
+transaction. A crash, cache outage, or lease expiry between those operations can permit a
+retry after some decisions were already written. Applications needing stronger guarantees
+must supply transactional persistence/outbox coordination appropriate to their store.
+If writing the completed marker fails after memory writes succeed, the current lease still
+suppresses retries until it expires; after expiry the turn may replay and repeat those writes.
 
 Recall is plain similarity search by default. You can blend in importance and recency:
 
@@ -338,6 +402,27 @@ Prune memories you no longer need:
 current_user.memory.forget_stale(older_than: 90 * 24 * 60 * 60, min_importance: 0.7)
 ```
 
+If you change embedder configuration (model/provider/dimensions), or if legacy memories were
+written without embedding provenance, run a scoped rebuild to refresh vectors and metadata:
+
+```bash
+bundle exec rake "engram:rebuild_embeddings[user:42]"
+```
+
+The task ships with the gem and is loaded by the Railtie, so it is available inside Rails
+applications and boots the app environment (initializers included) before running. Outside
+Rails, call `memory.rebuild_embeddings` directly.
+
+By default, only stale rows are rewritten. Set `STALE_ONLY=false` to rebuild all rows in
+the scope, and `BATCH_SIZE=<n>` to tune write size.
+
+The rebuild has no dry-run mode and updates each row as it goes. It assumes the store exposes
+a stable, finite, scope-isolated traversal; malformed adapters that repeat forever cannot be
+made deterministic by the rebuild. Rows without IDs are counted and skipped. Choose a batch
+size that fits provider and database limits, take a backup first, and monitor provider usage,
+errors, and database load. The final summary reports processed, updated, skipped, and failed
+counts; failures also list record IDs and cause the rake task to exit unsuccessfully.
+
 ## Observability
 
 When ActiveSupport is loaded, Engram emits `ActiveSupport::Notifications` events for the
@@ -478,8 +563,9 @@ gem unpack engram-*.gem --target /tmp/engram-package-check
 - v0.1 (done): recall + inject foundation, adapters, Rails + RubyLLM integration.
 - v0.2 (done): extract and consolidate (ADD / UPDATE / FORGET), background jobs.
 - v0.3 (done): idempotent observation, importance/recency recall, forgetting and decay.
-- v0.4 (in progress): memory kinds, persistence policy, typed recall filters, safer injection, and release-readiness docs.
-- later: real-provider eval ergonomics, additional storage backends, observability hooks, and larger eval benchmarks.
+- v0.4 (done): memory kinds, persistence policy, typed recall filters, safer injection, and observability hooks.
+- v0.5 (in progress): embedding provenance and scoped embedding rebuild operations.
+- later: additional storage backends and larger real-provider eval benchmarks.
 
 ## License
 

data/lib/engram/adapters/in_memory_processed_turns.rb

@@ -8,21 +8,53 @@ module Engram
     class InMemoryProcessedTurns
       include Ports::ProcessedTurns
 
-      def initialize
-        @keys = Set.new
+      def initialize(lease_ttl: 300, clock: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) })
+        @lease_ttl = lease_ttl
+        @clock = clock
+        @claims = {}
+        @completed = Set.new
+        @mutex = Mutex.new
       end
 
-      def seen?(key)
-        @keys.include?(key)
+      def claim(scope:, key:)
+        scoped_key = [scope, key]
+        @mutex.synchronize do
+          return if @completed.include?(scoped_key)
+          live = @claims[scoped_key]
+          return if live && live.last > @clock.call
+
+          token = Object.new
+          @claims[scoped_key] = [token, @clock.call + @lease_ttl]
+          token
+        end
+      end
+
+      def complete(scope:, key:, claim:)
+        @mutex.synchronize do
+          scoped_key = [scope, key]
+          return false unless @claims.dig(scoped_key, 0).equal?(claim)
+          @completed << scoped_key
+          @claims.delete(scoped_key)
+          true
+        end
+      end
+
+      def release(scope:, key:, claim:)
+        @mutex.synchronize do
+          scoped_key = [scope, key]
+          @claims.delete(scoped_key) if @claims.dig(scoped_key, 0).equal?(claim)
+        end
       end
 
-      def record(key)
-        @keys << key
-        key
+      def completed?(scope:, key:)
+        @mutex.synchronize { @completed.include?([scope, key]) }
       end
 
       def clear
-        @keys.clear
+        @mutex.synchronize do
+          @claims.clear
+          @completed.clear
+        end
       end
     end
   end

data/lib/engram/adapters/in_memory_store.rb

@@ -20,37 +20,52 @@ module Engram
         record
       end
 
-      def search(embedding:, scope:, limit:, kinds: nil)
+      def search(embedding:, scope:, limit:, kinds: nil, embedding_metadata: nil)
+        Engram::EmbeddingMetadata.validate_query!(embedding, embedding_metadata)
         allowed_kinds = normalize_kinds(kinds)
 
-        @records
+        results = @records
           .values
           .select { |r| searchable?(r, scope, allowed_kinds) }
           .map { |r| [r, Engram::Math.cosine_similarity(embedding, r.embedding)] }
           .sort_by { |(_, score)| -score }
           .first(limit)
           .map { |(record, _)| record }
+
+        results.each { |record| Engram::EmbeddingMetadata.validate_record!(record, embedding, embedding_metadata) }
+        results
       end
 
-      def all(scope:)
-        @records.values.select { |r| r.scope == scope }
+      def all(scope:, limit: nil, offset: 0, after_id: nil)
+        records = @records.values.select { |r| r.scope == scope }.sort_by { |record| record.id }
+        records = records.drop_while { |record| !after_id.nil? && record.id && record.id <= after_id }
+        records = records.drop(offset) if offset > 0
+        records = records.take(limit) if limit
+        records
       end
 
-      def update(id:, record:)
-        raise Engram::Error, "no memory with id #{id.inspect}" unless @records.key?(id)
+      def update(scope:, id:, record:)
+        existing = @records[id]
+        raise Engram::Error, "no memory with id #{id.inspect} in scope #{scope.inspect}" unless existing&.scope == scope
+        raise Engram::Error, "cannot move memory across scopes" unless record.scope == scope
 
         record.id = id
         @records[id] = record
       end
 
-      def delete(id:)
+      def delete(scope:, id:)
+        return 0 unless @records[id]&.scope == scope
+
         @records.delete(id)
+        1
       end
 
-      def touch(id:, at: Time.now)
+      def touch(scope:, id:, at: Time.now)
         record = @records[id]
-        record.last_accessed_at = at if record
-        record
+        return 0 unless record&.scope == scope
+
+        record.last_accessed_at = at
+        1
       end
 
       def clear

data/lib/engram/adapters/null_embedder.rb

@@ -16,6 +16,15 @@ module Engram
 
       attr_reader :dimensions
 
+      def embedding_metadata
+        Engram::EmbeddingMetadata.build(
+          adapter: self.class.name,
+          provider: "null",
+          model: "null-embedder-v1",
+          dimensions: dimensions
+        )
+      end
+
       def embed(text)
         seed = Digest::SHA256.hexdigest(text.to_s)
         Array.new(@dimensions) do |i|

data/lib/engram/adapters/pgvector_store.rb

@@ -28,7 +28,8 @@ module Engram
         to_record(row)
       end
 
-      def search(embedding:, scope:, limit:, kinds: nil)
+      def search(embedding:, scope:, limit:, kinds: nil, embedding_metadata: nil)
+        Engram::EmbeddingMetadata.validate_query!(embedding, embedding_metadata)
         query = model.where(scope: scope)
         normalized_kinds = normalize_kinds(kinds)
         query = query.where(kind: normalized_kinds) if normalized_kinds
@@ -37,30 +38,41 @@ module Engram
           .nearest_neighbors(:embedding, embedding, distance: "cosine")
           .limit(limit)
           .map { |row| to_record(row) }
+          .tap { |records| validate_query_records!(records, embedding, embedding_metadata) }
       end
 
-      def all(scope:)
-        model.where(scope: scope).map { |row| to_record(row) }
+      def all(scope:, limit: nil, offset: 0, after_id: nil)
+        query = model.where(scope: scope).order(:id)
+        query = query.where("id > ?", after_id) if after_id
+        query = query.limit(limit) if limit
+        query = query.offset(offset) if offset > 0
+        query.map { |row| to_record(row) }
       end
 
-      def update(id:, record:)
-        row = model.find(id)
-        row.update!(
-          content: record.content,
-          kind: record.kind.to_s,
-          importance: record.importance,
-          metadata: record.metadata,
-          embedding: record.embedding
-        )
-        to_record(row)
+      def update(scope:, id:, record:)
+        raise Engram::Error, "cannot move memory across scopes" unless record.scope == scope
+
+        model.transaction do
+          row = model.lock.find_by!(id: id, scope: scope)
+          row.update!(
+            content: record.content,
+            kind: record.kind.to_s,
+            importance: record.importance,
+            metadata: record.metadata,
+            embedding: record.embedding
+          )
+          to_record(row)
+        end
+      rescue ActiveRecord::RecordNotFound
+        raise Engram::Error, "no memory with id #{id.inspect} in scope #{scope.inspect}"
       end
 
-      def delete(id:)
-        model.where(id: id).delete_all
+      def delete(scope:, id:)
+        model.where(id: id, scope: scope).delete_all
       end
 
-      def touch(id:, at: Time.now)
-        model.where(id: id).update_all(last_accessed_at: at)
+      def touch(scope:, id:, at: Time.now)
+        model.where(id: id, scope: scope).update_all(last_accessed_at: at)
       end
 
       private
@@ -69,6 +81,12 @@ module Engram
         raise Engram::Error, "memory scope cannot be nil" if scope.nil?
       end
 
+      def validate_query_records!(records, embedding, embedding_metadata)
+        records.each do |record|
+          Engram::EmbeddingMetadata.validate_record!(record, embedding, embedding_metadata)
+        end
+      end
+
       def model
         @model ||= resolve_default_model
       end

data/lib/engram/adapters/ruby_llm_embedder.rb

@@ -10,16 +10,32 @@ module Engram
       DEFAULT_MODEL = "text-embedding-3-small"
       DEFAULT_DIMENSIONS = 1536
 
-      def initialize(model: DEFAULT_MODEL, dimensions: DEFAULT_DIMENSIONS)
+      def initialize(model: DEFAULT_MODEL, dimensions: nil)
+        if dimensions && (!dimensions.is_a?(Integer) || !dimensions.positive?)
+          raise ArgumentError, "dimensions must be a positive integer"
+        end
+
         @model = model
-        @dimensions = dimensions
+        @requested_dimensions = dimensions
+        @dimensions = dimensions || DEFAULT_DIMENSIONS
       end
 
-      attr_reader :dimensions
+      attr_reader :dimensions, :model
+
+      def embedding_metadata
+        Engram::EmbeddingMetadata.build(
+          adapter: self.class.name,
+          provider: "ruby_llm",
+          model: model,
+          dimensions: dimensions
+        )
+      end
 
       def embed(text)
         ensure_ruby_llm!
-        RubyLLM.embed(text, model: @model).vectors
+        vectors = request_embedding(text)
+        validate_dimensions!(vectors)
+        vectors
       end
 
       private
@@ -30,6 +46,31 @@ module Engram
         raise Engram::Error,
           "RubyLLMEmbedder requires the `ruby_llm` gem. Add it to your Gemfile and configure it."
       end
+
+      # Older RubyLLM versions do not accept the dimensions keyword.
+      def request_embedding(text)
+        if @requested_dimensions && embed_accepts_dimensions?
+          RubyLLM.embed(text, model: @model, dimensions: @requested_dimensions).vectors
+        else
+          RubyLLM.embed(text, model: @model).vectors
+        end
+      end
+
+      def embed_accepts_dimensions?
+        RubyLLM.method(:embed).parameters.any? do |type, name|
+          type == :keyrest || ([:key, :keyreq].include?(type) && name == :dimensions)
+        end
+      end
+
+      def validate_dimensions!(vectors)
+        length = vectors.respond_to?(:length) ? vectors.length : nil
+        return if length == @dimensions
+
+        raise Engram::Error,
+          "embedding model #{@model.inspect} returned #{length ? "#{length}-dimension" : "non-vector"} output " \
+            "but the embedder is configured with dimensions: #{@dimensions}; align the dimensions option " \
+            "(and your vector column) with the model's output"
+      end
     end
   end
 end

data/lib/engram/consolidators/heuristic_consolidator.rb

@@ -15,7 +15,13 @@ module Engram
 
       def reconcile_all(candidates:, scope:)
         Array(candidates).map do |candidate|
-          nearest = @store.search(embedding: candidate.embedding, scope: scope, limit: 1).first
+          nearest = Engram::EmbeddingMetadata.search(
+            @store,
+            embedding: candidate.embedding,
+            embedding_metadata: Engram::EmbeddingMetadata.extract(candidate.metadata),
+            scope: scope,
+            limit: 1
+          ).first
           similarity = nearest ? Engram::Math.cosine_similarity(candidate.embedding, nearest.embedding) : 0.0
 
           if nearest && similarity >= @similarity_threshold

data/lib/engram/consolidators/llm_consolidator.rb

@@ -56,23 +56,35 @@ module Engram
         candidates = Array(candidates)
         return [] if candidates.empty?
 
+        neighbors = neighbor_map(candidates, scope)
         result = @completion.complete(
           system: SYSTEM,
-          user: JSON.generate(payload(candidates, scope)),
+          user: JSON.generate(payload(candidates, neighbors)),
           schema: SCHEMA
         )
-        map_decisions(decisions(result), candidates)
+        map_decisions(decisions(result), candidates, neighbors)
       end
 
       private
 
-      def payload(candidates, scope)
+      def neighbor_map(candidates, scope)
+        candidates.map do |candidate|
+          Engram::EmbeddingMetadata.search(
+            @store,
+            embedding: candidate.embedding,
+            embedding_metadata: Engram::EmbeddingMetadata.extract(candidate.metadata),
+            scope: scope,
+            limit: @neighbors
+          )
+        end
+      end
+
+      def payload(candidates, neighbors)
         items = candidates.each_with_index.map do |candidate, index|
-          existing = @store.search(embedding: candidate.embedding, scope: scope, limit: @neighbors)
           {
             index: index,
             candidate: candidate.content,
-            existing: existing.map { |r| {id: r.id, content: r.content} }
+            existing: neighbors[index].map { |r| {id: r.id, content: r.content} }
           }
         end
         {candidates: items}
@@ -81,19 +93,34 @@ module Engram
       def decisions(result)
         return [] unless result.is_a?(Hash)
 
-        result["decisions"] || result[:decisions] || []
+        decisions = result["decisions"] || result[:decisions] || []
+        decisions.is_a?(Array) ? decisions : []
       end
 
-      def map_decisions(raw, candidates)
+      def map_decisions(raw, candidates, neighbors)
+        seen = Set.new
         raw.filter_map do |decision|
+          next unless decision.is_a?(Hash)
+
           decision = decision.transform_keys(&:to_s)
           index = decision["index"]
-          next unless index && candidates[index]
+          next unless index.is_a?(Integer) && index >= 0 && candidates[index]
+          next if seen.include?(index)
+
+          action = (decision["action"] || "noop").to_s
+          next unless Engram::Decision::ACTIONS.include?(action.to_sym)
+
+          target_id = decision["target_id"]
+          if %w[update forget].include?(action)
+            next if target_id.nil?
+            next unless neighbors[index].any? { |neighbor| neighbor.id == target_id }
+          end
 
+          seen.add(index)
           Engram::Decision.new(
-            action: (decision["action"] || "noop").to_sym,
+            action: action.to_sym,
             candidate: candidates[index],
-            target_id: decision["target_id"],
+            target_id: target_id,
             reason: decision["reason"]
           )
         end