engram 0.4.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 412e5e9bcb45b4889a24f5b6739a476b84057c3a62aca2bd767856cd4f725e3e
-  data.tar.gz: 26b9be259f6e937ba91a432bfa12f566c65a721ea49fefb7fa0a057c9fa435af
+  metadata.gz: c623f8198a9f71905e6e0c904b6cf60d5d6a0734b1e8fdaf877611a815f2a919
+  data.tar.gz: aca21e3da50239a5d02822306524adc6647883ef607af57747b05ce7ecb78827
 SHA512:
-  metadata.gz: fcd14fc54223897ed9d342ee574e279af725507b7653ff7545955f9b15e2815962edb40fdbf1c018fb99813fab84229ad061249fc666f9ff398f7826dede4da0
-  data.tar.gz: d27dfc3039f3c2e4dcd466be5c1481036dc823e6e1760e966eb8bd76dca4ee22ed2e71eae503ab2355a9696a05b000bf014efc6798e7b344ae4f975db48854f3
+  metadata.gz: 55f5e8335a86ec28cd28b17af162883fcc7c58dc56b903fc18b924c4278b593d9c8945cbf19cb28649a4c66de57ff38e6af46f3c50b15a6a96a8d729b90fc6ba
+  data.tar.gz: 519081acdc6f1b284df51bc664d274ca31befbc2d7a002ef9d62fee2f31c165f64c2896ab5f2dddca771b02c2052dcb630ef785e440776931d53d7f994de6e47

data/CHANGELOG.md

@@ -5,6 +5,119 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [0.6.0] - 2026-07-24
+
+### Added
+- `Engram::Memory#grounding_report` (and `Engram::UseCases::GroundingReport`) returns frozen,
+  scope-bound record counts by weakest source alignment, with legacy or unrecognized provenance
+  counted as unattributed and no record or source text exposed.
+- Add an immutable host-supplied source-text contract that validates provenance source identity
+  and Unicode-codepoint span bounds, and resolves authorized supporting text without retaining
+  source transcripts.
+- Optional `Engram::Extraction` results let custom extractors attach versioned source
+  provenance while remaining compatible with plain `Engram::Record` results.
+- `Engram::Record#provenance` exposes understood supporting source IDs, alignments, and spans
+  on recalled records while preserving tolerant reads for legacy and future schemas.
+- `Engram::Memory#memories_from_source` (and `Engram::UseCases::SourceImpact`) return the
+  records in a scope whose provenance references an exact host source. `source_id` and
+  `source_type` must each be a non-blank String and are matched exactly without trimming or
+  normalization. The lookup is scope-bound and returns only records, never source
+  text; source IDs are references, not an authorization boundary. Legacy, malformed, and
+  future-schema provenance do not match.
+
+### Changed
+- Persistence accepts records without provenance, rejects structurally ungrounded provenance
+  by default (configurable only with the exact boolean `allow_ungrounded: true`), and fails closed on malformed or
+  unknown future provenance during writes while keeping reads tolerant. Provenance validation
+  does not verify source text, and source IDs are references rather than authorization
+  boundaries. The built-in LLM extractor does not emit grounded provenance.
+- Existing `before_persist` hooks may continue to transform record content or embeddings, but
+  may neither add, remove, nor alter provenance; this restriction includes legacy records with
+  no provenance.
+- Extractors must return an `Array`; each result may be a plain `Engram::Record` or an
+  `Engram::Extraction` carrying provenance.
+- Custom consolidator decisions must reference the actual same-scope `Engram::Record` instance
+  supplied for reconciliation and must treat both the candidates array and records as read-only.
+  Engram now fails closed on collection replacement/reordering/iteration overrides and on
+  security-relevant nested value, identity/alias/topology, frozen-state, custom-behavior, or
+  hidden-state changes before destructive authorization. Arbitrary application metadata values
+  remain opaque and preserve their identity without invoking equality or serialization behavior;
+  provenance is independently parsed and integrity-protected.
+  Substituted, modified, missing, malformed, or cross-scope candidates fail closed.
+  `Observe` preflights the complete decision batch, including persistence policy and hook
+  transformations, before beginning store mutations.
+
+### Fixed
+- Provenance canonicalization now iterates untrusted metadata through core `Hash#keys`/`#values`
+  by position instead of `Hash#each_pair`. On Ruby 3.4 a caller-supplied hash left in a
+  delete-then-insert collision state could make `Hash#each_pair` invoke a key's `#eql?` mid-iteration,
+  letting a hostile `String`-subclass key run application code inside the behavior-free canonicalizer.
+  The adversarial hostile-key suite is now deterministic across Ruby 3.2–3.4.
+- Observation rejects extractor candidates with caller-supplied IDs, and `InMemoryStore#add`
+  always allocates a fresh ID like the pgvector adapter, preventing same- or cross-scope record
+  replacement through add semantics.
+- `forget` now separates destructive provenance authorization from write-content filtering and
+  redaction, so secret or transient memories can be deleted. Custom policies may implement
+  `allow_destructive?` with a strict boolean return; write-only policies no longer authorize
+  deletion through their `call` result.
+
+## [0.5.0] - 2026-07-17
+
+### Added
+- Embedding provenance metadata is stored with new memories so applications can detect model
+  and dimension drift during store search result validation.
+- `Memory#rebuild_embeddings` and `Engram::UseCases::RebuildEmbeddings` for scoped,
+  deterministic rebuilding of stale vectors plus provenance metadata.
+- Added a focused rake task `engram:rebuild_embeddings` with batch control and optional
+  forced full-scope rebuild mode for recovery after provider/model changes. The task is
+  packaged with the gem and loaded into host Rails apps by the Railtie, where it depends on
+  `:environment` so app initializers run first. `STALE_ONLY` accepts `false`, `0`, or `no`.
+
+### Changed
+- `MemoryStore` mutations now require an explicit `scope:` and enforce the `(scope, id)` boundary
+  for update, delete, and touch operations. Custom stores must adopt the new scoped signatures;
+  `update` returns the updated record or raises `Engram::Error`, while delete and touch return an
+  affected-row count.
+- **Breaking (pre-1.0):** the `ProcessedTurns` port migrated from the previous check/mark API
+  to `claim`, `complete`, `release`, and `completed?`; custom adapters must implement the new
+  claim lifecycle. A successful claim returns a truthy opaque token.
+- Observation idempotency now uses atomic, scope-aware claims with an in-progress lease and
+  completed state. `InMemoryProcessedTurns` is thread-safe and releases failures immediately.
+  Generic cache release is a no-op until lease expiry because ActiveSupport cache has no atomic
+  compare-and-delete; completion never deletes a possibly newer claim. Completed markers
+  suppress later calls only for their configured `ttl`. A failed completion write leaves the
+  lease to suppress work until expiry, after which already-applied work may replay. `lease_ttl`
+  should cover the longest observation while remaining much shorter than `ttl`, and
+  `Rails::CacheProcessedTurns` uses atomic `unless_exist` writes with separate claim and
+  completion keys. This coordinates concurrent work but does not make multi-decision memory
+  persistence and claim completion a crash-proof transaction. Lease expiry may permit overlap,
+  so claims do not guarantee ownership, fencing, or exactly-once execution.
+- Added API stability and migration posture documentation for pre-1.0 freeze planning, including public surface boundaries and legacy compatibility points.
+- Store search result validation now raises a clear `Engram::Error` when stored embedding
+  metadata or vector dimensions conflict with the active embedder, while legacy records without
+  metadata remain searchable when their vector dimensions match the active embedder.
+- Caller metadata keys named `_engram` are now reserved for Engram-owned embedding provenance;
+  rename any application metadata stored under that key before adding new memories.
+- `RubyLLMEmbedder` now requests explicitly configured `dimensions:` from the provider (on
+  RubyLLM versions that support the option) and validates every returned vector against the
+  configured dimensions, raising a clear `Engram::Error` on mismatch. Previously the option
+  was recorded in metadata but never sent or checked, so a mismatch surfaced later as opaque
+  pgvector insert failures. If you use a model whose native vector size differs from 1536,
+  set `dimensions:` to that model's actual output size.
+
+### Fixed
+- `Observe` now raises `Engram::ObservationInProgressError` when a turn is claimed but not
+  completed, rather than reporting a successful no-op. `ObserveJob` retries this error with
+  backoff that outlasts the default claim lease; direct callers should handle it as retryable.
+- Stale detection in `rebuild_embeddings` now compares against the embedder's declared
+  dimensions instead of the stored record's vector length, so a dimensions-only embedder
+  change marks existing rows stale. Previously the default stale-only rebuild silently
+  skipped every row after such a change. Embedders that do not declare dimensions keep the
+  previous record-length comparison.
+- `LLMConsolidator` now ignores malformed decisions and rejects `update`/`forget` targets
+  that were not shown to the model, preventing invalid model output from partially applying
+  a turn. Cross-scope updates from custom consolidators still raise in `Observe`.
+
 ## [0.4.0] - 2026-06-06
 
 ### Added

data/README.md

@@ -52,7 +52,7 @@ chat.ask("Why am I being rate limited?")
 
 ## Feature overview
 
-- Zero-dependency pure Ruby core with in-memory defaults for tests and local development.
+- Pure Ruby core with zero runtime dependencies and in-memory defaults for tests and local development.
 - Rails `has_memory` macro, install generator, and background `observe_later` job.
 - Postgres + pgvector storage through an optional ActiveRecord/neighbor adapter.
 - RubyLLM embedder and completion adapters for provider-backed embeddings and extraction.
@@ -61,6 +61,41 @@ chat.ask("Why am I being rate limited?")
 - Persistence policy that rejects obvious secrets and transient task-progress updates before storage.
 - Idempotent observation, recency/importance-aware ranking, recall touching, and stale-memory pruning.
 
+
+## API stability and migration posture
+
+Engram is intentionally marked pre-1.0; default behavior is to prioritize safety and compatibility, while still allowing occasional focused API additions.
+
+### Current public surface
+
+The following are part of the documented public surface and should remain stable except where
+explicitly version-gated:
+
+- Core types and facade: `Engram::Memory`, `Engram::Record`, `Engram::Decision`, `Engram::PersistencePolicy`, and `Engram.with_memory`.
+- Store and adapter ports: `Engram::Ports::MemoryStore`, `Engram::Ports::Embedder`, `Engram::Ports::Completion`.
+- Rails integration points: `has_memory`, `Memory#observe_later`, and generator outputs under `engram:` rake tasks.
+- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget_stale`, `rebuild_embeddings`, and `memories_from_source`.
+- RubyLLM adapter contract points and evaluator entrypoints (`rake eval`, `rake eval:real`).
+
+### Backward-compatibility commitments (pre-1.0)
+
+- Compatibility adjustments should be additive where possible.
+- Behavioral changes that could break callers should be documented under `CHANGELOG.md` and tested with migration scenarios.
+- Legacy compatibility notes already in effect:
+  - `kind: "semantic"` is normalized to `:fact` for read paths.
+  - Existing rows created without embedding provenance are still readable when dimensions match the active embedder.
+
+### 1.0 pre-freeze checklist
+
+Before the 1.0 release marker, freeze the public API by:
+
+1. Finalizing configuration and initializer keys in docs and examples.
+2. Writing upgrade notes for any remaining behavior-shifting defaults.
+3. Verifying migrations/rebuild flows for legacy rows remain observable and recoverable.
+4. Keeping security and persistence-policy boundaries explicit in host-app guidance.
+
+Use `CHANGELOG.md` as the authoritative source for breaking/compatibility changes while still in pre-1.0.
+
 ## Installation
 
 ```ruby
@@ -68,7 +103,7 @@ chat.ask("Why am I being rate limited?")
 gem "engram"
 ```
 
-The core has **zero runtime dependencies**. Optional adapters need host-app dependencies:
+The core has zero runtime dependencies and does not require Rails, ActiveRecord, or a database. Optional adapters need host-app dependencies:
 
 - `Engram::Adapters::PgvectorStore` → ActiveRecord + `neighbor` + Postgres/pgvector
 - `Engram::Adapters::RubyLLMEmbedder` and `Engram::Adapters::RubyLLMCompletion` → `ruby_llm`
@@ -112,15 +147,23 @@ current_user.memory.observe_later([
 ```
 
 `observe_later` uses ActiveJob, so configure the queue adapter you already use in
-production (Sidekiq, Solid Queue, GoodJob, etc.). For idempotency across retries and
-processes, use the Rails cache-backed processed-turn store:
+production (Sidekiq, Solid Queue, GoodJob, etc.). To coordinate observation claims across
+retries and processes, use the Rails cache-backed processed-turn store with a shared cache
+whose `write(..., unless_exist: true)` operation is atomic (for example Redis or Solid Cache):
 
 ```ruby
 Engram.configure do |config|
-  config.processed_turns = Engram::Rails::CacheProcessedTurns.new
+  config.processed_turns = Engram::Rails::CacheProcessedTurns.new(lease_ttl: 5.minutes)
 end
 ```
 
+The adapter rejects backends that do not return a boolean result for `unless_exist`, but it
+cannot detect a backend that accepts the option without implementing it atomically. Rails'
+memory cache coordinates threads in one process only, and `NullStore` is not suitable.
+Completed suppression is bounded by `ttl` (24 hours by default). Set `lease_ttl` longer than
+the longest expected observation, but normally much shorter than `ttl`, so crashed work can
+be retried promptly without weakening the completed-turn suppression window.
+
 ## Postgres + pgvector setup
 
 The Rails generator creates an `engram_memories` table with a `vector` extension and a
@@ -153,7 +196,9 @@ If you change embedding models, keep the database column dimension in sync with
 embedding vector length. A model that returns 768-dimensional vectors needs a 768-dimensional
 `vector` column; a 1536-dimensional migration will not be compatible with it. The install
 generator rejects non-positive or non-integer `--dimensions` values so an invalid vector
-size does not land in a migration.
+size does not land in a migration. `RubyLLMEmbedder` requests explicitly configured
+`dimensions:` from the provider (models like `text-embedding-3-*` support shortening) and
+raises a clear error when the model's output does not match the configured dimensions.
 
 For production recall performance, add one approximate vector index after the table has
 representative data. HNSW is the recommended default for read-heavy applications because it
@@ -246,6 +291,25 @@ memory.observe([
 # extracts "User is on the Pro plan", and if a "Free plan" memory exists, updates it
 ```
 
+### Custom consolidator candidate contract
+
+`reconcile_all(candidates:, scope:)` receives a plain `Array` of plain `Engram::Record`
+instances. A custom consolidator may inspect candidates and return decisions that reference
+those exact instances, but it must not replace, append, remove, or reorder collection entries;
+override collection iteration; add collection or record state or behavior; or mutate the
+records. Engram verifies record and collection integrity before it authorizes any add, update,
+or deletion and fails closed on a violation.
+
+Plain arrays and hashes in candidate state are detached without serializing their leaves.
+Core scalar/container state is integrity-protected without dispatching overridden traversal or
+equality. Arbitrary application metadata values (including value objects, procs, and IO) remain
+opaque, retain their identity, and are never compared or serialized by the integrity check.
+Their internal mutable state is consequently outside that check; provenance remains parsed and
+validated independently before any authorization. Containers must be acyclic, hashes cannot
+have default procs, and behavior-bearing core values remain unsupported.
+Extracted candidates must also have a nil `id`: IDs are store-owned and are allocated only
+when an add is persisted.
+
 ## Memory kinds and persistence policy
 
 Every memory has a normalized `kind`:
@@ -284,6 +348,107 @@ Engram.configure do |config|
 end
 ```
 
+Write-content filtering and transformation are not deletion authorization. `forget` still
+validates scope, target existence, candidate integrity, and provenance, but it does not run
+`before_persist` or the policy's `call` method; this allows secret, transient, and redacted
+memories to be removed. A custom policy can additionally control destructive decisions by
+implementing `allow_destructive?(record)` and returning exactly `true` or `false`. Policies
+that only implement `call` affect writes but do not prevent deletion. The built-in policy uses
+this separate hook to retain its ungrounded-provenance protection.
+
+### Extraction provenance
+
+Custom extractors remain compatible when they return an array of plain `Engram::Record`
+values. They may instead return `Engram::Extraction` values to attach optional structured
+`Engram::Provenance` to a candidate; arrays may contain either type. Records with no
+provenance remain accepted.
+
+Recalled records expose understood provenance directly, without loading source content:
+
+```ruby
+record = memory.recall("where did this preference come from?").first
+
+record.provenance&.sources&.each do |source|
+  puts [source.source_id, source.alignment, source.spans.map(&:to_h)].inspect
+end
+```
+
+When the host application has authorized and loaded the referenced message, it can validate
+the source identity and span bounds before presenting supporting text:
+
+```ruby
+source = record.provenance.sources.first
+source_text = Engram::Provenance::SourceText.new(
+  source_id: source.source_id,
+  source_type: source.source_type,
+  message_index: source.message_index,
+  role: source.role,
+  text: authorized_message_body
+)
+
+source.validate_source_text!(source_text)
+source.supporting_text(source_text) # text for each zero-based, end-exclusive codepoint span
+```
+
+Validation fails when any source identity field differs or a span exceeds that one message.
+Offsets count Unicode codepoints, not bytes or user-perceived grapheme clusters.
+`exact` and `normalized` remain extractor assertions: this API verifies identity and bounds but
+does not claim that supporting text semantically entails the memory.
+
+`Record#provenance` returns `nil` for legacy records and for malformed or future schemas so
+reads remain compatible. Treat the returned source IDs and spans as untrusted references.
+They do not prove source access or authorize retrieving source content; applications must
+enforce the record's scope and their own source authorization before resolving them.
+
+By default, the persistence policy rejects provenance containing a source marked
+`ungrounded`. Applications that intentionally accept it can opt out:
+
+```ruby
+Engram.configure do |config|
+  config.persistence_policy = Engram::PersistencePolicy.new(allow_ungrounded: true)
+end
+```
+
+Persistence validation is structural: Engram does not retain source text or verify that an
+alignment is truthful. Source IDs are host-owned references, not authorization boundaries.
+The built-in `Engram::Extractors::LLMExtractor` continues to return plain records and does not
+emit grounded provenance. Reads tolerate malformed or future provenance metadata, but writes
+fail closed when provenance is malformed or uses an unknown future schema version; upgrade an
+older writer before rewriting such records.
+
+### Source impact lookup
+
+To find which memories in a scope a host source produced, look them up by exact source
+reference:
+
+```ruby
+current_user.memory.memories_from_source(
+  source_id: "conversation:42",
+  source_type: "conversation"
+)
+```
+
+`source_id` and `source_type` must each be a non-blank String and are matched exactly, with no
+trimming or normalization. The lookup is bound to the scope and returns an `Array` of
+`Engram::Record` values (empty when nothing matches) — never source text. Result order follows
+the store adapter's enumeration and is not otherwise guaranteed. Source IDs are references,
+not an authorization boundary, so continue to enforce scope and your own source authorization.
+Legacy, malformed, and future-schema provenance do not match.
+
+### Grounding report
+
+To measure provenance grounding coverage without exposing record or source text, request the
+scope-bound report:
+
+```ruby
+current_user.memory.grounding_report
+# => {exact: 12, normalized: 3, inferred: 2, ungrounded: 1, unattributed: 4, total: 22}
+```
+
+Each record is counted once by its weakest source alignment (`exact`, `normalized`, `inferred`,
+then `ungrounded`). Records with no understood provenance, including legacy, malformed, and
+future-schema provenance, count as `unattributed`. The returned counts-only Hash is frozen.
+
 ## Prompt-injection and memory-injection safety
 
 Injected memories are rendered as typed XML-like elements with escaped content, which keeps
@@ -317,9 +482,28 @@ with the old `semantic` kind value.
 
 ## Tuning and maintenance
 
-Observation is idempotent per turn: observing the same messages twice does nothing the
-second time, so retries do not create duplicate memories or repeat LLM calls. In Rails,
-use a persistent processed-turn store so this also holds across job retries and processes.
+Observation uses a scope-and-turn claim before extraction. While a claim lease is live and
+the turn has not completed, calls for the same scope and turn raise
+`Engram::ObservationInProgressError` instead of reporting success without doing the work; a
+completed marker suppresses later calls (returning no decisions) until its configured `ttl`
+expires. `ObserveJob` retries `ObservationInProgressError` with polynomial backoff that
+outlasts the default lease. Direct `observe` callers should also treat the error as retryable.
+The in-memory adapter releases failures immediately. Generic Rails cache release
+is deliberately a no-op until lease expiry because ActiveSupport cache has no atomic
+compare-and-delete. In Rails, use a shared cache with atomic `unless_exist` writes for
+cross-process coordination. Set `lease_ttl` longer than the longest expected observation but
+much shorter than the completed-marker `ttl`; after a worker crash (or an overlong
+observation), the lease expires so a retry can proceed.
+
+Lease expiry can permit old and new workers to overlap; claims are not fencing tokens or an
+ownership guarantee. Successful work records completion but cannot safely delete a possibly
+newer cache claim. This is idempotency coordination, not crash-proof exactly-once persistence. An observation
+can apply multiple decisions, and the memory writes plus completed marker are not one
+transaction. A crash, cache outage, or lease expiry between those operations can permit a
+retry after some decisions were already written. Applications needing stronger guarantees
+must supply transactional persistence/outbox coordination appropriate to their store.
+If writing the completed marker fails after memory writes succeed, the current lease still
+suppresses retries until it expires; after expiry the turn may replay and repeat those writes.
 
 Recall is plain similarity search by default. You can blend in importance and recency:
 
@@ -338,6 +522,27 @@ Prune memories you no longer need:
 current_user.memory.forget_stale(older_than: 90 * 24 * 60 * 60, min_importance: 0.7)
 ```
 
+If you change embedder configuration (model/provider/dimensions), or if legacy memories were
+written without embedding provenance, run a scoped rebuild to refresh vectors and metadata:
+
+```bash
+bundle exec rake "engram:rebuild_embeddings[user:42]"
+```
+
+The task ships with the gem and is loaded by the Railtie, so it is available inside Rails
+applications and boots the app environment (initializers included) before running. Outside
+Rails, call `memory.rebuild_embeddings` directly.
+
+By default, only stale rows are rewritten. Set `STALE_ONLY=false` to rebuild all rows in
+the scope, and `BATCH_SIZE=<n>` to tune write size.
+
+The rebuild has no dry-run mode and updates each row as it goes. It assumes the store exposes
+a stable, finite, scope-isolated traversal; malformed adapters that repeat forever cannot be
+made deterministic by the rebuild. Rows without IDs are counted and skipped. Choose a batch
+size that fits provider and database limits, take a backup first, and monitor provider usage,
+errors, and database load. The final summary reports processed, updated, skipped, and failed
+counts; failures also list record IDs and cause the rake task to exit unsuccessfully.
+
 ## Observability
 
 When ActiveSupport is loaded, Engram emits `ActiveSupport::Notifications` events for the
@@ -478,8 +683,9 @@ gem unpack engram-*.gem --target /tmp/engram-package-check
 - v0.1 (done): recall + inject foundation, adapters, Rails + RubyLLM integration.
 - v0.2 (done): extract and consolidate (ADD / UPDATE / FORGET), background jobs.
 - v0.3 (done): idempotent observation, importance/recency recall, forgetting and decay.
-- v0.4 (in progress): memory kinds, persistence policy, typed recall filters, safer injection, and release-readiness docs.
-- later: real-provider eval ergonomics, additional storage backends, observability hooks, and larger eval benchmarks.
+- v0.4 (done): memory kinds, persistence policy, typed recall filters, safer injection, and observability hooks.
+- v0.5 (in progress): embedding provenance and scoped embedding rebuild operations.
+- later: additional storage backends and larger real-provider eval benchmarks.
 
 ## License
 

data/lib/engram/adapters/in_memory_processed_turns.rb

@@ -8,21 +8,53 @@ module Engram
     class InMemoryProcessedTurns
       include Ports::ProcessedTurns
 
-      def initialize
-        @keys = Set.new
+      def initialize(lease_ttl: 300, clock: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) })
+        @lease_ttl = lease_ttl
+        @clock = clock
+        @claims = {}
+        @completed = Set.new
+        @mutex = Mutex.new
       end
 
-      def seen?(key)
-        @keys.include?(key)
+      def claim(scope:, key:)
+        scoped_key = [scope, key]
+        @mutex.synchronize do
+          return if @completed.include?(scoped_key)
+          live = @claims[scoped_key]
+          return if live && live.last > @clock.call
+
+          token = Object.new
+          @claims[scoped_key] = [token, @clock.call + @lease_ttl]
+          token
+        end
+      end
+
+      def complete(scope:, key:, claim:)
+        @mutex.synchronize do
+          scoped_key = [scope, key]
+          return false unless @claims.dig(scoped_key, 0).equal?(claim)
+          @completed << scoped_key
+          @claims.delete(scoped_key)
+          true
+        end
+      end
+
+      def release(scope:, key:, claim:)
+        @mutex.synchronize do
+          scoped_key = [scope, key]
+          @claims.delete(scoped_key) if @claims.dig(scoped_key, 0).equal?(claim)
+        end
       end
 
-      def record(key)
-        @keys << key
-        key
+      def completed?(scope:, key:)
+        @mutex.synchronize { @completed.include?([scope, key]) }
       end
 
       def clear
-        @keys.clear
+        @mutex.synchronize do
+          @claims.clear
+          @completed.clear
+        end
       end
     end
   end

data/lib/engram/adapters/in_memory_store.rb

@@ -15,42 +15,61 @@ module Engram
       def add(record)
         validate_scope!(record.scope)
 
-        record.id ||= (@sequence += 1)
+        record.id = (@sequence += 1)
         @records[record.id] = record
         record
       end
 
-      def search(embedding:, scope:, limit:, kinds: nil)
+      def search(embedding:, scope:, limit:, kinds: nil, embedding_metadata: nil)
+        Engram::EmbeddingMetadata.validate_query!(embedding, embedding_metadata)
         allowed_kinds = normalize_kinds(kinds)
 
-        @records
+        results = @records
           .values
           .select { |r| searchable?(r, scope, allowed_kinds) }
           .map { |r| [r, Engram::Math.cosine_similarity(embedding, r.embedding)] }
           .sort_by { |(_, score)| -score }
           .first(limit)
           .map { |(record, _)| record }
+
+        results.each { |record| Engram::EmbeddingMetadata.validate_record!(record, embedding, embedding_metadata) }
+        results
+      end
+
+      def all(scope:, limit: nil, offset: 0, after_id: nil)
+        records = @records.values.select { |r| r.scope == scope }.sort_by { |record| record.id }
+        records = records.drop_while { |record| !after_id.nil? && record.id && record.id <= after_id }
+        records = records.drop(offset) if offset > 0
+        records = records.take(limit) if limit
+        records
       end
 
-      def all(scope:)
-        @records.values.select { |r| r.scope == scope }
+      def existing_ids(scope:, ids:)
+        ids.uniq.select { |id| @records[id]&.scope == scope }
       end
 
-      def update(id:, record:)
-        raise Engram::Error, "no memory with id #{id.inspect}" unless @records.key?(id)
+      def update(scope:, id:, record:)
+        existing = @records[id]
+        raise Engram::Error, "no memory with id #{id.inspect} in scope #{scope.inspect}" unless existing&.scope == scope
+        raise Engram::Error, "cannot move memory across scopes" unless record.scope == scope
 
         record.id = id
         @records[id] = record
       end
 
-      def delete(id:)
+      def delete(scope:, id:)
+        return 0 unless @records[id]&.scope == scope
+
         @records.delete(id)
+        1
       end
 
-      def touch(id:, at: Time.now)
+      def touch(scope:, id:, at: Time.now)
         record = @records[id]
-        record.last_accessed_at = at if record
-        record
+        return 0 unless record&.scope == scope
+
+        record.last_accessed_at = at
+        1
       end
 
       def clear

data/lib/engram/adapters/null_embedder.rb

@@ -16,6 +16,15 @@ module Engram
 
       attr_reader :dimensions
 
+      def embedding_metadata
+        Engram::EmbeddingMetadata.build(
+          adapter: self.class.name,
+          provider: "null",
+          model: "null-embedder-v1",
+          dimensions: dimensions
+        )
+      end
+
       def embed(text)
         seed = Digest::SHA256.hexdigest(text.to_s)
         Array.new(@dimensions) do |i|