engram 0.5.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8a38dfd456e4e89f37a9d4d8949a5bb77b9736d95621e9172ba1db72fb3a5771
-  data.tar.gz: 0b537ee13bea0313fd770c529024af45e9ace8fb84f7e49621c2dc7b9d6b888c
+  metadata.gz: f2d96222b1c43e68da7da579a6b9d1d39f85f5cbe2480be6cd39a19dcf36adc6
+  data.tar.gz: 53bc4261b347e498d20ea4171164b9478c55134067c33715641017b7572c78a7
 SHA512:
-  metadata.gz: 768d5d18a9ce04f60db58957127df76fbdc30482df4976d7d04b082af005850ba821ba45a35417c33946668af60331e869cf930e6239409ba10168a453f58665
-  data.tar.gz: 7ec7997ba40947deede93b6e970a549bf6caae32eed79f0e4c3922ebecbf05bc1545b5565afc521c9dab6c0f95b27898b3ae9bd9f3f57d9935a48d18e9211322
+  metadata.gz: a8001a49d7c280ded764b75e2f62695f3eeaf229e114e5455ee9bfa32f58d0c2a94fd9d70a1b563e246da0efcb570fe5f95ca6ecd61e69ca3cad758a0491abae
+  data.tar.gz: 29d1e339241f3edb54485925f20b78f81740770fc966c8e838cefaef839e191597080249f578f33c8f79c5eb2cc89563b0187cf897b296ab660ff4510a3b0d4d

data/CHANGELOG.md

@@ -5,6 +5,87 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [0.7.0] - 2026-09-13
+
+### Added
+- `Memory#forget(id:)` deletes one memory in the current scope, returning `1` when deleted
+  or `0` when missing. Explicit deletion bypasses persistence hooks and emits `forget.engram`
+  with the deleted count. Arrays, ranges, nil, and blank IDs are rejected.
+- Optional `min_similarity:` on recall, injection, and `Engram.with_memory`, with
+  `config.recall_min_similarity` as the facade default. Filters cosine similarity before
+  importance/recency ranking and touching; nil preserves existing top-k behavior.
+- Optional `max_bytes:` on injection and `Engram.with_memory`, with
+  `config.injection_max_bytes` as the default. Includes the complete appended memory block,
+  skips whole memories that do not fit, and preserves the original prompt.
+- Metrics for filtered candidates, injected/skipped memories, and appended bytes.
+- `MIN_SIMILARITY` eval control and negative-query false-positive/abstention measurements.
+
+### Fixed
+- Load `ObserveJob` on first use so `observe_later` works before another job has loaded
+  `ActiveJob::Base` in a Rails application.
+- Forward RubyLLM streaming blocks and retain the memory wrapper when fluent chat methods
+  return the underlying chat, so chained configuration does not bypass memory injection.
+
+### Changed
+- Recall rejects negative, fractional, nil, and string limits with `ArgumentError`; use an
+  integer. A zero limit skips embedding and search. A zero injection byte budget skips recall.
+
+## [0.6.0] - 2026-07-24
+
+### Added
+- `Engram::Memory#grounding_report` (and `Engram::UseCases::GroundingReport`) returns frozen,
+  scope-bound record counts by weakest source alignment, with legacy or unrecognized provenance
+  counted as unattributed and no record or source text exposed.
+- Add an immutable host-supplied source-text contract that validates provenance source identity
+  and Unicode-codepoint span bounds, and resolves authorized supporting text without retaining
+  source transcripts.
+- Optional `Engram::Extraction` results let custom extractors attach versioned source
+  provenance while remaining compatible with plain `Engram::Record` results.
+- `Engram::Record#provenance` exposes understood supporting source IDs, alignments, and spans
+  on recalled records while preserving tolerant reads for legacy and future schemas.
+- `Engram::Memory#memories_from_source` (and `Engram::UseCases::SourceImpact`) return the
+  records in a scope whose provenance references an exact host source. `source_id` and
+  `source_type` must each be a non-blank String and are matched exactly without trimming or
+  normalization. The lookup is scope-bound and returns only records, never source
+  text; source IDs are references, not an authorization boundary. Legacy, malformed, and
+  future-schema provenance do not match.
+
+### Changed
+- Persistence accepts records without provenance, rejects structurally ungrounded provenance
+  by default (configurable only with the exact boolean `allow_ungrounded: true`), and fails closed on malformed or
+  unknown future provenance during writes while keeping reads tolerant. Provenance validation
+  does not verify source text, and source IDs are references rather than authorization
+  boundaries. The built-in LLM extractor does not emit grounded provenance.
+- Existing `before_persist` hooks may continue to transform record content or embeddings, but
+  may neither add, remove, nor alter provenance; this restriction includes legacy records with
+  no provenance.
+- Extractors must return an `Array`; each result may be a plain `Engram::Record` or an
+  `Engram::Extraction` carrying provenance.
+- Custom consolidator decisions must reference the actual same-scope `Engram::Record` instance
+  supplied for reconciliation and must treat both the candidates array and records as read-only.
+  Engram now fails closed on collection replacement/reordering/iteration overrides and on
+  security-relevant nested value, identity/alias/topology, frozen-state, custom-behavior, or
+  hidden-state changes before destructive authorization. Arbitrary application metadata values
+  remain opaque and preserve their identity without invoking equality or serialization behavior;
+  provenance is independently parsed and integrity-protected.
+  Substituted, modified, missing, malformed, or cross-scope candidates fail closed.
+  `Observe` preflights the complete decision batch, including persistence policy and hook
+  transformations, before beginning store mutations.
+
+### Fixed
+- Provenance canonicalization now iterates untrusted metadata through core `Hash#keys`/`#values`
+  by position instead of `Hash#each_pair`. On Ruby 3.4 a caller-supplied hash left in a
+  delete-then-insert collision state could make `Hash#each_pair` invoke a key's `#eql?` mid-iteration,
+  letting a hostile `String`-subclass key run application code inside the behavior-free canonicalizer.
+  The adversarial hostile-key suite is now deterministic across Ruby 3.2–3.4.
+- Observation rejects extractor candidates with caller-supplied IDs, and `InMemoryStore#add`
+  always allocates a fresh ID like the pgvector adapter, preventing same- or cross-scope record
+  replacement through add semantics.
+- `forget` now separates destructive provenance authorization from write-content filtering and
+  redaction, so secret or transient memories can be deleted. Custom policies may implement
+  `allow_destructive?` with a strict boolean return; write-only policies no longer authorize
+  deletion through their `call` result.
+
 ## [0.5.0] - 2026-07-17
 
 ### Added

data/README.md

@@ -52,12 +52,14 @@ chat.ask("Why am I being rate limited?")
 
 ## Feature overview
 
-- Zero-dependency pure Ruby core with in-memory defaults for tests and local development.
+- Pure Ruby core with zero runtime dependencies and in-memory defaults for tests and local development.
 - Rails `has_memory` macro, install generator, and background `observe_later` job.
 - Postgres + pgvector storage through an optional ActiveRecord/neighbor adapter.
 - RubyLLM embedder and completion adapters for provider-backed embeddings and extraction.
 - Canonical memory kinds: `fact`, `preference`, `instruction`, and `episodic`.
 - Typed recall filters and typed, escaped memory injection.
+- Optional cosine relevance thresholds and byte budgets for injected context.
+- RubyLLM streaming and fluent chat configuration through the memory wrapper.
 - Persistence policy that rejects obvious secrets and transient task-progress updates before storage.
 - Idempotent observation, recency/importance-aware ranking, recall touching, and stale-memory pruning.
 
@@ -74,7 +76,7 @@ explicitly version-gated:
 - Core types and facade: `Engram::Memory`, `Engram::Record`, `Engram::Decision`, `Engram::PersistencePolicy`, and `Engram.with_memory`.
 - Store and adapter ports: `Engram::Ports::MemoryStore`, `Engram::Ports::Embedder`, `Engram::Ports::Completion`.
 - Rails integration points: `has_memory`, `Memory#observe_later`, and generator outputs under `engram:` rake tasks.
-- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget_stale`, and `rebuild_embeddings`.
+- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget`, `forget_stale`, `rebuild_embeddings`, and `memories_from_source`.
 - RubyLLM adapter contract points and evaluator entrypoints (`rake eval`, `rake eval:real`).
 
 ### Backward-compatibility commitments (pre-1.0)
@@ -103,7 +105,7 @@ Use `CHANGELOG.md` as the authoritative source for breaking/compatibility change
 gem "engram"
 ```
 
-The core has **zero runtime dependencies**. Optional adapters need host-app dependencies:
+The core has zero runtime dependencies and does not require Rails, ActiveRecord, or a database. Optional adapters need host-app dependencies:
 
 - `Engram::Adapters::PgvectorStore` → ActiveRecord + `neighbor` + Postgres/pgvector
 - `Engram::Adapters::RubyLLMEmbedder` and `Engram::Adapters::RubyLLMCompletion` → `ruby_llm`
@@ -223,6 +225,25 @@ class AddEngramMemoryEmbeddingIndex < ActiveRecord::Migration[8.0]
 end
 ```
 
+### Filtered vector search
+
+Keep the generated B-tree index on `scope`. With an approximate vector index, pgvector
+applies scope and kind filters after the index scan. This can return fewer than `limit`
+memories even when enough matching rows exist.
+
+On pgvector 0.8.0+, HNSW iterative scans can search further:
+
+```ruby
+Engram::MemoryRecord.transaction do
+  Engram::MemoryRecord.connection.execute("SET LOCAL hnsw.iterative_scan = strict_order")
+  current_user.memory.recall("billing preferences", limit: 5)
+end
+```
+
+The setting lasts for the transaction. Iterative scans add latency and still have scan
+limits. Compare them with exact search on your scoped data before enabling them.
+See [pgvector's filtering docs](https://github.com/pgvector/pgvector#filtering).
+
 ## Model/provider configuration
 
 Engram is model-provider agnostic. The core only depends on two ports:
@@ -272,6 +293,25 @@ chat.ask("why am I being rate limited?")
 # recall + inject happen automatically before the model sees the message
 ```
 
+Streaming and chained chat configuration work through the wrapper:
+
+```ruby
+chat = Engram.with_memory(
+  RubyLLM.chat,
+  memory: current_user.memory,
+  kinds: [:fact, :preference],
+  min_similarity: 0.5, # tune for your embedding model
+  max_bytes: 8_000
+)
+
+response = chat.with_instructions("Keep answers concise.").ask("Which plan am I on?") do |chunk|
+  print chunk.content
+end
+```
+
+`ask` forwards request options and the streaming block to RubyLLM and returns its final
+response. `with_memory` captures memory defaults when the wrapper is created.
+
 ## Automatic memory
 
 Instead of adding facts by hand, let engram derive them from a conversation turn. It
@@ -291,6 +331,25 @@ memory.observe([
 # extracts "User is on the Pro plan", and if a "Free plan" memory exists, updates it
 ```
 
+### Custom consolidator candidate contract
+
+`reconcile_all(candidates:, scope:)` receives a plain `Array` of plain `Engram::Record`
+instances. A custom consolidator may inspect candidates and return decisions that reference
+those exact instances, but it must not replace, append, remove, or reorder collection entries;
+override collection iteration; add collection or record state or behavior; or mutate the
+records. Engram verifies record and collection integrity before it authorizes any add, update,
+or deletion and fails closed on a violation.
+
+Plain arrays and hashes in candidate state are detached without serializing their leaves.
+Core scalar/container state is integrity-protected without dispatching overridden traversal or
+equality. Arbitrary application metadata values (including value objects, procs, and IO) remain
+opaque, retain their identity, and are never compared or serialized by the integrity check.
+Their internal mutable state is consequently outside that check; provenance remains parsed and
+validated independently before any authorization. Containers must be acyclic, hashes cannot
+have default procs, and behavior-bearing core values remain unsupported.
+Extracted candidates must also have a nil `id`: IDs are store-owned and are allocated only
+when an add is persisted.
+
 ## Memory kinds and persistence policy
 
 Every memory has a normalized `kind`:
@@ -329,6 +388,107 @@ Engram.configure do |config|
 end
 ```
 
+Write-content filtering and transformation are not deletion authorization. A consolidation
+`forget` decision still validates scope, target existence, candidate integrity, and provenance, but it does not run
+`before_persist` or the policy's `call` method; this allows secret, transient, and redacted
+memories to be removed. A custom policy can additionally control destructive decisions by
+implementing `allow_destructive?(record)` and returning exactly `true` or `false`. Policies
+that only implement `call` affect writes but do not prevent deletion. The built-in policy uses
+this separate hook to retain its ungrounded-provenance protection.
+
+### Extraction provenance
+
+Custom extractors remain compatible when they return an array of plain `Engram::Record`
+values. They may instead return `Engram::Extraction` values to attach optional structured
+`Engram::Provenance` to a candidate; arrays may contain either type. Records with no
+provenance remain accepted.
+
+Recalled records expose understood provenance directly, without loading source content:
+
+```ruby
+record = memory.recall("where did this preference come from?").first
+
+record.provenance&.sources&.each do |source|
+  puts [source.source_id, source.alignment, source.spans.map(&:to_h)].inspect
+end
+```
+
+When the host application has authorized and loaded the referenced message, it can validate
+the source identity and span bounds before presenting supporting text:
+
+```ruby
+source = record.provenance.sources.first
+source_text = Engram::Provenance::SourceText.new(
+  source_id: source.source_id,
+  source_type: source.source_type,
+  message_index: source.message_index,
+  role: source.role,
+  text: authorized_message_body
+)
+
+source.validate_source_text!(source_text)
+source.supporting_text(source_text) # text for each zero-based, end-exclusive codepoint span
+```
+
+Validation fails when any source identity field differs or a span exceeds that one message.
+Offsets count Unicode codepoints, not bytes or user-perceived grapheme clusters.
+`exact` and `normalized` remain extractor assertions: this API verifies identity and bounds but
+does not claim that supporting text semantically entails the memory.
+
+`Record#provenance` returns `nil` for legacy records and for malformed or future schemas so
+reads remain compatible. Treat the returned source IDs and spans as untrusted references.
+They do not prove source access or authorize retrieving source content; applications must
+enforce the record's scope and their own source authorization before resolving them.
+
+By default, the persistence policy rejects provenance containing a source marked
+`ungrounded`. Applications that intentionally accept it can opt out:
+
+```ruby
+Engram.configure do |config|
+  config.persistence_policy = Engram::PersistencePolicy.new(allow_ungrounded: true)
+end
+```
+
+Persistence validation is structural: Engram does not retain source text or verify that an
+alignment is truthful. Source IDs are host-owned references, not authorization boundaries.
+The built-in `Engram::Extractors::LLMExtractor` continues to return plain records and does not
+emit grounded provenance. Reads tolerate malformed or future provenance metadata, but writes
+fail closed when provenance is malformed or uses an unknown future schema version; upgrade an
+older writer before rewriting such records.
+
+### Source impact lookup
+
+To find which memories in a scope a host source produced, look them up by exact source
+reference:
+
+```ruby
+current_user.memory.memories_from_source(
+  source_id: "conversation:42",
+  source_type: "conversation"
+)
+```
+
+`source_id` and `source_type` must each be a non-blank String and are matched exactly, with no
+trimming or normalization. The lookup is bound to the scope and returns an `Array` of
+`Engram::Record` values (empty when nothing matches) — never source text. Result order follows
+the store adapter's enumeration and is not otherwise guaranteed. Source IDs are references,
+not an authorization boundary, so continue to enforce scope and your own source authorization.
+Legacy, malformed, and future-schema provenance do not match.
+
+### Grounding report
+
+To measure provenance grounding coverage without exposing record or source text, request the
+scope-bound report:
+
+```ruby
+current_user.memory.grounding_report
+# => {exact: 12, normalized: 3, inferred: 2, ungrounded: 1, unattributed: 4, total: 22}
+```
+
+Each record is counted once by its weakest source alignment (`exact`, `normalized`, `inferred`,
+then `ungrounded`). Records with no understood provenance, including legacy, malformed, and
+future-schema provenance, count as `unattributed`. The returned counts-only Hash is frozen.
+
 ## Prompt-injection and memory-injection safety
 
 Injected memories are rendered as typed XML-like elements with escaped content, which keeps
@@ -362,6 +522,39 @@ with the old `semantic` kind value.
 
 ## Tuning and maintenance
 
+### Recall thresholds and injection limits
+
+Recall returns the nearest matches even for unrelated queries. Set a minimum cosine
+similarity to exclude weak matches and a byte limit to keep injected context small:
+
+```ruby
+Engram.configure do |config|
+  config.recall_min_similarity = 0.5 # tune for your embedding model
+  config.injection_max_bytes = 8_000
+end
+
+memory.recall("billing preferences", min_similarity: 0.6)
+memory.inject_into(prompt, query: "billing preferences", max_bytes: 4_000)
+```
+
+`min_similarity` accepts a finite number in `[-1, 1]`. It filters candidates before
+importance/recency ranking and touching, and can return no matches. Filtering happens in
+Ruby; custom stores keep their existing search signature. It does not refill the candidate
+pool or suppress embedding compatibility errors.
+
+`max_bytes` limits the appended memory section, including escaped text, headers, and tags.
+It excludes the original prompt and chat history and counts bytes, not tokens. Memories
+are considered in recall order; those that do not fit are skipped without truncation.
+If none fit, the prompt is unchanged. With `touch_on_recall`, retrieved memories are touched
+even if the byte limit later excludes them.
+
+Both settings default to nil. Pass nil explicitly to override a configured value.
+`limit` and `max_bytes` require non-negative integers, except that `max_bytes: nil` removes
+the cap. `limit: 0` skips embedding and search; `max_bytes: 0` skips recall during injection.
+Invalid values raise `ArgumentError`.
+
+### Observation and maintenance
+
 Observation uses a scope-and-turn claim before extraction. While a claim lease is live and
 the turn has not completed, calls for the same scope and turn raise
 `Engram::ObservationInProgressError` instead of reporting success without doing the work; a
@@ -395,7 +588,24 @@ Engram.configure do |config|
 end
 ```
 
-Prune memories you no longer need:
+Delete a specific memory:
+
+```ruby
+record = current_user.memory.add("Prefers tea")
+current_user.memory.forget(id: record.id) # => 1
+current_user.memory.forget(id: record.id) # => 0 (already deleted)
+```
+
+`forget` deletes one record within the memory's scope and returns the affected row count.
+Missing records and IDs belonging to another scope return `0`. Pass the ID returned by
+the store; integers and non-blank strings are accepted. Arrays, ranges, and nil raise
+`ArgumentError`.
+
+This is an explicit application operation: authorize the request before calling it.
+It does not run extraction, consolidation, or persistence hooks, including `allow_destructive?`.
+Source messages and processed-turn markers are kept; a later observation can recreate the memory.
+
+Prune stale memories:
 
 ```ruby
 # Forget memories untouched for 90 days, but keep anything important
@@ -429,6 +639,7 @@ When ActiveSupport is loaded, Engram emits `ActiveSupport::Notifications` events
 main memory pipeline:
 
 - `add.engram`
+- `forget.engram`
 - `recall.engram`
 - `inject.engram`
 - `observe.engram`
@@ -460,6 +671,11 @@ ActiveSupport::Notifications.subscribe(/\.engram\z/) do |name, _started, _finish
 end
 ```
 
+Recall events include `candidate_count`, `filtered_count`, `result_count`, and
+`min_similarity` when set. Injection events include the input `memory_count`,
+`injected_count`, `skipped_count`, `injected_bytes`, and `max_bytes` when set.
+`forget.engram` reports `deleted_count` without the record ID or content.
+
 Avoid adding memory content or raw prompts to subscriber logs; recalled content is
 user-derived and should be treated as sensitive application data.
 
@@ -472,7 +688,8 @@ user-derived and should be treated as sensitive application data.
 - Configure ActiveJob for `observe_later`; keep automatic observation off the request path.
 - Configure `Engram::Rails::CacheProcessedTurns` or another persistent processed-turns adapter for retries.
 - Review persistence policy settings and add app-specific redaction/denylist patterns.
-- Set recall limits and `kinds:` filters appropriate for your prompt budget and threat model.
+- Calibrate `recall_min_similarity` and set `injection_max_bytes`, recall limits, and `kinds:` filters.
+- Check filtered pgvector recall coverage and latency on representative tenant sizes.
 - Run the deterministic test/eval suite plus pgvector integration tests before release.
 
 ## How it works
@@ -547,9 +764,10 @@ safe to run in CI as a smoke test.
 The harness reports recall@k over labelled relevant memories, a labelled precision
 proxy@k, near-distractor retrieval rate, contradiction-pair full recall, extraction
 structured-output parsing cases, consolidation decision cases, and a heuristic duplicate-add
-baseline. Negative queries are printed for inspection, but top-k recall currently has no
-similarity threshold, so the harness does not report a hallucination rate. Treat the default
-NullEmbedder recall numbers as a mechanics check, not as a semantic retrieval benchmark.
+baseline. Set `MIN_SIMILARITY=0.5` on either eval task to test a threshold. With a semantic
+embedder, it also reports how often unrelated queries retrieve memories and how often
+recall returns nothing. Compare these against positive-query recall when tuning the
+threshold. NullEmbedder results only check mechanics; they do not measure semantic quality.
 
 Before opening a release PR, also verify the gem package:
 
@@ -564,7 +782,9 @@ gem unpack engram-*.gem --target /tmp/engram-package-check
 - v0.2 (done): extract and consolidate (ADD / UPDATE / FORGET), background jobs.
 - v0.3 (done): idempotent observation, importance/recency recall, forgetting and decay.
 - v0.4 (done): memory kinds, persistence policy, typed recall filters, safer injection, and observability hooks.
-- v0.5 (in progress): embedding provenance and scoped embedding rebuild operations.
+- v0.5 (done): embedding provenance and scoped embedding rebuild operations.
+- v0.6 (done): structured provenance, source impact lookup, and grounding reports.
+- v0.7 (done): recall thresholds, injection limits, scoped deletion, and RubyLLM streaming fixes.
 - later: additional storage backends and larger real-provider eval benchmarks.
 
 ## License

data/lib/engram/adapters/in_memory_store.rb

@@ -15,7 +15,7 @@ module Engram
       def add(record)
         validate_scope!(record.scope)
 
-        record.id ||= (@sequence += 1)
+        record.id = (@sequence += 1)
         @records[record.id] = record
         record
       end
@@ -44,6 +44,10 @@ module Engram
         records
       end
 
+      def existing_ids(scope:, ids:)
+        ids.uniq.select { |id| @records[id]&.scope == scope }
+      end
+
       def update(scope:, id:, record:)
         existing = @records[id]
         raise Engram::Error, "no memory with id #{id.inspect} in scope #{scope.inspect}" unless existing&.scope == scope

data/lib/engram/adapters/pgvector_store.rb

@@ -49,6 +49,21 @@ module Engram
         query.map { |row| to_record(row) }
       end
 
+      def existing_ids(scope:, ids:)
+        persisted_ids = model.where(scope: scope, id: ids).pluck(:id)
+        id_type = model.type_for_attribute("id")
+        persisted_lookup = persisted_ids.each_with_object({}) { |id, lookup| lookup[id] = true }
+        matched_ids = {}
+
+        ids.select do |id|
+          cast_id = id_type.cast(id)
+          next false unless persisted_lookup[cast_id]
+          next false if matched_ids[cast_id]
+
+          matched_ids[cast_id] = true
+        end
+      end
+
       def update(scope:, id:, record:)
         raise Engram::Error, "cannot move memory across scopes" unless record.scope == scope
 

data/lib/engram/configuration.rb

@@ -8,13 +8,16 @@ module Engram
     attr_accessor :store, :embedder, :completion, :default_limit,
       :consolidator, :extraction_min_confidence, :processed_turns,
       :importance_weight, :recency_weight, :recency_halflife, :touch_on_recall,
-      :persistence_policy, :before_persist, :instrumentation_scope_identifier
+      :persistence_policy, :before_persist, :instrumentation_scope_identifier,
+      :recall_min_similarity, :injection_max_bytes
 
     def initialize
       @store = Adapters::InMemoryStore.new
       @embedder = Adapters::NullEmbedder.new
       @completion = nil # required for observe (extract/consolidate); nil until configured
       @default_limit = 5
+      @recall_min_similarity = nil # opt-in cosine floor, before importance/recency ranking
+      @injection_max_bytes = nil # cap the complete appended memory block, not the host prompt
       @consolidator = :heuristic # :heuristic (deterministic) or :llm (LLM-as-judge)
       @extraction_min_confidence = 0.5
       @processed_turns = Adapters::InMemoryProcessedTurns.new # idempotency for observe

data/lib/engram/extraction.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module Engram
+  # An immutable extractor result that carries a candidate Record and its source provenance.
+  # Converting it creates a new record and leaves the caller-owned record and metadata untouched.
+  class Extraction
+    attr_reader :record, :provenance
+
+    def initialize(record:, provenance:)
+      is_record = Object.instance_method(:is_a?).bind_call(record, Engram::Record)
+      raise ArgumentError, "record must be an Engram::Record" unless is_record
+
+      is_provenance = Object.instance_method(:is_a?).bind_call(provenance, Engram::Provenance)
+      raise ArgumentError, "provenance must be an Engram::Provenance" unless is_provenance
+
+      @record = record
+      @provenance = provenance
+      freeze
+    end
+
+    def to_record
+      source = Engram::Internal::CandidateIntegrity.new.detach(record)
+      attributes = Engram::Record::STATE_READERS.to_h do |attribute|
+        [attribute, Engram::Record.instance_method(attribute).bind_call(source)]
+      end
+      attributes[:metadata] = Engram::Provenance.attach(attributes.fetch(:metadata), provenance)
+      Engram::Record.new(**attributes)
+    end
+  end
+end

data/lib/engram/integrations/ruby_llm.rb

@@ -9,21 +9,27 @@ module Engram
       #   chat = Engram.with_memory(RubyLLM.chat, memory: current_user.memory)
       #   chat.ask("why am I rate limited?")  # recall + inject happen automatically
       class MemoryChat
-        def initialize(chat, memory:, limit: Engram.config.default_limit)
+        def initialize(chat, memory:, limit: Engram.config.default_limit, kinds: nil,
+          min_similarity: Engram.config.recall_min_similarity, max_bytes: Engram.config.injection_max_bytes)
           @chat = chat
           @memory = memory
           @limit = limit
+          @kinds = kinds
+          @min_similarity = min_similarity
+          @max_bytes = max_bytes
         end
 
-        def ask(message, **opts)
-          augmented = @memory.inject_into(message.to_s, query: message.to_s, limit: @limit)
-          @chat.ask(augmented, **opts)
+        def ask(message, **opts, &block)
+          augmented = @memory.inject_into(message.to_s, query: message.to_s, limit: @limit,
+            kinds: @kinds, min_similarity: @min_similarity, max_bytes: @max_bytes)
+          @chat.ask(augmented, **opts, &block)
         end
 
         def method_missing(name, *args, **kwargs, &block)
           return super unless @chat.respond_to?(name)
 
-          @chat.public_send(name, *args, **kwargs, &block)
+          result = @chat.public_send(name, *args, **kwargs, &block)
+          result.equal?(@chat) ? self : result
         end
 
         def respond_to_missing?(name, include_private = false)
@@ -34,7 +40,9 @@ module Engram
   end
 
   # Convenience entrypoint.
-  def self.with_memory(chat, memory:, limit: config.default_limit)
-    Integrations::RubyLLM::MemoryChat.new(chat, memory: memory, limit: limit)
+  def self.with_memory(chat, memory:, limit: config.default_limit, kinds: nil,
+    min_similarity: config.recall_min_similarity, max_bytes: config.injection_max_bytes)
+    Integrations::RubyLLM::MemoryChat.new(chat, memory: memory, limit: limit, kinds: kinds,
+      min_similarity: min_similarity, max_bytes: max_bytes)
   end
 end