engram 0.6.0 → 0.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c623f8198a9f71905e6e0c904b6cf60d5d6a0734b1e8fdaf877611a815f2a919
-  data.tar.gz: aca21e3da50239a5d02822306524adc6647883ef607af57747b05ce7ecb78827
+  metadata.gz: b3d370df000e9c12a40df932919f0859d9eb7623270ed5aeaa781747f239d865
+  data.tar.gz: 2c49fd3e8126a510f49b711a9ccf9d159ac01b6e540ef2ab168e7e51fa6509f3
 SHA512:
-  metadata.gz: 55f5e8335a86ec28cd28b17af162883fcc7c58dc56b903fc18b924c4278b593d9c8945cbf19cb28649a4c66de57ff38e6af46f3c50b15a6a96a8d729b90fc6ba
-  data.tar.gz: 519081acdc6f1b284df51bc664d274ca31befbc2d7a002ef9d62fee2f31c165f64c2896ab5f2dddca771b02c2052dcb630ef785e440776931d53d7f994de6e47
+  metadata.gz: 161c8e9c199352d663e0ce670f3a6dc76ac98aba0b429566a1c3c96fa8ee5c7db260a449355f449b11dfdb25de4b85f207ca263fcffa6a76248d1269632ecddc
+  data.tar.gz: 1c3867577e6a86128fa4a4036b64cc2790754336c4cc462495284f25f5a7ac63ac56be24694ffb9578d2b50999bde325ca4ced68b6d8e0aed9143234108b0888

data/CHANGELOG.md

@@ -5,6 +5,39 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [0.8.0] - 2026-09-15
+
+### Added
+
+- Optional `expires_at` on memories. Expired records are excluded from search and injection while remaining available for inspection and deletion.
+- `engram:expiry` generator for existing Rails apps, with a concurrent index for scoped expiry queries.
+- Scoped `forget_expired` cleanup with batching, dry runs, and a Rails task. Deletion rechecks the deadline so concurrent expiry extensions are preserved.
+
+## [0.7.0] - 2026-09-13
+
+### Added
+- `Memory#forget(id:)` deletes one memory in the current scope, returning `1` when deleted
+  or `0` when missing. Explicit deletion bypasses persistence hooks and emits `forget.engram`
+  with the deleted count. Arrays, ranges, nil, and blank IDs are rejected.
+- Optional `min_similarity:` on recall, injection, and `Engram.with_memory`, with
+  `config.recall_min_similarity` as the facade default. Filters cosine similarity before
+  importance/recency ranking and touching; nil preserves existing top-k behavior.
+- Optional `max_bytes:` on injection and `Engram.with_memory`, with
+  `config.injection_max_bytes` as the default. Includes the complete appended memory block,
+  skips whole memories that do not fit, and preserves the original prompt.
+- Metrics for filtered candidates, injected/skipped memories, and appended bytes.
+- `MIN_SIMILARITY` eval control and negative-query false-positive/abstention measurements.
+
+### Fixed
+- Load `ObserveJob` on first use so `observe_later` works before another job has loaded
+  `ActiveJob::Base` in a Rails application.
+- Forward RubyLLM streaming blocks and retain the memory wrapper when fluent chat methods
+  return the underlying chat, so chained configuration does not bypass memory injection.
+
+### Changed
+- Recall rejects negative, fractional, nil, and string limits with `ArgumentError`; use an
+  integer. A zero limit skips embedding and search. A zero injection byte budget skips recall.
+
 ## [0.6.0] - 2026-07-24
 
 ### Added

data/README.md

@@ -58,6 +58,8 @@ chat.ask("Why am I being rate limited?")
 - RubyLLM embedder and completion adapters for provider-backed embeddings and extraction.
 - Canonical memory kinds: `fact`, `preference`, `instruction`, and `episodic`.
 - Typed recall filters and typed, escaped memory injection.
+- Optional cosine relevance thresholds and byte budgets for injected context.
+- RubyLLM streaming and fluent chat configuration through the memory wrapper.
 - Persistence policy that rejects obvious secrets and transient task-progress updates before storage.
 - Idempotent observation, recency/importance-aware ranking, recall touching, and stale-memory pruning.
 
@@ -74,7 +76,7 @@ explicitly version-gated:
 - Core types and facade: `Engram::Memory`, `Engram::Record`, `Engram::Decision`, `Engram::PersistencePolicy`, and `Engram.with_memory`.
 - Store and adapter ports: `Engram::Ports::MemoryStore`, `Engram::Ports::Embedder`, `Engram::Ports::Completion`.
 - Rails integration points: `has_memory`, `Memory#observe_later`, and generator outputs under `engram:` rake tasks.
-- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget_stale`, `rebuild_embeddings`, and `memories_from_source`.
+- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget`, `forget_stale`, `forget_expired`, `rebuild_embeddings`, and `memories_from_source`.
 - RubyLLM adapter contract points and evaluator entrypoints (`rake eval`, `rake eval:real`).
 
 ### Backward-compatibility commitments (pre-1.0)
@@ -223,6 +225,25 @@ class AddEngramMemoryEmbeddingIndex < ActiveRecord::Migration[8.0]
 end
 ```
 
+### Filtered vector search
+
+Keep the generated B-tree index on `scope`. With an approximate vector index, pgvector
+applies scope, kind, and expiry filters after the index scan. This can return fewer than `limit`
+memories even when enough matching rows exist.
+
+On pgvector 0.8.0+, HNSW iterative scans can search further:
+
+```ruby
+Engram::MemoryRecord.transaction do
+  Engram::MemoryRecord.connection.execute("SET LOCAL hnsw.iterative_scan = strict_order")
+  current_user.memory.recall("billing preferences", limit: 5)
+end
+```
+
+The setting lasts for the transaction. Iterative scans add latency and still have scan
+limits. Compare them with exact search on your scoped data before enabling them.
+See [pgvector's filtering docs](https://github.com/pgvector/pgvector#filtering).
+
 ## Model/provider configuration
 
 Engram is model-provider agnostic. The core only depends on two ports:
@@ -272,6 +293,25 @@ chat.ask("why am I being rate limited?")
 # recall + inject happen automatically before the model sees the message
 ```
 
+Streaming and chained chat configuration work through the wrapper:
+
+```ruby
+chat = Engram.with_memory(
+  RubyLLM.chat,
+  memory: current_user.memory,
+  kinds: [:fact, :preference],
+  min_similarity: 0.5, # tune for your embedding model
+  max_bytes: 8_000
+)
+
+response = chat.with_instructions("Keep answers concise.").ask("Which plan am I on?") do |chunk|
+  print chunk.content
+end
+```
+
+`ask` forwards request options and the streaming block to RubyLLM and returns its final
+response. `with_memory` captures memory defaults when the wrapper is created.
+
 ## Automatic memory
 
 Instead of adding facts by hand, let engram derive them from a conversation turn. It
@@ -348,8 +388,8 @@ Engram.configure do |config|
 end
 ```
 
-Write-content filtering and transformation are not deletion authorization. `forget` still
-validates scope, target existence, candidate integrity, and provenance, but it does not run
+Write-content filtering and transformation are not deletion authorization. A consolidation
+`forget` decision still validates scope, target existence, candidate integrity, and provenance, but it does not run
 `before_persist` or the policy's `call` method; this allows secret, transient, and redacted
 memories to be removed. A custom policy can additionally control destructive decisions by
 implementing `allow_destructive?(record)` and returning exactly `true` or `false`. Policies
@@ -482,6 +522,100 @@ with the old `semantic` kind value.
 
 ## Tuning and maintenance
 
+### Recall thresholds and injection limits
+
+Recall returns the nearest matches even for unrelated queries. Set a minimum cosine
+similarity to exclude weak matches and a byte limit to keep injected context small:
+
+```ruby
+Engram.configure do |config|
+  config.recall_min_similarity = 0.5 # tune for your embedding model
+  config.injection_max_bytes = 8_000
+end
+
+memory.recall("billing preferences", min_similarity: 0.6)
+memory.inject_into(prompt, query: "billing preferences", max_bytes: 4_000)
+```
+
+`min_similarity` accepts a finite number in `[-1, 1]`. It filters candidates before
+importance/recency ranking and touching, and can return no matches. Filtering happens in
+Ruby; custom stores keep their existing search signature. It does not refill the candidate
+pool or suppress embedding compatibility errors.
+
+`max_bytes` limits the appended memory section, including escaped text, headers, and tags.
+It excludes the original prompt and chat history and counts bytes, not tokens. Memories
+are considered in recall order; those that do not fit are skipped without truncation.
+If none fit, the prompt is unchanged. With `touch_on_recall`, retrieved memories are touched
+even if the byte limit later excludes them.
+
+Both settings default to nil. Pass nil explicitly to override a configured value.
+`limit` and `max_bytes` require non-negative integers, except that `max_bytes: nil` removes
+the cap. `limit: 0` skips embedding and search; `max_bytes: 0` skips recall during injection.
+Invalid values raise `ArgumentError`.
+
+### Memory expiry
+
+Set a deadline for temporary memories:
+
+```ruby
+user.memory.add("Trial access is active", expires_at: 7.days.from_now)
+```
+
+`expires_at` accepts a Time (including Rails time-zone values) or nil. Nil is the default
+and never expires. A memory is expired at `expires_at <= Time.now`. Both built-in stores
+exclude expired records before the search limit, so they no longer participate in recall
+or consolidation. Injection checks expiry again when rendering.
+
+Expiry does not delete rows. `memory.all` still returns them, and `memory.forget(id:)` can
+delete them. A new observation may recreate the same fact from retained source data.
+Custom extractors or `before_persist` can set expiry with `record.with(expires_at: deadline)`;
+Engram does not infer deadlines from conversation text. Store updates replace expiry with
+the supplied record's value, including nil.
+
+New Rails installs include the expiry column and index. For a table created by 0.7.0 or
+earlier, generate and run the upgrade migration:
+
+```sh
+bin/rails generate engram:expiry
+bin/rails db:migrate
+```
+
+The migration adds a nullable datetime column and builds a partial index on scope and
+expiry concurrently. Existing rows keep their data and do not expire. Reads and
+non-expiring writes continue to work before migration; expiring writes raise an error
+instead of losing the deadline. Restart app processes after migrating so ActiveRecord
+refreshes its cached schema. If you use a custom table, adjust the generated migration.
+
+Custom stores must persist `expires_at` and exclude expired records before limiting search
+results. Recall also discards any expired records a custom store returns, but cannot fill
+the gaps left by a store that filters after its limit.
+
+To remove expired rows, preview the count, then run cleanup:
+
+```ruby
+user.memory.forget_expired(dry_run: true)
+# => {matched: 12, deleted: 0, dry_run: true}
+user.memory.forget_expired(batch_size: 500)
+# => {matched: 12, deleted: 12, dry_run: false}
+```
+
+The Rails task requires a scope:
+
+```sh
+DRY_RUN=true BATCH_SIZE=500 bin/rails 'engram:forget_expired[user:42]'
+BATCH_SIZE=500 bin/rails 'engram:forget_expired[user:42]'
+```
+
+Cleanup uses one cutoff for the run and reads only IDs in batches. It rechecks scope and
+expiry when deleting, so a concurrent extension or removal of the deadline is preserved.
+`matched` can exceed `deleted` if records change during the run; dry-run counts are advisory.
+Like `forget(id:)`, this is explicit deletion and bypasses persistence hooks and policies.
+Errors stop the run; completed batches stay deleted and a retry can process the remainder.
+Custom stores need the optional `expired_ids` and `delete_expired` methods for cleanup.
+The `forget_expired.engram` event reports counts without memory content or IDs.
+
+### Observation and maintenance
+
 Observation uses a scope-and-turn claim before extraction. While a claim lease is live and
 the turn has not completed, calls for the same scope and turn raise
 `Engram::ObservationInProgressError` instead of reporting success without doing the work; a
@@ -515,7 +649,24 @@ Engram.configure do |config|
 end
 ```
 
-Prune memories you no longer need:
+Delete a specific memory:
+
+```ruby
+record = current_user.memory.add("Prefers tea")
+current_user.memory.forget(id: record.id) # => 1
+current_user.memory.forget(id: record.id) # => 0 (already deleted)
+```
+
+`forget` deletes one record within the memory's scope and returns the affected row count.
+Missing records and IDs belonging to another scope return `0`. Pass the ID returned by
+the store; integers and non-blank strings are accepted. Arrays, ranges, and nil raise
+`ArgumentError`.
+
+This is an explicit application operation: authorize the request before calling it.
+It does not run extraction, consolidation, or persistence hooks, including `allow_destructive?`.
+Source messages and processed-turn markers are kept; a later observation can recreate the memory.
+
+Prune stale memories:
 
 ```ruby
 # Forget memories untouched for 90 days, but keep anything important
@@ -549,6 +700,8 @@ When ActiveSupport is loaded, Engram emits `ActiveSupport::Notifications` events
 main memory pipeline:
 
 - `add.engram`
+- `forget.engram`
+- `forget_expired.engram`
 - `recall.engram`
 - `inject.engram`
 - `observe.engram`
@@ -580,6 +733,11 @@ ActiveSupport::Notifications.subscribe(/\.engram\z/) do |name, _started, _finish
 end
 ```
 
+Recall events include `candidate_count`, `filtered_count`, `result_count`, and
+`min_similarity` when set. Injection events include the input `memory_count`,
+`injected_count`, `skipped_count`, `injected_bytes`, and `max_bytes` when set.
+`forget.engram` reports `deleted_count` without the record ID or content.
+
 Avoid adding memory content or raw prompts to subscriber logs; recalled content is
 user-derived and should be treated as sensitive application data.
 
@@ -592,7 +750,8 @@ user-derived and should be treated as sensitive application data.
 - Configure ActiveJob for `observe_later`; keep automatic observation off the request path.
 - Configure `Engram::Rails::CacheProcessedTurns` or another persistent processed-turns adapter for retries.
 - Review persistence policy settings and add app-specific redaction/denylist patterns.
-- Set recall limits and `kinds:` filters appropriate for your prompt budget and threat model.
+- Calibrate `recall_min_similarity` and set `injection_max_bytes`, recall limits, and `kinds:` filters.
+- Check filtered pgvector recall coverage and latency on representative tenant sizes.
 - Run the deterministic test/eval suite plus pgvector integration tests before release.
 
 ## How it works
@@ -667,9 +826,10 @@ safe to run in CI as a smoke test.
 The harness reports recall@k over labelled relevant memories, a labelled precision
 proxy@k, near-distractor retrieval rate, contradiction-pair full recall, extraction
 structured-output parsing cases, consolidation decision cases, and a heuristic duplicate-add
-baseline. Negative queries are printed for inspection, but top-k recall currently has no
-similarity threshold, so the harness does not report a hallucination rate. Treat the default
-NullEmbedder recall numbers as a mechanics check, not as a semantic retrieval benchmark.
+baseline. Set `MIN_SIMILARITY=0.5` on either eval task to test a threshold. With a semantic
+embedder, it also reports how often unrelated queries retrieve memories and how often
+recall returns nothing. Compare these against positive-query recall when tuning the
+threshold. NullEmbedder results only check mechanics; they do not measure semantic quality.
 
 Before opening a release PR, also verify the gem package:
 
@@ -684,7 +844,10 @@ gem unpack engram-*.gem --target /tmp/engram-package-check
 - v0.2 (done): extract and consolidate (ADD / UPDATE / FORGET), background jobs.
 - v0.3 (done): idempotent observation, importance/recency recall, forgetting and decay.
 - v0.4 (done): memory kinds, persistence policy, typed recall filters, safer injection, and observability hooks.
-- v0.5 (in progress): embedding provenance and scoped embedding rebuild operations.
+- v0.5 (done): embedding provenance and scoped embedding rebuild operations.
+- v0.6 (done): structured provenance, source impact lookup, and grounding reports.
+- v0.7 (done): recall thresholds, injection limits, scoped deletion, and RubyLLM streaming fixes.
+- v0.8 (done): explicit memory expiry, Rails upgrade migration, and scoped cleanup with dry runs.
 - later: additional storage backends and larger real-provider eval benchmarks.
 
 ## License

data/lib/engram/adapters/in_memory_store.rb

@@ -10,23 +10,26 @@ module Engram
       def initialize
         @records = {}
         @sequence = 0
+        @mutex = Mutex.new
       end
 
       def add(record)
         validate_scope!(record.scope)
 
-        record.id = (@sequence += 1)
-        @records[record.id] = record
-        record
+        @mutex.synchronize do
+          record.id = (@sequence += 1)
+          @records[record.id] = record
+          record
+        end
       end
 
       def search(embedding:, scope:, limit:, kinds: nil, embedding_metadata: nil)
         Engram::EmbeddingMetadata.validate_query!(embedding, embedding_metadata)
         allowed_kinds = normalize_kinds(kinds)
+        now = Time.now
 
-        results = @records
-          .values
-          .select { |r| searchable?(r, scope, allowed_kinds) }
+        results = @mutex.synchronize { @records.values }
+          .select { |r| searchable?(r, scope, allowed_kinds) && !r.expired?(at: now) }
           .map { |r| [r, Engram::Math.cosine_similarity(embedding, r.embedding)] }
           .sort_by { |(_, score)| -score }
           .first(limit)
@@ -37,7 +40,7 @@ module Engram
       end
 
       def all(scope:, limit: nil, offset: 0, after_id: nil)
-        records = @records.values.select { |r| r.scope == scope }.sort_by { |record| record.id }
+        records = @mutex.synchronize { @records.values }.select { |r| r.scope == scope }.sort_by { |record| record.id }
         records = records.drop_while { |record| !after_id.nil? && record.id && record.id <= after_id }
         records = records.drop(offset) if offset > 0
         records = records.take(limit) if limit
@@ -45,36 +48,65 @@ module Engram
       end
 
       def existing_ids(scope:, ids:)
-        ids.uniq.select { |id| @records[id]&.scope == scope }
+        @mutex.synchronize { ids.uniq.select { |id| @records[id]&.scope == scope } }
       end
 
       def update(scope:, id:, record:)
-        existing = @records[id]
-        raise Engram::Error, "no memory with id #{id.inspect} in scope #{scope.inspect}" unless existing&.scope == scope
-        raise Engram::Error, "cannot move memory across scopes" unless record.scope == scope
-
-        record.id = id
-        @records[id] = record
+        @mutex.synchronize do
+          existing = @records[id]
+          raise Engram::Error, "no memory with id #{id.inspect} in scope #{scope.inspect}" unless existing&.scope == scope
+          raise Engram::Error, "cannot move memory across scopes" unless record.scope == scope
+
+          record.id = id
+          @records[id] = record
+        end
       end
 
       def delete(scope:, id:)
-        return 0 unless @records[id]&.scope == scope
+        @mutex.synchronize do
+          return 0 unless @records[id]&.scope == scope
 
-        @records.delete(id)
-        1
+          @records.delete(id)
+          1
+        end
       end
 
       def touch(scope:, id:, at: Time.now)
-        record = @records[id]
-        return 0 unless record&.scope == scope
+        @mutex.synchronize do
+          record = @records[id]
+          return 0 unless record&.scope == scope
+
+          record.last_accessed_at = at
+          1
+        end
+      end
 
-        record.last_accessed_at = at
-        1
+      def expired_ids(scope:, at:, limit:, after_id: nil)
+        # IDs follow insertion order; the hash retains it across updates/deletes.
+        @mutex.synchronize do
+          @records.each_value.lazy
+            .select { |record| record.scope == scope && (after_id.nil? || record.id > after_id) && record.expired?(at: at) }
+            .map(&:id).take(limit).force
+        end
+      end
+
+      def delete_expired(scope:, ids:, at:)
+        @mutex.synchronize do
+          ids.uniq.count do |id|
+            record = @records[id]
+            next false unless record&.scope == scope && record.expired?(at: at)
+
+            @records.delete(id)
+            true
+          end
+        end
       end
 
       def clear
-        @records.clear
-        @sequence = 0
+        @mutex.synchronize do
+          @records.clear
+          @sequence = 0
+        end
       end
 
       private

data/lib/engram/adapters/pgvector_store.rb

@@ -18,6 +18,7 @@ module Engram
         validate_scope!(record.scope)
 
         row = model.create!(
+          **expiry_attributes(record),
           content: record.content,
           scope: record.scope,
           kind: record.kind.to_s,
@@ -31,6 +32,10 @@ module Engram
       def search(embedding:, scope:, limit:, kinds: nil, embedding_metadata: nil)
         Engram::EmbeddingMetadata.validate_query!(embedding, embedding_metadata)
         query = model.where(scope: scope)
+        if expiry_column?
+          expiry = model.arel_table[:expires_at]
+          query = query.where(expiry.eq(nil).or(expiry.gt(Time.now)))
+        end
         normalized_kinds = normalize_kinds(kinds)
         query = query.where(kind: normalized_kinds) if normalized_kinds
 
@@ -70,6 +75,7 @@ module Engram
         model.transaction do
           row = model.lock.find_by!(id: id, scope: scope)
           row.update!(
+            **expiry_attributes(record),
             content: record.content,
             kind: record.kind.to_s,
             importance: record.importance,
@@ -90,8 +96,37 @@ module Engram
         model.where(id: id, scope: scope).update_all(last_accessed_at: at)
       end
 
+      def expired_ids(scope:, at:, limit:, after_id: nil)
+        return [] unless expiry_column?
+
+        query = expired_scope(scope, at)
+        query = query.where(model.arel_table[:id].gt(after_id)) unless after_id.nil?
+        query.order(:id).limit(limit).pluck(:id)
+      end
+
+      def delete_expired(scope:, ids:, at:)
+        return 0 if ids.empty? || !expiry_column?
+
+        expired_scope(scope, at).where(id: ids).delete_all
+      end
+
       private
 
+      def expired_scope(scope, at)
+        model.where(scope: scope).where(model.arel_table[:expires_at].lteq(at))
+      end
+
+      def expiry_column?
+        model.column_names.include?("expires_at")
+      end
+
+      def expiry_attributes(record)
+        return {expires_at: record.expires_at} if expiry_column?
+        return {} if record.expires_at.nil?
+
+        raise Engram::Error, "PgvectorStore needs a nullable expires_at datetime column to store expiring memories"
+      end
+
       def validate_scope!(scope)
         raise Engram::Error, "memory scope cannot be nil" if scope.nil?
       end
@@ -124,7 +159,8 @@ module Engram
           importance: row.importance || 1.0,
           metadata: row.metadata || {},
           created_at: row.created_at,
-          last_accessed_at: row.try(:last_accessed_at)
+          last_accessed_at: row.try(:last_accessed_at),
+          expires_at: row.try(:expires_at)
         )
       end
 

data/lib/engram/configuration.rb

@@ -8,13 +8,16 @@ module Engram
     attr_accessor :store, :embedder, :completion, :default_limit,
       :consolidator, :extraction_min_confidence, :processed_turns,
       :importance_weight, :recency_weight, :recency_halflife, :touch_on_recall,
-      :persistence_policy, :before_persist, :instrumentation_scope_identifier
+      :persistence_policy, :before_persist, :instrumentation_scope_identifier,
+      :recall_min_similarity, :injection_max_bytes
 
     def initialize
       @store = Adapters::InMemoryStore.new
       @embedder = Adapters::NullEmbedder.new
       @completion = nil # required for observe (extract/consolidate); nil until configured
       @default_limit = 5
+      @recall_min_similarity = nil # opt-in cosine floor, before importance/recency ranking
+      @injection_max_bytes = nil # cap the complete appended memory block, not the host prompt
       @consolidator = :heuristic # :heuristic (deterministic) or :llm (LLM-as-judge)
       @extraction_min_confidence = 0.5
       @processed_turns = Adapters::InMemoryProcessedTurns.new # idempotency for observe

data/lib/engram/integrations/ruby_llm.rb

@@ -9,21 +9,27 @@ module Engram
       #   chat = Engram.with_memory(RubyLLM.chat, memory: current_user.memory)
       #   chat.ask("why am I rate limited?")  # recall + inject happen automatically
       class MemoryChat
-        def initialize(chat, memory:, limit: Engram.config.default_limit)
+        def initialize(chat, memory:, limit: Engram.config.default_limit, kinds: nil,
+          min_similarity: Engram.config.recall_min_similarity, max_bytes: Engram.config.injection_max_bytes)
           @chat = chat
           @memory = memory
           @limit = limit
+          @kinds = kinds
+          @min_similarity = min_similarity
+          @max_bytes = max_bytes
         end
 
-        def ask(message, **opts)
-          augmented = @memory.inject_into(message.to_s, query: message.to_s, limit: @limit)
-          @chat.ask(augmented, **opts)
+        def ask(message, **opts, &block)
+          augmented = @memory.inject_into(message.to_s, query: message.to_s, limit: @limit,
+            kinds: @kinds, min_similarity: @min_similarity, max_bytes: @max_bytes)
+          @chat.ask(augmented, **opts, &block)
         end
 
         def method_missing(name, *args, **kwargs, &block)
           return super unless @chat.respond_to?(name)
 
-          @chat.public_send(name, *args, **kwargs, &block)
+          result = @chat.public_send(name, *args, **kwargs, &block)
+          result.equal?(@chat) ? self : result
         end
 
         def respond_to_missing?(name, include_private = false)
@@ -34,7 +40,9 @@ module Engram
   end
 
   # Convenience entrypoint.
-  def self.with_memory(chat, memory:, limit: config.default_limit)
-    Integrations::RubyLLM::MemoryChat.new(chat, memory: memory, limit: limit)
+  def self.with_memory(chat, memory:, limit: config.default_limit, kinds: nil,
+    min_similarity: config.recall_min_similarity, max_bytes: config.injection_max_bytes)
+    Integrations::RubyLLM::MemoryChat.new(chat, memory: memory, limit: limit, kinds: kinds,
+      min_similarity: min_similarity, max_bytes: max_bytes)
   end
 end

data/lib/engram/internal/candidate_integrity.rb

@@ -29,7 +29,8 @@ module Engram
         kind: [Symbol],
         importance: [Integer, Float],
         created_at: [Time],
-        last_accessed_at: [NilClass, Time]
+        last_accessed_at: [NilClass, Time],
+        expires_at: [NilClass, Time]
       }.freeze
 
       Snapshot = Struct.new(:collection, :collection_state, :records)

data/lib/engram/memory.rb

@@ -14,23 +14,23 @@ module Engram
 
     # Persist a memory record of the given kind. Returns nil when the configured
     # persistence policy rejects the record.
-    def add(content, kind: :fact, importance: 1.0, metadata: {})
+    def add(content, kind: :fact, importance: 1.0, metadata: {}, expires_at: nil)
       Engram::Instrumentation.instrument("add", Engram::Instrumentation.payload(scope: scope, store: @store, kind: kind)) do
-        embedding = @embedder.embed(content)
         record = Record.new(
           content: content,
           scope: scope,
-          embedding: embedding,
           kind: kind,
           importance: importance,
-          metadata: metadata
+          metadata: metadata,
+          expires_at: expires_at
         )
-        persist(record)
+        persist(record.with(embedding: @embedder.embed(content)))
       end
     end
 
     # Return the most relevant memories for a query.
-    def recall(query, limit: Engram.config.default_limit, kinds: nil)
+    def recall(query, limit: Engram.config.default_limit, kinds: nil,
+      min_similarity: Engram.config.recall_min_similarity)
       UseCases::Recall.new(
         store: @store,
         embedder: @embedder,
@@ -38,13 +38,15 @@ module Engram
         recency_weight: Engram.config.recency_weight,
         recency_halflife: Engram.config.recency_halflife,
         touch: Engram.config.touch_on_recall
-      ).call(query, scope: scope, limit: limit, kinds: kinds)
+      ).call(query, scope: scope, limit: limit, kinds: kinds, min_similarity: min_similarity)
     end
 
     # Recall, then inject into a prompt string.
-    def inject_into(prompt, query:, limit: Engram.config.default_limit, kinds: nil)
-      memories = recall(query, limit: limit, kinds: kinds)
-      UseCases::Inject.new.call(prompt: prompt, memories: memories)
+    def inject_into(prompt, query:, limit: Engram.config.default_limit, kinds: nil,
+      min_similarity: Engram.config.recall_min_similarity, max_bytes: Engram.config.injection_max_bytes)
+      UseCases::Inject.validate_max_bytes!(max_bytes)
+      memories = (max_bytes == 0) ? [] : recall(query, limit: limit, kinds: kinds, min_similarity: min_similarity)
+      UseCases::Inject.new.call(prompt: prompt, memories: memories, max_bytes: max_bytes)
     end
 
     # Derive memories from a conversation turn and consolidate them (v0.2).
@@ -86,6 +88,18 @@ module Engram
       @store.all(scope: scope)
     end
 
+    # Delete one memory in this scope. Returns 1 when deleted, or 0 when missing.
+    # Explicit deletion bypasses extraction, consolidation, and persistence hooks.
+    def forget(id:)
+      valid_id = id.instance_of?(Integer) || (id.instance_of?(String) && id.valid_encoding? && !id.strip.empty?)
+      raise ArgumentError, "id must be an Integer or a non-empty String" unless valid_id
+
+      payload = Engram::Instrumentation.payload(scope: scope, store: @store)
+      Engram::Instrumentation.instrument("forget", payload) do
+        payload[:deleted_count] = @store.delete(scope: scope, id: id)
+      end
+    end
+
     # Recompute embeddings (and embedding metadata) for memories in the scope.
     # When `stale_only` is true, only records whose current metadata does not match the
     # active embedder are rebuilt.
@@ -104,6 +118,13 @@ module Engram
         .call(scope: scope, older_than: older_than, min_importance: min_importance)
     end
 
+    # Delete expired records without loading their content or embeddings.
+    # Returns matched/deleted counts; dry runs only count eligible records.
+    def forget_expired(batch_size: 100, dry_run: false)
+      UseCases::ForgetExpired.new(store: @store)
+        .call(scope: scope, batch_size: batch_size, dry_run: dry_run)
+    end
+
     # Return the memories in this scope whose provenance references the exact host
     # source. `source_id` and `source_type` must each be non-blank Strings and are
     # matched exactly, without normalization. Source IDs are references, not an

data/lib/engram/ports/memory_store.rb

@@ -12,12 +12,13 @@ module Engram
 
       # Return up to `limit` Records in `scope` nearest to `embedding`,
       # ordered most-relevant first. When `kinds` is provided, only records with
-      # those canonical memory kinds are eligible.
+      # those canonical memory kinds are eligible. Exclude expired records before
+      # ordering and limiting; expires_at <= Time.now is expired, nil never expires.
       def search(embedding:, scope:, limit:, kinds: nil, embedding_metadata: nil)
         raise NotImplementedError, "#{self.class} must implement #search"
       end
 
-      # All Records for a scope (mostly for inspection/tests).
+      # All Records for a scope, including expired records (inspection/maintenance).
       # Supports optional `limit` and `offset` for batching large sweeps.
       # Returned records are sorted in stable `id` order when batching is used.
       # Use `after_id` for keyset pagination.
@@ -49,6 +50,19 @@ module Engram
         raise NotImplementedError, "#{self.class} must implement #delete"
       end
 
+      # Optional expiry maintenance: up to `limit` IDs with expires_at <= at,
+      # ordered by ID, strictly after after_id. Never includes nil expiry.
+      def expired_ids(scope:, at:, limit:, after_id: nil)
+        raise NotImplementedError, "#{self.class} does not implement #expired_ids"
+      end
+
+      # Optional expiry maintenance: delete only the requested IDs in scope that
+      # are still expired at the same cutoff. Recheck expiry atomically at deletion
+      # so a concurrent deadline extension is preserved. Returns the deleted count.
+      def delete_expired(scope:, ids:, at:)
+        raise NotImplementedError, "#{self.class} does not implement #delete_expired"
+      end
+
       # Update the last-accessed timestamp of a memory. Used by recency-aware recall.
       # Returns the number of affected rows: 1 when touched, 0 when the scoped record
       # does not exist.

data/lib/engram/rails/tasks.rake

@@ -3,6 +3,21 @@
 # Loaded by the Railtie in Rails apps (where :environment boots the app and its
 # initializers) and by the gem's own Rakefile (which defines a no-op :environment).
 namespace :engram do
+  desc "Delete expired memories in a scope. Usage: bundle exec rake 'engram:forget_expired[user:1]'"
+  task :forget_expired, [:scope] => :environment do |_, args|
+    scope = args[:scope]
+    raise ArgumentError, "forget_expired requires a scope argument" if scope.to_s.empty?
+
+    dry_run = case ENV.fetch("DRY_RUN", "false").downcase
+    when "true", "1", "yes" then true
+    when "false", "0", "no" then false
+    else raise ArgumentError, "DRY_RUN must be true or false"
+    end
+    batch_size = Integer(ENV.fetch("BATCH_SIZE", "100"))
+    result = Engram::Memory.new(scope: scope).forget_expired(batch_size: batch_size, dry_run: dry_run)
+    puts "matched=#{result[:matched]} deleted=#{result[:deleted]} dry_run=#{result[:dry_run]}"
+  end
+
   desc "Rebuild embeddings in a memory scope. Usage: bundle exec rake 'engram:rebuild_embeddings[user:1]'"
   task :rebuild_embeddings, [:scope] => :environment do |_, args|
     scope = args[:scope]

data/lib/engram/railtie.rb

@@ -15,9 +15,7 @@ module Engram
     end
 
     initializer "engram.active_job" do
-      ActiveSupport.on_load(:active_job) do
-        require "engram/rails/observe_job"
-      end
+      Engram.autoload :ObserveJob, "engram/rails/observe_job" if defined?(::ActiveJob::Base)
     end
 
     rake_tasks do

data/lib/engram/record.rb

@@ -9,15 +9,18 @@ module Engram
   # `semantic` kind is normalized to `fact` for compatibility with pre-1.0 records.
   class Record
     STATE_READERS = %i[
-      id content scope embedding kind importance metadata created_at last_accessed_at
+      id content scope embedding kind importance metadata created_at last_accessed_at expires_at
     ].freeze
 
     attr_accessor :id, :last_accessed_at
     attr_reader :content, :embedding, :scope, :kind, :importance, :metadata,
-      :created_at
+      :created_at, :expires_at
 
     def initialize(content:, scope:, id: nil, embedding: nil, kind: :fact,
-      importance: 1.0, metadata: {}, created_at: nil, last_accessed_at: nil)
+      importance: 1.0, metadata: {}, created_at: nil, last_accessed_at: nil, expires_at: nil)
+      unless expires_at.nil? || expires_at.is_a?(Time)
+        raise ArgumentError, "expires_at must be a Time or nil"
+      end
       @id = id
       @content = content
       @scope = scope
@@ -27,12 +30,17 @@ module Engram
       @metadata = metadata
       @created_at = created_at || Time.now
       @last_accessed_at = last_accessed_at
+      @expires_at = expires_at&.getutc
     end
 
     def with(**attributes)
       self.class.new(**to_h.merge(attributes))
     end
 
+    def expired?(at: Time.now)
+      !expires_at.nil? && expires_at <= at
+    end
+
     # Return structured supporting-source metadata when this record carries a
     # provenance schema understood by the installed Engram version.
     # Legacy, malformed, and future-schema records remain readable and return nil.

data/lib/engram/use_cases/forget_expired.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module Engram
+  module UseCases
+    # Deletes expired memories in bounded batches using one cutoff for the whole run.
+    class ForgetExpired
+      def initialize(store:)
+        @store = store
+      end
+
+      def call(scope:, batch_size: 100, dry_run: false, now: Time.now)
+        unless batch_size.is_a?(Integer) && batch_size.positive?
+          raise ArgumentError, "batch_size must be a positive integer"
+        end
+        unless dry_run.equal?(true) || dry_run.equal?(false)
+          raise ArgumentError, "dry_run must be true or false"
+        end
+        unless @store.respond_to?(:expired_ids) && @store.respond_to?(:delete_expired)
+          raise Engram::Error, "forget_expired requires a store implementing expired_ids and delete_expired"
+        end
+
+        counts = {matched: 0, deleted: 0, dry_run: dry_run}
+        payload = Engram::Instrumentation.payload(scope: scope, store: @store, **counts)
+        Engram::Instrumentation.instrument("forget_expired", payload) do
+          after_id = nil
+          loop do
+            ids = @store.expired_ids(scope: scope, at: now, limit: batch_size, after_id: after_id)
+            break if ids.empty?
+            raise Engram::Error, "expired_ids must advance its cursor" if ids.last == after_id
+
+            counts[:matched] += ids.length
+            counts[:deleted] += @store.delete_expired(scope: scope, ids: ids, at: now) unless dry_run
+            payload.merge!(counts)
+            after_id = ids.last
+          end
+          counts
+        end
+      rescue NotImplementedError
+        raise Engram::Error, "forget_expired requires a store implementing expired_ids and delete_expired"
+      end
+    end
+  end
+end

data/lib/engram/use_cases/inject.rb

@@ -12,14 +12,42 @@ module Engram
         @header = header
       end
 
-      # Returns a new prompt string. If there are no memories, the prompt is unchanged.
-      def call(prompt:, memories:)
-        payload = {memory_count: memories&.size.to_i}
+      def self.validate_max_bytes!(value)
+        return if value.nil? || (value.is_a?(Integer) && value >= 0)
+
+        raise ArgumentError, "max_bytes must be a non-negative integer, or nil"
+      end
+
+      # The budget includes the header, delimiters, escaping, and separators, but
+      # excludes the original prompt. Skip whole memories that do not fit.
+      def call(prompt:, memories:, max_bytes: nil)
+        self.class.validate_max_bytes!(max_bytes)
+        payload = {memory_count: memories&.size.to_i, injected_count: 0,
+                   skipped_count: memories&.size.to_i, injected_bytes: 0, max_bytes: max_bytes}.compact
         Engram::Instrumentation.instrument("inject", payload) do
-          next prompt if memories.nil? || memories.empty?
+          next prompt if memories.nil? || memories.empty? || max_bytes == 0
+
+          prefix = "\n\n#{@header}:\n<engram-memories>\n"
+          suffix = "\n</engram-memories>"
+          bytes = prefix.bytesize + suffix.bytesize
+          lines = []
+          now = Time.now
+          memories.each do |memory|
+            next if memory.expired?(at: now)
+
+            line = render_memory(memory)
+            added_bytes = line.bytesize + (lines.empty? ? 0 : 1)
+            next if max_bytes && bytes + added_bytes > max_bytes
+
+            lines << line
+            bytes += added_bytes
+          end
+          next prompt if lines.empty?
 
-          block = memories.map { |memory| render_memory(memory) }.join("\n")
-          "#{prompt}\n\n#{@header}:\n<engram-memories>\n#{block}\n</engram-memories>"
+          payload[:injected_count] = lines.size
+          payload[:skipped_count] -= lines.size
+          payload[:injected_bytes] = bytes
+          "#{prompt}#{prefix}#{lines.join("\n")}#{suffix}"
         end
       end
 

data/lib/engram/use_cases/recall.rb

@@ -24,17 +24,27 @@ module Engram
       end
 
       # Returns Array<Record>, most relevant first.
-      def call(query, scope:, limit: Engram.config.default_limit, kinds: nil)
+      def call(query, scope:, limit: Engram.config.default_limit, kinds: nil, min_similarity: nil)
         raise ArgumentError, "query must be a non-empty string" if query.to_s.strip.empty?
+        unless limit.is_a?(Integer) && limit >= 0
+          raise ArgumentError, "limit must be a non-negative integer"
+        end
+        validate_min_similarity!(min_similarity)
 
         payload = Engram::Instrumentation.payload(
           scope: scope,
           store: @store,
           limit: limit,
           kinds: Array(kinds).map(&:to_s),
-          reranking: reranking?
+          reranking: reranking?,
+          min_similarity: min_similarity,
+          candidate_count: 0,
+          filtered_count: 0,
+          result_count: 0
         )
         Engram::Instrumentation.instrument("recall", payload) do
+          next [] if limit.zero?
+
           embedding = @embedder.embed(query)
           embedding_metadata = Engram::EmbeddingMetadata.for_embedder(@embedder, embedding: embedding)
           pool_limit = reranking? ? limit * @pool_factor : limit
@@ -47,16 +57,43 @@ module Engram
             kinds: kinds
           )
 
-          results = (reranking? ? rerank(pool, embedding) : pool).first(limit)
+          now = Time.now
+          candidates = pool.select do |record|
+            !record.expired?(at: now) &&
+              (min_similarity.nil? || meets_similarity?(record.embedding, embedding, min_similarity))
+          end
+          results = (reranking? ? rerank(candidates, embedding) : candidates).first(limit)
           touch(results, scope) if @touch
           payload[:result_count] = results.size
           payload[:candidate_count] = pool.size
+          payload[:filtered_count] = pool.size - candidates.size
           results
         end
       end
 
       private
 
+      def validate_min_similarity!(value)
+        return if value.nil?
+        return if value.is_a?(Numeric) && value.real? && value.finite? && value.between?(-1, 1)
+
+        raise ArgumentError, "min_similarity must be a finite number between -1 and 1, or nil"
+      end
+
+      def meets_similarity?(embedding, query_embedding, minimum)
+        return false unless comparable_vector?(embedding) && comparable_vector?(query_embedding)
+        return false unless embedding.length == query_embedding.length
+
+        similarity = Engram::Math.cosine_similarity(query_embedding, embedding)
+        similarity.finite? && similarity.clamp(-1.0, 1.0) >= minimum
+      end
+
+      def comparable_vector?(vector)
+        vector.is_a?(Array) && !vector.empty? &&
+          vector.all? { |value| value.is_a?(Numeric) && value.real? && value.finite? } &&
+          vector.any? { |value| !value.zero? }
+      end
+
       def reranking?
         !@importance_weight.zero? || !@recency_weight.zero?
       end

data/lib/engram/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Engram
-  VERSION = "0.6.0"
+  VERSION = "0.8.0"
 end

data/lib/engram.rb

@@ -31,6 +31,7 @@ require_relative "engram/use_cases/recall"
 require_relative "engram/use_cases/inject"
 require_relative "engram/use_cases/observe"
 require_relative "engram/use_cases/forget"
+require_relative "engram/use_cases/forget_expired"
 require_relative "engram/use_cases/rebuild_embeddings"
 require_relative "engram/use_cases/source_impact"
 require_relative "engram/use_cases/grounding_report"

data/lib/generators/engram/expiry_generator.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+
+module Engram
+  module Generators
+    # Adds expiry to a table created by Engram 0.7 or earlier.
+    class ExpiryGenerator < ::Rails::Generators::Base
+      include ::Rails::Generators::Migration
+
+      source_root File.expand_path("templates", __dir__)
+
+      def create_migration_file
+        migration_template "add_expiry_to_engram_memories.rb.tt",
+          "db/migrate/add_expiry_to_engram_memories.rb"
+      end
+
+      def self.next_migration_number(dir)
+        ::ActiveRecord::Generators::Base.next_migration_number(dir)
+      end
+
+      private
+
+      def migration_version
+        "#{::ActiveRecord::VERSION::MAJOR}.#{::ActiveRecord::VERSION::MINOR}"
+      end
+    end
+  end
+end

data/lib/generators/engram/templates/add_expiry_to_engram_memories.rb.tt

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+class AddExpiryToEngramMemories < ActiveRecord::Migration[<%= migration_version %>]
+  disable_ddl_transaction!
+
+  def change
+    add_column :engram_memories, :expires_at, :datetime
+    add_index :engram_memories, [:scope, :expires_at],
+      name: "index_engram_memories_on_scope_and_expiry",
+      where: "expires_at IS NOT NULL", algorithm: :concurrently
+  end
+end

data/lib/generators/engram/templates/create_engram_memories.rb.tt

@@ -12,11 +12,15 @@ class CreateEngramMemories < ActiveRecord::Migration[<%= migration_version %>]
       t.jsonb :metadata, null: false, default: {}
       t.vector :embedding, limit: <%= dimensions %>
       t.datetime :last_accessed_at
+      t.datetime :expires_at
 
       t.timestamps
     end
 
     add_index :engram_memories, :scope
+    add_index :engram_memories, [:scope, :expires_at],
+      name: "index_engram_memories_on_scope_and_expiry",
+      where: "expires_at IS NOT NULL"
 
     # For larger datasets, choose one approximate vector index after backfilling data.
     # HNSW is a strong default for read-heavy recall workloads with frequent inserts.

data/lib/generators/engram/templates/initializer.rb.tt

@@ -9,4 +9,9 @@ Engram.configure do |config|
 
   # How many memories to recall by default.
   config.default_limit = 5
+
+  # Minimum cosine similarity. Tune for your embedding model.
+  # config.recall_min_similarity = 0.5
+  # Cap the appended memory block in bytes, including escaped text and delimiters.
+  # config.injection_max_bytes = 8_000
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: engram
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.8.0
 platform: ruby
 authors:
 - Alexandr Kholodniak
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2026-07-28 00:00:00.000000000 Z
+date: 2026-09-15 00:00:00.000000000 Z
 dependencies: []
 description: |
   Engram gives AI agents durable, long-term memory. It recalls relevant facts about a
@@ -65,6 +65,7 @@ files:
 - lib/engram/reserved_metadata.rb
 - lib/engram/turn_digest.rb
 - lib/engram/use_cases/forget.rb
+- lib/engram/use_cases/forget_expired.rb
 - lib/engram/use_cases/grounding_report.rb
 - lib/engram/use_cases/inject.rb
 - lib/engram/use_cases/observe.rb
@@ -72,7 +73,9 @@ files:
 - lib/engram/use_cases/recall.rb
 - lib/engram/use_cases/source_impact.rb
 - lib/engram/version.rb
+- lib/generators/engram/expiry_generator.rb
 - lib/generators/engram/install_generator.rb
+- lib/generators/engram/templates/add_expiry_to_engram_memories.rb.tt
 - lib/generators/engram/templates/create_engram_memories.rb.tt
 - lib/generators/engram/templates/initializer.rb.tt
 - lib/generators/engram/templates/memory_record.rb.tt