engram 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8a38dfd456e4e89f37a9d4d8949a5bb77b9736d95621e9172ba1db72fb3a5771
-  data.tar.gz: 0b537ee13bea0313fd770c529024af45e9ace8fb84f7e49621c2dc7b9d6b888c
+  metadata.gz: c623f8198a9f71905e6e0c904b6cf60d5d6a0734b1e8fdaf877611a815f2a919
+  data.tar.gz: aca21e3da50239a5d02822306524adc6647883ef607af57747b05ce7ecb78827
 SHA512:
-  metadata.gz: 768d5d18a9ce04f60db58957127df76fbdc30482df4976d7d04b082af005850ba821ba45a35417c33946668af60331e869cf930e6239409ba10168a453f58665
-  data.tar.gz: 7ec7997ba40947deede93b6e970a549bf6caae32eed79f0e4c3922ebecbf05bc1545b5565afc521c9dab6c0f95b27898b3ae9bd9f3f57d9935a48d18e9211322
+  metadata.gz: 55f5e8335a86ec28cd28b17af162883fcc7c58dc56b903fc18b924c4278b593d9c8945cbf19cb28649a4c66de57ff38e6af46f3c50b15a6a96a8d729b90fc6ba
+  data.tar.gz: 519081acdc6f1b284df51bc664d274ca31befbc2d7a002ef9d62fee2f31c165f64c2896ab5f2dddca771b02c2052dcb630ef785e440776931d53d7f994de6e47

data/CHANGELOG.md

@@ -5,6 +5,62 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [0.6.0] - 2026-07-24
+
+### Added
+- `Engram::Memory#grounding_report` (and `Engram::UseCases::GroundingReport`) returns frozen,
+  scope-bound record counts by weakest source alignment, with legacy or unrecognized provenance
+  counted as unattributed and no record or source text exposed.
+- Add an immutable host-supplied source-text contract that validates provenance source identity
+  and Unicode-codepoint span bounds, and resolves authorized supporting text without retaining
+  source transcripts.
+- Optional `Engram::Extraction` results let custom extractors attach versioned source
+  provenance while remaining compatible with plain `Engram::Record` results.
+- `Engram::Record#provenance` exposes understood supporting source IDs, alignments, and spans
+  on recalled records while preserving tolerant reads for legacy and future schemas.
+- `Engram::Memory#memories_from_source` (and `Engram::UseCases::SourceImpact`) return the
+  records in a scope whose provenance references an exact host source. `source_id` and
+  `source_type` must each be a non-blank String and are matched exactly without trimming or
+  normalization. The lookup is scope-bound and returns only records, never source
+  text; source IDs are references, not an authorization boundary. Legacy, malformed, and
+  future-schema provenance do not match.
+
+### Changed
+- Persistence accepts records without provenance, rejects structurally ungrounded provenance
+  by default (configurable only with the exact boolean `allow_ungrounded: true`), and fails closed on malformed or
+  unknown future provenance during writes while keeping reads tolerant. Provenance validation
+  does not verify source text, and source IDs are references rather than authorization
+  boundaries. The built-in LLM extractor does not emit grounded provenance.
+- Existing `before_persist` hooks may continue to transform record content or embeddings, but
+  may neither add, remove, nor alter provenance; this restriction includes legacy records with
+  no provenance.
+- Extractors must return an `Array`; each result may be a plain `Engram::Record` or an
+  `Engram::Extraction` carrying provenance.
+- Custom consolidator decisions must reference the actual same-scope `Engram::Record` instance
+  supplied for reconciliation and must treat both the candidates array and records as read-only.
+  Engram now fails closed on collection replacement/reordering/iteration overrides and on
+  security-relevant nested value, identity/alias/topology, frozen-state, custom-behavior, or
+  hidden-state changes before destructive authorization. Arbitrary application metadata values
+  remain opaque and preserve their identity without invoking equality or serialization behavior;
+  provenance is independently parsed and integrity-protected.
+  Substituted, modified, missing, malformed, or cross-scope candidates fail closed.
+  `Observe` preflights the complete decision batch, including persistence policy and hook
+  transformations, before beginning store mutations.
+
+### Fixed
+- Provenance canonicalization now iterates untrusted metadata through core `Hash#keys`/`#values`
+  by position instead of `Hash#each_pair`. On Ruby 3.4 a caller-supplied hash left in a
+  delete-then-insert collision state could make `Hash#each_pair` invoke a key's `#eql?` mid-iteration,
+  letting a hostile `String`-subclass key run application code inside the behavior-free canonicalizer.
+  The adversarial hostile-key suite is now deterministic across Ruby 3.2–3.4.
+- Observation rejects extractor candidates with caller-supplied IDs, and `InMemoryStore#add`
+  always allocates a fresh ID like the pgvector adapter, preventing same- or cross-scope record
+  replacement through add semantics.
+- `forget` now separates destructive provenance authorization from write-content filtering and
+  redaction, so secret or transient memories can be deleted. Custom policies may implement
+  `allow_destructive?` with a strict boolean return; write-only policies no longer authorize
+  deletion through their `call` result.
+
 ## [0.5.0] - 2026-07-17
 
 ### Added

data/README.md

@@ -52,7 +52,7 @@ chat.ask("Why am I being rate limited?")
 
 ## Feature overview
 
-- Zero-dependency pure Ruby core with in-memory defaults for tests and local development.
+- Pure Ruby core with zero runtime dependencies and in-memory defaults for tests and local development.
 - Rails `has_memory` macro, install generator, and background `observe_later` job.
 - Postgres + pgvector storage through an optional ActiveRecord/neighbor adapter.
 - RubyLLM embedder and completion adapters for provider-backed embeddings and extraction.
@@ -74,7 +74,7 @@ explicitly version-gated:
 - Core types and facade: `Engram::Memory`, `Engram::Record`, `Engram::Decision`, `Engram::PersistencePolicy`, and `Engram.with_memory`.
 - Store and adapter ports: `Engram::Ports::MemoryStore`, `Engram::Ports::Embedder`, `Engram::Ports::Completion`.
 - Rails integration points: `has_memory`, `Memory#observe_later`, and generator outputs under `engram:` rake tasks.
-- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget_stale`, and `rebuild_embeddings`.
+- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget_stale`, `rebuild_embeddings`, and `memories_from_source`.
 - RubyLLM adapter contract points and evaluator entrypoints (`rake eval`, `rake eval:real`).
 
 ### Backward-compatibility commitments (pre-1.0)
@@ -103,7 +103,7 @@ Use `CHANGELOG.md` as the authoritative source for breaking/compatibility change
 gem "engram"
 ```
 
-The core has **zero runtime dependencies**. Optional adapters need host-app dependencies:
+The core has zero runtime dependencies and does not require Rails, ActiveRecord, or a database. Optional adapters need host-app dependencies:
 
 - `Engram::Adapters::PgvectorStore` → ActiveRecord + `neighbor` + Postgres/pgvector
 - `Engram::Adapters::RubyLLMEmbedder` and `Engram::Adapters::RubyLLMCompletion` → `ruby_llm`
@@ -291,6 +291,25 @@ memory.observe([
 # extracts "User is on the Pro plan", and if a "Free plan" memory exists, updates it
 ```
 
+### Custom consolidator candidate contract
+
+`reconcile_all(candidates:, scope:)` receives a plain `Array` of plain `Engram::Record`
+instances. A custom consolidator may inspect candidates and return decisions that reference
+those exact instances, but it must not replace, append, remove, or reorder collection entries;
+override collection iteration; add collection or record state or behavior; or mutate the
+records. Engram verifies record and collection integrity before it authorizes any add, update,
+or deletion and fails closed on a violation.
+
+Plain arrays and hashes in candidate state are detached without serializing their leaves.
+Core scalar/container state is integrity-protected without dispatching overridden traversal or
+equality. Arbitrary application metadata values (including value objects, procs, and IO) remain
+opaque, retain their identity, and are never compared or serialized by the integrity check.
+Their internal mutable state is consequently outside that check; provenance remains parsed and
+validated independently before any authorization. Containers must be acyclic, hashes cannot
+have default procs, and behavior-bearing core values remain unsupported.
+Extracted candidates must also have a nil `id`: IDs are store-owned and are allocated only
+when an add is persisted.
+
 ## Memory kinds and persistence policy
 
 Every memory has a normalized `kind`:
@@ -329,6 +348,107 @@ Engram.configure do |config|
 end
 ```
 
+Write-content filtering and transformation are not deletion authorization. `forget` still
+validates scope, target existence, candidate integrity, and provenance, but it does not run
+`before_persist` or the policy's `call` method; this allows secret, transient, and redacted
+memories to be removed. A custom policy can additionally control destructive decisions by
+implementing `allow_destructive?(record)` and returning exactly `true` or `false`. Policies
+that only implement `call` affect writes but do not prevent deletion. The built-in policy uses
+this separate hook to retain its ungrounded-provenance protection.
+
+### Extraction provenance
+
+Custom extractors remain compatible when they return an array of plain `Engram::Record`
+values. They may instead return `Engram::Extraction` values to attach optional structured
+`Engram::Provenance` to a candidate; arrays may contain either type. Records with no
+provenance remain accepted.
+
+Recalled records expose understood provenance directly, without loading source content:
+
+```ruby
+record = memory.recall("where did this preference come from?").first
+
+record.provenance&.sources&.each do |source|
+  puts [source.source_id, source.alignment, source.spans.map(&:to_h)].inspect
+end
+```
+
+When the host application has authorized and loaded the referenced message, it can validate
+the source identity and span bounds before presenting supporting text:
+
+```ruby
+source = record.provenance.sources.first
+source_text = Engram::Provenance::SourceText.new(
+  source_id: source.source_id,
+  source_type: source.source_type,
+  message_index: source.message_index,
+  role: source.role,
+  text: authorized_message_body
+)
+
+source.validate_source_text!(source_text)
+source.supporting_text(source_text) # text for each zero-based, end-exclusive codepoint span
+```
+
+Validation fails when any source identity field differs or a span exceeds that one message.
+Offsets count Unicode codepoints, not bytes or user-perceived grapheme clusters.
+`exact` and `normalized` remain extractor assertions: this API verifies identity and bounds but
+does not claim that supporting text semantically entails the memory.
+
+`Record#provenance` returns `nil` for legacy records and for malformed or future schemas so
+reads remain compatible. Treat the returned source IDs and spans as untrusted references.
+They do not prove source access or authorize retrieving source content; applications must
+enforce the record's scope and their own source authorization before resolving them.
+
+By default, the persistence policy rejects provenance containing a source marked
+`ungrounded`. Applications that intentionally accept it can opt out:
+
+```ruby
+Engram.configure do |config|
+  config.persistence_policy = Engram::PersistencePolicy.new(allow_ungrounded: true)
+end
+```
+
+Persistence validation is structural: Engram does not retain source text or verify that an
+alignment is truthful. Source IDs are host-owned references, not authorization boundaries.
+The built-in `Engram::Extractors::LLMExtractor` continues to return plain records and does not
+emit grounded provenance. Reads tolerate malformed or future provenance metadata, but writes
+fail closed when provenance is malformed or uses an unknown future schema version; upgrade an
+older writer before rewriting such records.
+
+### Source impact lookup
+
+To find which memories in a scope a host source produced, look them up by exact source
+reference:
+
+```ruby
+current_user.memory.memories_from_source(
+  source_id: "conversation:42",
+  source_type: "conversation"
+)
+```
+
+`source_id` and `source_type` must each be a non-blank String and are matched exactly, with no
+trimming or normalization. The lookup is bound to the scope and returns an `Array` of
+`Engram::Record` values (empty when nothing matches) — never source text. Result order follows
+the store adapter's enumeration and is not otherwise guaranteed. Source IDs are references,
+not an authorization boundary, so continue to enforce scope and your own source authorization.
+Legacy, malformed, and future-schema provenance do not match.
+
+### Grounding report
+
+To measure provenance grounding coverage without exposing record or source text, request the
+scope-bound report:
+
+```ruby
+current_user.memory.grounding_report
+# => {exact: 12, normalized: 3, inferred: 2, ungrounded: 1, unattributed: 4, total: 22}
+```
+
+Each record is counted once by its weakest source alignment (`exact`, `normalized`, `inferred`,
+then `ungrounded`). Records with no understood provenance, including legacy, malformed, and
+future-schema provenance, count as `unattributed`. The returned counts-only Hash is frozen.
+
 ## Prompt-injection and memory-injection safety
 
 Injected memories are rendered as typed XML-like elements with escaped content, which keeps

data/lib/engram/adapters/in_memory_store.rb

@@ -15,7 +15,7 @@ module Engram
       def add(record)
         validate_scope!(record.scope)
 
-        record.id ||= (@sequence += 1)
+        record.id = (@sequence += 1)
         @records[record.id] = record
         record
       end
@@ -44,6 +44,10 @@ module Engram
         records
       end
 
+      def existing_ids(scope:, ids:)
+        ids.uniq.select { |id| @records[id]&.scope == scope }
+      end
+
       def update(scope:, id:, record:)
         existing = @records[id]
         raise Engram::Error, "no memory with id #{id.inspect} in scope #{scope.inspect}" unless existing&.scope == scope

data/lib/engram/adapters/pgvector_store.rb

@@ -49,6 +49,21 @@ module Engram
         query.map { |row| to_record(row) }
       end
 
+      def existing_ids(scope:, ids:)
+        persisted_ids = model.where(scope: scope, id: ids).pluck(:id)
+        id_type = model.type_for_attribute("id")
+        persisted_lookup = persisted_ids.each_with_object({}) { |id, lookup| lookup[id] = true }
+        matched_ids = {}
+
+        ids.select do |id|
+          cast_id = id_type.cast(id)
+          next false unless persisted_lookup[cast_id]
+          next false if matched_ids[cast_id]
+
+          matched_ids[cast_id] = true
+        end
+      end
+
       def update(scope:, id:, record:)
         raise Engram::Error, "cannot move memory across scopes" unless record.scope == scope
 

data/lib/engram/extraction.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module Engram
+  # An immutable extractor result that carries a candidate Record and its source provenance.
+  # Converting it creates a new record and leaves the caller-owned record and metadata untouched.
+  class Extraction
+    attr_reader :record, :provenance
+
+    def initialize(record:, provenance:)
+      is_record = Object.instance_method(:is_a?).bind_call(record, Engram::Record)
+      raise ArgumentError, "record must be an Engram::Record" unless is_record
+
+      is_provenance = Object.instance_method(:is_a?).bind_call(provenance, Engram::Provenance)
+      raise ArgumentError, "provenance must be an Engram::Provenance" unless is_provenance
+
+      @record = record
+      @provenance = provenance
+      freeze
+    end
+
+    def to_record
+      source = Engram::Internal::CandidateIntegrity.new.detach(record)
+      attributes = Engram::Record::STATE_READERS.to_h do |attribute|
+        [attribute, Engram::Record.instance_method(attribute).bind_call(source)]
+      end
+      attributes[:metadata] = Engram::Provenance.attach(attributes.fetch(:metadata), provenance)
+      Engram::Record.new(**attributes)
+    end
+  end
+end