engram 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +25 -0
- data/README.md +109 -9
- data/lib/engram/configuration.rb +4 -1
- data/lib/engram/integrations/ruby_llm.rb +15 -7
- data/lib/engram/memory.rb +20 -5
- data/lib/engram/railtie.rb +1 -3
- data/lib/engram/use_cases/inject.rb +32 -7
- data/lib/engram/use_cases/recall.rb +40 -3
- data/lib/engram/version.rb +1 -1
- data/lib/generators/engram/templates/initializer.rb.tt +5 -0
- metadata +2 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: f2d96222b1c43e68da7da579a6b9d1d39f85f5cbe2480be6cd39a19dcf36adc6
|
|
4
|
+
data.tar.gz: 53bc4261b347e498d20ea4171164b9478c55134067c33715641017b7572c78a7
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a8001a49d7c280ded764b75e2f62695f3eeaf229e114e5455ee9bfa32f58d0c2a94fd9d70a1b563e246da0efcb570fe5f95ca6ecd61e69ca3cad758a0491abae
|
|
7
|
+
data.tar.gz: 29d1e339241f3edb54485925f20b78f81740770fc966c8e838cefaef839e191597080249f578f33c8f79c5eb2cc89563b0187cf897b296ab660ff4510a3b0d4d
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,31 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [0.7.0] - 2026-09-13
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
- `Memory#forget(id:)` deletes one memory in the current scope, returning `1` when deleted
|
|
12
|
+
or `0` when missing. Explicit deletion bypasses persistence hooks and emits `forget.engram`
|
|
13
|
+
with the deleted count. Arrays, ranges, nil, and blank IDs are rejected.
|
|
14
|
+
- Optional `min_similarity:` on recall, injection, and `Engram.with_memory`, with
|
|
15
|
+
`config.recall_min_similarity` as the facade default. Filters cosine similarity before
|
|
16
|
+
importance/recency ranking and touching; nil preserves existing top-k behavior.
|
|
17
|
+
- Optional `max_bytes:` on injection and `Engram.with_memory`, with
|
|
18
|
+
`config.injection_max_bytes` as the default. Includes the complete appended memory block,
|
|
19
|
+
skips whole memories that do not fit, and preserves the original prompt.
|
|
20
|
+
- Metrics for filtered candidates, injected/skipped memories, and appended bytes.
|
|
21
|
+
- `MIN_SIMILARITY` eval control and negative-query false-positive/abstention measurements.
|
|
22
|
+
|
|
23
|
+
### Fixed
|
|
24
|
+
- Load `ObserveJob` on first use so `observe_later` works before another job has loaded
|
|
25
|
+
`ActiveJob::Base` in a Rails application.
|
|
26
|
+
- Forward RubyLLM streaming blocks and retain the memory wrapper when fluent chat methods
|
|
27
|
+
return the underlying chat, so chained configuration does not bypass memory injection.
|
|
28
|
+
|
|
29
|
+
### Changed
|
|
30
|
+
- Recall rejects negative, fractional, nil, and string limits with `ArgumentError`; use an
|
|
31
|
+
integer. A zero limit skips embedding and search. A zero injection byte budget skips recall.
|
|
32
|
+
|
|
8
33
|
## [0.6.0] - 2026-07-24
|
|
9
34
|
|
|
10
35
|
### Added
|
data/README.md
CHANGED
|
@@ -58,6 +58,8 @@ chat.ask("Why am I being rate limited?")
|
|
|
58
58
|
- RubyLLM embedder and completion adapters for provider-backed embeddings and extraction.
|
|
59
59
|
- Canonical memory kinds: `fact`, `preference`, `instruction`, and `episodic`.
|
|
60
60
|
- Typed recall filters and typed, escaped memory injection.
|
|
61
|
+
- Optional cosine relevance thresholds and byte budgets for injected context.
|
|
62
|
+
- RubyLLM streaming and fluent chat configuration through the memory wrapper.
|
|
61
63
|
- Persistence policy that rejects obvious secrets and transient task-progress updates before storage.
|
|
62
64
|
- Idempotent observation, recency/importance-aware ranking, recall touching, and stale-memory pruning.
|
|
63
65
|
|
|
@@ -74,7 +76,7 @@ explicitly version-gated:
|
|
|
74
76
|
- Core types and facade: `Engram::Memory`, `Engram::Record`, `Engram::Decision`, `Engram::PersistencePolicy`, and `Engram.with_memory`.
|
|
75
77
|
- Store and adapter ports: `Engram::Ports::MemoryStore`, `Engram::Ports::Embedder`, `Engram::Ports::Completion`.
|
|
76
78
|
- Rails integration points: `has_memory`, `Memory#observe_later`, and generator outputs under `engram:` rake tasks.
|
|
77
|
-
- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget_stale`, `rebuild_embeddings`, and `memories_from_source`.
|
|
79
|
+
- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget`, `forget_stale`, `rebuild_embeddings`, and `memories_from_source`.
|
|
78
80
|
- RubyLLM adapter contract points and evaluator entrypoints (`rake eval`, `rake eval:real`).
|
|
79
81
|
|
|
80
82
|
### Backward-compatibility commitments (pre-1.0)
|
|
@@ -223,6 +225,25 @@ class AddEngramMemoryEmbeddingIndex < ActiveRecord::Migration[8.0]
|
|
|
223
225
|
end
|
|
224
226
|
```
|
|
225
227
|
|
|
228
|
+
### Filtered vector search
|
|
229
|
+
|
|
230
|
+
Keep the generated B-tree index on `scope`. With an approximate vector index, pgvector
|
|
231
|
+
applies scope and kind filters after the index scan. This can return fewer than `limit`
|
|
232
|
+
memories even when enough matching rows exist.
|
|
233
|
+
|
|
234
|
+
On pgvector 0.8.0+, HNSW iterative scans can search further:
|
|
235
|
+
|
|
236
|
+
```ruby
|
|
237
|
+
Engram::MemoryRecord.transaction do
|
|
238
|
+
Engram::MemoryRecord.connection.execute("SET LOCAL hnsw.iterative_scan = strict_order")
|
|
239
|
+
current_user.memory.recall("billing preferences", limit: 5)
|
|
240
|
+
end
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
The setting lasts for the transaction. Iterative scans add latency and still have scan
|
|
244
|
+
limits. Compare them with exact search on your scoped data before enabling them.
|
|
245
|
+
See [pgvector's filtering docs](https://github.com/pgvector/pgvector#filtering).
|
|
246
|
+
|
|
226
247
|
## Model/provider configuration
|
|
227
248
|
|
|
228
249
|
Engram is model-provider agnostic. The core only depends on two ports:
|
|
@@ -272,6 +293,25 @@ chat.ask("why am I being rate limited?")
|
|
|
272
293
|
# recall + inject happen automatically before the model sees the message
|
|
273
294
|
```
|
|
274
295
|
|
|
296
|
+
Streaming and chained chat configuration work through the wrapper:
|
|
297
|
+
|
|
298
|
+
```ruby
|
|
299
|
+
chat = Engram.with_memory(
|
|
300
|
+
RubyLLM.chat,
|
|
301
|
+
memory: current_user.memory,
|
|
302
|
+
kinds: [:fact, :preference],
|
|
303
|
+
min_similarity: 0.5, # tune for your embedding model
|
|
304
|
+
max_bytes: 8_000
|
|
305
|
+
)
|
|
306
|
+
|
|
307
|
+
response = chat.with_instructions("Keep answers concise.").ask("Which plan am I on?") do |chunk|
|
|
308
|
+
print chunk.content
|
|
309
|
+
end
|
|
310
|
+
```
|
|
311
|
+
|
|
312
|
+
`ask` forwards request options and the streaming block to RubyLLM and returns its final
|
|
313
|
+
response. `with_memory` captures memory defaults when the wrapper is created.
|
|
314
|
+
|
|
275
315
|
## Automatic memory
|
|
276
316
|
|
|
277
317
|
Instead of adding facts by hand, let engram derive them from a conversation turn. It
|
|
@@ -348,8 +388,8 @@ Engram.configure do |config|
|
|
|
348
388
|
end
|
|
349
389
|
```
|
|
350
390
|
|
|
351
|
-
Write-content filtering and transformation are not deletion authorization.
|
|
352
|
-
validates scope, target existence, candidate integrity, and provenance, but it does not run
|
|
391
|
+
Write-content filtering and transformation are not deletion authorization. A consolidation
|
|
392
|
+
`forget` decision still validates scope, target existence, candidate integrity, and provenance, but it does not run
|
|
353
393
|
`before_persist` or the policy's `call` method; this allows secret, transient, and redacted
|
|
354
394
|
memories to be removed. A custom policy can additionally control destructive decisions by
|
|
355
395
|
implementing `allow_destructive?(record)` and returning exactly `true` or `false`. Policies
|
|
@@ -482,6 +522,39 @@ with the old `semantic` kind value.
|
|
|
482
522
|
|
|
483
523
|
## Tuning and maintenance
|
|
484
524
|
|
|
525
|
+
### Recall thresholds and injection limits
|
|
526
|
+
|
|
527
|
+
Recall returns the nearest matches even for unrelated queries. Set a minimum cosine
|
|
528
|
+
similarity to exclude weak matches and a byte limit to keep injected context small:
|
|
529
|
+
|
|
530
|
+
```ruby
|
|
531
|
+
Engram.configure do |config|
|
|
532
|
+
config.recall_min_similarity = 0.5 # tune for your embedding model
|
|
533
|
+
config.injection_max_bytes = 8_000
|
|
534
|
+
end
|
|
535
|
+
|
|
536
|
+
memory.recall("billing preferences", min_similarity: 0.6)
|
|
537
|
+
memory.inject_into(prompt, query: "billing preferences", max_bytes: 4_000)
|
|
538
|
+
```
|
|
539
|
+
|
|
540
|
+
`min_similarity` accepts a finite number in `[-1, 1]`. It filters candidates before
|
|
541
|
+
importance/recency ranking and touching, and can return no matches. Filtering happens in
|
|
542
|
+
Ruby; custom stores keep their existing search signature. It does not refill the candidate
|
|
543
|
+
pool or suppress embedding compatibility errors.
|
|
544
|
+
|
|
545
|
+
`max_bytes` limits the appended memory section, including escaped text, headers, and tags.
|
|
546
|
+
It excludes the original prompt and chat history and counts bytes, not tokens. Memories
|
|
547
|
+
are considered in recall order; those that do not fit are skipped without truncation.
|
|
548
|
+
If none fit, the prompt is unchanged. With `touch_on_recall`, retrieved memories are touched
|
|
549
|
+
even if the byte limit later excludes them.
|
|
550
|
+
|
|
551
|
+
Both settings default to nil. Pass nil explicitly to override a configured value.
|
|
552
|
+
`limit` and `max_bytes` require non-negative integers, except that `max_bytes: nil` removes
|
|
553
|
+
the cap. `limit: 0` skips embedding and search; `max_bytes: 0` skips recall during injection.
|
|
554
|
+
Invalid values raise `ArgumentError`.
|
|
555
|
+
|
|
556
|
+
### Observation and maintenance
|
|
557
|
+
|
|
485
558
|
Observation uses a scope-and-turn claim before extraction. While a claim lease is live and
|
|
486
559
|
the turn has not completed, calls for the same scope and turn raise
|
|
487
560
|
`Engram::ObservationInProgressError` instead of reporting success without doing the work; a
|
|
@@ -515,7 +588,24 @@ Engram.configure do |config|
|
|
|
515
588
|
end
|
|
516
589
|
```
|
|
517
590
|
|
|
518
|
-
|
|
591
|
+
Delete a specific memory:
|
|
592
|
+
|
|
593
|
+
```ruby
|
|
594
|
+
record = current_user.memory.add("Prefers tea")
|
|
595
|
+
current_user.memory.forget(id: record.id) # => 1
|
|
596
|
+
current_user.memory.forget(id: record.id) # => 0 (already deleted)
|
|
597
|
+
```
|
|
598
|
+
|
|
599
|
+
`forget` deletes one record within the memory's scope and returns the affected row count.
|
|
600
|
+
Missing records and IDs belonging to another scope return `0`. Pass the ID returned by
|
|
601
|
+
the store; integers and non-blank strings are accepted. Arrays, ranges, and nil raise
|
|
602
|
+
`ArgumentError`.
|
|
603
|
+
|
|
604
|
+
This is an explicit application operation: authorize the request before calling it.
|
|
605
|
+
It does not run extraction, consolidation, or persistence hooks, including `allow_destructive?`.
|
|
606
|
+
Source messages and processed-turn markers are kept; a later observation can recreate the memory.
|
|
607
|
+
|
|
608
|
+
Prune stale memories:
|
|
519
609
|
|
|
520
610
|
```ruby
|
|
521
611
|
# Forget memories untouched for 90 days, but keep anything important
|
|
@@ -549,6 +639,7 @@ When ActiveSupport is loaded, Engram emits `ActiveSupport::Notifications` events
|
|
|
549
639
|
main memory pipeline:
|
|
550
640
|
|
|
551
641
|
- `add.engram`
|
|
642
|
+
- `forget.engram`
|
|
552
643
|
- `recall.engram`
|
|
553
644
|
- `inject.engram`
|
|
554
645
|
- `observe.engram`
|
|
@@ -580,6 +671,11 @@ ActiveSupport::Notifications.subscribe(/\.engram\z/) do |name, _started, _finish
|
|
|
580
671
|
end
|
|
581
672
|
```
|
|
582
673
|
|
|
674
|
+
Recall events include `candidate_count`, `filtered_count`, `result_count`, and
|
|
675
|
+
`min_similarity` when set. Injection events include the input `memory_count`,
|
|
676
|
+
`injected_count`, `skipped_count`, `injected_bytes`, and `max_bytes` when set.
|
|
677
|
+
`forget.engram` reports `deleted_count` without the record ID or content.
|
|
678
|
+
|
|
583
679
|
Avoid adding memory content or raw prompts to subscriber logs; recalled content is
|
|
584
680
|
user-derived and should be treated as sensitive application data.
|
|
585
681
|
|
|
@@ -592,7 +688,8 @@ user-derived and should be treated as sensitive application data.
|
|
|
592
688
|
- Configure ActiveJob for `observe_later`; keep automatic observation off the request path.
|
|
593
689
|
- Configure `Engram::Rails::CacheProcessedTurns` or another persistent processed-turns adapter for retries.
|
|
594
690
|
- Review persistence policy settings and add app-specific redaction/denylist patterns.
|
|
595
|
-
-
|
|
691
|
+
- Calibrate `recall_min_similarity` and set `injection_max_bytes`, recall limits, and `kinds:` filters.
|
|
692
|
+
- Check filtered pgvector recall coverage and latency on representative tenant sizes.
|
|
596
693
|
- Run the deterministic test/eval suite plus pgvector integration tests before release.
|
|
597
694
|
|
|
598
695
|
## How it works
|
|
@@ -667,9 +764,10 @@ safe to run in CI as a smoke test.
|
|
|
667
764
|
The harness reports recall@k over labelled relevant memories, a labelled precision
|
|
668
765
|
proxy@k, near-distractor retrieval rate, contradiction-pair full recall, extraction
|
|
669
766
|
structured-output parsing cases, consolidation decision cases, and a heuristic duplicate-add
|
|
670
|
-
baseline.
|
|
671
|
-
|
|
672
|
-
|
|
767
|
+
baseline. Set `MIN_SIMILARITY=0.5` on either eval task to test a threshold. With a semantic
|
|
768
|
+
embedder, it also reports how often unrelated queries retrieve memories and how often
|
|
769
|
+
recall returns nothing. Compare these against positive-query recall when tuning the
|
|
770
|
+
threshold. NullEmbedder results only check mechanics; they do not measure semantic quality.
|
|
673
771
|
|
|
674
772
|
Before opening a release PR, also verify the gem package:
|
|
675
773
|
|
|
@@ -684,7 +782,9 @@ gem unpack engram-*.gem --target /tmp/engram-package-check
|
|
|
684
782
|
- v0.2 (done): extract and consolidate (ADD / UPDATE / FORGET), background jobs.
|
|
685
783
|
- v0.3 (done): idempotent observation, importance/recency recall, forgetting and decay.
|
|
686
784
|
- v0.4 (done): memory kinds, persistence policy, typed recall filters, safer injection, and observability hooks.
|
|
687
|
-
- v0.5 (
|
|
785
|
+
- v0.5 (done): embedding provenance and scoped embedding rebuild operations.
|
|
786
|
+
- v0.6 (done): structured provenance, source impact lookup, and grounding reports.
|
|
787
|
+
- v0.7 (done): recall thresholds, injection limits, scoped deletion, and RubyLLM streaming fixes.
|
|
688
788
|
- later: additional storage backends and larger real-provider eval benchmarks.
|
|
689
789
|
|
|
690
790
|
## License
|
data/lib/engram/configuration.rb
CHANGED
|
@@ -8,13 +8,16 @@ module Engram
|
|
|
8
8
|
attr_accessor :store, :embedder, :completion, :default_limit,
|
|
9
9
|
:consolidator, :extraction_min_confidence, :processed_turns,
|
|
10
10
|
:importance_weight, :recency_weight, :recency_halflife, :touch_on_recall,
|
|
11
|
-
:persistence_policy, :before_persist, :instrumentation_scope_identifier
|
|
11
|
+
:persistence_policy, :before_persist, :instrumentation_scope_identifier,
|
|
12
|
+
:recall_min_similarity, :injection_max_bytes
|
|
12
13
|
|
|
13
14
|
def initialize
|
|
14
15
|
@store = Adapters::InMemoryStore.new
|
|
15
16
|
@embedder = Adapters::NullEmbedder.new
|
|
16
17
|
@completion = nil # required for observe (extract/consolidate); nil until configured
|
|
17
18
|
@default_limit = 5
|
|
19
|
+
@recall_min_similarity = nil # opt-in cosine floor, before importance/recency ranking
|
|
20
|
+
@injection_max_bytes = nil # cap the complete appended memory block, not the host prompt
|
|
18
21
|
@consolidator = :heuristic # :heuristic (deterministic) or :llm (LLM-as-judge)
|
|
19
22
|
@extraction_min_confidence = 0.5
|
|
20
23
|
@processed_turns = Adapters::InMemoryProcessedTurns.new # idempotency for observe
|
|
@@ -9,21 +9,27 @@ module Engram
|
|
|
9
9
|
# chat = Engram.with_memory(RubyLLM.chat, memory: current_user.memory)
|
|
10
10
|
# chat.ask("why am I rate limited?") # recall + inject happen automatically
|
|
11
11
|
class MemoryChat
|
|
12
|
-
def initialize(chat, memory:, limit: Engram.config.default_limit
|
|
12
|
+
def initialize(chat, memory:, limit: Engram.config.default_limit, kinds: nil,
|
|
13
|
+
min_similarity: Engram.config.recall_min_similarity, max_bytes: Engram.config.injection_max_bytes)
|
|
13
14
|
@chat = chat
|
|
14
15
|
@memory = memory
|
|
15
16
|
@limit = limit
|
|
17
|
+
@kinds = kinds
|
|
18
|
+
@min_similarity = min_similarity
|
|
19
|
+
@max_bytes = max_bytes
|
|
16
20
|
end
|
|
17
21
|
|
|
18
|
-
def ask(message, **opts)
|
|
19
|
-
augmented = @memory.inject_into(message.to_s, query: message.to_s, limit: @limit
|
|
20
|
-
|
|
22
|
+
def ask(message, **opts, &block)
|
|
23
|
+
augmented = @memory.inject_into(message.to_s, query: message.to_s, limit: @limit,
|
|
24
|
+
kinds: @kinds, min_similarity: @min_similarity, max_bytes: @max_bytes)
|
|
25
|
+
@chat.ask(augmented, **opts, &block)
|
|
21
26
|
end
|
|
22
27
|
|
|
23
28
|
def method_missing(name, *args, **kwargs, &block)
|
|
24
29
|
return super unless @chat.respond_to?(name)
|
|
25
30
|
|
|
26
|
-
@chat.public_send(name, *args, **kwargs, &block)
|
|
31
|
+
result = @chat.public_send(name, *args, **kwargs, &block)
|
|
32
|
+
result.equal?(@chat) ? self : result
|
|
27
33
|
end
|
|
28
34
|
|
|
29
35
|
def respond_to_missing?(name, include_private = false)
|
|
@@ -34,7 +40,9 @@ module Engram
|
|
|
34
40
|
end
|
|
35
41
|
|
|
36
42
|
# Convenience entrypoint.
|
|
37
|
-
def self.with_memory(chat, memory:, limit: config.default_limit
|
|
38
|
-
|
|
43
|
+
def self.with_memory(chat, memory:, limit: config.default_limit, kinds: nil,
|
|
44
|
+
min_similarity: config.recall_min_similarity, max_bytes: config.injection_max_bytes)
|
|
45
|
+
Integrations::RubyLLM::MemoryChat.new(chat, memory: memory, limit: limit, kinds: kinds,
|
|
46
|
+
min_similarity: min_similarity, max_bytes: max_bytes)
|
|
39
47
|
end
|
|
40
48
|
end
|
data/lib/engram/memory.rb
CHANGED
|
@@ -30,7 +30,8 @@ module Engram
|
|
|
30
30
|
end
|
|
31
31
|
|
|
32
32
|
# Return the most relevant memories for a query.
|
|
33
|
-
def recall(query, limit: Engram.config.default_limit, kinds: nil
|
|
33
|
+
def recall(query, limit: Engram.config.default_limit, kinds: nil,
|
|
34
|
+
min_similarity: Engram.config.recall_min_similarity)
|
|
34
35
|
UseCases::Recall.new(
|
|
35
36
|
store: @store,
|
|
36
37
|
embedder: @embedder,
|
|
@@ -38,13 +39,15 @@ module Engram
|
|
|
38
39
|
recency_weight: Engram.config.recency_weight,
|
|
39
40
|
recency_halflife: Engram.config.recency_halflife,
|
|
40
41
|
touch: Engram.config.touch_on_recall
|
|
41
|
-
).call(query, scope: scope, limit: limit, kinds: kinds)
|
|
42
|
+
).call(query, scope: scope, limit: limit, kinds: kinds, min_similarity: min_similarity)
|
|
42
43
|
end
|
|
43
44
|
|
|
44
45
|
# Recall, then inject into a prompt string.
|
|
45
|
-
def inject_into(prompt, query:, limit: Engram.config.default_limit, kinds: nil
|
|
46
|
-
|
|
47
|
-
UseCases::Inject.
|
|
46
|
+
def inject_into(prompt, query:, limit: Engram.config.default_limit, kinds: nil,
|
|
47
|
+
min_similarity: Engram.config.recall_min_similarity, max_bytes: Engram.config.injection_max_bytes)
|
|
48
|
+
UseCases::Inject.validate_max_bytes!(max_bytes)
|
|
49
|
+
memories = (max_bytes == 0) ? [] : recall(query, limit: limit, kinds: kinds, min_similarity: min_similarity)
|
|
50
|
+
UseCases::Inject.new.call(prompt: prompt, memories: memories, max_bytes: max_bytes)
|
|
48
51
|
end
|
|
49
52
|
|
|
50
53
|
# Derive memories from a conversation turn and consolidate them (v0.2).
|
|
@@ -86,6 +89,18 @@ module Engram
|
|
|
86
89
|
@store.all(scope: scope)
|
|
87
90
|
end
|
|
88
91
|
|
|
92
|
+
# Delete one memory in this scope. Returns 1 when deleted, or 0 when missing.
|
|
93
|
+
# Explicit deletion bypasses extraction, consolidation, and persistence hooks.
|
|
94
|
+
def forget(id:)
|
|
95
|
+
valid_id = id.instance_of?(Integer) || (id.instance_of?(String) && id.valid_encoding? && !id.strip.empty?)
|
|
96
|
+
raise ArgumentError, "id must be an Integer or a non-empty String" unless valid_id
|
|
97
|
+
|
|
98
|
+
payload = Engram::Instrumentation.payload(scope: scope, store: @store)
|
|
99
|
+
Engram::Instrumentation.instrument("forget", payload) do
|
|
100
|
+
payload[:deleted_count] = @store.delete(scope: scope, id: id)
|
|
101
|
+
end
|
|
102
|
+
end
|
|
103
|
+
|
|
89
104
|
# Recompute embeddings (and embedding metadata) for memories in the scope.
|
|
90
105
|
# When `stale_only` is true, only records whose current metadata does not match the
|
|
91
106
|
# active embedder are rebuilt.
|
data/lib/engram/railtie.rb
CHANGED
|
@@ -15,9 +15,7 @@ module Engram
|
|
|
15
15
|
end
|
|
16
16
|
|
|
17
17
|
initializer "engram.active_job" do
|
|
18
|
-
|
|
19
|
-
require "engram/rails/observe_job"
|
|
20
|
-
end
|
|
18
|
+
Engram.autoload :ObserveJob, "engram/rails/observe_job" if defined?(::ActiveJob::Base)
|
|
21
19
|
end
|
|
22
20
|
|
|
23
21
|
rake_tasks do
|
|
@@ -12,14 +12,39 @@ module Engram
|
|
|
12
12
|
@header = header
|
|
13
13
|
end
|
|
14
14
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
15
|
+
def self.validate_max_bytes!(value)
|
|
16
|
+
return if value.nil? || (value.is_a?(Integer) && value >= 0)
|
|
17
|
+
|
|
18
|
+
raise ArgumentError, "max_bytes must be a non-negative integer, or nil"
|
|
19
|
+
end
|
|
20
20
|
|
|
21
|
-
|
|
22
|
-
|
|
21
|
+
# The budget includes the header, delimiters, escaping, and separators, but
|
|
22
|
+
# excludes the original prompt. Skip whole memories that do not fit.
|
|
23
|
+
def call(prompt:, memories:, max_bytes: nil)
|
|
24
|
+
self.class.validate_max_bytes!(max_bytes)
|
|
25
|
+
payload = {memory_count: memories&.size.to_i, injected_count: 0,
|
|
26
|
+
skipped_count: memories&.size.to_i, injected_bytes: 0, max_bytes: max_bytes}.compact
|
|
27
|
+
Engram::Instrumentation.instrument("inject", payload) do
|
|
28
|
+
next prompt if memories.nil? || memories.empty? || max_bytes == 0
|
|
29
|
+
|
|
30
|
+
prefix = "\n\n#{@header}:\n<engram-memories>\n"
|
|
31
|
+
suffix = "\n</engram-memories>"
|
|
32
|
+
bytes = prefix.bytesize + suffix.bytesize
|
|
33
|
+
lines = []
|
|
34
|
+
memories.each do |memory|
|
|
35
|
+
line = render_memory(memory)
|
|
36
|
+
added_bytes = line.bytesize + (lines.empty? ? 0 : 1)
|
|
37
|
+
next if max_bytes && bytes + added_bytes > max_bytes
|
|
38
|
+
|
|
39
|
+
lines << line
|
|
40
|
+
bytes += added_bytes
|
|
41
|
+
end
|
|
42
|
+
next prompt if lines.empty?
|
|
43
|
+
|
|
44
|
+
payload[:injected_count] = lines.size
|
|
45
|
+
payload[:skipped_count] -= lines.size
|
|
46
|
+
payload[:injected_bytes] = bytes
|
|
47
|
+
"#{prompt}#{prefix}#{lines.join("\n")}#{suffix}"
|
|
23
48
|
end
|
|
24
49
|
end
|
|
25
50
|
|
|
@@ -24,17 +24,27 @@ module Engram
|
|
|
24
24
|
end
|
|
25
25
|
|
|
26
26
|
# Returns Array<Record>, most relevant first.
|
|
27
|
-
def call(query, scope:, limit: Engram.config.default_limit, kinds: nil)
|
|
27
|
+
def call(query, scope:, limit: Engram.config.default_limit, kinds: nil, min_similarity: nil)
|
|
28
28
|
raise ArgumentError, "query must be a non-empty string" if query.to_s.strip.empty?
|
|
29
|
+
unless limit.is_a?(Integer) && limit >= 0
|
|
30
|
+
raise ArgumentError, "limit must be a non-negative integer"
|
|
31
|
+
end
|
|
32
|
+
validate_min_similarity!(min_similarity)
|
|
29
33
|
|
|
30
34
|
payload = Engram::Instrumentation.payload(
|
|
31
35
|
scope: scope,
|
|
32
36
|
store: @store,
|
|
33
37
|
limit: limit,
|
|
34
38
|
kinds: Array(kinds).map(&:to_s),
|
|
35
|
-
reranking: reranking
|
|
39
|
+
reranking: reranking?,
|
|
40
|
+
min_similarity: min_similarity,
|
|
41
|
+
candidate_count: 0,
|
|
42
|
+
filtered_count: 0,
|
|
43
|
+
result_count: 0
|
|
36
44
|
)
|
|
37
45
|
Engram::Instrumentation.instrument("recall", payload) do
|
|
46
|
+
next [] if limit.zero?
|
|
47
|
+
|
|
38
48
|
embedding = @embedder.embed(query)
|
|
39
49
|
embedding_metadata = Engram::EmbeddingMetadata.for_embedder(@embedder, embedding: embedding)
|
|
40
50
|
pool_limit = reranking? ? limit * @pool_factor : limit
|
|
@@ -47,16 +57,43 @@ module Engram
|
|
|
47
57
|
kinds: kinds
|
|
48
58
|
)
|
|
49
59
|
|
|
50
|
-
|
|
60
|
+
candidates = if min_similarity.nil?
|
|
61
|
+
pool
|
|
62
|
+
else
|
|
63
|
+
pool.select { |record| meets_similarity?(record.embedding, embedding, min_similarity) }
|
|
64
|
+
end
|
|
65
|
+
results = (reranking? ? rerank(candidates, embedding) : candidates).first(limit)
|
|
51
66
|
touch(results, scope) if @touch
|
|
52
67
|
payload[:result_count] = results.size
|
|
53
68
|
payload[:candidate_count] = pool.size
|
|
69
|
+
payload[:filtered_count] = pool.size - candidates.size
|
|
54
70
|
results
|
|
55
71
|
end
|
|
56
72
|
end
|
|
57
73
|
|
|
58
74
|
private
|
|
59
75
|
|
|
76
|
+
def validate_min_similarity!(value)
|
|
77
|
+
return if value.nil?
|
|
78
|
+
return if value.is_a?(Numeric) && value.real? && value.finite? && value.between?(-1, 1)
|
|
79
|
+
|
|
80
|
+
raise ArgumentError, "min_similarity must be a finite number between -1 and 1, or nil"
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
def meets_similarity?(embedding, query_embedding, minimum)
|
|
84
|
+
return false unless comparable_vector?(embedding) && comparable_vector?(query_embedding)
|
|
85
|
+
return false unless embedding.length == query_embedding.length
|
|
86
|
+
|
|
87
|
+
similarity = Engram::Math.cosine_similarity(query_embedding, embedding)
|
|
88
|
+
similarity.finite? && similarity.clamp(-1.0, 1.0) >= minimum
|
|
89
|
+
end
|
|
90
|
+
|
|
91
|
+
def comparable_vector?(vector)
|
|
92
|
+
vector.is_a?(Array) && !vector.empty? &&
|
|
93
|
+
vector.all? { |value| value.is_a?(Numeric) && value.real? && value.finite? } &&
|
|
94
|
+
vector.any? { |value| !value.zero? }
|
|
95
|
+
end
|
|
96
|
+
|
|
60
97
|
def reranking?
|
|
61
98
|
!@importance_weight.zero? || !@recency_weight.zero?
|
|
62
99
|
end
|
data/lib/engram/version.rb
CHANGED
|
@@ -9,4 +9,9 @@ Engram.configure do |config|
|
|
|
9
9
|
|
|
10
10
|
# How many memories to recall by default.
|
|
11
11
|
config.default_limit = 5
|
|
12
|
+
|
|
13
|
+
# Minimum cosine similarity. Tune for your embedding model.
|
|
14
|
+
# config.recall_min_similarity = 0.5
|
|
15
|
+
# Cap the appended memory block in bytes, including escaped text and delimiters.
|
|
16
|
+
# config.injection_max_bytes = 8_000
|
|
12
17
|
end
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: engram
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.7.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Alexandr Kholodniak
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-
|
|
11
|
+
date: 2026-09-13 00:00:00.000000000 Z
|
|
12
12
|
dependencies: []
|
|
13
13
|
description: |
|
|
14
14
|
Engram gives AI agents durable, long-term memory. It recalls relevant facts about a
|