engram 0.7.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +8 -0
- data/README.md +65 -2
- data/lib/engram/adapters/in_memory_store.rb +55 -23
- data/lib/engram/adapters/pgvector_store.rb +37 -1
- data/lib/engram/internal/candidate_integrity.rb +2 -1
- data/lib/engram/memory.rb +11 -5
- data/lib/engram/ports/memory_store.rb +16 -2
- data/lib/engram/rails/tasks.rake +15 -0
- data/lib/engram/record.rb +11 -3
- data/lib/engram/use_cases/forget_expired.rb +43 -0
- data/lib/engram/use_cases/inject.rb +3 -0
- data/lib/engram/use_cases/recall.rb +4 -4
- data/lib/engram/version.rb +1 -1
- data/lib/engram.rb +1 -0
- data/lib/generators/engram/expiry_generator.rb +30 -0
- data/lib/generators/engram/templates/add_expiry_to_engram_memories.rb.tt +12 -0
- data/lib/generators/engram/templates/create_engram_memories.rb.tt +4 -0
- metadata +5 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: b3d370df000e9c12a40df932919f0859d9eb7623270ed5aeaa781747f239d865
|
|
4
|
+
data.tar.gz: 2c49fd3e8126a510f49b711a9ccf9d159ac01b6e540ef2ab168e7e51fa6509f3
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 161c8e9c199352d663e0ce670f3a6dc76ac98aba0b429566a1c3c96fa8ee5c7db260a449355f449b11dfdb25de4b85f207ca263fcffa6a76248d1269632ecddc
|
|
7
|
+
data.tar.gz: 1c3867577e6a86128fa4a4036b64cc2790754336c4cc462495284f25f5a7ac63ac56be24694ffb9578d2b50999bde325ca4ced68b6d8e0aed9143234108b0888
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|
|
5
5
|
|
|
6
6
|
## [Unreleased]
|
|
7
7
|
|
|
8
|
+
## [0.8.0] - 2026-09-15
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- Optional `expires_at` on memories. Expired records are excluded from search and injection while remaining available for inspection and deletion.
|
|
13
|
+
- `engram:expiry` generator for existing Rails apps, with a concurrent index for scoped expiry queries.
|
|
14
|
+
- Scoped `forget_expired` cleanup with batching, dry runs, and a Rails task. Deletion rechecks the deadline so concurrent expiry extensions are preserved.
|
|
15
|
+
|
|
8
16
|
## [0.7.0] - 2026-09-13
|
|
9
17
|
|
|
10
18
|
### Added
|
data/README.md
CHANGED
|
@@ -76,7 +76,7 @@ explicitly version-gated:
|
|
|
76
76
|
- Core types and facade: `Engram::Memory`, `Engram::Record`, `Engram::Decision`, `Engram::PersistencePolicy`, and `Engram.with_memory`.
|
|
77
77
|
- Store and adapter ports: `Engram::Ports::MemoryStore`, `Engram::Ports::Embedder`, `Engram::Ports::Completion`.
|
|
78
78
|
- Rails integration points: `has_memory`, `Memory#observe_later`, and generator outputs under `engram:` rake tasks.
|
|
79
|
-
- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget`, `forget_stale`, `rebuild_embeddings`, and `memories_from_source`.
|
|
79
|
+
- Lifecycle methods in `Engram::Memory`: `add`, `recall`, `inject_into`, `observe`, `observe_later`, `forget`, `forget_stale`, `forget_expired`, `rebuild_embeddings`, and `memories_from_source`.
|
|
80
80
|
- RubyLLM adapter contract points and evaluator entrypoints (`rake eval`, `rake eval:real`).
|
|
81
81
|
|
|
82
82
|
### Backward-compatibility commitments (pre-1.0)
|
|
@@ -228,7 +228,7 @@ end
|
|
|
228
228
|
### Filtered vector search
|
|
229
229
|
|
|
230
230
|
Keep the generated B-tree index on `scope`. With an approximate vector index, pgvector
|
|
231
|
-
applies scope and
|
|
231
|
+
applies scope, kind, and expiry filters after the index scan. This can return fewer than `limit`
|
|
232
232
|
memories even when enough matching rows exist.
|
|
233
233
|
|
|
234
234
|
On pgvector 0.8.0+, HNSW iterative scans can search further:
|
|
@@ -553,6 +553,67 @@ Both settings default to nil. Pass nil explicitly to override a configured value
|
|
|
553
553
|
the cap. `limit: 0` skips embedding and search; `max_bytes: 0` skips recall during injection.
|
|
554
554
|
Invalid values raise `ArgumentError`.
|
|
555
555
|
|
|
556
|
+
### Memory expiry
|
|
557
|
+
|
|
558
|
+
Set a deadline for temporary memories:
|
|
559
|
+
|
|
560
|
+
```ruby
|
|
561
|
+
user.memory.add("Trial access is active", expires_at: 7.days.from_now)
|
|
562
|
+
```
|
|
563
|
+
|
|
564
|
+
`expires_at` accepts a Time (including Rails time-zone values) or nil. Nil is the default
|
|
565
|
+
and never expires. A memory is expired at `expires_at <= Time.now`. Both built-in stores
|
|
566
|
+
exclude expired records before the search limit, so they no longer participate in recall
|
|
567
|
+
or consolidation. Injection checks expiry again when rendering.
|
|
568
|
+
|
|
569
|
+
Expiry does not delete rows. `memory.all` still returns them, and `memory.forget(id:)` can
|
|
570
|
+
delete them. A new observation may recreate the same fact from retained source data.
|
|
571
|
+
Custom extractors or `before_persist` can set expiry with `record.with(expires_at: deadline)`;
|
|
572
|
+
Engram does not infer deadlines from conversation text. Store updates replace expiry with
|
|
573
|
+
the supplied record's value, including nil.
|
|
574
|
+
|
|
575
|
+
New Rails installs include the expiry column and index. For a table created by 0.7.0 or
|
|
576
|
+
earlier, generate and run the upgrade migration:
|
|
577
|
+
|
|
578
|
+
```sh
|
|
579
|
+
bin/rails generate engram:expiry
|
|
580
|
+
bin/rails db:migrate
|
|
581
|
+
```
|
|
582
|
+
|
|
583
|
+
The migration adds a nullable datetime column and builds a partial index on scope and
|
|
584
|
+
expiry concurrently. Existing rows keep their data and do not expire. Reads and
|
|
585
|
+
non-expiring writes continue to work before migration; expiring writes raise an error
|
|
586
|
+
instead of losing the deadline. Restart app processes after migrating so ActiveRecord
|
|
587
|
+
refreshes its cached schema. If you use a custom table, adjust the generated migration.
|
|
588
|
+
|
|
589
|
+
Custom stores must persist `expires_at` and exclude expired records before limiting search
|
|
590
|
+
results. Recall also discards any expired records a custom store returns, but cannot fill
|
|
591
|
+
the gaps left by a store that filters after its limit.
|
|
592
|
+
|
|
593
|
+
To remove expired rows, preview the count, then run cleanup:
|
|
594
|
+
|
|
595
|
+
```ruby
|
|
596
|
+
user.memory.forget_expired(dry_run: true)
|
|
597
|
+
# => {matched: 12, deleted: 0, dry_run: true}
|
|
598
|
+
user.memory.forget_expired(batch_size: 500)
|
|
599
|
+
# => {matched: 12, deleted: 12, dry_run: false}
|
|
600
|
+
```
|
|
601
|
+
|
|
602
|
+
The Rails task requires a scope:
|
|
603
|
+
|
|
604
|
+
```sh
|
|
605
|
+
DRY_RUN=true BATCH_SIZE=500 bin/rails 'engram:forget_expired[user:42]'
|
|
606
|
+
BATCH_SIZE=500 bin/rails 'engram:forget_expired[user:42]'
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
Cleanup uses one cutoff for the run and reads only IDs in batches. It rechecks scope and
|
|
610
|
+
expiry when deleting, so a concurrent extension or removal of the deadline is preserved.
|
|
611
|
+
`matched` can exceed `deleted` if records change during the run; dry-run counts are advisory.
|
|
612
|
+
Like `forget(id:)`, this is explicit deletion and bypasses persistence hooks and policies.
|
|
613
|
+
Errors stop the run; completed batches stay deleted and a retry can process the remainder.
|
|
614
|
+
Custom stores need the optional `expired_ids` and `delete_expired` methods for cleanup.
|
|
615
|
+
The `forget_expired.engram` event reports counts without memory content or IDs.
|
|
616
|
+
|
|
556
617
|
### Observation and maintenance
|
|
557
618
|
|
|
558
619
|
Observation uses a scope-and-turn claim before extraction. While a claim lease is live and
|
|
@@ -640,6 +701,7 @@ main memory pipeline:
|
|
|
640
701
|
|
|
641
702
|
- `add.engram`
|
|
642
703
|
- `forget.engram`
|
|
704
|
+
- `forget_expired.engram`
|
|
643
705
|
- `recall.engram`
|
|
644
706
|
- `inject.engram`
|
|
645
707
|
- `observe.engram`
|
|
@@ -785,6 +847,7 @@ gem unpack engram-*.gem --target /tmp/engram-package-check
|
|
|
785
847
|
- v0.5 (done): embedding provenance and scoped embedding rebuild operations.
|
|
786
848
|
- v0.6 (done): structured provenance, source impact lookup, and grounding reports.
|
|
787
849
|
- v0.7 (done): recall thresholds, injection limits, scoped deletion, and RubyLLM streaming fixes.
|
|
850
|
+
- v0.8 (done): explicit memory expiry, Rails upgrade migration, and scoped cleanup with dry runs.
|
|
788
851
|
- later: additional storage backends and larger real-provider eval benchmarks.
|
|
789
852
|
|
|
790
853
|
## License
|
|
@@ -10,23 +10,26 @@ module Engram
|
|
|
10
10
|
def initialize
|
|
11
11
|
@records = {}
|
|
12
12
|
@sequence = 0
|
|
13
|
+
@mutex = Mutex.new
|
|
13
14
|
end
|
|
14
15
|
|
|
15
16
|
def add(record)
|
|
16
17
|
validate_scope!(record.scope)
|
|
17
18
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
19
|
+
@mutex.synchronize do
|
|
20
|
+
record.id = (@sequence += 1)
|
|
21
|
+
@records[record.id] = record
|
|
22
|
+
record
|
|
23
|
+
end
|
|
21
24
|
end
|
|
22
25
|
|
|
23
26
|
def search(embedding:, scope:, limit:, kinds: nil, embedding_metadata: nil)
|
|
24
27
|
Engram::EmbeddingMetadata.validate_query!(embedding, embedding_metadata)
|
|
25
28
|
allowed_kinds = normalize_kinds(kinds)
|
|
29
|
+
now = Time.now
|
|
26
30
|
|
|
27
|
-
results = @records
|
|
28
|
-
.
|
|
29
|
-
.select { |r| searchable?(r, scope, allowed_kinds) }
|
|
31
|
+
results = @mutex.synchronize { @records.values }
|
|
32
|
+
.select { |r| searchable?(r, scope, allowed_kinds) && !r.expired?(at: now) }
|
|
30
33
|
.map { |r| [r, Engram::Math.cosine_similarity(embedding, r.embedding)] }
|
|
31
34
|
.sort_by { |(_, score)| -score }
|
|
32
35
|
.first(limit)
|
|
@@ -37,7 +40,7 @@ module Engram
|
|
|
37
40
|
end
|
|
38
41
|
|
|
39
42
|
def all(scope:, limit: nil, offset: 0, after_id: nil)
|
|
40
|
-
records = @records.values.select { |r| r.scope == scope }.sort_by { |record| record.id }
|
|
43
|
+
records = @mutex.synchronize { @records.values }.select { |r| r.scope == scope }.sort_by { |record| record.id }
|
|
41
44
|
records = records.drop_while { |record| !after_id.nil? && record.id && record.id <= after_id }
|
|
42
45
|
records = records.drop(offset) if offset > 0
|
|
43
46
|
records = records.take(limit) if limit
|
|
@@ -45,36 +48,65 @@ module Engram
|
|
|
45
48
|
end
|
|
46
49
|
|
|
47
50
|
def existing_ids(scope:, ids:)
|
|
48
|
-
ids.uniq.select { |id| @records[id]&.scope == scope }
|
|
51
|
+
@mutex.synchronize { ids.uniq.select { |id| @records[id]&.scope == scope } }
|
|
49
52
|
end
|
|
50
53
|
|
|
51
54
|
def update(scope:, id:, record:)
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
55
|
+
@mutex.synchronize do
|
|
56
|
+
existing = @records[id]
|
|
57
|
+
raise Engram::Error, "no memory with id #{id.inspect} in scope #{scope.inspect}" unless existing&.scope == scope
|
|
58
|
+
raise Engram::Error, "cannot move memory across scopes" unless record.scope == scope
|
|
59
|
+
|
|
60
|
+
record.id = id
|
|
61
|
+
@records[id] = record
|
|
62
|
+
end
|
|
58
63
|
end
|
|
59
64
|
|
|
60
65
|
def delete(scope:, id:)
|
|
61
|
-
|
|
66
|
+
@mutex.synchronize do
|
|
67
|
+
return 0 unless @records[id]&.scope == scope
|
|
62
68
|
|
|
63
|
-
|
|
64
|
-
|
|
69
|
+
@records.delete(id)
|
|
70
|
+
1
|
|
71
|
+
end
|
|
65
72
|
end
|
|
66
73
|
|
|
67
74
|
def touch(scope:, id:, at: Time.now)
|
|
68
|
-
|
|
69
|
-
|
|
75
|
+
@mutex.synchronize do
|
|
76
|
+
record = @records[id]
|
|
77
|
+
return 0 unless record&.scope == scope
|
|
78
|
+
|
|
79
|
+
record.last_accessed_at = at
|
|
80
|
+
1
|
|
81
|
+
end
|
|
82
|
+
end
|
|
70
83
|
|
|
71
|
-
|
|
72
|
-
|
|
84
|
+
def expired_ids(scope:, at:, limit:, after_id: nil)
|
|
85
|
+
# IDs follow insertion order; the hash retains it across updates/deletes.
|
|
86
|
+
@mutex.synchronize do
|
|
87
|
+
@records.each_value.lazy
|
|
88
|
+
.select { |record| record.scope == scope && (after_id.nil? || record.id > after_id) && record.expired?(at: at) }
|
|
89
|
+
.map(&:id).take(limit).force
|
|
90
|
+
end
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
def delete_expired(scope:, ids:, at:)
|
|
94
|
+
@mutex.synchronize do
|
|
95
|
+
ids.uniq.count do |id|
|
|
96
|
+
record = @records[id]
|
|
97
|
+
next false unless record&.scope == scope && record.expired?(at: at)
|
|
98
|
+
|
|
99
|
+
@records.delete(id)
|
|
100
|
+
true
|
|
101
|
+
end
|
|
102
|
+
end
|
|
73
103
|
end
|
|
74
104
|
|
|
75
105
|
def clear
|
|
76
|
-
@
|
|
77
|
-
|
|
106
|
+
@mutex.synchronize do
|
|
107
|
+
@records.clear
|
|
108
|
+
@sequence = 0
|
|
109
|
+
end
|
|
78
110
|
end
|
|
79
111
|
|
|
80
112
|
private
|
|
@@ -18,6 +18,7 @@ module Engram
|
|
|
18
18
|
validate_scope!(record.scope)
|
|
19
19
|
|
|
20
20
|
row = model.create!(
|
|
21
|
+
**expiry_attributes(record),
|
|
21
22
|
content: record.content,
|
|
22
23
|
scope: record.scope,
|
|
23
24
|
kind: record.kind.to_s,
|
|
@@ -31,6 +32,10 @@ module Engram
|
|
|
31
32
|
def search(embedding:, scope:, limit:, kinds: nil, embedding_metadata: nil)
|
|
32
33
|
Engram::EmbeddingMetadata.validate_query!(embedding, embedding_metadata)
|
|
33
34
|
query = model.where(scope: scope)
|
|
35
|
+
if expiry_column?
|
|
36
|
+
expiry = model.arel_table[:expires_at]
|
|
37
|
+
query = query.where(expiry.eq(nil).or(expiry.gt(Time.now)))
|
|
38
|
+
end
|
|
34
39
|
normalized_kinds = normalize_kinds(kinds)
|
|
35
40
|
query = query.where(kind: normalized_kinds) if normalized_kinds
|
|
36
41
|
|
|
@@ -70,6 +75,7 @@ module Engram
|
|
|
70
75
|
model.transaction do
|
|
71
76
|
row = model.lock.find_by!(id: id, scope: scope)
|
|
72
77
|
row.update!(
|
|
78
|
+
**expiry_attributes(record),
|
|
73
79
|
content: record.content,
|
|
74
80
|
kind: record.kind.to_s,
|
|
75
81
|
importance: record.importance,
|
|
@@ -90,8 +96,37 @@ module Engram
|
|
|
90
96
|
model.where(id: id, scope: scope).update_all(last_accessed_at: at)
|
|
91
97
|
end
|
|
92
98
|
|
|
99
|
+
def expired_ids(scope:, at:, limit:, after_id: nil)
|
|
100
|
+
return [] unless expiry_column?
|
|
101
|
+
|
|
102
|
+
query = expired_scope(scope, at)
|
|
103
|
+
query = query.where(model.arel_table[:id].gt(after_id)) unless after_id.nil?
|
|
104
|
+
query.order(:id).limit(limit).pluck(:id)
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
def delete_expired(scope:, ids:, at:)
|
|
108
|
+
return 0 if ids.empty? || !expiry_column?
|
|
109
|
+
|
|
110
|
+
expired_scope(scope, at).where(id: ids).delete_all
|
|
111
|
+
end
|
|
112
|
+
|
|
93
113
|
private
|
|
94
114
|
|
|
115
|
+
def expired_scope(scope, at)
|
|
116
|
+
model.where(scope: scope).where(model.arel_table[:expires_at].lteq(at))
|
|
117
|
+
end
|
|
118
|
+
|
|
119
|
+
def expiry_column?
|
|
120
|
+
model.column_names.include?("expires_at")
|
|
121
|
+
end
|
|
122
|
+
|
|
123
|
+
def expiry_attributes(record)
|
|
124
|
+
return {expires_at: record.expires_at} if expiry_column?
|
|
125
|
+
return {} if record.expires_at.nil?
|
|
126
|
+
|
|
127
|
+
raise Engram::Error, "PgvectorStore needs a nullable expires_at datetime column to store expiring memories"
|
|
128
|
+
end
|
|
129
|
+
|
|
95
130
|
def validate_scope!(scope)
|
|
96
131
|
raise Engram::Error, "memory scope cannot be nil" if scope.nil?
|
|
97
132
|
end
|
|
@@ -124,7 +159,8 @@ module Engram
|
|
|
124
159
|
importance: row.importance || 1.0,
|
|
125
160
|
metadata: row.metadata || {},
|
|
126
161
|
created_at: row.created_at,
|
|
127
|
-
last_accessed_at: row.try(:last_accessed_at)
|
|
162
|
+
last_accessed_at: row.try(:last_accessed_at),
|
|
163
|
+
expires_at: row.try(:expires_at)
|
|
128
164
|
)
|
|
129
165
|
end
|
|
130
166
|
|
|
@@ -29,7 +29,8 @@ module Engram
|
|
|
29
29
|
kind: [Symbol],
|
|
30
30
|
importance: [Integer, Float],
|
|
31
31
|
created_at: [Time],
|
|
32
|
-
last_accessed_at: [NilClass, Time]
|
|
32
|
+
last_accessed_at: [NilClass, Time],
|
|
33
|
+
expires_at: [NilClass, Time]
|
|
33
34
|
}.freeze
|
|
34
35
|
|
|
35
36
|
Snapshot = Struct.new(:collection, :collection_state, :records)
|
data/lib/engram/memory.rb
CHANGED
|
@@ -14,18 +14,17 @@ module Engram
|
|
|
14
14
|
|
|
15
15
|
# Persist a memory record of the given kind. Returns nil when the configured
|
|
16
16
|
# persistence policy rejects the record.
|
|
17
|
-
def add(content, kind: :fact, importance: 1.0, metadata: {})
|
|
17
|
+
def add(content, kind: :fact, importance: 1.0, metadata: {}, expires_at: nil)
|
|
18
18
|
Engram::Instrumentation.instrument("add", Engram::Instrumentation.payload(scope: scope, store: @store, kind: kind)) do
|
|
19
|
-
embedding = @embedder.embed(content)
|
|
20
19
|
record = Record.new(
|
|
21
20
|
content: content,
|
|
22
21
|
scope: scope,
|
|
23
|
-
embedding: embedding,
|
|
24
22
|
kind: kind,
|
|
25
23
|
importance: importance,
|
|
26
|
-
metadata: metadata
|
|
24
|
+
metadata: metadata,
|
|
25
|
+
expires_at: expires_at
|
|
27
26
|
)
|
|
28
|
-
persist(record)
|
|
27
|
+
persist(record.with(embedding: @embedder.embed(content)))
|
|
29
28
|
end
|
|
30
29
|
end
|
|
31
30
|
|
|
@@ -119,6 +118,13 @@ module Engram
|
|
|
119
118
|
.call(scope: scope, older_than: older_than, min_importance: min_importance)
|
|
120
119
|
end
|
|
121
120
|
|
|
121
|
+
# Delete expired records without loading their content or embeddings.
|
|
122
|
+
# Returns matched/deleted counts; dry runs only count eligible records.
|
|
123
|
+
def forget_expired(batch_size: 100, dry_run: false)
|
|
124
|
+
UseCases::ForgetExpired.new(store: @store)
|
|
125
|
+
.call(scope: scope, batch_size: batch_size, dry_run: dry_run)
|
|
126
|
+
end
|
|
127
|
+
|
|
122
128
|
# Return the memories in this scope whose provenance references the exact host
|
|
123
129
|
# source. `source_id` and `source_type` must each be non-blank Strings and are
|
|
124
130
|
# matched exactly, without normalization. Source IDs are references, not an
|
|
@@ -12,12 +12,13 @@ module Engram
|
|
|
12
12
|
|
|
13
13
|
# Return up to `limit` Records in `scope` nearest to `embedding`,
|
|
14
14
|
# ordered most-relevant first. When `kinds` is provided, only records with
|
|
15
|
-
# those canonical memory kinds are eligible.
|
|
15
|
+
# those canonical memory kinds are eligible. Exclude expired records before
|
|
16
|
+
# ordering and limiting; expires_at <= Time.now is expired, nil never expires.
|
|
16
17
|
def search(embedding:, scope:, limit:, kinds: nil, embedding_metadata: nil)
|
|
17
18
|
raise NotImplementedError, "#{self.class} must implement #search"
|
|
18
19
|
end
|
|
19
20
|
|
|
20
|
-
# All Records for a scope
|
|
21
|
+
# All Records for a scope, including expired records (inspection/maintenance).
|
|
21
22
|
# Supports optional `limit` and `offset` for batching large sweeps.
|
|
22
23
|
# Returned records are sorted in stable `id` order when batching is used.
|
|
23
24
|
# Use `after_id` for keyset pagination.
|
|
@@ -49,6 +50,19 @@ module Engram
|
|
|
49
50
|
raise NotImplementedError, "#{self.class} must implement #delete"
|
|
50
51
|
end
|
|
51
52
|
|
|
53
|
+
# Optional expiry maintenance: up to `limit` IDs with expires_at <= at,
|
|
54
|
+
# ordered by ID, strictly after after_id. Never includes nil expiry.
|
|
55
|
+
def expired_ids(scope:, at:, limit:, after_id: nil)
|
|
56
|
+
raise NotImplementedError, "#{self.class} does not implement #expired_ids"
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
# Optional expiry maintenance: delete only the requested IDs in scope that
|
|
60
|
+
# are still expired at the same cutoff. Recheck expiry atomically at deletion
|
|
61
|
+
# so a concurrent deadline extension is preserved. Returns the deleted count.
|
|
62
|
+
def delete_expired(scope:, ids:, at:)
|
|
63
|
+
raise NotImplementedError, "#{self.class} does not implement #delete_expired"
|
|
64
|
+
end
|
|
65
|
+
|
|
52
66
|
# Update the last-accessed timestamp of a memory. Used by recency-aware recall.
|
|
53
67
|
# Returns the number of affected rows: 1 when touched, 0 when the scoped record
|
|
54
68
|
# does not exist.
|
data/lib/engram/rails/tasks.rake
CHANGED
|
@@ -3,6 +3,21 @@
|
|
|
3
3
|
# Loaded by the Railtie in Rails apps (where :environment boots the app and its
|
|
4
4
|
# initializers) and by the gem's own Rakefile (which defines a no-op :environment).
|
|
5
5
|
namespace :engram do
|
|
6
|
+
desc "Delete expired memories in a scope. Usage: bundle exec rake 'engram:forget_expired[user:1]'"
|
|
7
|
+
task :forget_expired, [:scope] => :environment do |_, args|
|
|
8
|
+
scope = args[:scope]
|
|
9
|
+
raise ArgumentError, "forget_expired requires a scope argument" if scope.to_s.empty?
|
|
10
|
+
|
|
11
|
+
dry_run = case ENV.fetch("DRY_RUN", "false").downcase
|
|
12
|
+
when "true", "1", "yes" then true
|
|
13
|
+
when "false", "0", "no" then false
|
|
14
|
+
else raise ArgumentError, "DRY_RUN must be true or false"
|
|
15
|
+
end
|
|
16
|
+
batch_size = Integer(ENV.fetch("BATCH_SIZE", "100"))
|
|
17
|
+
result = Engram::Memory.new(scope: scope).forget_expired(batch_size: batch_size, dry_run: dry_run)
|
|
18
|
+
puts "matched=#{result[:matched]} deleted=#{result[:deleted]} dry_run=#{result[:dry_run]}"
|
|
19
|
+
end
|
|
20
|
+
|
|
6
21
|
desc "Rebuild embeddings in a memory scope. Usage: bundle exec rake 'engram:rebuild_embeddings[user:1]'"
|
|
7
22
|
task :rebuild_embeddings, [:scope] => :environment do |_, args|
|
|
8
23
|
scope = args[:scope]
|
data/lib/engram/record.rb
CHANGED
|
@@ -9,15 +9,18 @@ module Engram
|
|
|
9
9
|
# `semantic` kind is normalized to `fact` for compatibility with pre-1.0 records.
|
|
10
10
|
class Record
|
|
11
11
|
STATE_READERS = %i[
|
|
12
|
-
id content scope embedding kind importance metadata created_at last_accessed_at
|
|
12
|
+
id content scope embedding kind importance metadata created_at last_accessed_at expires_at
|
|
13
13
|
].freeze
|
|
14
14
|
|
|
15
15
|
attr_accessor :id, :last_accessed_at
|
|
16
16
|
attr_reader :content, :embedding, :scope, :kind, :importance, :metadata,
|
|
17
|
-
:created_at
|
|
17
|
+
:created_at, :expires_at
|
|
18
18
|
|
|
19
19
|
def initialize(content:, scope:, id: nil, embedding: nil, kind: :fact,
|
|
20
|
-
importance: 1.0, metadata: {}, created_at: nil, last_accessed_at: nil)
|
|
20
|
+
importance: 1.0, metadata: {}, created_at: nil, last_accessed_at: nil, expires_at: nil)
|
|
21
|
+
unless expires_at.nil? || expires_at.is_a?(Time)
|
|
22
|
+
raise ArgumentError, "expires_at must be a Time or nil"
|
|
23
|
+
end
|
|
21
24
|
@id = id
|
|
22
25
|
@content = content
|
|
23
26
|
@scope = scope
|
|
@@ -27,12 +30,17 @@ module Engram
|
|
|
27
30
|
@metadata = metadata
|
|
28
31
|
@created_at = created_at || Time.now
|
|
29
32
|
@last_accessed_at = last_accessed_at
|
|
33
|
+
@expires_at = expires_at&.getutc
|
|
30
34
|
end
|
|
31
35
|
|
|
32
36
|
def with(**attributes)
|
|
33
37
|
self.class.new(**to_h.merge(attributes))
|
|
34
38
|
end
|
|
35
39
|
|
|
40
|
+
def expired?(at: Time.now)
|
|
41
|
+
!expires_at.nil? && expires_at <= at
|
|
42
|
+
end
|
|
43
|
+
|
|
36
44
|
# Return structured supporting-source metadata when this record carries a
|
|
37
45
|
# provenance schema understood by the installed Engram version.
|
|
38
46
|
# Legacy, malformed, and future-schema records remain readable and return nil.
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Engram
|
|
4
|
+
module UseCases
|
|
5
|
+
# Deletes expired memories in bounded batches using one cutoff for the whole run.
|
|
6
|
+
class ForgetExpired
|
|
7
|
+
def initialize(store:)
|
|
8
|
+
@store = store
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
def call(scope:, batch_size: 100, dry_run: false, now: Time.now)
|
|
12
|
+
unless batch_size.is_a?(Integer) && batch_size.positive?
|
|
13
|
+
raise ArgumentError, "batch_size must be a positive integer"
|
|
14
|
+
end
|
|
15
|
+
unless dry_run.equal?(true) || dry_run.equal?(false)
|
|
16
|
+
raise ArgumentError, "dry_run must be true or false"
|
|
17
|
+
end
|
|
18
|
+
unless @store.respond_to?(:expired_ids) && @store.respond_to?(:delete_expired)
|
|
19
|
+
raise Engram::Error, "forget_expired requires a store implementing expired_ids and delete_expired"
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
counts = {matched: 0, deleted: 0, dry_run: dry_run}
|
|
23
|
+
payload = Engram::Instrumentation.payload(scope: scope, store: @store, **counts)
|
|
24
|
+
Engram::Instrumentation.instrument("forget_expired", payload) do
|
|
25
|
+
after_id = nil
|
|
26
|
+
loop do
|
|
27
|
+
ids = @store.expired_ids(scope: scope, at: now, limit: batch_size, after_id: after_id)
|
|
28
|
+
break if ids.empty?
|
|
29
|
+
raise Engram::Error, "expired_ids must advance its cursor" if ids.last == after_id
|
|
30
|
+
|
|
31
|
+
counts[:matched] += ids.length
|
|
32
|
+
counts[:deleted] += @store.delete_expired(scope: scope, ids: ids, at: now) unless dry_run
|
|
33
|
+
payload.merge!(counts)
|
|
34
|
+
after_id = ids.last
|
|
35
|
+
end
|
|
36
|
+
counts
|
|
37
|
+
end
|
|
38
|
+
rescue NotImplementedError
|
|
39
|
+
raise Engram::Error, "forget_expired requires a store implementing expired_ids and delete_expired"
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
end
|
|
43
|
+
end
|
|
@@ -31,7 +31,10 @@ module Engram
|
|
|
31
31
|
suffix = "\n</engram-memories>"
|
|
32
32
|
bytes = prefix.bytesize + suffix.bytesize
|
|
33
33
|
lines = []
|
|
34
|
+
now = Time.now
|
|
34
35
|
memories.each do |memory|
|
|
36
|
+
next if memory.expired?(at: now)
|
|
37
|
+
|
|
35
38
|
line = render_memory(memory)
|
|
36
39
|
added_bytes = line.bytesize + (lines.empty? ? 0 : 1)
|
|
37
40
|
next if max_bytes && bytes + added_bytes > max_bytes
|
|
@@ -57,10 +57,10 @@ module Engram
|
|
|
57
57
|
kinds: kinds
|
|
58
58
|
)
|
|
59
59
|
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
60
|
+
now = Time.now
|
|
61
|
+
candidates = pool.select do |record|
|
|
62
|
+
!record.expired?(at: now) &&
|
|
63
|
+
(min_similarity.nil? || meets_similarity?(record.embedding, embedding, min_similarity))
|
|
64
64
|
end
|
|
65
65
|
results = (reranking? ? rerank(candidates, embedding) : candidates).first(limit)
|
|
66
66
|
touch(results, scope) if @touch
|
data/lib/engram/version.rb
CHANGED
data/lib/engram.rb
CHANGED
|
@@ -31,6 +31,7 @@ require_relative "engram/use_cases/recall"
|
|
|
31
31
|
require_relative "engram/use_cases/inject"
|
|
32
32
|
require_relative "engram/use_cases/observe"
|
|
33
33
|
require_relative "engram/use_cases/forget"
|
|
34
|
+
require_relative "engram/use_cases/forget_expired"
|
|
34
35
|
require_relative "engram/use_cases/rebuild_embeddings"
|
|
35
36
|
require_relative "engram/use_cases/source_impact"
|
|
36
37
|
require_relative "engram/use_cases/grounding_report"
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
require "rails/generators"
|
|
4
|
+
require "rails/generators/active_record"
|
|
5
|
+
|
|
6
|
+
module Engram
|
|
7
|
+
module Generators
|
|
8
|
+
# Adds expiry to a table created by Engram 0.7 or earlier.
|
|
9
|
+
class ExpiryGenerator < ::Rails::Generators::Base
|
|
10
|
+
include ::Rails::Generators::Migration
|
|
11
|
+
|
|
12
|
+
source_root File.expand_path("templates", __dir__)
|
|
13
|
+
|
|
14
|
+
def create_migration_file
|
|
15
|
+
migration_template "add_expiry_to_engram_memories.rb.tt",
|
|
16
|
+
"db/migrate/add_expiry_to_engram_memories.rb"
|
|
17
|
+
end
|
|
18
|
+
|
|
19
|
+
def self.next_migration_number(dir)
|
|
20
|
+
::ActiveRecord::Generators::Base.next_migration_number(dir)
|
|
21
|
+
end
|
|
22
|
+
|
|
23
|
+
private
|
|
24
|
+
|
|
25
|
+
def migration_version
|
|
26
|
+
"#{::ActiveRecord::VERSION::MAJOR}.#{::ActiveRecord::VERSION::MINOR}"
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
class AddExpiryToEngramMemories < ActiveRecord::Migration[<%= migration_version %>]
|
|
4
|
+
disable_ddl_transaction!
|
|
5
|
+
|
|
6
|
+
def change
|
|
7
|
+
add_column :engram_memories, :expires_at, :datetime
|
|
8
|
+
add_index :engram_memories, [:scope, :expires_at],
|
|
9
|
+
name: "index_engram_memories_on_scope_and_expiry",
|
|
10
|
+
where: "expires_at IS NOT NULL", algorithm: :concurrently
|
|
11
|
+
end
|
|
12
|
+
end
|
|
@@ -12,11 +12,15 @@ class CreateEngramMemories < ActiveRecord::Migration[<%= migration_version %>]
|
|
|
12
12
|
t.jsonb :metadata, null: false, default: {}
|
|
13
13
|
t.vector :embedding, limit: <%= dimensions %>
|
|
14
14
|
t.datetime :last_accessed_at
|
|
15
|
+
t.datetime :expires_at
|
|
15
16
|
|
|
16
17
|
t.timestamps
|
|
17
18
|
end
|
|
18
19
|
|
|
19
20
|
add_index :engram_memories, :scope
|
|
21
|
+
add_index :engram_memories, [:scope, :expires_at],
|
|
22
|
+
name: "index_engram_memories_on_scope_and_expiry",
|
|
23
|
+
where: "expires_at IS NOT NULL"
|
|
20
24
|
|
|
21
25
|
# For larger datasets, choose one approximate vector index after backfilling data.
|
|
22
26
|
# HNSW is a strong default for read-heavy recall workloads with frequent inserts.
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: engram
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.8.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Alexandr Kholodniak
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2026-09-
|
|
11
|
+
date: 2026-09-15 00:00:00.000000000 Z
|
|
12
12
|
dependencies: []
|
|
13
13
|
description: |
|
|
14
14
|
Engram gives AI agents durable, long-term memory. It recalls relevant facts about a
|
|
@@ -65,6 +65,7 @@ files:
|
|
|
65
65
|
- lib/engram/reserved_metadata.rb
|
|
66
66
|
- lib/engram/turn_digest.rb
|
|
67
67
|
- lib/engram/use_cases/forget.rb
|
|
68
|
+
- lib/engram/use_cases/forget_expired.rb
|
|
68
69
|
- lib/engram/use_cases/grounding_report.rb
|
|
69
70
|
- lib/engram/use_cases/inject.rb
|
|
70
71
|
- lib/engram/use_cases/observe.rb
|
|
@@ -72,7 +73,9 @@ files:
|
|
|
72
73
|
- lib/engram/use_cases/recall.rb
|
|
73
74
|
- lib/engram/use_cases/source_impact.rb
|
|
74
75
|
- lib/engram/version.rb
|
|
76
|
+
- lib/generators/engram/expiry_generator.rb
|
|
75
77
|
- lib/generators/engram/install_generator.rb
|
|
78
|
+
- lib/generators/engram/templates/add_expiry_to_engram_memories.rb.tt
|
|
76
79
|
- lib/generators/engram/templates/create_engram_memories.rb.tt
|
|
77
80
|
- lib/generators/engram/templates/initializer.rb.tt
|
|
78
81
|
- lib/generators/engram/templates/memory_record.rb.tt
|