echo-mem 0.5.2__tar.gz → 0.5.4__tar.gz

{echo_mem-0.5.2/src/echo_mem.egg-info → echo_mem-0.5.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: echo-mem
-Version: 0.5.2
+Version: 0.5.4
 Summary: A long-horizon memory architecture for AI agents
 License-Expression: BUSL-1.1
 Requires-Python: >=3.11
@@ -80,6 +80,27 @@ echo-memory install --global
 > hosted product. The import package and the CLI are both `echo_memory` / `echo-memory`;
 > only the distribution name differs.
 
+## Upgrading
+
+```bash
+pipx upgrade echo-mem
+echo-memory init-db                # apply any new migrations
+echo-memory reindex                # rebuild what the new code derives
+```
+
+Both steps, in that order, and neither is optional on a store that predates the version
+you just installed.
+
+**Skipping `init-db` fails loudly.** A read against a schema older than the code raises an
+undefined column, which is the good case: it stops rather than answering from something it
+half understands.
+
+**Skipping `reindex` fails silently, which is worse.** Derived data that the new code reads
+differently is still the old data, and nothing errors. Concretely, on an upgrade to 0.5.3
+the lexical corpus statistics are treated as stale until rebuilt, so BM25 stays off and
+your queries keep ranking by `ts_rank` while the setting reports as on. `reindex` rebuilds
+the embeddings and those statistics together. `echo-memory health` tells you afterwards.
+
 ## Usage
 
 ```bash
@@ -94,9 +115,13 @@ echo-memory export                 # everything, as JSON
 echo-memory install --for cursor   # wire one client, project scoped
 echo-memory adopt                  # wire every MCP client on the machine, each with its own id
 
+echo-memory infer-causal-hints     # type the facts whose own sentence states a cause (dry run)
+
 echo-memory eval                   # retrieval quality against your own store
 echo-memory eval --context         # what a recall costs against injecting everything
 echo-memory eval --context --sweep # the same, as a curve across corpus size
+echo-memory eval-external locomo <path>       # LoCoMo, the corpus the published figures use
+echo-memory eval-external longmemeval <path>  # LongMemEval, the same
 echo-memory calibrate              # is entity resolution trustworthy on your data
 echo-memory benchmark              # write, query and digest latency
 ```
@@ -106,7 +131,7 @@ echo-memory benchmark              # write, query and digest latency
 | Tool | What it does |
 |---|---|
 | `write_episode` | Store entities and the facts connecting them. No model call. |
-| `query_memory` | Hybrid vector and full text retrieval, fused by reciprocal rank. |
+| `query_memory` | Hybrid vector and full text retrieval, fused by reciprocal rank. Full text ranks by BM25. Takes `as_of` to read the store as it stood at an instant, and `about` to return the facts recorded against one or two named entities. |
 | `trace_cause` | Causal chains through the entities a subject matches, not a ranked list. |
 | `record_recall_save` | Mark that a recalled fact saved re explaining something. Refuses a fact no read returned. |
 | `get_audit_log` | Every change to memory, with a plain language reason. |
@@ -126,6 +151,59 @@ no summarisation, and retrieval today walks every active fact in the scope. It i
 described in [`docs/designs/`](docs/designs/) and listed below as v1c, and this paragraph
 used to claim it in the present tense.
 
+**A ranked answer that says why each fact is in it.** Every returned fact carries
+`score`, its cosine similarity to the query, `rank`, its position, and `matched`, the
+channels that found it. `score` is deliberately not the fusion number: reciprocal rank
+values are sums of 1/(k+rank) and mean nothing from one query to the next, while a
+similarity means the same thing every time, which is what a caller thresholding on it
+needs. It is computed for every fact returned, including the ones only full text search
+found, because which channel retrieved a fact is an implementation detail and has no
+business reaching a field callers read as relevance.
+
+**The full text channel ranks by BM25.** Postgres's `ts_rank` counts term occurrences and
+nothing else: no inverse document frequency, so a word in every fact of a scope counts as
+much as one in three; no saturation, so repetition scales linearly; no length
+normalisation, so a long fact is punished for being long. On 324 real prompts the rank 3
+and rank 4 scores were identical in 59% of them, and reciprocal rank fusion reads rank
+position and never the score underneath it, so a channel ordered by scan order handed the
+fusion a coin flip.
+
+BM25 is computed in SQL rather than installed, so no extension is added to your database.
+Corpus statistics are rebuilt amortised on write inside the transaction that already holds
+the scope's lock, and by `echo-memory reindex`; a scope that has none falls back to
+`ts_rank` rather than ranking on statistics it does not have. `ECHO_MEMORY_LEXICAL_BM25=0`
+returns to `ts_rank` everywhere.
+
+**Facts about an entity, which resemblance cannot express.** `query_memory` takes
+`about`, one or two entity names, and returns only the facts recorded against them. One
+name gives the edges incident to that node; two give the edges between them in either
+direction, which is "both names in the same fact" without going near the fact text.
+
+This is not a ranking improvement and it is not a substitute for one. A query about one
+team returns a semantically close fact about a different team because the wrong fact
+genuinely does resemble the question, and no amount of better ranking removes it.
+Identity is not similarity, and a fact is an edge between two nodes, so asking which
+entity a fact is about is structural and exact.
+
+Name matching is exact and case insensitive, never a substring, because a short name is a
+substring of longer unrelated words. A name nothing is recorded under returns no facts
+rather than the nearest thing, and so does a pair with no fact joining them: that is real
+absence and it means nobody recorded this, not here is something adjacent. More than two
+names is refused rather than answered empty, because an edge has two endpoints and a fact
+about three entities is a question this model cannot express.
+
+**The history the store has always kept is readable.** Every fact has carried `t_valid`
+and `t_invalid` since the first migration, and the read path had only ever asked whether a
+fact is current, so the store paid to keep the whole record of what a scope believed and
+could not answer the first question a post mortem asks. `query_memory` takes `as_of` in
+unix seconds and every channel honours it: vector, full text, the graph hop, the digest.
+
+Supersession is what makes that more than a curiosity. Writing the same source, target and
+relation again does not edit the old fact, it ends it: the old edge takes a `t_invalid` and
+stays queryable. Nothing is ever rewritten, so the history is real rather than
+reconstructed, which is the property the memory reconsolidation literature gives up and
+the reason this project refuses that idea.
+
 **Provenance on every fact.** Who wrote it, which tool, which project, when, and which
 reads returned it. A superseded fact is never deleted. It stops being drawn and stays
 reachable with its history.
@@ -138,13 +216,30 @@ this happen" is an ordered chain and similarity cannot produce one. "A led_to B"
 caused_by A" are one claim written from opposite ends, and both assemble into the same
 chain.
 
-A fact with no hint is associative, which is the default and usually correct. Facts
-written before 0.5.0 have no hint and are not a gap to fix. An empty answer from
-`trace_cause` says which kind of empty it is: nobody recorded a cause here is a
-different fact about a store than there are no chains.
+A fact with no hint is associative, which is the default and usually correct. An empty
+answer from `trace_cause` says which kind of empty it is: nobody recorded a cause here is
+a different fact about a store than there are no chains.
+
+**Filling it in, for a store that predates it.** Facts written before 0.5.0 carry no
+hint, so on an existing store `trace_cause` has nothing to walk.
+`echo-memory infer-causal-hints` re-reads the fact text already stored with your own
+model (`ECHO_MEMORY_LLM_API_KEY`, `ECHO_MEMORY_LLM_MODEL`) and types the edges whose own
+sentence states the relation. Dry run by default; `--write` applies what it printed and
+`--clear --write` takes it back.
+
+This is extraction done late, not causal discovery, and the difference is enforced rather
+than promised. A sentence with no causal connective is never sent to a model, so
+co-occurrence is refused before it costs anything. Every proposal has to quote the words
+that state the relation, and a quote that is not literally in the fact is dropped, so a
+model reasoning from the world instead of reading the sentence gets nothing stored. Each
+hint written is audited and marked as extracted late, which is what makes `--clear`
+able to remove exactly these and never one you wrote at write time.
 
 **No inference on the write path.** Extraction happens in the calling agent, so storing
-a memory invokes no model on the server. The cost moved rather than vanished: the agent
+a memory invokes no model on the server. `infer-causal-hints` above is not an exception
+to that and is worth being precise about: it is a command a person types against facts
+already stored, it never runs as part of a write, a migration or a hook, and a store that
+never runs it never causes a model call. The cost moved rather than vanished: the agent
 has to arrive with entities and facts already extracted, which is what the
 [tool contract](docs/DEVELOPMENT.md) spells out. The comparison that makes this matter is
 Zep/Graphiti, the closest architectural match, whose own description of ingestion is that
@@ -169,8 +264,9 @@ difference nobody sized is not a result.
 |---|---|---|
 | Context per recall vs injecting everything | **96.7% less**, hit@10 0.872 over 1,190 questions | `echo-memory eval --context` |
 | The same saving across 8x of corpus growth | 75.5% at 32 facts rising to **96.4% at 261**, hit@10 0.900 to 0.946 | `echo-memory eval --context --sweep` |
-| LoCoMo retrieval, 1,982 questions, 5,882 turns | recall@10 **0.601**, hit@10 0.658, MRR 0.460 | `scripts/locomo-bench.py` |
-| LongMemEval retrieval, 90 questions, 15 per type | session@10 **0.940**, turn@10 0.727 | `scripts/longmemeval-bench.py --per-type 15` |
+| LoCoMo retrieval, 1,982 questions, 5,882 turns | recall@10 **0.601**, hit@10 0.658, MRR 0.460, session@10 0.850 | `echo-memory eval-external locomo` |
+| LongMemEval retrieval, 90 questions, 15 per type | session@10 **0.937**, recall@10 0.727, MRR 0.428 | `echo-memory eval-external longmemeval --per-type 15` |
+| Lexical BM25 against `ts_rank` | **+0.0271** MRR on entity_pair [+0.0083, +0.0481] and on multihop [+0.0166, +0.0381], the other two shapes inside noise, none worse | `echo-memory eval --ablate`, the `without BM25` row with its sign reversed |
 | Server side model calls per write | **0** | `echo-memory benchmark` |
 | Write, query, digest latency (median) | 15ms, 8ms, 1ms | `echo-memory benchmark` |
 | Entity resolution AUC | 0.666, 95% CI [0.421, 0.881] | `echo-memory calibrate` |
@@ -280,7 +376,8 @@ not a thing to withhold from the person whose data it is.
 organisation wide shared graph, with no forced migration later. The novel work is the
 memory structure and the read/write algorithm on top of it, not a new database engine.
 
-**Retrieval** Hybrid vector and full text search fused by reciprocal rank in v1a.
+**Retrieval** Hybrid vector and full text search fused by reciprocal rank in v1a, with
+the full text side ranked by BM25 computed in SQL over per scope corpus statistics.
 Personalised PageRank via `networkx` lands in v1b for multi hop associative retrieval.
 
 **Interface** [Model Context Protocol](https://modelcontextprotocol.io), so any compliant

{echo_mem-0.5.2 → echo_mem-0.5.4}/README.md

@@ -57,6 +57,27 @@ echo-memory install --global
 > hosted product. The import package and the CLI are both `echo_memory` / `echo-memory`;
 > only the distribution name differs.
 
+## Upgrading
+
+```bash
+pipx upgrade echo-mem
+echo-memory init-db                # apply any new migrations
+echo-memory reindex                # rebuild what the new code derives
+```
+
+Both steps, in that order, and neither is optional on a store that predates the version
+you just installed.
+
+**Skipping `init-db` fails loudly.** A read against a schema older than the code raises an
+undefined column, which is the good case: it stops rather than answering from something it
+half understands.
+
+**Skipping `reindex` fails silently, which is worse.** Derived data that the new code reads
+differently is still the old data, and nothing errors. Concretely, on an upgrade to 0.5.3
+the lexical corpus statistics are treated as stale until rebuilt, so BM25 stays off and
+your queries keep ranking by `ts_rank` while the setting reports as on. `reindex` rebuilds
+the embeddings and those statistics together. `echo-memory health` tells you afterwards.
+
 ## Usage
 
 ```bash
@@ -71,9 +92,13 @@ echo-memory export                 # everything, as JSON
 echo-memory install --for cursor   # wire one client, project scoped
 echo-memory adopt                  # wire every MCP client on the machine, each with its own id
 
+echo-memory infer-causal-hints     # type the facts whose own sentence states a cause (dry run)
+
 echo-memory eval                   # retrieval quality against your own store
 echo-memory eval --context         # what a recall costs against injecting everything
 echo-memory eval --context --sweep # the same, as a curve across corpus size
+echo-memory eval-external locomo <path>       # LoCoMo, the corpus the published figures use
+echo-memory eval-external longmemeval <path>  # LongMemEval, the same
 echo-memory calibrate              # is entity resolution trustworthy on your data
 echo-memory benchmark              # write, query and digest latency
 ```
@@ -83,7 +108,7 @@ echo-memory benchmark              # write, query and digest latency
 | Tool | What it does |
 |---|---|
 | `write_episode` | Store entities and the facts connecting them. No model call. |
-| `query_memory` | Hybrid vector and full text retrieval, fused by reciprocal rank. |
+| `query_memory` | Hybrid vector and full text retrieval, fused by reciprocal rank. Full text ranks by BM25. Takes `as_of` to read the store as it stood at an instant, and `about` to return the facts recorded against one or two named entities. |
 | `trace_cause` | Causal chains through the entities a subject matches, not a ranked list. |
 | `record_recall_save` | Mark that a recalled fact saved re explaining something. Refuses a fact no read returned. |
 | `get_audit_log` | Every change to memory, with a plain language reason. |
@@ -103,6 +128,59 @@ no summarisation, and retrieval today walks every active fact in the scope. It i
 described in [`docs/designs/`](docs/designs/) and listed below as v1c, and this paragraph
 used to claim it in the present tense.
 
+**A ranked answer that says why each fact is in it.** Every returned fact carries
+`score`, its cosine similarity to the query, `rank`, its position, and `matched`, the
+channels that found it. `score` is deliberately not the fusion number: reciprocal rank
+values are sums of 1/(k+rank) and mean nothing from one query to the next, while a
+similarity means the same thing every time, which is what a caller thresholding on it
+needs. It is computed for every fact returned, including the ones only full text search
+found, because which channel retrieved a fact is an implementation detail and has no
+business reaching a field callers read as relevance.
+
+**The full text channel ranks by BM25.** Postgres's `ts_rank` counts term occurrences and
+nothing else: no inverse document frequency, so a word in every fact of a scope counts as
+much as one in three; no saturation, so repetition scales linearly; no length
+normalisation, so a long fact is punished for being long. On 324 real prompts the rank 3
+and rank 4 scores were identical in 59% of them, and reciprocal rank fusion reads rank
+position and never the score underneath it, so a channel ordered by scan order handed the
+fusion a coin flip.
+
+BM25 is computed in SQL rather than installed, so no extension is added to your database.
+Corpus statistics are rebuilt amortised on write inside the transaction that already holds
+the scope's lock, and by `echo-memory reindex`; a scope that has none falls back to
+`ts_rank` rather than ranking on statistics it does not have. `ECHO_MEMORY_LEXICAL_BM25=0`
+returns to `ts_rank` everywhere.
+
+**Facts about an entity, which resemblance cannot express.** `query_memory` takes
+`about`, one or two entity names, and returns only the facts recorded against them. One
+name gives the edges incident to that node; two give the edges between them in either
+direction, which is "both names in the same fact" without going near the fact text.
+
+This is not a ranking improvement and it is not a substitute for one. A query about one
+team returns a semantically close fact about a different team because the wrong fact
+genuinely does resemble the question, and no amount of better ranking removes it.
+Identity is not similarity, and a fact is an edge between two nodes, so asking which
+entity a fact is about is structural and exact.
+
+Name matching is exact and case insensitive, never a substring, because a short name is a
+substring of longer unrelated words. A name nothing is recorded under returns no facts
+rather than the nearest thing, and so does a pair with no fact joining them: that is real
+absence and it means nobody recorded this, not here is something adjacent. More than two
+names is refused rather than answered empty, because an edge has two endpoints and a fact
+about three entities is a question this model cannot express.
+
+**The history the store has always kept is readable.** Every fact has carried `t_valid`
+and `t_invalid` since the first migration, and the read path had only ever asked whether a
+fact is current, so the store paid to keep the whole record of what a scope believed and
+could not answer the first question a post mortem asks. `query_memory` takes `as_of` in
+unix seconds and every channel honours it: vector, full text, the graph hop, the digest.
+
+Supersession is what makes that more than a curiosity. Writing the same source, target and
+relation again does not edit the old fact, it ends it: the old edge takes a `t_invalid` and
+stays queryable. Nothing is ever rewritten, so the history is real rather than
+reconstructed, which is the property the memory reconsolidation literature gives up and
+the reason this project refuses that idea.
+
 **Provenance on every fact.** Who wrote it, which tool, which project, when, and which
 reads returned it. A superseded fact is never deleted. It stops being drawn and stays
 reachable with its history.
@@ -115,13 +193,30 @@ this happen" is an ordered chain and similarity cannot produce one. "A led_to B"
 caused_by A" are one claim written from opposite ends, and both assemble into the same
 chain.
 
-A fact with no hint is associative, which is the default and usually correct. Facts
-written before 0.5.0 have no hint and are not a gap to fix. An empty answer from
-`trace_cause` says which kind of empty it is: nobody recorded a cause here is a
-different fact about a store than there are no chains.
+A fact with no hint is associative, which is the default and usually correct. An empty
+answer from `trace_cause` says which kind of empty it is: nobody recorded a cause here is
+a different fact about a store than there are no chains.
+
+**Filling it in, for a store that predates it.** Facts written before 0.5.0 carry no
+hint, so on an existing store `trace_cause` has nothing to walk.
+`echo-memory infer-causal-hints` re-reads the fact text already stored with your own
+model (`ECHO_MEMORY_LLM_API_KEY`, `ECHO_MEMORY_LLM_MODEL`) and types the edges whose own
+sentence states the relation. Dry run by default; `--write` applies what it printed and
+`--clear --write` takes it back.
+
+This is extraction done late, not causal discovery, and the difference is enforced rather
+than promised. A sentence with no causal connective is never sent to a model, so
+co-occurrence is refused before it costs anything. Every proposal has to quote the words
+that state the relation, and a quote that is not literally in the fact is dropped, so a
+model reasoning from the world instead of reading the sentence gets nothing stored. Each
+hint written is audited and marked as extracted late, which is what makes `--clear`
+able to remove exactly these and never one you wrote at write time.
 
 **No inference on the write path.** Extraction happens in the calling agent, so storing
-a memory invokes no model on the server. The cost moved rather than vanished: the agent
+a memory invokes no model on the server. `infer-causal-hints` above is not an exception
+to that and is worth being precise about: it is a command a person types against facts
+already stored, it never runs as part of a write, a migration or a hook, and a store that
+never runs it never causes a model call. The cost moved rather than vanished: the agent
 has to arrive with entities and facts already extracted, which is what the
 [tool contract](docs/DEVELOPMENT.md) spells out. The comparison that makes this matter is
 Zep/Graphiti, the closest architectural match, whose own description of ingestion is that
@@ -146,8 +241,9 @@ difference nobody sized is not a result.
 |---|---|---|
 | Context per recall vs injecting everything | **96.7% less**, hit@10 0.872 over 1,190 questions | `echo-memory eval --context` |
 | The same saving across 8x of corpus growth | 75.5% at 32 facts rising to **96.4% at 261**, hit@10 0.900 to 0.946 | `echo-memory eval --context --sweep` |
-| LoCoMo retrieval, 1,982 questions, 5,882 turns | recall@10 **0.601**, hit@10 0.658, MRR 0.460 | `scripts/locomo-bench.py` |
-| LongMemEval retrieval, 90 questions, 15 per type | session@10 **0.940**, turn@10 0.727 | `scripts/longmemeval-bench.py --per-type 15` |
+| LoCoMo retrieval, 1,982 questions, 5,882 turns | recall@10 **0.601**, hit@10 0.658, MRR 0.460, session@10 0.850 | `echo-memory eval-external locomo` |
+| LongMemEval retrieval, 90 questions, 15 per type | session@10 **0.937**, recall@10 0.727, MRR 0.428 | `echo-memory eval-external longmemeval --per-type 15` |
+| Lexical BM25 against `ts_rank` | **+0.0271** MRR on entity_pair [+0.0083, +0.0481] and on multihop [+0.0166, +0.0381], the other two shapes inside noise, none worse | `echo-memory eval --ablate`, the `without BM25` row with its sign reversed |
 | Server side model calls per write | **0** | `echo-memory benchmark` |
 | Write, query, digest latency (median) | 15ms, 8ms, 1ms | `echo-memory benchmark` |
 | Entity resolution AUC | 0.666, 95% CI [0.421, 0.881] | `echo-memory calibrate` |
@@ -257,7 +353,8 @@ not a thing to withhold from the person whose data it is.
 organisation wide shared graph, with no forced migration later. The novel work is the
 memory structure and the read/write algorithm on top of it, not a new database engine.
 
-**Retrieval** Hybrid vector and full text search fused by reciprocal rank in v1a.
+**Retrieval** Hybrid vector and full text search fused by reciprocal rank in v1a, with
+the full text side ranked by BM25 computed in SQL over per scope corpus statistics.
 Personalised PageRank via `networkx` lands in v1b for multi hop associative retrieval.
 
 **Interface** [Model Context Protocol](https://modelcontextprotocol.io), so any compliant

{echo_mem-0.5.2 → echo_mem-0.5.4}/pyproject.toml

@@ -6,7 +6,7 @@
 # (`-m echo_memory.server`) and every existing install reference it, and
 # renaming the module would break live configuration to fix a cosmetic mismatch.
 name = "echo-mem"
-version = "0.5.2"
+version = "0.5.4"
 description = "A long-horizon memory architecture for AI agents"
 readme = "README.md"
 # BUSL-1.1 is the SPDX identifier for the Business Source License 1.1, so

{echo_mem-0.5.2 → echo_mem-0.5.4/src/echo_mem.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: echo-mem
-Version: 0.5.2
+Version: 0.5.4
 Summary: A long-horizon memory architecture for AI agents
 License-Expression: BUSL-1.1
 Requires-Python: >=3.11
@@ -80,6 +80,27 @@ echo-memory install --global
 > hosted product. The import package and the CLI are both `echo_memory` / `echo-memory`;
 > only the distribution name differs.
 
+## Upgrading
+
+```bash
+pipx upgrade echo-mem
+echo-memory init-db                # apply any new migrations
+echo-memory reindex                # rebuild what the new code derives
+```
+
+Both steps, in that order, and neither is optional on a store that predates the version
+you just installed.
+
+**Skipping `init-db` fails loudly.** A read against a schema older than the code raises an
+undefined column, which is the good case: it stops rather than answering from something it
+half understands.
+
+**Skipping `reindex` fails silently, which is worse.** Derived data that the new code reads
+differently is still the old data, and nothing errors. Concretely, on an upgrade to 0.5.3
+the lexical corpus statistics are treated as stale until rebuilt, so BM25 stays off and
+your queries keep ranking by `ts_rank` while the setting reports as on. `reindex` rebuilds
+the embeddings and those statistics together. `echo-memory health` tells you afterwards.
+
 ## Usage
 
 ```bash
@@ -94,9 +115,13 @@ echo-memory export                 # everything, as JSON
 echo-memory install --for cursor   # wire one client, project scoped
 echo-memory adopt                  # wire every MCP client on the machine, each with its own id
 
+echo-memory infer-causal-hints     # type the facts whose own sentence states a cause (dry run)
+
 echo-memory eval                   # retrieval quality against your own store
 echo-memory eval --context         # what a recall costs against injecting everything
 echo-memory eval --context --sweep # the same, as a curve across corpus size
+echo-memory eval-external locomo <path>       # LoCoMo, the corpus the published figures use
+echo-memory eval-external longmemeval <path>  # LongMemEval, the same
 echo-memory calibrate              # is entity resolution trustworthy on your data
 echo-memory benchmark              # write, query and digest latency
 ```
@@ -106,7 +131,7 @@ echo-memory benchmark              # write, query and digest latency
 | Tool | What it does |
 |---|---|
 | `write_episode` | Store entities and the facts connecting them. No model call. |
-| `query_memory` | Hybrid vector and full text retrieval, fused by reciprocal rank. |
+| `query_memory` | Hybrid vector and full text retrieval, fused by reciprocal rank. Full text ranks by BM25. Takes `as_of` to read the store as it stood at an instant, and `about` to return the facts recorded against one or two named entities. |
 | `trace_cause` | Causal chains through the entities a subject matches, not a ranked list. |
 | `record_recall_save` | Mark that a recalled fact saved re explaining something. Refuses a fact no read returned. |
 | `get_audit_log` | Every change to memory, with a plain language reason. |
@@ -126,6 +151,59 @@ no summarisation, and retrieval today walks every active fact in the scope. It i
 described in [`docs/designs/`](docs/designs/) and listed below as v1c, and this paragraph
 used to claim it in the present tense.
 
+**A ranked answer that says why each fact is in it.** Every returned fact carries
+`score`, its cosine similarity to the query, `rank`, its position, and `matched`, the
+channels that found it. `score` is deliberately not the fusion number: reciprocal rank
+values are sums of 1/(k+rank) and mean nothing from one query to the next, while a
+similarity means the same thing every time, which is what a caller thresholding on it
+needs. It is computed for every fact returned, including the ones only full text search
+found, because which channel retrieved a fact is an implementation detail and has no
+business reaching a field callers read as relevance.
+
+**The full text channel ranks by BM25.** Postgres's `ts_rank` counts term occurrences and
+nothing else: no inverse document frequency, so a word in every fact of a scope counts as
+much as one in three; no saturation, so repetition scales linearly; no length
+normalisation, so a long fact is punished for being long. On 324 real prompts the rank 3
+and rank 4 scores were identical in 59% of them, and reciprocal rank fusion reads rank
+position and never the score underneath it, so a channel ordered by scan order handed the
+fusion a coin flip.
+
+BM25 is computed in SQL rather than installed, so no extension is added to your database.
+Corpus statistics are rebuilt amortised on write inside the transaction that already holds
+the scope's lock, and by `echo-memory reindex`; a scope that has none falls back to
+`ts_rank` rather than ranking on statistics it does not have. `ECHO_MEMORY_LEXICAL_BM25=0`
+returns to `ts_rank` everywhere.
+
+**Facts about an entity, which resemblance cannot express.** `query_memory` takes
+`about`, one or two entity names, and returns only the facts recorded against them. One
+name gives the edges incident to that node; two give the edges between them in either
+direction, which is "both names in the same fact" without going near the fact text.
+
+This is not a ranking improvement and it is not a substitute for one. A query about one
+team returns a semantically close fact about a different team because the wrong fact
+genuinely does resemble the question, and no amount of better ranking removes it.
+Identity is not similarity, and a fact is an edge between two nodes, so asking which
+entity a fact is about is structural and exact.
+
+Name matching is exact and case insensitive, never a substring, because a short name is a
+substring of longer unrelated words. A name nothing is recorded under returns no facts
+rather than the nearest thing, and so does a pair with no fact joining them: that is real
+absence and it means nobody recorded this, not here is something adjacent. More than two
+names is refused rather than answered empty, because an edge has two endpoints and a fact
+about three entities is a question this model cannot express.
+
+**The history the store has always kept is readable.** Every fact has carried `t_valid`
+and `t_invalid` since the first migration, and the read path had only ever asked whether a
+fact is current, so the store paid to keep the whole record of what a scope believed and
+could not answer the first question a post mortem asks. `query_memory` takes `as_of` in
+unix seconds and every channel honours it: vector, full text, the graph hop, the digest.
+
+Supersession is what makes that more than a curiosity. Writing the same source, target and
+relation again does not edit the old fact, it ends it: the old edge takes a `t_invalid` and
+stays queryable. Nothing is ever rewritten, so the history is real rather than
+reconstructed, which is the property the memory reconsolidation literature gives up and
+the reason this project refuses that idea.
+
 **Provenance on every fact.** Who wrote it, which tool, which project, when, and which
 reads returned it. A superseded fact is never deleted. It stops being drawn and stays
 reachable with its history.
@@ -138,13 +216,30 @@ this happen" is an ordered chain and similarity cannot produce one. "A led_to B"
 caused_by A" are one claim written from opposite ends, and both assemble into the same
 chain.
 
-A fact with no hint is associative, which is the default and usually correct. Facts
-written before 0.5.0 have no hint and are not a gap to fix. An empty answer from
-`trace_cause` says which kind of empty it is: nobody recorded a cause here is a
-different fact about a store than there are no chains.
+A fact with no hint is associative, which is the default and usually correct. An empty
+answer from `trace_cause` says which kind of empty it is: nobody recorded a cause here is
+a different fact about a store than there are no chains.
+
+**Filling it in, for a store that predates it.** Facts written before 0.5.0 carry no
+hint, so on an existing store `trace_cause` has nothing to walk.
+`echo-memory infer-causal-hints` re-reads the fact text already stored with your own
+model (`ECHO_MEMORY_LLM_API_KEY`, `ECHO_MEMORY_LLM_MODEL`) and types the edges whose own
+sentence states the relation. Dry run by default; `--write` applies what it printed and
+`--clear --write` takes it back.
+
+This is extraction done late, not causal discovery, and the difference is enforced rather
+than promised. A sentence with no causal connective is never sent to a model, so
+co-occurrence is refused before it costs anything. Every proposal has to quote the words
+that state the relation, and a quote that is not literally in the fact is dropped, so a
+model reasoning from the world instead of reading the sentence gets nothing stored. Each
+hint written is audited and marked as extracted late, which is what makes `--clear`
+able to remove exactly these and never one you wrote at write time.
 
 **No inference on the write path.** Extraction happens in the calling agent, so storing
-a memory invokes no model on the server. The cost moved rather than vanished: the agent
+a memory invokes no model on the server. `infer-causal-hints` above is not an exception
+to that and is worth being precise about: it is a command a person types against facts
+already stored, it never runs as part of a write, a migration or a hook, and a store that
+never runs it never causes a model call. The cost moved rather than vanished: the agent
 has to arrive with entities and facts already extracted, which is what the
 [tool contract](docs/DEVELOPMENT.md) spells out. The comparison that makes this matter is
 Zep/Graphiti, the closest architectural match, whose own description of ingestion is that
@@ -169,8 +264,9 @@ difference nobody sized is not a result.
 |---|---|---|
 | Context per recall vs injecting everything | **96.7% less**, hit@10 0.872 over 1,190 questions | `echo-memory eval --context` |
 | The same saving across 8x of corpus growth | 75.5% at 32 facts rising to **96.4% at 261**, hit@10 0.900 to 0.946 | `echo-memory eval --context --sweep` |
-| LoCoMo retrieval, 1,982 questions, 5,882 turns | recall@10 **0.601**, hit@10 0.658, MRR 0.460 | `scripts/locomo-bench.py` |
-| LongMemEval retrieval, 90 questions, 15 per type | session@10 **0.940**, turn@10 0.727 | `scripts/longmemeval-bench.py --per-type 15` |
+| LoCoMo retrieval, 1,982 questions, 5,882 turns | recall@10 **0.601**, hit@10 0.658, MRR 0.460, session@10 0.850 | `echo-memory eval-external locomo` |
+| LongMemEval retrieval, 90 questions, 15 per type | session@10 **0.937**, recall@10 0.727, MRR 0.428 | `echo-memory eval-external longmemeval --per-type 15` |
+| Lexical BM25 against `ts_rank` | **+0.0271** MRR on entity_pair [+0.0083, +0.0481] and on multihop [+0.0166, +0.0381], the other two shapes inside noise, none worse | `echo-memory eval --ablate`, the `without BM25` row with its sign reversed |
 | Server side model calls per write | **0** | `echo-memory benchmark` |
 | Write, query, digest latency (median) | 15ms, 8ms, 1ms | `echo-memory benchmark` |
 | Entity resolution AUC | 0.666, 95% CI [0.421, 0.881] | `echo-memory calibrate` |
@@ -280,7 +376,8 @@ not a thing to withhold from the person whose data it is.
 organisation wide shared graph, with no forced migration later. The novel work is the
 memory structure and the read/write algorithm on top of it, not a new database engine.
 
-**Retrieval** Hybrid vector and full text search fused by reciprocal rank in v1a.
+**Retrieval** Hybrid vector and full text search fused by reciprocal rank in v1a, with
+the full text side ranked by BM25 computed in SQL over per scope corpus statistics.
 Personalised PageRank via `networkx` lands in v1b for multi hop associative retrieval.
 
 **Interface** [Model Context Protocol](https://modelcontextprotocol.io), so any compliant

{echo_mem-0.5.2 → echo_mem-0.5.4}/src/echo_mem.egg-info/SOURCES.txt

@@ -25,9 +25,11 @@ src/echo_memory/cli/export.py
 src/echo_memory/cli/graph.py
 src/echo_memory/cli/health.py
 src/echo_memory/cli/hooks.py
+src/echo_memory/cli/infer_causal.py
 src/echo_memory/cli/initdb.py
 src/echo_memory/cli/install.py
 src/echo_memory/cli/judge.py
+src/echo_memory/cli/llm.py
 src/echo_memory/cli/main.py
 src/echo_memory/cli/merge.py
 src/echo_memory/cli/queue.py
@@ -45,6 +47,7 @@ src/echo_memory/cli/trial.py
 src/echo_memory/cli/unmerge.py
 src/echo_memory/cli/why.py
 src/echo_memory/eval/__init__.py
+src/echo_memory/eval/external.py
 src/echo_memory/eval/independent.py
 src/echo_memory/eval/retrieval.py
 src/echo_memory/eval/sweep.py
@@ -89,10 +92,15 @@ src/echo_memory/migrations/versions/0021_delivery_evidence.py
 src/echo_memory/migrations/versions/0022_independent_evaluation.py
 src/echo_memory/migrations/versions/0023_second_judge.py
 src/echo_memory/migrations/versions/0024_node_by_id.py
+src/echo_memory/migrations/versions/0025_lexical_statistics.py
+src/echo_memory/migrations/versions/0026_late_causal_extraction.py
+src/echo_memory/migrations/versions/0027_lexical_staleness_without_a_scan.py
 src/echo_memory/retrieval/__init__.py
+src/echo_memory/retrieval/bm25.py
 src/echo_memory/retrieval/causality.py
 src/echo_memory/retrieval/fusion.py
 src/echo_memory/retrieval/query_memory.py
+src/echo_memory/retrieval/salience.py
 src/echo_memory/skill/SKILL.md
 src/echo_memory/trial/__init__.py
 src/echo_memory/trial/check.py