memleaf 0.2.30__tar.gz → 0.2.31__tar.gz

{memleaf-0.2.30 → memleaf-0.2.31}/CHANGELOG.md

@@ -2,6 +2,13 @@
 
 All notable changes to memleaf are documented here.
 
+## 0.2.31 — 2026-09-07
+
+- Unify Core and the copied Hermes provider on one idempotent UTF-8 evidence budget: at most 64 source records, 32 KiB per body, and 128 KiB of aggregate body text. Complete matched sources above the former 2,000-character/eight-record limits remain available within those bounds.
+- Keep loss diagnostics bounded to 64 marker identities plus one aggregate marker. Per-record, aggregate, and marker overflow remains explicit incomplete evidence and cannot authorize a write or successful cleanup.
+- Preserve metadata/off, attachment, redaction, and call-ID behavior, and consume metadata-mode pending evidence after a successful capture using the same effective policy on pending and inbox sides.
+- Add provider-copy, capture-to-process, loss-defer, marker-capacity and lifecycle regressions. Deterministic tests do not claim real-model semantic quality, real-session replay, or customer acceptance.
+
 ## 0.2.30 — 2026-09-07
 
 - Add a physical evidence projection for the Gate: the complete local inventory remains available for provenance, replay and audit, while only user-origin units and complete external observations become bindable model evidence. Assistant synthesis, retrieved memory, incomplete observations and metadata-only records remain context or unresolved ledger state.

{memleaf-0.2.30/src/memleaf.egg-info → memleaf-0.2.31}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: memleaf
-Version: 0.2.30
+Version: 0.2.31
 Summary: A local-first Markdown memory core for AI agents
 Author: memleaf contributors
 License-Expression: MIT
@@ -23,8 +23,8 @@ Dynamic: license-file
 
 [English](README.en.md) · [PyPI](https://pypi.org/project/memleaf/) · [GitHub](https://github.com/miffyblueboo/memleaf)
 
-> **版本：0.2.30。**
-> 核心库、Vault、stdio MCP Server、初始化 CLI、模型路由、提炼流程、受控检索协议和宿主适配器已经实现。本版将完整证据清单与 Gate 可绑定的 physical evidence projection 分开，统一 `candidates`、`coverage`、`evidence_bindings` 协议，保留 legacy candidate-only 的 exact/bound compatibility，对未决证据安全收口并提供 allowlisted `evidence_check` 诊断，同时保持 source-neutral 语义、Markdown 唯一事实源和无 SQLite 运行时依赖。真实模型语义效果仍需结合本地模型和代表性样本验收。
+> **版本：0.2.31。**
+> 核心库、Vault、stdio MCP Server、初始化 CLI、模型路由、提炼流程、受控检索协议和宿主适配器已经实现。本版统一 Core 与 Hermes Provider 的 UTF-8 证据留存预算：最多 64 条 source records、每条正文 32 KiB、正文总量 128 KiB，并保留有界 loss markers 与幂等规范化；同时将 metadata 模式下成功捕获的超限 pending 证据按相同有效策略消费。Gate 继续使用 physical evidence projection、`candidates`、`coverage`、`evidence_bindings` 和 allowlisted `evidence_check` 诊断，保持 source-neutral 语义、Markdown 唯一事实源和无 SQLite 运行时依赖。真实模型语义效果仍需结合本地模型和代表性样本验收。
 > **当前版本支持 Hermes 和 Codex。** Antigravity（反重力）不检测、不安装、不配置。
 
 ## 项目定位
@@ -490,7 +490,7 @@ MIT，见 [LICENSE](LICENSE)。
 *Your memories, in files you own.*
 
 
-## 通用处理与只读验收（0.2.30）
+## 通用处理与只读验收（0.2.31）
 
 邮件、日历、工单、文件、浏览器与普通对话共用证据准入、覆盖检查和写入路径。
 自动摘要只能使用获准引用的原文；助手复述和旧记忆回读不能单独授权新增写入。

{memleaf-0.2.30 → memleaf-0.2.31}/README.en.md

@@ -4,8 +4,8 @@
 
 [中文](README.md) · [PyPI](https://pypi.org/project/memleaf/) · [GitHub](https://github.com/miffyblueboo/memleaf)
 
-> **Version: 0.2.30.**
-> The core library, Vault, stdio MCP server, initialization CLI, model routing, memory extraction, controlled retrieval protocol, and host adapters are implemented. This release separates the complete evidence inventory from the Gate's bindable physical evidence projection, unifies the `candidates`, `coverage`, and `evidence_bindings` contract, preserves legacy candidate-only exact/bound compatibility, keeps unresolved evidence safe, and adds allowlisted `evidence_check` diagnostics while preserving source-neutral semantics, Markdown as the sole source of truth, and zero SQLite runtime dependencies. Real-model semantics still require local acceptance with the selected model and representative inputs.
+> **Version: 0.2.31.**
+> The core library, Vault, stdio MCP server, initialization CLI, model routing, memory extraction, controlled retrieval protocol, and host adapters are implemented. This release unifies Core and the Hermes provider on an idempotent UTF-8 evidence budget: at most 64 source records, 32 KiB per body, and 128 KiB of aggregate body text, with bounded loss markers; metadata-mode pending evidence from an oversized observation is consumed after successful capture under the same effective policy. The Gate still uses the physical evidence projection, the `candidates`, `coverage`, and `evidence_bindings` contract, and allowlisted `evidence_check` diagnostics while preserving source-neutral semantics, Markdown as the sole source of truth, and zero SQLite runtime dependencies. Real-model semantics still require local acceptance with the selected model and representative inputs.
 > **The current release supports Hermes and Codex.** Antigravity is not detected, installed, or configured.
 
 ## Project scope
@@ -474,7 +474,7 @@ MIT; see [LICENSE](LICENSE).
 *Your memories, in files you own.*
 
 
-## General processing and read-only inspection (0.2.30)
+## General processing and read-only inspection (0.2.31)
 
 Dialogue, calendars, tickets, files, web results and other tools share the evidence, coverage and write path.
 Models interpret semantics; Core validates physical provenance and exact original quotations.

{memleaf-0.2.30 → memleaf-0.2.31}/README.md

@@ -4,8 +4,8 @@
 
 [English](README.en.md) · [PyPI](https://pypi.org/project/memleaf/) · [GitHub](https://github.com/miffyblueboo/memleaf)
 
-> **版本：0.2.30。**
-> 核心库、Vault、stdio MCP Server、初始化 CLI、模型路由、提炼流程、受控检索协议和宿主适配器已经实现。本版将完整证据清单与 Gate 可绑定的 physical evidence projection 分开，统一 `candidates`、`coverage`、`evidence_bindings` 协议，保留 legacy candidate-only 的 exact/bound compatibility，对未决证据安全收口并提供 allowlisted `evidence_check` 诊断，同时保持 source-neutral 语义、Markdown 唯一事实源和无 SQLite 运行时依赖。真实模型语义效果仍需结合本地模型和代表性样本验收。
+> **版本：0.2.31。**
+> 核心库、Vault、stdio MCP Server、初始化 CLI、模型路由、提炼流程、受控检索协议和宿主适配器已经实现。本版统一 Core 与 Hermes Provider 的 UTF-8 证据留存预算：最多 64 条 source records、每条正文 32 KiB、正文总量 128 KiB，并保留有界 loss markers 与幂等规范化；同时将 metadata 模式下成功捕获的超限 pending 证据按相同有效策略消费。Gate 继续使用 physical evidence projection、`candidates`、`coverage`、`evidence_bindings` 和 allowlisted `evidence_check` 诊断，保持 source-neutral 语义、Markdown 唯一事实源和无 SQLite 运行时依赖。真实模型语义效果仍需结合本地模型和代表性样本验收。
 > **当前版本支持 Hermes 和 Codex。** Antigravity（反重力）不检测、不安装、不配置。
 
 ## 项目定位
@@ -471,7 +471,7 @@ MIT，见 [LICENSE](LICENSE)。
 *Your memories, in files you own.*
 
 
-## 通用处理与只读验收（0.2.30）
+## 通用处理与只读验收（0.2.31）
 
 邮件、日历、工单、文件、浏览器与普通对话共用证据准入、覆盖检查和写入路径。
 自动摘要只能使用获准引用的原文；助手复述和旧记忆回读不能单独授权新增写入。

memleaf-0.2.31/docs/capture-budget-design.md

@@ -0,0 +1,74 @@
+# Tool evidence capture budget
+
+## Problem and ownership
+
+Capture permissions and capture capacity are separate contracts. `metadata`
+deliberately removes source bodies; `off` removes observations. Neither setting
+can produce new external facts for extraction. Enabling `bounded` authorizes
+bounded body retention, but does not reconstruct previously discarded data.
+
+The previous implementation also applied independent eight-record and
+2,000-character limits in the Hermes adapter, provenance normalization and
+pending-state processing. Early discovery and skill results could consume the
+entire record allowance before later source results arrived. Larger plain-text
+results were truncated before Core could account for their complete content.
+This is an ingestion-capacity problem; changing Gate semantics cannot recover
+those bytes.
+
+## Required implementation contract
+
+- Keep one shared, deterministic budget implementation for Core and the copied
+  Hermes provider. The provider must continue to work without importing Core
+  from the host's Python environment.
+- Bound record count, each body and aggregate body size. Publish the limits as
+  engineering capacity, not a guarantee of complete capture for arbitrary turns.
+  The implementation target is 64 source records, 32 KiB of UTF-8 text per body
+  and 128 KiB of total body text. Omission markers have a separate allowance of
+  64 call identities and one aggregate marker. They do not consume source-record
+  capacity or count themselves as newly lost observations on a later normalization
+  pass. Marker bodies are fixed diagnostics, never retained source excerpts.
+- Preserve complete, matched source results within those limits. Do not rank
+  business topics or tool names, infer facts from assistant text, split arbitrary
+  stdout into invented document records, or promote truncated prefixes to
+  complete observations.
+- Normalize retained evidence idempotently. Cache, capture, inbox read and
+  planning must not each discard another part of an already bounded inventory.
+- Keep explicit omission/incompleteness accounting when a limit is exceeded.
+  Policy-authorized but incomplete observations remain unresolved and cannot
+  authorize a write or successful source cleanup.
+- Apply `metadata`, `off`, attachment exclusions and redaction before persistent
+  writes. A capacity increase must not silently change capture permission.
+- Keep the existing Gate/Summarize semantic responsibility and Markdown storage
+  model. Larger source capacity is not a claim of successful model extraction.
+
+## Acceptance
+
+Use synthetic host messages shaped like an ordinary discovery/read turn: early
+tool discovery, skills and memory search followed by nine later results, with
+individual bodies ranging from hundreds to tens of thousands of characters.
+The old path loses all nine later results; the revised path must retain complete
+results when the full input is within its declared capacity.
+
+Verify the same evidence across adapter, pending cache, capture and inbox read;
+then verify a supported later-source candidate can pass the actual processing
+path in an isolated Vault. Check record/body/aggregate overflow separately,
+including repeated reads, accurate loss accounting, no write from incomplete
+evidence, and no cleanup of unresolved turns. Retain permission, redaction,
+cross-turn isolation and standalone installation coverage.
+
+Tests must use synthetic data and a deterministic backend. Live model semantics,
+changes to a user's capture permission, installation and release are separate
+operations with separately reported results.
+
+## Model capacity remains a separate limit
+
+The capture limits bound retained source data, not the final prompt or model
+token count. Evidence annotations, conversation context, retrieved context and
+the output allowance also consume model capacity. The current `llm.context_window`
+setting is not a pre-call tokenizer check. A source inventory within the capture
+budget can still exceed a selected model's usable context.
+
+Do not silently drop evidence to make that call succeed. Existing model-failure
+handling must preserve the failed turn and prevent cleanup. Automatic batching
+or model-specific token accounting needs its own design and is outside this
+capture-budget change.

{memleaf-0.2.30 → memleaf-0.2.31}/docs/evidence-retention.md

@@ -17,12 +17,20 @@ legacy boolean (false/absent -> metadata, true -> bounded). An explicit new mode
 wins over the legacy boolean. Invalid strings and non-boolean flags fail closed.
 Loading config does not rewrite it. Normal saves make the effective mode explicit.
 
-The inherited limits remain eight records, 2,000 characters per body, and 320 per
-metadata field. Oversized eligible evidence remains incomplete; its prefix is
-never promoted into a complete fact. Policy-excluded observations are marked
+The shared capture budget allows 64 source records, 32 KiB of UTF-8 text per body,
+128 KiB of total body text, and 320 characters per metadata field. Loss markers
+have a separate allowance of 64 call identities plus one aggregate marker, with
+fixed diagnostic bodies. Reapplying normalization does not reduce an
+already bounded inventory or count existing omissions again. Oversized eligible
+evidence remains incomplete; its prefix is never promoted into a complete fact.
+Policy-excluded observations are marked
 retention=metadata, have no content, and are not retried as missing evidence.
 No later relaxation recreates discarded original content.
 
+These are source-retention limits, not a guarantee that every configured model
+can process the resulting prompt. See [capture budget design](capture-budget-design.md)
+for the ingestion boundary, model-capacity limitation and acceptance contract.
+
 Document/attachment bodies require include_attachments=true and bounded mode.
 HostRuntime and the standalone Hermes adapter classify structural file arguments
 using the same tested contract (path/file_path/file_id/attachment_id/file URI,

{memleaf-0.2.30 → memleaf-0.2.31}/docs/general-processing.md

@@ -80,12 +80,15 @@ labels them NO_CHANGE. Incomplete turns retain their source instead of being
 cleaned after the usual grace period. Scope-filtered retries may revisit them;
 no endless automatic model retry or extra external tool call is introduced.
 
-Tool evidence is bounded to eight records with at most 2,000 content characters
-per captured result record and 320-character metadata fields, with redaction at
-Core capture. Large unambiguous top-level record collections retain complete
-records and enclosing context within that budget. Per-record provenance takes
-precedence over common source metadata. An overflow slot reports omitted
-records. Arbitrary large prose is not split into falsely complete facts;
+Tool evidence uses one shared budget: 64 source records, at most 32 KiB of UTF-8
+body text per record and 128 KiB of total body text, plus 320-character metadata
+fields, with redaction at Core capture. Cache and inbox normalization are
+idempotent under this budget. Large unambiguous top-level record collections
+retain complete records and enclosing context within that budget. Per-record provenance takes
+precedence over common source metadata. Separate loss markers report omitted
+records, with at most 64 call identities plus one aggregate marker. These markers
+have fixed diagnostic bodies and cannot supply external facts.
+Arbitrary large prose is not split into falsely complete facts;
 unsupported/incomplete content needs a supported complete source excerpt or a
 later source input. Switching from `metadata` to `bounded` does not reconstruct
 body content that was never captured; bounded retention can still produce an

{memleaf-0.2.30 → memleaf-0.2.31}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "memleaf"
-version = "0.2.30"
+version = "0.2.31"
 description = "A local-first Markdown memory core for AI agents"
 readme = "README.md"
 requires-python = ">=3.11"

{memleaf-0.2.30 → memleaf-0.2.31}/src/memleaf/__init__.py

@@ -1,6 +1,6 @@
 """Local-first Markdown memory core for AI agents."""
 
-__version__ = "0.2.30"
+__version__ = "0.2.31"
 
 from .config import DEFAULT_CONFIG, default_config, load_config, save_config
 from .frontmatter import FrontmatterError, dump_frontmatter, dump_yaml, load_yaml, parse_frontmatter

memleaf-0.2.31/src/memleaf/evidence_budget.py

@@ -0,0 +1,294 @@
+"""Pure standard-library evidence capture budget primitives.
+
+The Core package and the copied Hermes provider use this module as the one
+shared retention boundary.  Budgets are measured in UTF-8 bytes for bodies;
+the loss marker is metadata and is deliberately outside both budgets.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Any, Mapping, Sequence
+
+
+DEFAULT_MAX_RECORDS = 64
+DEFAULT_MAX_RECORD_BYTES = 32 * 1024
+DEFAULT_MAX_TOTAL_BYTES = 128 * 1024
+
+_OVERFLOW_TOOL = "evidence.inventory"
+_OVERFLOW_CALL_IDS = frozenset({"overflow", "retention-overflow"})
+_LOSS_MARKER_CONTENT = "Additional tool observations exceeded the capture budget."
+_MARKER_FIELDS = frozenset({
+    "call_id", "execution_status", "schema_version", "retention",
+})
+
+
+
+@dataclass(frozen=True)
+class EvidenceBudget:
+    """The bounded evidence shape shared by Core and the Hermes copy."""
+
+    max_records: int = DEFAULT_MAX_RECORDS
+    max_record_bytes: int = DEFAULT_MAX_RECORD_BYTES
+    max_total_bytes: int = DEFAULT_MAX_TOTAL_BYTES
+
+    def __post_init__(self) -> None:
+        for field in ("max_records", "max_record_bytes", "max_total_bytes"):
+            value = getattr(self, field)
+            if isinstance(value, bool) or not isinstance(value, int) or value <= 0:
+                raise ValueError(f"invalid evidence budget {field}")
+
+
+DEFAULT_BUDGET = EvidenceBudget()
+
+
+def utf8_size(value: str) -> int:
+    """Return the exact UTF-8 byte size of one body string."""
+
+    if not isinstance(value, str):
+        raise TypeError("evidence body must be text")
+    return len(value.encode("utf-8"))
+
+
+def truncate_utf8(value: str, limit: int) -> tuple[str, bool]:
+    """Safely truncate text to ``limit`` UTF-8 bytes at a codepoint boundary."""
+
+    if not isinstance(value, str):
+        raise TypeError("evidence body must be text")
+    if isinstance(limit, bool) or not isinstance(limit, int) or limit < 0:
+        raise ValueError("invalid UTF-8 byte limit")
+    encoded = value.encode("utf-8")
+    if len(encoded) <= limit:
+        return value, False
+    # ``ignore`` removes at most the incomplete final codepoint created by the
+    # byte slice.  It never removes a complete codepoint before the boundary.
+    return encoded[:limit].decode("utf-8", "ignore"), True
+
+
+def is_overflow_marker(value: Mapping[str, Any]) -> bool:
+    """Recognize the canonical and legacy loss markers without body matching."""
+
+    if not isinstance(value, Mapping):
+        return False
+    if value.get("tool_name") == _OVERFLOW_TOOL:
+        call_id = value.get("call_id")
+        if call_id in _OVERFLOW_CALL_IDS:
+            return True
+    return value.get("record_id") == "overflow" and (
+        value.get("omitted_count") is not None or value.get("omitted_bytes") is not None
+    )
+
+
+def marker_count(value: Mapping[str, Any]) -> int:
+    """Return the audited number of observations represented by a marker."""
+
+    raw = value.get("omitted_count", "0")
+    if isinstance(raw, str) and raw.isascii() and raw.isdigit():
+        return int(raw)
+    if isinstance(raw, int) and not isinstance(raw, bool) and raw >= 0:
+        return raw
+    return 0
+
+
+def marker_bytes(value: Mapping[str, Any]) -> int:
+    """Return the audited number of omitted body bytes represented by a marker."""
+
+    raw = value.get("omitted_bytes", "0")
+    if isinstance(raw, str) and raw.isascii() and raw.isdigit():
+        return int(raw)
+    if isinstance(raw, int) and not isinstance(raw, bool) and raw >= 0:
+        return raw
+    return 0
+
+
+def logical_observation_count(value: Mapping[str, Any]) -> int:
+    """Count one ordinary record or the observations named by a loss marker."""
+
+    return marker_count(value) if is_overflow_marker(value) else 1
+
+
+def _body_bytes(value: Mapping[str, Any]) -> int:
+    body = value.get("content")
+    return utf8_size(body) if isinstance(body, str) else 0
+
+
+def _marker_from(
+    markers: Sequence[Mapping[str, Any]],
+    *,
+    omitted_count: int,
+    omitted_bytes: int,
+) -> dict[str, str]:
+    """Merge loss markers while preserving the first marker's audit identity."""
+
+    marker: dict[str, str] = {}
+    if markers:
+        first = markers[0]
+        for field in _MARKER_FIELDS:
+            value = first.get(field)
+            if isinstance(value, str) and value:
+                marker[field] = value[:320]
+        marker["call_id"] = str(first.get("call_id") or "overflow")[:320]
+    else:
+        marker["call_id"] = "overflow"
+    total_count = sum(marker_count(item) for item in markers) + omitted_count
+    total_bytes = sum(marker_bytes(item) for item in markers) + omitted_bytes
+    # A marker is always an untrusted inventory diagnostic, even if a caller
+    # supplied an ``overflow`` row with authority-looking fields.  Keep only a
+    # small fixed identity and force the non-authoritative state.
+    marker["tool_name"] = _OVERFLOW_TOOL
+    marker["record_id"] = "overflow"
+    marker["kind"] = "unknown"
+    marker["result_status"] = "truncated"
+    marker["completeness"] = "partial"
+    marker["omitted_count"] = str(total_count)
+    if total_bytes or any("omitted_bytes" in item for item in markers):
+        marker["omitted_bytes"] = str(total_bytes)
+    # Metadata mode deliberately removes the body.  Do not recreate one on a
+    # later normalization pass; bounded markers use one fixed diagnostic body.
+    if markers and any("content" in item for item in markers) or omitted_count or omitted_bytes:
+        marker["content"] = _LOSS_MARKER_CONTENT
+    return marker
+
+
+def apply_evidence_budget(
+    value: Sequence[Mapping[str, Any]],
+    *,
+    budget: EvidenceBudget = DEFAULT_BUDGET,
+) -> list[dict[str, str]]:
+    """Apply one idempotent record/body/aggregate evidence budget.
+
+    The input is expected to contain already validated records.  This helper
+    only owns budget accounting: ordinary metadata stays untouched, body text
+    is bounded in UTF-8 bytes, and loss markers are excluded from both record
+    and body budgets.  Passing its output through again is stable.
+    """
+
+    if not isinstance(value, (list, tuple)):
+        raise ValueError("tool evidence must be a list")
+    if not isinstance(budget, EvidenceBudget):
+        raise TypeError("budget must be an EvidenceBudget")
+
+    markers: list[Mapping[str, Any]] = []
+    records: list[dict[str, str]] = []
+    for raw in value:
+        if not isinstance(raw, Mapping):
+            continue
+        if is_overflow_marker(raw):
+            markers.append(raw)
+        else:
+            records.append(dict(raw))
+
+    omitted_count = 0
+    omitted_bytes = 0
+    if len(records) > budget.max_records:
+        for row in records[budget.max_records :]:
+            omitted_count += logical_observation_count(row)
+            omitted_bytes += _body_bytes(row)
+        records = records[: budget.max_records]
+
+    retained: list[dict[str, str]] = []
+    remaining = budget.max_total_bytes
+    for row in records:
+        body = row.get("content")
+        if not isinstance(body, str):
+            retained.append(row)
+            continue
+        # Core historically strips evidence fields before persistence.  Keep
+        # that canonical boundary in the shared helper so a provider pass and
+        # a Core pass cannot disagree at a trailing-space truncation edge.
+        body = body.strip()
+        if not body:
+            row.pop("content", None)
+            retained.append(row)
+            continue
+        original_bytes = utf8_size(body)
+        body, per_record_loss = truncate_utf8(body, budget.max_record_bytes)
+        if per_record_loss:
+            canonical_body = body.strip()
+            body = canonical_body
+            row["content"] = body
+            row["result_status"] = "truncated"
+            row["completeness"] = "partial"
+            omitted_bytes += original_bytes - utf8_size(body)
+            if not body:
+                omitted_count += logical_observation_count(row)
+                continue
+
+        body_bytes = utf8_size(body)
+        if body_bytes <= remaining:
+            row["content"] = body
+            retained.append(row)
+            remaining -= body_bytes
+            continue
+
+        if remaining > 0:
+            body, aggregate_loss = truncate_utf8(body, remaining)
+            canonical_body = body.strip()
+            body = canonical_body
+            if not body:
+                omitted_count += logical_observation_count(row)
+                omitted_bytes += body_bytes
+                continue
+            row["content"] = body
+            if aggregate_loss:
+                row["result_status"] = "truncated"
+                row["completeness"] = "partial"
+            omitted_bytes += body_bytes - utf8_size(body)
+            retained.append(row)
+            remaining = 0
+        else:
+            omitted_count += logical_observation_count(row)
+            omitted_bytes += body_bytes
+
+    if markers or omitted_count or omitted_bytes:
+        # Keep distinct host call markers distinct so a later repeated hook
+        # can be deduplicated by ``call_id``.  Markers with the same identity
+        # are merged.  Marker identities have their own fixed allowance: keep
+        # at most one row per ordinary record slot and summarize every excess
+        # marker into one global row.  This prevents an unbounded stream of
+        # distinct host call IDs from bypassing the capture budget.
+        groups: dict[tuple[Any, Any], list[Mapping[str, Any]]] = {}
+        for marker in markers:
+            key = (marker.get("call_id"), marker.get("record_id"))
+            groups.setdefault(key, []).append(marker)
+        global_groups: list[Mapping[str, Any]] = []
+        call_groups: list[list[Mapping[str, Any]]] = []
+        for key, group in groups.items():
+            if key[0] in _OVERFLOW_CALL_IDS:
+                global_groups.extend(group)
+            else:
+                call_groups.append(group)
+        retained_groups = call_groups[:budget.max_records]
+        dropped_groups = call_groups[budget.max_records:]
+        dropped_count = sum(marker_count(item) for group in dropped_groups for item in group)
+        dropped_bytes = sum(marker_bytes(item) for group in dropped_groups for item in group)
+        marker_rows = [
+            _marker_from(group, omitted_count=0, omitted_bytes=0)
+            for group in retained_groups
+        ]
+        if global_groups or dropped_groups or omitted_count or omitted_bytes:
+            marker_rows.append(
+                _marker_from(
+                    global_groups,
+                    omitted_count=dropped_count + omitted_count,
+                    omitted_bytes=dropped_bytes + omitted_bytes,
+                )
+            )
+        retained.extend(marker_rows)
+    return retained
+
+
+__all__ = [
+    "DEFAULT_BUDGET",
+    "DEFAULT_MAX_RECORD_BYTES",
+    "DEFAULT_MAX_RECORDS",
+    "DEFAULT_MAX_TOTAL_BYTES",
+    "EvidenceBudget",
+    "apply_evidence_budget",
+    "is_overflow_marker",
+    "logical_observation_count",
+    "marker_bytes",
+    "marker_count",
+    "truncate_utf8",
+    "utf8_size",
+]

{memleaf-0.2.30 → memleaf-0.2.31}/src/memleaf/hermes_provider/__init__.py

@@ -9,12 +9,14 @@ same local ``~/.memleaf`` vault as the standalone MCP server.
 from __future__ import annotations
 
 import json
+import importlib.util
 import logging
 import os
 import re
 import queue
 import shutil
 import subprocess
+import sys
 import threading
 import time
 from collections import OrderedDict, deque
@@ -24,6 +26,23 @@ from typing import Any, Deque, Dict, List, Mapping, Optional, Tuple
 
 from agent.memory_provider import MemoryProvider, RecallStatus
 
+try:
+    from .evidence_budget import apply_evidence_budget
+except (ImportError, ValueError):
+    # Hermes installs this file as a standalone plugin directory.  Load the
+    # adjacent copied module directly so the provider never imports Core (or
+    # Hermes' package initializer) just to apply the capture budget.
+    _BUDGET_SPEC = importlib.util.spec_from_file_location(
+        "_memleaf_hermes_evidence_budget",
+        Path(__file__).with_name("evidence_budget.py"),
+    )
+    if _BUDGET_SPEC is None or _BUDGET_SPEC.loader is None:
+        raise ImportError("Hermes evidence budget module is unavailable")
+    _BUDGET_MODULE = importlib.util.module_from_spec(_BUDGET_SPEC)
+    sys.modules[_BUDGET_SPEC.name] = _BUDGET_MODULE
+    _BUDGET_SPEC.loader.exec_module(_BUDGET_MODULE)
+    apply_evidence_budget = _BUDGET_MODULE.apply_evidence_budget
+
 logger = logging.getLogger(__name__)
 
 _DEFAULT_VAULT = "~/.memleaf"
@@ -1179,8 +1198,9 @@ def _bounded_current_tool_evidence(messages: Optional[List[Dict[str, Any]]], *,
     """Match current-turn results strictly by call ID, not tool name/order.
 
     Kept standard-library-only: Hermes can load this copied provider while the
-    core runs in a separate environment. Core capture redacts and validates
-    these bounded records again before persistence.
+    core runs in a separate environment. The adjacent shared budget module is
+    the only body/record boundary; Core redacts and validates these records
+    again before persistence using the same idempotent rule.
     """
     if not isinstance(messages, list):
         return []
@@ -1220,12 +1240,11 @@ def _bounded_current_tool_evidence(messages: Optional[List[Dict[str, Any]]], *,
                 return None
             if not text or "\x00" in text:
                 return None
-            partial = len(text) > 2000
             item = {"tool_name": name[:320], "call_id": cid[:320], "kind": kind,
                     "execution_status": "error" if execution_error else "success",
-                    "completeness": "partial" if partial else "complete", "schema_version": "2",
-                    "result_status": "truncated" if partial else "error" if execution_error else "success",
-                    "content": text[:2000],
+                    "completeness": "complete", "schema_version": "2",
+                    "result_status": "error" if execution_error else "success",
+                    "content": text,
                     "source_type": "document" if _has_document_arguments(call.get("arguments")) else "tool_result"}
             if isinstance(value, Mapping):
                 for key in ("record_id", "title", "message_id", "subject", "sender", "domain"):
@@ -1240,7 +1259,7 @@ def _bounded_current_tool_evidence(messages: Optional[List[Dict[str, Any]]], *,
         if original is None:
             continue
         collection, context = None, {}
-        if original["completeness"] == "partial" and not execution_error:
+        if not execution_error:
             if isinstance(payload, list):
                 collection = payload
             elif isinstance(payload, Mapping):
@@ -1249,7 +1268,7 @@ def _bounded_current_tool_evidence(messages: Optional[List[Dict[str, Any]]], *,
                     collection = payload[keys[0]]
                     context = {key: value for key, value in payload.items() if key != keys[0]}
         if collection:
-            for index, value in enumerate(collection[:8]):
+            for index, value in enumerate(collection):
                 item = record({"context": context, "record": value}, f"result-record-{index}")
                 if item is not None:
                     for field in ("message_id", "subject", "sender", "domain", "title"):
@@ -1259,20 +1278,10 @@ def _bounded_current_tool_evidence(messages: Optional[List[Dict[str, Any]]], *,
                         if isinstance(field_value, str) and field_value.strip() and not any(ch in field_value for ch in "\x00\r\n"):
                             item[field] = field_value[:320]
                     output.append(item)
-            if len(collection) > 8:
-                output.append({"tool_name": "evidence.inventory", "call_id": cid[:320],
-                    "record_id": "overflow", "kind": "unknown", "result_status": "truncated",
-                    "completeness": "partial", "execution_status": "success", "schema_version": "2",
-                    "omitted_count": str(len(collection) - 8), "content": "Additional structured observations exceeded the capture budget."})
         else:
             output.append(original)
         seen.add(cid)
-    if len(output) > 8:
-        omitted = sum(int(row.get("omitted_count", "1")) for row in output[7:])
-        output = output[:7] + [{"tool_name": "evidence.inventory", "call_id": "overflow",
-            "kind": "unknown", "result_status": "truncated", "completeness": "partial",
-            "omitted_count": str(omitted), "content": "Additional host observations exceeded the capture budget."}]
-    return output
+    return apply_evidence_budget(output)
 
 
 class MemleafMemoryProvider(MemoryProvider):