foldkeep 0.9.2__tar.gz

foldkeep-0.9.2/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 foldkeep contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

foldkeep-0.9.2/PKG-INFO

@@ -0,0 +1,117 @@
+Metadata-Version: 2.4
+Name: foldkeep
+Version: 0.9.2
+Summary: Zero-dependency lossless context folding for agent sessions: pinned facts, tolerance routing, byte-for-byte expand.
+Author: foldkeep contributors
+License: MIT License
+        
+        Copyright (c) 2026 foldkeep contributors
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+Keywords: llm,agent,context-window,memory,compression
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# foldkeep
+
+**Fold the context. Keep the facts.**
+
+foldkeep is a zero-dependency, deterministic context-compression engine for
+long agent conversations. It rolls old turns into compressed form while a
+pinned fact layer keeps every decision reachable — and a lossless base lets
+you expand ANY turn back byte-for-byte, even after rolling and merging.
+
+Pure Python stdlib. No LLM calls. Millisecond latency. SQLite-backed.
+
+## Why
+
+Agent transcripts grow until the model drowns. Naive fixes lose the past:
+tail-truncation drops early decisions; naive compression mangles numbers,
+code and tool payloads. foldkeep keeps three guarantees:
+
+1. **Pinned facts survive** — every turn pins a salient evidence line;
+   a set-cover knapsack picks which lines fill the leftover budget so
+   entity coverage is maximized, not clustered.
+2. **Zero-tolerance content is never mangled** — code fences, JSON payloads
+   and tool-call shapes are routed away from lossy compression.
+3. **Nothing is ever lost** — originals persist on disk; `expand(name, seq)`
+   recovers any turn byte-for-byte after rollup, consolidation or render
+   drops.
+
+## Quickstart
+
+```python
+from foldkeep import session as sess
+
+sess.new("my-session")
+for role, text in conversation:          # push turns as they happen
+    sess.push("my-session", role, text)  # auto-rolls when long
+
+out, used, dropped = sess.render("my-session", budget=800)
+# -> compressed body + key-facts registry + pinned-fact block, <= 800 tok
+
+orig = sess.expand("my-session", 3)      # byte-for-byte original of turn 3
+hits = sess.search("my-session", "TKT-9001")   # recall across rolled turns
+md = sess.export_md("my-session", "vault.md")  # inspectable Markdown vault
+```
+
+CLI:
+
+```bash
+python -m foldkeep selftest        # 16-case correctness battery
+```
+
+## Proven, not promised
+
+| Suite | Result |
+|---|---|
+| selftest (correctness) | 16/16 |
+| stress_audit (adversarial probes) | 22/22 |
+| marathon (long-session checkpoints) | 9/9, push ≤ 12 ms, render ≤ 80 ms |
+| agent_selftest (real build-session replay, equal budgets) | wins 5/6 cells vs tail-only & head-trunc |
+| chat_selftest (this project's own chat replay) | 12/12, recall 9/9 under forced rolling |
+
+Benchmarks live in this repo (`agent_selftest.py`, `chat_selftest.py`,
+`stress_audit.py`, `long_session_test.py`) — every number reproducible.
+
+## Design
+
+- **Rolling compression**: newest stays verbatim, oldest compresses into a
+  head; hierarchical re-merge keeps the head shrinking.
+- **Three-layer render**: compressed body + one-line entity registry +
+  set-cover pinned-fact block, all inside your token budget.
+- **Loss-tolerance routing**: fences / JSON / tool-call shapes bypass
+  lossy paths entirely.
+- **Lossless base**: `orig_text` column + `originals` table + `expand()`.
+- **Operator knobs**: `OC_REGISTRY_MAX` env caps registry injection.
+
+## Status
+
+v0.9.2. Research notes (`RESEARCH.md`) document the competitive landscape
+(LLMLingua, leanctx, mnesis, FoldAgent, pi-fold, ECC, hermes-agent) and what
+we learned from each.
+
+MIT licensed. Windows/macOS/Linux, Python 3.10+.

foldkeep-0.9.2/README.md

@@ -0,0 +1,80 @@
+# foldkeep
+
+**Fold the context. Keep the facts.**
+
+foldkeep is a zero-dependency, deterministic context-compression engine for
+long agent conversations. It rolls old turns into compressed form while a
+pinned fact layer keeps every decision reachable — and a lossless base lets
+you expand ANY turn back byte-for-byte, even after rolling and merging.
+
+Pure Python stdlib. No LLM calls. Millisecond latency. SQLite-backed.
+
+## Why
+
+Agent transcripts grow until the model drowns. Naive fixes lose the past:
+tail-truncation drops early decisions; naive compression mangles numbers,
+code and tool payloads. foldkeep keeps three guarantees:
+
+1. **Pinned facts survive** — every turn pins a salient evidence line;
+   a set-cover knapsack picks which lines fill the leftover budget so
+   entity coverage is maximized, not clustered.
+2. **Zero-tolerance content is never mangled** — code fences, JSON payloads
+   and tool-call shapes are routed away from lossy compression.
+3. **Nothing is ever lost** — originals persist on disk; `expand(name, seq)`
+   recovers any turn byte-for-byte after rollup, consolidation or render
+   drops.
+
+## Quickstart
+
+```python
+from foldkeep import session as sess
+
+sess.new("my-session")
+for role, text in conversation:          # push turns as they happen
+    sess.push("my-session", role, text)  # auto-rolls when long
+
+out, used, dropped = sess.render("my-session", budget=800)
+# -> compressed body + key-facts registry + pinned-fact block, <= 800 tok
+
+orig = sess.expand("my-session", 3)      # byte-for-byte original of turn 3
+hits = sess.search("my-session", "TKT-9001")   # recall across rolled turns
+md = sess.export_md("my-session", "vault.md")  # inspectable Markdown vault
+```
+
+CLI:
+
+```bash
+python -m foldkeep selftest        # 16-case correctness battery
+```
+
+## Proven, not promised
+
+| Suite | Result |
+|---|---|
+| selftest (correctness) | 16/16 |
+| stress_audit (adversarial probes) | 22/22 |
+| marathon (long-session checkpoints) | 9/9, push ≤ 12 ms, render ≤ 80 ms |
+| agent_selftest (real build-session replay, equal budgets) | wins 5/6 cells vs tail-only & head-trunc |
+| chat_selftest (this project's own chat replay) | 12/12, recall 9/9 under forced rolling |
+
+Benchmarks live in this repo (`agent_selftest.py`, `chat_selftest.py`,
+`stress_audit.py`, `long_session_test.py`) — every number reproducible.
+
+## Design
+
+- **Rolling compression**: newest stays verbatim, oldest compresses into a
+  head; hierarchical re-merge keeps the head shrinking.
+- **Three-layer render**: compressed body + one-line entity registry +
+  set-cover pinned-fact block, all inside your token budget.
+- **Loss-tolerance routing**: fences / JSON / tool-call shapes bypass
+  lossy paths entirely.
+- **Lossless base**: `orig_text` column + `originals` table + `expand()`.
+- **Operator knobs**: `OC_REGISTRY_MAX` env caps registry injection.
+
+## Status
+
+v0.9.2. Research notes (`RESEARCH.md`) document the competitive landscape
+(LLMLingua, leanctx, mnesis, FoldAgent, pi-fold, ECC, hermes-agent) and what
+we learned from each.
+
+MIT licensed. Windows/macOS/Linux, Python 3.10+.

foldkeep-0.9.2/foldkeep/__init__.py

@@ -0,0 +1,16 @@
+"""FoldKeep -- Agent Context Infrastructure.
+
+Layers:
+  compressor : FastLingua engine (entity-preserving heuristic compression)
+  estimate   : token & cost estimation across popular models
+  store      : named compressed-context snapshots for agent workflows
+
+Layers: core.compressor | estimate | store(SQLite memory bank) | packing | mcp_server.
+"""
+__version__ = "0.6.0"
+
+from .core.compressor import FastLingua  # noqa: F401
+from .benchmark.suite import run_bench, evolve  # noqa: F401
+from .estimate import estimate_tokens, estimate_cost, savings  # noqa: F401
+from .memory import store  # noqa: F401
+from .packing import pack, render  # noqa: F401

foldkeep-0.9.2/foldkeep/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+main()

foldkeep-0.9.2/foldkeep/cli.py

@@ -0,0 +1,246 @@
+"""FoldKeep unified CLI: oc compress / estimate / store / bench / evolve."""
+import argparse
+import json
+import os
+import sys
+
+from . import __version__
+from .core.compressor import FastLingua, GENES
+from .benchmark.suite import run_bench, evolve, EVOLVE_FILE
+from .memory import store
+from .estimate import estimate_cost, savings
+from .packing import pack, render
+from .diff import commit as ctx_commit, log as ctx_log, show as ctx_show
+from . import session as sess
+
+def _read(src):
+    if src == "-":
+        raw = sys.stdin.buffer.read()
+        try:
+            return raw.decode("utf-8")
+        except UnicodeDecodeError:
+            return raw.decode("gbk", "replace")
+    with open(src, encoding="utf-8-sig") as f:
+        return f.read().lstrip("\ufeff")
+
+def _adapt(fl, stats, target):
+    try:
+        cur = {}
+        if os.path.exists(EVOLVE_FILE):
+            with open(EVOLVE_FILE, encoding="utf-8") as f:
+                cur = json.load(f)
+        err = stats["ratio"] - target
+        if abs(err) > 0.05:
+            lo, hi = GENES["dup_th"]
+            d = float(cur.get("dup_th", 0.72)) - (0.01 if err > 0 else -0.008)
+            cur["dup_th"] = round(min(hi, max(lo, d)), 4)
+            lo, hi = GENES["ent_w"]
+            d = float(cur.get("ent_w", 3.0)) - (0.08 if err > 0 else -0.10)
+            cur["ent_w"] = round(min(hi, max(lo, d)), 4)
+            lo, hi = GENES["cont_w"]
+            d = float(cur.get("cont_w", 1.0)) - (0.05 if err > 0 else -0.04)
+            cur["cont_w"] = round(min(hi, max(lo, d)), 4)
+            with open(EVOLVE_FILE, "w", encoding="utf-8") as f:
+                json.dump(cur, f, indent=1)
+    except Exception:
+        pass
+
+def main(argv=None):
+    ap = argparse.ArgumentParser(
+        prog="foldkeep",
+        description="FoldKeep: Agent Context Infrastructure (compress | estimate | store | bench | evolve)")
+    ap.add_argument("--version", action="version", version="foldkeep " + __version__)
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    c = sub.add_parser("compress", help="compress text via FastLingua engine")
+    c.add_argument("input", help="file path or '-' for stdin")
+    c.add_argument("--ratio", type=float, default=0.5)
+    c.add_argument("--mode", choices=FastLingua.MODES, default="balanced")
+    c.add_argument("--anchors", action="store_true")
+    c.add_argument("--collapse", action="store_true")
+    c.add_argument("--adapt", action="store_true")
+    c.add_argument("--no-auto", action="store_true")
+    c.add_argument("--save", metavar="NAME", help="also store snapshot for later reuse")
+    c.add_argument("-o", "--output")
+
+    e = sub.add_parser("estimate", help="token & cost estimate")
+    e.add_argument("input")
+    e.add_argument("--model", default="claude-sonnet")
+    e.add_argument("--compress", action="store_true", help="also show post-compression savings")
+
+    st = sub.add_parser("store", help="snapshot management")
+    st.add_argument("op", choices=["list", "get", "delete"])
+    st.add_argument("name", nargs="?")
+
+    b = sub.add_parser("bench", help="run benchmark")
+    b.add_argument("--full", action="store_true", help="1000-sample synthetic suite")
+
+    pk = sub.add_parser("pack", help="fit multiple files into a token budget")
+    pk.add_argument("inputs", nargs="+")
+    pk.add_argument("--budget", type=int, default=8000)
+    pk.add_argument("--ratio", type=float, default=0.5)
+    pk.add_argument("--model", default=None, help="also show cost for this model")
+    pk.add_argument("--save", metavar="NAME")
+    pk.add_argument("-o", "--output")
+    sub.add_parser("selftest", help="run the 16-case edge battery")
+
+    ss = sub.add_parser("session", help="long-conversation rolling compression")
+    ss.add_argument("op", choices=["new", "push", "show", "stat"])
+    ss.add_argument("name")
+    ss.add_argument("-r", "--role", default="user", choices=["user", "assistant", "system"])
+    ss.add_argument("-t", "--text")
+    ss.add_argument("-f", "--file")
+    ss.add_argument("--budget", type=int, default=None)
+    ss.add_argument("--roll-threshold", type=int, default=None,
+                    help="override auto-rollup token threshold")
+    cx = sub.add_parser("ctx", help="context version chain: commit/log/show/restore")
+    cx.add_argument("op", choices=["commit", "log", "show", "restore"])
+    cx.add_argument("target", nargs="?", help="file path (commit) or version id")
+    cx.add_argument("-p", "--project", default="default")
+    cx.add_argument("-l", "--label", default="")
+    cx.add_argument("-o", "--output")
+    ev = sub.add_parser("evolve", help="self-evolve compressor genes")
+    ev.add_argument("--gens", type=int, default=8)
+    ev.add_argument("--pop", type=int, default=8)
+    ev.add_argument("--seed", type=int, default=42)
+
+    a = ap.parse_args(argv)
+
+    if a.cmd == "compress":
+        text = _read(a.input)
+        fl = FastLingua(a.mode, a.ratio, a.anchors,
+                        auto=not a.no_auto, collapse=a.collapse)
+        comp, stats = fl.compress(text)
+        if a.adapt:
+            _adapt(fl, stats, a.ratio)
+        out = comp  # engine already renders [F#] anchors when enabled
+        if a.output:
+            with open(a.output, "w", encoding="utf-8") as f:
+                f.write(out)
+        else:
+            print(out)
+        if a.save:
+            p = store.save(a.save, text, comp,
+                           {"ratio": stats["ratio"], "mode": a.mode})
+            print("[stored] %s" % p, file=sys.stderr)
+
+    elif a.cmd == "estimate":
+        text = _read(a.input)
+        info = estimate_cost(text, a.model)
+        print("input : %6d tokens | $%.6f  (%s)" % (info["tokens"], info["usd"], a.model))
+        if a.compress:
+            comp, _st = FastLingua("balanced", 0.5).compress(text)
+            after = estimate_cost(comp, a.model)
+            sv = savings(text, comp, a.model)
+            print("after : %6d tokens | $%.6f" % (after["tokens"], after["usd"]))
+            print("saved : %6d tokens | $%.6f  | compressed to %.1f%%"
+                  % (sv["tokens_saved"], sv["usd_saved"], sv["ratio"] * 100))
+
+    elif a.cmd == "store":
+        if a.op == "list":
+            rows = store.list_all()
+            if not rows:
+                print("(empty)")
+            for r in rows:
+                print("%-24s %7d -> %7d chars" % (r["name"], r["orig"], r["comp"]))
+        elif a.op == "get":
+            r = store.get(a.name or "")
+            print(r["text"] if r else "(not found)")
+        else:
+            print("deleted" if store.delete(a.name or "") else "(not found)")
+
+    elif a.cmd == "bench":
+        run_bench(getattr(a, "full", False))
+
+    elif a.cmd == "pack":
+        items = []
+        for src in a.inputs:
+            try:
+                items.append((os.path.basename(src), _read(src)))
+            except Exception as ex:
+                print("[skip] %s: %s" % (src, ex), file=sys.stderr)
+        if not items:
+            print("(no readable inputs)", file=sys.stderr)
+            return
+        res = pack(items, a.budget, a.ratio)
+        out = render(res)
+        if a.model:
+            from .estimate import estimate_cost
+            info = estimate_cost(out, a.model)
+            out += "\n=== cost: %d tokens | $%.6f (%s) ===" % (
+                info["tokens"], info["usd"], a.model)
+        if a.output:
+            with open(a.output, "w", encoding="utf-8") as f:
+                f.write(out)
+        else:
+            print(out)
+        if a.save:
+            pth = store.save(a.save, "".join(t for _, t in items), out,
+                             {"budget": a.budget, "used": res["used"]})
+            print("[stored] %s" % pth, file=sys.stderr)
+
+    elif a.cmd == "selftest":
+        from .selftest import run
+        R, fails = run()
+        print("---")
+        print("TOTAL %d | PASS %d | FAIL %d" % (len(R), len(R) - len(fails), len(fails)))
+        if fails:
+            raise SystemExit(1)
+
+    elif a.cmd == "session":
+        if a.op == "new":
+            sess.new(a.name)
+            print("session '%s' ready" % a.name)
+        elif a.op == "push":
+            text = a.text if a.text is not None else (_read(a.file) if a.file else None)
+            if not text:
+                print("need -t TEXT or -f FILE", file=sys.stderr)
+                return
+            kw = {} if a.roll_threshold is None else {"roll_threshold": a.roll_threshold}
+            r = sess.push(a.name, a.role, text, **kw)
+            st = sess.stat(a.name)
+            print("pushed #%d (%s) | rolled %d | session: %d turns, ~%d tok"
+                  % (r["seq"], a.role, r["rolled"], st["turns"], st["est_tokens"]))
+        elif a.op == "show":
+            out, used, dropped = sess.render(a.name, a.budget)
+            print(out)
+        else:
+            print(sess.stat(a.name))
+
+    elif a.cmd == "ctx":
+        if a.op == "commit":
+            if not a.target:
+                print("need file path", file=sys.stderr)
+                return
+            text = _read(a.target)
+            r = ctx_commit(a.project, text, a.label)
+            print("committed v%d [%s] %d -> %d chars (%.1f%%), stored %s"
+                  % (r["id"], a.project, r["orig"], r["comp"],
+                     r["ratio"] * 100, r["stored"]))
+        elif a.op == "log":
+            rows = ctx_log(a.project)
+            if not rows:
+                print("(empty)")
+            for r in rows:
+                print("v%-4d %-16s parent=%-4s %6d->%6d %s"
+                      % (r["id"], r["label"] or "-", r["parent"] or "-",
+                         r["orig"], r["comp"], "FULL" if r["full"] else "delta"))
+        else:
+            if not a.target or not a.target.isdigit():
+                print("need version id", file=sys.stderr)
+                return
+            text = ctx_show(int(a.target))
+            if text is None:
+                print("(version not found)", file=sys.stderr)
+                return
+            if a.output:
+                with open(a.output, "w", encoding="utf-8") as f:
+                    f.write(text)
+            else:
+                print(text)
+
+    elif a.cmd == "evolve":
+        evolve(a.gens, a.pop, a.seed)
+
+if __name__ == "__main__":
+    main()