bigbrain-memory 1.1.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 BigBrain Contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,420 @@
1
+ Metadata-Version: 2.4
2
+ Name: bigbrain-memory
3
+ Version: 1.1.1
4
+ Summary: Deterministic, filesystem-first memory and context-management engine for LLM coding agents.
5
+ Author: BigBrain Contributors
6
+ License: MIT
7
+ Requires-Python: >=3.9
8
+ Description-Content-Type: text/markdown
9
+ License-File: LICENSE
10
+ Requires-Dist: PyYAML>=6.0
11
+ Dynamic: license-file
12
+
13
+ # BigBrain
14
+
15
+ Deterministic, filesystem-first memory and context-management engine for LLM
16
+ coding agents.
17
+
18
+ Version 1.1 adds a local, rebuildable **source index** alongside the existing
19
+ Markdown memory. Agents can browse, search and read current source through
20
+ either the CLI or MCP. Existing memory commands and formats are unchanged.
21
+
22
+ Version 1.1.1 adds live CLI progress and indexed chunk cleanup for large
23
+ repositories. Existing 1.1 caches upgrade their derived metadata automatically;
24
+ no memory migration or manual cache deletion is required.
25
+
26
+ ## New installation
27
+
28
+ Requirements: Python 3.9+, Git, and Python's SQLite with FTS5 support (included
29
+ in the standard Windows/macOS Python installers and common Linux packages).
30
+
31
+ ```bash
32
+ python -m pip install "git+https://github.com/corbybender/bigbrain-memory.git"
33
+ cd /path/to/your/project
34
+ bigbrain install
35
+ bigbrain source-status
36
+ ```
37
+
38
+ Run these commands in the project you want BigBrain to understand. Memory
39
+ works without Git; source indexing requires a Git working tree. The first
40
+ source command creates the index automatically, and subsequent source commands
41
+ refresh it automatically. No enable flag, API key, scheduled task or watcher
42
+ setup is required. The status report should show `complete: true` and an
43
+ appropriate `indexed_files` count for the eligible files in your project.
44
+
45
+ ## Upgrade an existing project to 1.1.1
46
+
47
+ ```bash
48
+ python -m pip install --upgrade "git+https://github.com/corbybender/bigbrain-memory.git"
49
+ cd /path/to/your/project
50
+ bigbrain install --force
51
+ bigbrain source-status
52
+ ```
53
+
54
+ Use the Python environment that supplies your `bigbrain` command. Repeat
55
+ `bigbrain install --force` in each existing project to refresh its agent
56
+ instructions, MCP registration and tool schema. Existing memory and
57
+ configuration settings are preserved; no memory migration or manual source
58
+ index creation is needed. Restart/reconnect the MCP host afterward so it
59
+ discovers the five new source tools. If you move or clone a project, rerun
60
+ `bigbrain install --force` there to update its absolute MCP project root.
61
+
62
+ See [setup and troubleshooting](docs/setup.md) for Windows interpreter checks,
63
+ verification, automatic update behavior and recovery.
64
+
65
+ ## Source context
66
+
67
+ ```bash
68
+ bigbrain source-status
69
+ bigbrain source-files --prefix bigbrain/ --limit 20
70
+ bigbrain source-search "source cache recovery" --top 5
71
+ bigbrain source-read bigbrain/source.py --start-line 1 --max-lines 60
72
+ bigbrain source-refresh --rebuild
73
+ ```
74
+
75
+ Use `--path <project-root>` before or after the command to target another
76
+ project. Git must be installed and the project must be inside a Git working
77
+ tree; an explicitly selected subdirectory indexes only that subtree. No Git
78
+ commit, model, API key, vector service or background daemon is needed.
79
+
80
+ Source tools discover tracked and untracked files, honoring Git ignore rules
81
+ (including rules matching tracked files). They reconcile the working tree
82
+ before every query, so edits, renames, deletions, ignore changes and branch
83
+ switches are reflected without manual refresh. Eligible files are hashed on
84
+ every scan; unchanged files reuse their indexed chunks. This detects edits
85
+ even when file size and modification time are unchanged.
86
+
87
+ CLI source commands print discovery, scanning, indexing and commit progress
88
+ to stderr; stdout remains JSON for scripts. Large first builds can take time.
89
+ Ctrl+C cancels the operation and rolls back uncommitted source-cache changes
90
+ without changing memory. MCP calls do not print CLI progress into their
91
+ protocol stream; use the CLI for the initial build of a large repository.
92
+
93
+ The source index is disposable SQLite/FTS5 data under `.bigbrain/cache/`,
94
+ automatically excluded from Git by a cache-local ignore file. Markdown memory
95
+ remains authoritative for decisions and intent; source results provide current
96
+ implementation evidence. Source operations never rewrite memory. Rebuild a
97
+ damaged or incompatible cache with `source-refresh --rebuild`.
98
+
99
+ Results include a scope/freshness report, paths, SHA-256 hashes, and bounded
100
+ excerpts or line ranges. Search returns the best chunk per distinct file, with
101
+ deterministic lexical BM25 ranking and camelCase/snake_case token splitting.
102
+ Reads are limited to 200 lines and 16,000 characters; search excerpts to 1,600
103
+ characters per file and 50 files per request. Inventory is paginated up to 200
104
+ files per request. A hash identifies the bytes read during reconciliation;
105
+ files can change after that scan. Inventory pages can change between calls.
106
+
107
+ By default, indexing skips non-UTF-8/binary files, files larger than 512 KiB,
108
+ symlinks and Windows reparse points, dependency/cache directories, common
109
+ credential filenames/extensions, and `dist`, `build`, `generated`, `coverage`
110
+ and `vendor`. These filename exclusions are **not a secret scanner**; add
111
+ project-specific sensitive paths to Git ignores or source exclusions. Text
112
+ documentation and configuration are eligible alongside code.
113
+
114
+ Optional `.bigbrain/config.yml` settings:
115
+
116
+ ```yaml
117
+ source:
118
+ max_file_bytes: 524288 # 1 through 10485760
119
+ exclude: # replaces the default generated-output exclusions
120
+ - dist
121
+ - build
122
+ - generated
123
+ - coverage
124
+ - vendor
125
+ - private-settings.json
126
+ - docs/internal/*
127
+ ```
128
+
129
+ Patterns match a full project-relative path or any path component, using
130
+ case-sensitive shell wildcards (`*` can span `/` in full-path matching).
131
+ Git ignores, credential and dependency/cache exclusions, and link restrictions
132
+ still apply when `exclude: []` is configured. Files skipped by policy are
133
+ counted by reason. Read failures are counted (with up to 50 details) and mark coverage
134
+ `complete: false`; CLI commands exit 1 for incomplete coverage. MCP clients
135
+ must inspect this report before treating an empty result as evidence of absence.
136
+
137
+ The MCP tools are `source_status`, `list_source_files`,
138
+ `refresh_source_index`, `search_source`, and `read_source`. Use memory tools
139
+ to retrieve rationale, then source tools to check its current implementation.
140
+ Run `bigbrain install --force` after upgrading to refresh tool schemas and
141
+ agent instructions; existing configuration settings are preserved. Restart/reconnect the MCP host
142
+ to discover new tools. New MCP registrations use an absolute project root,
143
+ independent of the host's working directory.
144
+
145
+ This release uses query-time reconciliation rather than a watcher. It hashes
146
+ all eligible files per request, so latency grows with repository size. Source
147
+ indexing requires SQLite FTS5 and Git. Symbol resolution, call graphs,
148
+ cross-language dependency analysis, embeddings and automatic memory/source
149
+ contradiction resolution are not implemented.
150
+
151
+ ### Source validation
152
+
153
+ ```bash
154
+ python tests/test_core.py
155
+ python -m unittest discover -s tests -p test_source.py -v
156
+ python benchmarks/source_retrieval.py
157
+ ```
158
+
159
+ CI runs these checks on Windows and Linux with Python 3.9 and 3.13. The benchmark
160
+ uses six fixed keyword questions over an isolated copy of BigBrain's package
161
+ source and reports recall@5, indexing/query times, response sizes and cache size.
162
+ It is a reproducible smoke benchmark, not proof of improved agent task success
163
+ or performance on large repositories.
164
+
165
+ ## TL;DR
166
+
167
+ - **What it is:** persistent project memory as plain Markdown files under
168
+ `.bigbrain/` — no vector DB, no embeddings, human-readable, git-friendly.
169
+ - **Why:** so an AI coding agent can carry decisions, architecture notes, and
170
+ task state across sessions instead of re-deriving them (or forgetting them)
171
+ every time.
172
+ - **Works with:** any AI coding tool — Claude Code, Cursor, Windsurf, Gemini
173
+ CLI, Copilot, Aider, whatever — via plain shell commands, with a bonus MCP
174
+ server for hosts that support native tool-calling.
175
+
176
+ ### Setup
177
+
178
+ ```bash
179
+ python -m pip install git+https://github.com/corbybender/bigbrain-memory.git
180
+ cd /path/to/your/project
181
+ bigbrain install
182
+ ```
183
+
184
+ That's it. `bigbrain install` scaffolds `.bigbrain/`, wires instructions into
185
+ whatever AI-instructions files your tools use (`AGENTS.md`, `CLAUDE.md`,
186
+ `.cursor/rules/`, etc.), registers an MCP server for hosts that support it,
187
+ and sets up auto-compile hooks (git pre-commit + Claude Code session hooks)
188
+ so nobody has to remember to run anything. Re-running it later is safe —
189
+ it's idempotent.
190
+
191
+ Verify it worked:
192
+
193
+ ```bash
194
+ bigbrain status # should report initialized: true
195
+ bigbrain index # should print a Markdown table of memory files
196
+ ```
197
+
198
+ ---
199
+
200
+ ## Detailed guide
201
+
202
+ See [`plan.md`](plan.md) for the original design spec this implements.
203
+
204
+ ### Core idea
205
+
206
+ Information lives in standard Markdown files with YAML frontmatter. An agent
207
+ reads a lightweight `INDEX.md` catalog at session start instead of ingesting
208
+ the whole memory directory, and pulls specific files on demand. New facts get
209
+ appended to an append-only transaction log (`log.md`) during a session, then
210
+ reconciled into the real memory files by a compile step — either a free
211
+ deterministic merge, or an optional LLM-assisted pass that actually dedupes
212
+ and resolves contradictions.
213
+
214
+ ### Install into a repo
215
+
216
+ ```bash
217
+ python -m pip install git+https://github.com/corbybender/bigbrain-memory.git
218
+ cd /path/to/your/project
219
+ bigbrain install
220
+ ```
221
+
222
+ `bigbrain install`:
223
+
224
+ 1. Scaffolds `.bigbrain/` (index, log, default memory files) at the repo root.
225
+ 2. Writes `.bigbrain/tools.json` — the raw tool schema, for hosts that do
226
+ custom function-calling.
227
+ 3. Appends a marked BigBrain instructions block to every well-known
228
+ agent-instructions file it finds or creates: `AGENTS.md`, `CLAUDE.md`,
229
+ `GEMINI.md`, `.windsurfrules`, `.github/copilot-instructions.md`, and a
230
+ Cursor rule at `.cursor/rules/bigbrain.mdc`. That block tells the agent
231
+ how to drive BigBrain through the `bigbrain` shell command — which needs
232
+ no protocol support at all, so it works with literally any agent that can
233
+ run a shell command.
234
+ 4. Registers a BigBrain MCP server in `.mcp.json` and `.cursor/mcp.json` for
235
+ hosts with native MCP tool-calling (Claude Code, Cursor, ...), so those
236
+ hosts get real tool calls instead of shelling out.
237
+ 5. Wires **auto-compile**, so nobody has to remember to run it:
238
+ - a `.git/hooks/pre-commit` hook that compiles before every commit (any
239
+ host, no-op in ~10ms if there's nothing pending), and
240
+ - for Claude Code specifically, `SessionStart` / `SessionEnd` /
241
+ `PreCompact` hooks in `.claude/settings.json` — `SessionStart` also
242
+ auto-injects `INDEX.md` into context, so the agent never has to
243
+ remember to read it either.
244
+ 6. Drops `.bigbrain/config.yml` with commented defaults for the LLM CLI
245
+ override, archive rotation size and optional source settings. Existing
246
+ configuration is preserved, including with `--force`.
247
+
248
+ Source indexing initializes lazily on the first source command or MCP source
249
+ tool call; `install` does not crawl the repository. Every source query then
250
+ reconciles changes automatically. Memory compile hooks only compile memory;
251
+ they do not run source indexing in the background.
252
+
253
+ Re-running `bigbrain install` is idempotent (existing hooks/files aren't
254
+ duplicated). Pass `--force` to refresh the wiring blocks after upgrading
255
+ BigBrain. If a target file already has other content (an existing pre-commit
256
+ hook, an existing `AGENTS.md`), BigBrain appends a marked block rather than
257
+ overwriting it.
258
+
259
+ ### Directory layout
260
+
261
+ ```text
262
+ .bigbrain/
263
+ ├── INDEX.md # compiled catalog, ingested at session start
264
+ ├── log.md # append-only transaction log (pending entries)
265
+ ├── log.archive.md # compiled/processed log entries (rotates when it gets big)
266
+ ├── log.archive.<ts>.md # rotated-out archive segments, once log.archive.md exceeds archive_max_bytes
267
+ ├── config.yml # llm_command override, archive_max_bytes
268
+ ├── tools.json # raw tool schemas for function-calling hosts
269
+ ├── cache/ # disposable, Git-ignored; created by first source query
270
+ │ ├── .gitignore # excludes this cache directory's contents
271
+ │ └── source.sqlite3 # source inventory and FTS5 search chunks
272
+ └── memory/
273
+ ├── system_invariants.md
274
+ ├── architecture.md
275
+ ├── active_tasks.md
276
+ ├── decisions.md
277
+ └── domain_knowledge.md
278
+ └── ... # compile can create new memory files on demand
279
+ ```
280
+
281
+ ### CLI
282
+
283
+ ```bash
284
+ bigbrain init # scaffold .bigbrain/ only
285
+ bigbrain install # scaffold + wire into AI tooling (recommended)
286
+ bigbrain index # print INDEX.md
287
+ bigbrain read <file_id> # print one memory file
288
+ bigbrain grep <term> # exact-substring fallback search when you don't know the file_id
289
+ bigbrain search <query> # ranked fuzzy/word-overlap search when you don't know the exact wording either
290
+ bigbrain source-status # reconcile source and report scope/freshness/errors
291
+ bigbrain source-files # browse eligible source inventory (paginated)
292
+ bigbrain source-search "query" # reconcile and search current source/docs
293
+ bigbrain source-read <path> # reconcile and read bounded source lines
294
+ bigbrain source-refresh --rebuild # recover/rebuild disposable source cache only
295
+ bigbrain doctor [--fix] # validate .bigbrain/ (stale locks, bad frontmatter, stale index); repair with --fix
296
+ bigbrain log --type DECISION --target decisions \
297
+ --summary "Chose SQLite for log indexing" \
298
+ --details "ACID writes matter under concurrent CLI runs."
299
+ bigbrain compile --reason "session end" # reconcile log.md -> memory/*.md, rebuild INDEX.md
300
+ bigbrain compile --reason "session end" --llm # same, but LLM-assisted dedup/contradiction-resolution
301
+ bigbrain status # JSON snapshot of engine state
302
+ bigbrain serve-mcp # run the MCP stdio server
303
+ ```
304
+
305
+ `--type` must be one of `DECISION`, `TASK_UPDATE`, `LEARNING`, `INVARIANT`.
306
+
307
+ ### Deterministic vs. `--llm` compile
308
+
309
+ Plain `bigbrain compile` never calls a model: pending log entries are grouped
310
+ by target file and appended under a dated "Log Reconciliation" section. It's
311
+ free, instant, and can't fail in a surprising way — but memory files only
312
+ ever grow; nothing gets deduped or rewritten to resolve contradictions.
313
+
314
+ `bigbrain compile --llm` pipes each target file + its pending entries to a
315
+ configured LLM CLI and asks it to properly rewrite the file: merge related
316
+ facts, drop obsolete statements, and resolve contradictions in favor of the
317
+ newer entry — matching the reconciliation behavior in the original design
318
+ (`plan.md` §4, Phase 3). It needs a local LLM CLI:
319
+
320
+ - auto-detected from PATH (tries `claude -p`, then `llm`), or
321
+ - pinned explicitly in `.bigbrain/config.yml`:
322
+ ```yaml
323
+ llm_command: "claude -p"
324
+ ```
325
+
326
+ `--llm` never risks your data: if the CLI is missing, times out, errors, or
327
+ returns output that doesn't parse as a valid memory file (frontmatter with an
328
+ `id` + non-empty body), that *one file* silently falls back to the
329
+ deterministic merge and a warning is printed — compile always succeeds.
330
+
331
+ ### Archive rotation
332
+
333
+ `log.archive.md` accumulates every compiled log entry. Once it exceeds
334
+ `archive_max_bytes` (default 2MB, configurable in `.bigbrain/config.yml`), the
335
+ next compile rotates it to `log.archive.<timestamp>.md` and starts fresh, so
336
+ it doesn't grow without bound.
337
+
338
+ ### `grep` vs `search`
339
+
340
+ - `bigbrain grep <term>` — exact, case-insensitive substring match. Use when
341
+ you know the precise word/name/quote.
342
+ - `bigbrain search <query>` — deterministic word-overlap ranking across every
343
+ memory file's body/title/tags, with a phrase-match bonus. Use when you know
344
+ roughly what you're looking for but not the exact wording or which file it's
345
+ in. This is **not** embeddings/vector search — BigBrain's core principle is
346
+ no opaque embeddings — just explainable keyword scoring, good enough to
347
+ replace "guess the file_id" at the scale a flat-file memory store is meant
348
+ for. If your project outgrows this, that's a signal to split memory more,
349
+ not to bolt on a vector DB.
350
+
351
+ ### Concurrency
352
+
353
+ Every write path (`log`, `compile`) takes the same advisory lock
354
+ (`.bigbrain/.lock`), so two agent sessions (or an agent and the pre-commit
355
+ hook) writing at the same time serialize instead of interleaving/corrupting
356
+ files. A lock older than 120s is treated as abandoned and reclaimed
357
+ automatically; `bigbrain status` shows the current lock's pid and age if one
358
+ is held, and `bigbrain doctor --fix` will clear a stuck one manually.
359
+
360
+ ### `doctor` (validation, repair, and schema migration)
361
+
362
+ `bigbrain doctor` checks for: a stale lock file, memory files missing
363
+ required frontmatter (`id`, `domain`, `last_updated`), duplicate `id`s across
364
+ files, a `log.md` containing content that doesn't match the expected entry
365
+ format (e.g. from hand-editing), an invalid `config.yml`, and a stale
366
+ `INDEX.md`. Run it plain to see what's wrong (exits 1 if anything is); run
367
+ `--fix` to repair everything that's safely auto-fixable (exits 0 once clean —
368
+ a fixed issue is never also reported as still-broken). The one thing it never
369
+ auto-fixes is malformed `log.md` content, since guessing what a corrupted
370
+ entry meant risks losing data — that always needs a human look.
371
+
372
+ Each memory file also carries a `schema_version` field; `doctor` stamps files
373
+ onto the current version. There's only ever been one version so far, but this
374
+ is the hook a future BigBrain release would register a real migration
375
+ against — `doctor --fix` is the migration command.
376
+
377
+ ### How an agent is expected to use it
378
+
379
+ 1. At session start, read `.bigbrain/INDEX.md` (`bigbrain index`) instead of
380
+ guessing at project structure or past decisions.
381
+ 2. Pull a specific memory file on demand (`bigbrain read architecture`) when
382
+ it needs deep context on a domain.
383
+ For current implementation, browse with `list_source_files`, search with
384
+ `search_source`, then inspect with `read_source` (or their `source-*` CLI
385
+ equivalents). Those tools create/refresh the source index automatically.
386
+ Check the coverage report and cite source paths/lines; use memory for intent
387
+ and decisions rather than assuming it describes unchanged code.
388
+ 3. Log non-trivial decisions, task-state changes, learnings, and invariants
389
+ as they happen (`bigbrain log ...`), rather than trying to remember them.
390
+ 4. Trigger compilation at milestones or session end (`bigbrain compile`) to
391
+ reconcile the log into `memory/*.md` and rebuild `INDEX.md`.
392
+
393
+ ### MCP server
394
+
395
+ For hosts with native MCP tool-calling, `bigbrain serve-mcp` exposes six memory
396
+ tools and five source tools matching `bigbrain/schemas.py` / `.bigbrain/tools.json`:
397
+ `read_index`, `read_memory_file`, `append_transaction_log`, `compile_memory`
398
+ (with an optional `use_llm` argument), `search_memory` (ranked fuzzy
399
+ discovery), and `grep_memory` (exact substring). Source tools are
400
+ `source_status`, `list_source_files`, `refresh_source_index`, `search_source`
401
+ and `read_source`. It speaks newline-delimited
402
+ JSON-RPC 2.0 over stdio — no MCP SDK dependency.
403
+
404
+ ### Library usage
405
+
406
+ ```python
407
+ from pathlib import Path
408
+ from bigbrain import core
409
+
410
+ core.init_engine(Path("."))
411
+ core.append_log(Path("."), "LEARNING", "domain_knowledge", "Payments API v2", "Uses cursor pagination.")
412
+ core.compile_memory(Path("."), reason="milestone")
413
+ ```
414
+
415
+ ### Tests
416
+
417
+ ```bash
418
+ python -m pip install -e .
419
+ python tests/test_core.py
420
+ ```