mempipeline 0.8.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- mempipeline-0.8.0/LICENSE +201 -0
- mempipeline-0.8.0/PKG-INFO +317 -0
- mempipeline-0.8.0/README.md +302 -0
- mempipeline-0.8.0/mempipeline/__init__.py +9 -0
- mempipeline-0.8.0/mempipeline/access_log.py +102 -0
- mempipeline-0.8.0/mempipeline/act.py +203 -0
- mempipeline-0.8.0/mempipeline/audit.py +206 -0
- mempipeline-0.8.0/mempipeline/auto_commit.py +173 -0
- mempipeline-0.8.0/mempipeline/bridge.py +162 -0
- mempipeline-0.8.0/mempipeline/crossref.py +156 -0
- mempipeline-0.8.0/mempipeline/engine.py +228 -0
- mempipeline-0.8.0/mempipeline/governance/__init__.py +42 -0
- mempipeline-0.8.0/mempipeline/governance/_health.py +57 -0
- mempipeline-0.8.0/mempipeline/governance/_scan.py +210 -0
- mempipeline-0.8.0/mempipeline/governance/_score.py +79 -0
- mempipeline-0.8.0/mempipeline/governance/_state.py +220 -0
- mempipeline-0.8.0/mempipeline/ingest.py +173 -0
- mempipeline-0.8.0/mempipeline/inject.py +241 -0
- mempipeline-0.8.0/mempipeline/panel.py +597 -0
- mempipeline-0.8.0/mempipeline/panel_ops.py +231 -0
- mempipeline-0.8.0/mempipeline/protocol.py +308 -0
- mempipeline-0.8.0/mempipeline/recall.py +551 -0
- mempipeline-0.8.0/mempipeline/recall_golden.py +196 -0
- mempipeline-0.8.0/mempipeline/semantic.py +352 -0
- mempipeline-0.8.0/mempipeline/timegrap.py +465 -0
- mempipeline-0.8.0/mempipeline/trust_rank.py +90 -0
- mempipeline-0.8.0/mempipeline/writer_contracts.py +157 -0
- mempipeline-0.8.0/mempipeline.egg-info/PKG-INFO +317 -0
- mempipeline-0.8.0/mempipeline.egg-info/SOURCES.txt +32 -0
- mempipeline-0.8.0/mempipeline.egg-info/dependency_links.txt +1 -0
- mempipeline-0.8.0/mempipeline.egg-info/entry_points.txt +5 -0
- mempipeline-0.8.0/mempipeline.egg-info/top_level.txt +1 -0
- mempipeline-0.8.0/pyproject.toml +28 -0
- mempipeline-0.8.0/setup.cfg +4 -0
|
@@ -0,0 +1,201 @@
|
|
|
1
|
+
Apache License
|
|
2
|
+
Version 2.0, January 2004
|
|
3
|
+
http://www.apache.org/licenses/
|
|
4
|
+
|
|
5
|
+
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
|
6
|
+
|
|
7
|
+
1. Definitions.
|
|
8
|
+
|
|
9
|
+
"License" shall mean the terms and conditions for use, reproduction,
|
|
10
|
+
and distribution as defined by Sections 1 through 9 of this document.
|
|
11
|
+
|
|
12
|
+
"Licensor" shall mean the copyright owner or entity authorized by
|
|
13
|
+
the copyright owner that is granting the License.
|
|
14
|
+
|
|
15
|
+
"Legal Entity" shall mean the union of the acting entity and all
|
|
16
|
+
other entities that control, are controlled by, or are under common
|
|
17
|
+
control with that entity. For the purposes of this definition,
|
|
18
|
+
"control" means (i) the power, direct or indirect, to cause the
|
|
19
|
+
direction or management of such entity, whether by contract or
|
|
20
|
+
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
|
21
|
+
outstanding shares, or (iii) beneficial ownership of such entity.
|
|
22
|
+
|
|
23
|
+
"You" (or "Your") shall mean an individual or Legal Entity
|
|
24
|
+
exercising permissions granted by this License.
|
|
25
|
+
|
|
26
|
+
"Source" form shall mean the preferred form for making modifications,
|
|
27
|
+
including but not limited to software source code, documentation
|
|
28
|
+
source, and configuration files.
|
|
29
|
+
|
|
30
|
+
"Object" form shall mean any form resulting from mechanical
|
|
31
|
+
transformation or translation of a Source form, including but
|
|
32
|
+
not limited to compiled object code, generated documentation,
|
|
33
|
+
and conversions to other media types.
|
|
34
|
+
|
|
35
|
+
"Work" shall mean the work of authorship, whether in Source or
|
|
36
|
+
Object form, made available under the License, as indicated by a
|
|
37
|
+
copyright notice that is included in or attached to the work
|
|
38
|
+
(an example is provided in the Appendix below).
|
|
39
|
+
|
|
40
|
+
"Derivative Works" shall mean any work, whether in Source or Object
|
|
41
|
+
form, that is based on (or derived from) the Work and for which the
|
|
42
|
+
editorial revisions, annotations, elaborations, or other modifications
|
|
43
|
+
represent, as a whole, an original work of authorship. For the purposes
|
|
44
|
+
of this License, Derivative Works shall not include works that remain
|
|
45
|
+
separable from, or merely link (or bind by name) to the interfaces of,
|
|
46
|
+
the Work and Derivative Works thereof.
|
|
47
|
+
|
|
48
|
+
"Contribution" shall mean any work of authorship, including
|
|
49
|
+
the original version of the Work and any modifications or additions
|
|
50
|
+
to that Work or Derivative Works thereof, that is intentionally
|
|
51
|
+
submitted to Licensor for inclusion in the Work by the copyright owner
|
|
52
|
+
or by an individual or Legal Entity authorized to submit on behalf of
|
|
53
|
+
the copyright owner. For the purposes of this definition, "submitted"
|
|
54
|
+
means any form of electronic, verbal, or written communication sent
|
|
55
|
+
to the Licensor or its representatives, including but not limited to
|
|
56
|
+
communication on electronic mailing lists, source code control systems,
|
|
57
|
+
and issue tracking systems that are managed by, or on behalf of, the
|
|
58
|
+
Licensor for the purpose of discussing and improving the Work, but
|
|
59
|
+
excluding communication that is conspicuously marked or otherwise
|
|
60
|
+
designated in writing by the copyright owner as "Not a Contribution."
|
|
61
|
+
|
|
62
|
+
"Contributor" shall mean Licensor and any individual or Legal Entity
|
|
63
|
+
on behalf of whom a Contribution has been received by Licensor and
|
|
64
|
+
subsequently incorporated within the Work.
|
|
65
|
+
|
|
66
|
+
2. Grant of Copyright License. Subject to the terms and conditions of
|
|
67
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
68
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
69
|
+
copyright license to reproduce, prepare Derivative Works of,
|
|
70
|
+
publicly display, publicly perform, sublicense, and distribute the
|
|
71
|
+
Work and such Derivative Works in Source or Object form.
|
|
72
|
+
|
|
73
|
+
3. Grant of Patent License. Subject to the terms and conditions of
|
|
74
|
+
this License, each Contributor hereby grants to You a perpetual,
|
|
75
|
+
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
|
76
|
+
(except as stated in this section) patent license to make, have made,
|
|
77
|
+
use, offer to sell, sell, import, and otherwise transfer the Work,
|
|
78
|
+
where such license applies only to those patent claims licensable
|
|
79
|
+
by such Contributor that are necessarily infringed by their
|
|
80
|
+
Contribution(s) alone or by combination of their Contribution(s)
|
|
81
|
+
with the Work to which such Contribution(s) was submitted. If You
|
|
82
|
+
institute patent litigation against any entity (including a
|
|
83
|
+
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
|
84
|
+
or a Contribution incorporated within the Work constitutes direct
|
|
85
|
+
or contributory patent infringement, then any patent licenses
|
|
86
|
+
granted to You under this License for that Work shall terminate
|
|
87
|
+
as of the date such litigation is filed.
|
|
88
|
+
|
|
89
|
+
4. Redistribution. You may reproduce and distribute copies of the
|
|
90
|
+
Work or Derivative Works thereof in any medium, with or without
|
|
91
|
+
modifications, and in Source or Object form, provided that You
|
|
92
|
+
meet the following conditions:
|
|
93
|
+
|
|
94
|
+
(a) You must give any other recipients of the Work or
|
|
95
|
+
Derivative Works a copy of this License; and
|
|
96
|
+
|
|
97
|
+
(b) You must cause any modified files to carry prominent notices
|
|
98
|
+
stating that You changed the files; and
|
|
99
|
+
|
|
100
|
+
(c) You must retain, in the Source form of any Derivative Works
|
|
101
|
+
that You distribute, all copyright, patent, trademark, and
|
|
102
|
+
attribution notices from the Source form of the Work,
|
|
103
|
+
excluding those notices that do not pertain to any part of
|
|
104
|
+
the Derivative Works; and
|
|
105
|
+
|
|
106
|
+
(d) If the Work includes a "NOTICE" text file as part of its
|
|
107
|
+
distribution, then any Derivative Works that You distribute must
|
|
108
|
+
include a readable copy of the attribution notices contained
|
|
109
|
+
within such NOTICE file, excluding those notices that do not
|
|
110
|
+
pertain to any part of the Derivative Works, in at least one
|
|
111
|
+
of the following places: within a NOTICE text file distributed
|
|
112
|
+
as part of the Derivative Works; within the Source form or
|
|
113
|
+
documentation, if provided along with the Derivative Works; or,
|
|
114
|
+
within a display generated by the Derivative Works, if and
|
|
115
|
+
wherever such third-party notices normally appear. The contents
|
|
116
|
+
of the NOTICE file are for informational purposes only and
|
|
117
|
+
do not modify the License. You may add Your own attribution
|
|
118
|
+
notices within Derivative Works that You distribute, alongside
|
|
119
|
+
or as an addendum to the NOTICE text from the Work, provided
|
|
120
|
+
that such additional attribution notices cannot be construed
|
|
121
|
+
as modifying the License.
|
|
122
|
+
|
|
123
|
+
You may add Your own copyright statement to Your modifications and
|
|
124
|
+
may provide additional or different license terms and conditions
|
|
125
|
+
for use, reproduction, or distribution of Your modifications, or
|
|
126
|
+
for any such Derivative Works as a whole, provided Your use,
|
|
127
|
+
reproduction, and distribution of the Work otherwise complies with
|
|
128
|
+
the conditions stated in this License.
|
|
129
|
+
|
|
130
|
+
5. Submission of Contributions. Unless You explicitly state otherwise,
|
|
131
|
+
any Contribution intentionally submitted for inclusion in the Work
|
|
132
|
+
by You to the Licensor shall be under the terms and conditions of
|
|
133
|
+
this License, without any additional terms or conditions.
|
|
134
|
+
Notwithstanding the above, nothing herein shall supersede or modify
|
|
135
|
+
the terms of any separate license agreement you may have executed
|
|
136
|
+
with Licensor regarding such Contributions.
|
|
137
|
+
|
|
138
|
+
6. Trademarks. This License does not grant permission to use the trade
|
|
139
|
+
names, trademarks, service marks, or product names of the Licensor,
|
|
140
|
+
except as required for reasonable and customary use in describing the
|
|
141
|
+
origin of the Work and reproducing the content of the NOTICE file.
|
|
142
|
+
|
|
143
|
+
7. Disclaimer of Warranty. Unless required by applicable law or
|
|
144
|
+
agreed to in writing, Licensor provides the Work (and each
|
|
145
|
+
Contributor provides its Contributions) on an "AS IS" BASIS,
|
|
146
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
|
147
|
+
implied, including, without limitation, any warranties or conditions
|
|
148
|
+
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
|
149
|
+
PARTICULAR PURPOSE. You are solely responsible for determining the
|
|
150
|
+
appropriateness of using or redistributing the Work and assume any
|
|
151
|
+
risks associated with Your exercise of permissions under this License.
|
|
152
|
+
|
|
153
|
+
8. Limitation of Liability. In no event and under no legal theory,
|
|
154
|
+
whether in tort (including negligence), contract, or otherwise,
|
|
155
|
+
unless required by applicable law (such as deliberate and grossly
|
|
156
|
+
negligent acts) or agreed to in writing, shall any Contributor be
|
|
157
|
+
liable to You for damages, including any direct, indirect, special,
|
|
158
|
+
incidental, or consequential damages of any character arising as a
|
|
159
|
+
result of this License or out of the use or inability to use the
|
|
160
|
+
Work (including but not limited to damages for loss of goodwill,
|
|
161
|
+
work stoppage, computer failure or malfunction, or any and all
|
|
162
|
+
other commercial damages or losses), even if such Contributor
|
|
163
|
+
has been advised of the possibility of such damages.
|
|
164
|
+
|
|
165
|
+
9. Accepting Warranty or Additional Liability. While redistributing
|
|
166
|
+
the Work or Derivative Works thereof, You may choose to offer,
|
|
167
|
+
and charge a fee for, acceptance of support, warranty, indemnity,
|
|
168
|
+
or other liability obligations and/or rights consistent with this
|
|
169
|
+
License. However, in accepting such obligations, You may act only
|
|
170
|
+
on Your own behalf and on Your sole responsibility, not on behalf
|
|
171
|
+
of any other Contributor, and only if You agree to indemnify,
|
|
172
|
+
defend, and hold each Contributor harmless for any liability
|
|
173
|
+
incurred by, or claims asserted against, such Contributor by reason
|
|
174
|
+
of your accepting any such warranty or additional liability.
|
|
175
|
+
|
|
176
|
+
END OF TERMS AND CONDITIONS
|
|
177
|
+
|
|
178
|
+
APPENDIX: How to apply the Apache License to your work.
|
|
179
|
+
|
|
180
|
+
To apply the Apache License to your work, attach the following
|
|
181
|
+
boilerplate notice, with the fields enclosed by brackets "[]"
|
|
182
|
+
replaced with your own identifying information. (Don't include
|
|
183
|
+
the brackets!) The text should be enclosed in the appropriate
|
|
184
|
+
comment syntax for the file format. We also recommend that a
|
|
185
|
+
file or class name and description of purpose be included on the
|
|
186
|
+
same "printed page" as the copyright notice for easier
|
|
187
|
+
identification within third-party archives.
|
|
188
|
+
|
|
189
|
+
Copyright [yyyy] [name of copyright owner]
|
|
190
|
+
|
|
191
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
|
192
|
+
you may not use this file except in compliance with the License.
|
|
193
|
+
You may obtain a copy of the License at
|
|
194
|
+
|
|
195
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
|
196
|
+
|
|
197
|
+
Unless required by applicable law or agreed to in writing, software
|
|
198
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
|
199
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
200
|
+
See the License for the specific language governing permissions and
|
|
201
|
+
limitations under the License.
|
|
@@ -0,0 +1,317 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: mempipeline
|
|
3
|
+
Version: 0.8.0
|
|
4
|
+
Summary: 1-writer + N-contributors + unlimited-readers shared memory write pipeline: atomic idempotent writes, pluggable audit, staging ingest gate.
|
|
5
|
+
Author: mempipeline contributors
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Keywords: memory,agent,pipeline,audit,idempotent,staging
|
|
8
|
+
Classifier: Operating System :: OS Independent
|
|
9
|
+
Classifier: Programming Language :: Python :: 3
|
|
10
|
+
Classifier: Topic :: Software Development :: Libraries
|
|
11
|
+
Requires-Python: >=3.10
|
|
12
|
+
Description-Content-Type: text/markdown
|
|
13
|
+
License-File: LICENSE
|
|
14
|
+
Dynamic: license-file
|
|
15
|
+
|
|
16
|
+
# mempipeline
|
|
17
|
+
|
|
18
|
+
**Local-first shared memory for AI agents.** One writer distills, N contributors stage, unlimited readers recall — every write atomic, idempotent, audited, version-chained. **Your memory files never leave your machine.**
|
|
19
|
+
|
|
20
|
+
**本地方优先的 AI Agent 共享记忆管线**:单一写者蒸馏、N 投稿者闸门投稿、无限读者召回。每次写入原子、幂等、可审计、版本链可回滚——**记忆正本永不出本机**。
|
|
21
|
+
|
|
22
|
+

|
|
23
|
+

|
|
24
|
+
|
|
25
|
+
## Why mempipeline
|
|
26
|
+
|
|
27
|
+
Most agent-memory solutions are cloud APIs: convenient, but your memory — the distilled record of your decisions — lives on someone else's server. mempipeline takes the opposite bet: memory is plain Markdown on your own disk, with the discipline of a production database.
|
|
28
|
+
|
|
29
|
+
| | mempipeline | Cloud APIs (Mem0 / Zep / Letta Cloud) |
|
|
30
|
+
|---|---|---|
|
|
31
|
+
| **Data residency** | plain Markdown files on your disk; nothing leaves the machine | memory stored in vendor cloud |
|
|
32
|
+
| **Auditability** | append-only audit log + SHA-256 version chain, rollback-ready | opaque |
|
|
33
|
+
| **Idempotent writes** | identical stable body is never rewritten (content-hash skip) | N/A |
|
|
34
|
+
| **Feedback loop** | distill → recall closed in-repo; adopt/judge trust-overlay wiring left to your orchestration layer (APIs provided) | store/retrieve only |
|
|
35
|
+
| **Retrieval** | ngram TF-IDF + SQLite inverted index; optional local bge-m3 hybrid | managed embeddings |
|
|
36
|
+
| **Best fit** | single machine, single operator, privacy-first | teams, multi-tenant, hosted scale |
|
|
37
|
+
|
|
38
|
+
**Honest trade-offs**: no hosted dashboard, no multi-tenant sync, no managed embeddings. If you need vendor-managed scale across a team, use the clouds. If you need your memory to stay *yours* — auditable, portable, version-chained — keep reading.
|
|
39
|
+
|
|
40
|
+
## 30-second quick start
|
|
41
|
+
|
|
42
|
+
Install from source (PyPI package coming soon):
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
git clone https://github.com/18DU1019/mempipeline.git
|
|
46
|
+
cd mempipeline
|
|
47
|
+
python -c "import mempipeline; print('ok')"
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Write a memory (writer side), fully self-contained:
|
|
51
|
+
|
|
52
|
+
```python
|
|
53
|
+
from pathlib import Path
|
|
54
|
+
from tempfile import mkdtemp
|
|
55
|
+
from mempipeline.audit import FileAudit
|
|
56
|
+
from mempipeline.engine import write_atomic
|
|
57
|
+
from mempipeline.protocol import Note
|
|
58
|
+
|
|
59
|
+
mem_root = Path(mkdtemp()) / "mem" # any directory you own
|
|
60
|
+
audit = FileAudit(mem_root / ".." / "audit" / "log.md",
|
|
61
|
+
mem_root / ".." / "audit" / "manifest.json", mem_root)
|
|
62
|
+
note = Note(title="Rule", summary="demo", tier="medium", importance=0.6, body="body")
|
|
63
|
+
out = mem_root / "02-中期记忆" / "项目会话-规则-ab1234cd.md"
|
|
64
|
+
out.parent.mkdir(parents=True, exist_ok=True)
|
|
65
|
+
status, _ = write_atomic(out, note.to_frontmatter() + "\n\n" + note.body + "\n", audit)
|
|
66
|
+
# status == "wrote" (first write); rewrite the same body -> "skipped" (idempotent)
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Recall it (reader side):
|
|
70
|
+
|
|
71
|
+
```python
|
|
72
|
+
from mempipeline.recall import MemoryRecall
|
|
73
|
+
hits = MemoryRecall(mem_root).recall("rule keywords") # [(path, score), ...]
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
---
|
|
77
|
+
|
|
78
|
+
**Shared-memory *write* pipeline for the 1-writer / N-contributor /
|
|
79
|
+
unlimited-reader model.** Atomic idempotent writes, a pluggable audit
|
|
80
|
+
backend, and a staging ingest gate — with zero private paths baked in.
|
|
81
|
+
|
|
82
|
+
`mempipeline` is a **library**: `engine` (atomic idempotent writes),
|
|
83
|
+
`ingest` (staging gate), `audit` (pluggable backends), plus read-side
|
|
84
|
+
modules — ngram TF-IDF recall
|
|
85
|
+
(`mempipeline.recall`, with an optional SQLite inverted-index fast path), semantic
|
|
86
|
+
hybrid retrieval (`semantic.py`), timeline graph (`timegrap.py`),
|
|
87
|
+
bidirectional cross-referencing (`crossref.py`), the PDCA Check signal
|
|
88
|
+
layer (`recall_golden.py`), and a recall-degradation diagnosis & Act
|
|
89
|
+
suggestion surface (`act.py`, human-triggered, Act off by default).
|
|
90
|
+
CLI orchestration (one-command recall with a judge feedback loop) sits
|
|
91
|
+
in the operator's own automation layer — build yours on top of these APIs.
|
|
92
|
+
|
|
93
|
+
## Model
|
|
94
|
+
|
|
95
|
+
| Role | Capability | Channel |
|
|
96
|
+
|-------------|------------------------------------------|----------------------|
|
|
97
|
+
| **Writer** | the single one who distills into mirror | engine |
|
|
98
|
+
| **Contributor** | drops authored note to staging gate | ingest |
|
|
99
|
+
| **Reader** | reads mirror, never writes back | any read-only API |
|
|
100
|
+
|
|
101
|
+
All paths are injected by the caller; nothing is hard-coded.
|
|
102
|
+
|
|
103
|
+
## Install / import
|
|
104
|
+
|
|
105
|
+
```bash
|
|
106
|
+
pip install mempipeline
|
|
107
|
+
# or: clone and add the folder to sys.path, then `import mempipeline`
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
## Quick start
|
|
111
|
+
|
|
112
|
+
```python
|
|
113
|
+
from pathlib import Path
|
|
114
|
+
from mempipeline.audit import FileAudit
|
|
115
|
+
from mempipeline.engine import write_atomic
|
|
116
|
+
from mempipeline.protocol import Note
|
|
117
|
+
|
|
118
|
+
mem_root = Path("./example_data/mem")
|
|
119
|
+
audit = FileAudit(
|
|
120
|
+
log_path=mem_root / ".." / "audit" / "log.md",
|
|
121
|
+
manifest_path=mem_root / ".." / "audit" / "manifest.json",
|
|
122
|
+
root_dir=mem_root,
|
|
123
|
+
)
|
|
124
|
+
|
|
125
|
+
note = Note(title="Rule", summary="demo", tier="medium", importance=0.6, body="body")
|
|
126
|
+
out = mem_root / "02-中期记忆" / "项目会话-规则-ab1234cd.md"
|
|
127
|
+
status, wrote = write_atomic(out, note.to_frontmatter() + "\n\n" + note.body + "\n", audit)
|
|
128
|
+
assert status == "wrote" # first write
|
|
129
|
+
status, _ = write_atomic(out, note.to_frontmatter() + "\n\n" + note.body + "\n", audit)
|
|
130
|
+
assert status == "skipped" # idempotent: same stable body, no rewrite
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
> Run the zero-dependency end-to-end suite locally: `python test_mempipeline.py`
|
|
134
|
+
|
|
135
|
+
### N contributors via staging
|
|
136
|
+
|
|
137
|
+
Drop authored Markdown (with `title` / `summary` / `memory_tier` /
|
|
138
|
+
`source_agent` frontmatter) into a staging dir, then:
|
|
139
|
+
|
|
140
|
+
```python
|
|
141
|
+
from mempipeline.ingest import ingest
|
|
142
|
+
ingest(
|
|
143
|
+
staging_dir=Path("./example_data/staging"),
|
|
144
|
+
mem_root=mem_root,
|
|
145
|
+
tier_dirs={"long": "01-长期记忆", "medium": "02-中期记忆"},
|
|
146
|
+
audit=audit,
|
|
147
|
+
on_ingest=lambda out: print("git add --", out),
|
|
148
|
+
)
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
## Design pillars
|
|
152
|
+
|
|
153
|
+
- **Atomic + idempotent** — writes go through a temp file + `os.replace`;
|
|
154
|
+
identical stable body (timestamps stripped) is skipped, never rewritten.
|
|
155
|
+
- **Audit is a pluggable backend** — default `FileAudit` (append-only log +
|
|
156
|
+
external fingerprint manifest); swap in a DB/API backend behind the same
|
|
157
|
+
`AuditBackend` interface.
|
|
158
|
+
- **Data-decoupled** — `protocol` owns the frontmatter contract; every path
|
|
159
|
+
is injected. Nothing here knows your vault.
|
|
160
|
+
|
|
161
|
+
## SQLite inverted-index fast path (P1-A2)
|
|
162
|
+
|
|
163
|
+
`recall.MemoryRecall` does a full-disk read + full IDF recompute on every
|
|
164
|
+
query. For large mirrors that is O(N) per query. `recall.TFIDFIndex` is a
|
|
165
|
+
zero-embedding SQLite inverted index that moves that cost to an explicit
|
|
166
|
+
`build()`:
|
|
167
|
+
|
|
168
|
+
```python
|
|
169
|
+
from pathlib import Path
|
|
170
|
+
from mempipeline.recall import MemoryRecall, TFIDFIndex
|
|
171
|
+
|
|
172
|
+
idx = TFIDFIndex(Path("./recall.db"))
|
|
173
|
+
idx.build(mem_root) # one-time scan; idempotent (skips indexed docs)
|
|
174
|
+
m = MemoryRecall(mem_root, index=idx) # fast path: no normalization, or index normalized with the same synonym table
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
- `recall()` returns the same `(path, score)` shape as `MemoryRecall`, so the
|
|
178
|
+
index is a drop-in fast path, not a second retrieval strategy. On any query
|
|
179
|
+
error it falls back to the full scan.
|
|
180
|
+
- When `synonyms` normalization is active on `MemoryRecall`, the fast path
|
|
181
|
+
survives only if the index was built with the same synonym table
|
|
182
|
+
(`TFIDFIndex` normalizes documents at `build()` time and queries at recall
|
|
183
|
+
time, so a same-table injection stays isomorphic to the full scan);
|
|
184
|
+
a foreign-table index falls back to the full scan. The built-in
|
|
185
|
+
`recall.DEFAULT_SYNONYMS` table is what the panel/semantic gates use.
|
|
186
|
+
- `governance.governance_health()` is a separate read-only snapshot (status
|
|
187
|
+
distribution, candidate backlog & oldest-stale age) for rendering the
|
|
188
|
+
governance-review pane.
|
|
189
|
+
|
|
190
|
+
`mempipeline-panel` exposes the optional `--tfidf-db` flag to feed the index
|
|
191
|
+
into the served recall path.
|
|
192
|
+
|
|
193
|
+
## Time-dimension graph (P2)
|
|
194
|
+
|
|
195
|
+
`mempipeline.timegrap` is a **read-only** scanner that clusters notes into
|
|
196
|
+
timelines and traces how a topic evolves over time.
|
|
197
|
+
|
|
198
|
+
- **Subject clustering** - `subject_key(title)` normalizes a title into a
|
|
199
|
+
stable cluster key (strips whitespace / punctuation / stop chars), so the
|
|
200
|
+
same topic keeps one theme even when titles differ in wording or
|
|
201
|
+
punctuation. Summaries never participate in the key, so *drifted*
|
|
202
|
+
notes (whose conclusion changed) still land in the same timeline and
|
|
203
|
+
can be seen as drift.
|
|
204
|
+
- **Timeline playback** - each cluster yields a chronologically ascending
|
|
205
|
+
`TopicTimeline`; `recall()` returns the paths in time order,
|
|
206
|
+
`spans_days()` gives the span.
|
|
207
|
+
- **Evolution signals** - per node, derived read-only: `first` /
|
|
208
|
+
`continued` / `revived` (gap >= `gap_days`, default 90) / `same_day`, plus
|
|
209
|
+
a `relation` of `duplicate` / `drift` / `stable` from bigram-Jaccard
|
|
210
|
+
summary similarity against the previous note (thresholds
|
|
211
|
+
`dup_threshold` / `drift_threshold`, both configurable and recorded on
|
|
212
|
+
each signal).
|
|
213
|
+
- **Time benchmark fallback** - `updated`/`created` frontmatter win; missing
|
|
214
|
+
fields fall back to file mtime with an explicit `time_src` tag for
|
|
215
|
+
auditability.
|
|
216
|
+
- **Project isolation** - pass `projects=[...]` to constrain the scan; no
|
|
217
|
+
cross-project leakage.
|
|
218
|
+
|
|
219
|
+
Example:
|
|
220
|
+
|
|
221
|
+
```python
|
|
222
|
+
from mempipeline.timegrap import build_timeline
|
|
223
|
+
timelines = build_timeline(mem_root) # {subject_key: TopicTimeline}
|
|
224
|
+
for sk, tl in timelines.items():
|
|
225
|
+
print(sk, tl.spans_days(), [sig["flag"] for sig in tl.signals])
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
## Timeline signals into governance candidates (P3-③)
|
|
229
|
+
|
|
230
|
+
`governance.scan_stale_notes(mem_root, ..., timeline=...)` optionally accepts a
|
|
231
|
+
`timeline` built by `build_timeline`; the lifecycle signals are then merged
|
|
232
|
+
into the same P1 candidate list (still **no** automatic state change - Act
|
|
233
|
+
stays human-triggered):
|
|
234
|
+
|
|
235
|
+
- `conclusion_drift` (previous summary bigram similarity below `drift_threshold`)
|
|
236
|
+
surfaces as a `re-review` candidate;
|
|
237
|
+
- `revived` (resumed after a gap >= `gap_days`) surfaces as a `review`
|
|
238
|
+
candidate, with a concurrent conclusion drift folded in as detail.
|
|
239
|
+
|
|
240
|
+
Last mile: the caller builds the timeline and passes it in, so candidates are
|
|
241
|
+
deduped by path against the stale-archive base list. No new metric system is
|
|
242
|
+
introduced - only the time-dimension signals already validated in P2.
|
|
243
|
+
|
|
244
|
+
---
|
|
245
|
+
|
|
246
|
+
## 中文使用说明
|
|
247
|
+
|
|
248
|
+
### 定位与三种身份
|
|
249
|
+
|
|
250
|
+
`mempipeline` 是「1 写者 + N 投稿 + 无限读」模型的**写侧**管线。镜像是一个分层的 Markdown 目录;读侧能力(TF-IDF 召回 / 语义混合检索 / 时间图谱 / 交叉引用 / golden 检核)在本仓库内,外部蒸馏与控层脚本在运行台脚本目录(distill / sediment / audit_core)。
|
|
251
|
+
|
|
252
|
+
| 身份 | 能力 | 通道 | 落点 |
|
|
253
|
+
|---|---|---|---|
|
|
254
|
+
| **写者** | 唯一有资格直接蒸馏进镜像 | `engine.write_atomic` | 幂等原子写 |
|
|
255
|
+
| **投稿者** | 把带 frontmatter 的笔记丢到闸门 | `ingest.ingest` | staging 熔合 |
|
|
256
|
+
| **读者** | 只读镜像、永不回写 | `distill-mem` 的 inject/recall | 任意只读接口 |
|
|
257
|
+
|
|
258
|
+
所有路径由调用方注入,**零硬编码私有路径**。
|
|
259
|
+
|
|
260
|
+
### 快速接入(四步)
|
|
261
|
+
|
|
262
|
+
1. **配置路径**:复制 `config.example.py` 为 `config.py`,改 `MEM_ROOT` / `STAGING_DIR` / `STAGING_ROOT` / `AUDIT_LOG` / `MANIFEST` / `TIER_DIRS`。其中 `STAGING_ROOT` 为面板投稿位根,缺省回落 `MEMPIPELINE_STAGING` 环境变量或中性默认值;`config.py` 已被 `.gitignore` 排除,仅 `config.example.py` 样例入库。
|
|
263
|
+
2. **写者**(蒸馏进镜像):
|
|
264
|
+
```python
|
|
265
|
+
from pathlib import Path
|
|
266
|
+
from mempipeline.audit import FileAudit
|
|
267
|
+
from mempipeline.engine import write_atomic
|
|
268
|
+
from mempipeline.protocol import Note
|
|
269
|
+
|
|
270
|
+
mem = Path("./example_data/mem")
|
|
271
|
+
audit = FileAudit(mem / ".." / "audit" / "log.md",
|
|
272
|
+
mem / ".." / "audit" / "manifest.json", mem)
|
|
273
|
+
note = Note(title="规则", summary="单笔风险≤1ATR", tier="long",
|
|
274
|
+
importance=0.9, body="正文。")
|
|
275
|
+
out = mem / "01-长期记忆" / "项目会话-规则-ab12cd34.md"
|
|
276
|
+
s, _ = write_atomic(out, note.to_frontmatter() + "\n\n" + note.body + "\n", audit)
|
|
277
|
+
# s == "wrote"(首次)→ 再写同稳定正文 → s == "skipped"(幂等跳过)
|
|
278
|
+
```
|
|
279
|
+
3. **投稿者**(staging 熔合):把带 `title`/`summary`/`memory_tier`/`source_agent` frontmatter 的裸 md 放入 staging 目录,再
|
|
280
|
+
```python
|
|
281
|
+
from mempipeline.ingest import ingest
|
|
282
|
+
ingest(staging, mem, {"long": "01-长期记忆", "medium": "02-中期记忆"},
|
|
283
|
+
audit, on_ingest=lambda o: print("git add --", o))
|
|
284
|
+
```
|
|
285
|
+
4. **读者**:用本仓库 `recall.MemoryRecall`(ngram TF-IDF + 同义归一)或 `semantic.hybrid_recall`(bge-m3 + RRF 融合)做召回;时间演化用 `timegrap.build_timeline`(P2);交叉引用用 `crossref.find_related`。大镜像可先用 `recall.TFIDFIndex.build()` 建 SQLite 倒排,再注入 `MemoryRecall(index=...)` 走快路径(无归一、或索引与实例用同一张同义表双侧归一时保持快路径;异表索引回落全扫,口径一致)。
|
|
286
|
+
|
|
287
|
+
### DQ 转义契约(2026-08-27 起生效)
|
|
288
|
+
|
|
289
|
+
写侧序列化时,**frontmatter 的字符串字段一律输出 YAML 双引号标量(无条件引号)**:
|
|
290
|
+
|
|
291
|
+
```text
|
|
292
|
+
---
|
|
293
|
+
title: "2026" # 纯数字字符串,仍读回 str,不会被解析成 int
|
|
294
|
+
summary: "a # b:含冒号" # 含 # 的串,不会被当注释截断
|
|
295
|
+
source_agent: "writer"
|
|
296
|
+
---
|
|
297
|
+
importance: 0.9 # 数值字段保持裸值
|
|
298
|
+
```
|
|
299
|
+
|
|
300
|
+
两点原因:
|
|
301
|
+
|
|
302
|
+
1. **指示符歧义**:plain 标量内 `#`(行首或前有空白)会被 YAML 当注释起始,`title: a # b` 解析成 `a`,值被静默截断。
|
|
303
|
+
2. **schema 类型失真**:YAML 1.2 core schema 把未加引号的纯数字/布尔字符串解析成 int/bool——`title: 2026` 读回是 int 而非 str。
|
|
304
|
+
|
|
305
|
+
配套:`ingest._parse_fm` 解析投稿 frontmatter 时会**剥匹配的外层引号**,故投稿端写引号或不写引号都兼容。转义顺序为 `\\` → `\"` → `\n\r\t`,全部合法 Double-Quoted 序列。
|
|
306
|
+
|
|
307
|
+
### 安全与幂等契约(2026-09-01 起生效)
|
|
308
|
+
|
|
309
|
+
**`project_id` 安全边界**:落盘路径由 `safe_project_id` 校验,仅放行 `[\w.-]+`;含 `/`、`\`、`..` 或其它非法字符的 `project_id` 一律回落 `legacy`/`global`,防止路径穿越与绝对路径逃逸。投稿端若依赖带子目录的 `project_id`,须改用合法字符。
|
|
310
|
+
|
|
311
|
+
**`content_key` 幂等判据**:稳定正文 SHA-256 截断由 8 位提高至 12 位 hex,降低碰撞概率。迁移影响:升级前已用 8 位 key 落库的笔记,重写同一正文时因 key 变化会被判为新笔记而重复写入;如有需要,可对既有笔记按稳定正文重建 key 一次,或接受单次重复后由人工清理。
|
|
312
|
+
|
|
313
|
+
**`stable_body` 豁免字段(2026-09-11 P3-0 起)**:稳定正文剔除 `updated:` 与 `created:` 两行。`updated` 是实时字段,参入会让幂等失效;`created` 是后加字段(P3-0 起写者落盘前回填),参入会让存量笔记首次带 created 重写时被判为新笔记。其余 frontmatter 字段(含 `status`)参与稳定正文——一次状态迁移会改变 content_key,属设计语义:状态是实质变化,该次写回为预期写入,不适用幂等跳过。
|
|
314
|
+
|
|
315
|
+
## License
|
|
316
|
+
|
|
317
|
+
Apache-2.0. See [LICENSE](LICENSE).
|