portable-memory 0.1.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- portable_memory-0.1.2/.github/ISSUE_TEMPLATE/adapter_or_implementation.yml +18 -0
- portable_memory-0.1.2/.github/ISSUE_TEMPLATE/bug_report.yml +39 -0
- portable_memory-0.1.2/.github/ISSUE_TEMPLATE/config.yml +5 -0
- portable_memory-0.1.2/.github/ISSUE_TEMPLATE/spec_change.yml +32 -0
- portable_memory-0.1.2/.github/PULL_REQUEST_TEMPLATE.md +23 -0
- portable_memory-0.1.2/.github/workflows/ci.yml +37 -0
- portable_memory-0.1.2/.github/workflows/publish.yml +65 -0
- portable_memory-0.1.2/.gitignore +17 -0
- portable_memory-0.1.2/CHANGELOG.md +70 -0
- portable_memory-0.1.2/CITATION.cff +28 -0
- portable_memory-0.1.2/CODE_OF_CONDUCT.md +50 -0
- portable_memory-0.1.2/CONTRIBUTING.md +69 -0
- portable_memory-0.1.2/Conformance/README.md +65 -0
- portable_memory-0.1.2/Conformance/fixtures/sample.mem/CHECKSUMS +5 -0
- portable_memory-0.1.2/Conformance/fixtures/sample.mem/audit/tombstones.jsonl +1 -0
- portable_memory-0.1.2/Conformance/fixtures/sample.mem/items/edge.jsonl +1 -0
- portable_memory-0.1.2/Conformance/fixtures/sample.mem/items/entity.jsonl +1 -0
- portable_memory-0.1.2/Conformance/fixtures/sample.mem/items/episode.jsonl +2 -0
- portable_memory-0.1.2/Conformance/fixtures/sample.mem/items/vendorThing.jsonl +1 -0
- portable_memory-0.1.2/Conformance/fixtures/sample.mem/manifest.json +1 -0
- portable_memory-0.1.2/Conformance/vectors/README.md +65 -0
- portable_memory-0.1.2/Conformance/vectors/canonical-json.json +230 -0
- portable_memory-0.1.2/Conformance/vectors/generate.py +132 -0
- portable_memory-0.1.2/Conformance/vectors/signed.mem/CHECKSUMS +2 -0
- portable_memory-0.1.2/Conformance/vectors/signed.mem/items/entity.jsonl +1 -0
- portable_memory-0.1.2/Conformance/vectors/signed.mem/items/episode.jsonl +2 -0
- portable_memory-0.1.2/Conformance/vectors/signed.mem/manifest.json +1 -0
- portable_memory-0.1.2/Conformance/vectors/signed.mem/manifest.sig +1 -0
- portable_memory-0.1.2/Conformance/vectors/signing-test-key.json +6 -0
- portable_memory-0.1.2/GOVERNANCE.md +56 -0
- portable_memory-0.1.2/LICENSE +21 -0
- portable_memory-0.1.2/PKG-INFO +256 -0
- portable_memory-0.1.2/README.md +223 -0
- portable_memory-0.1.2/SECURITY.md +52 -0
- portable_memory-0.1.2/Schemas/category.schema.json +13 -0
- portable_memory-0.1.2/Schemas/chunk.schema.json +17 -0
- portable_memory-0.1.2/Schemas/community.schema.json +15 -0
- portable_memory-0.1.2/Schemas/context.schema.json +16 -0
- portable_memory-0.1.2/Schemas/core.schema.json +15 -0
- portable_memory-0.1.2/Schemas/edge.schema.json +21 -0
- portable_memory-0.1.2/Schemas/entity.schema.json +18 -0
- portable_memory-0.1.2/Schemas/episode.schema.json +35 -0
- portable_memory-0.1.2/Schemas/episodeLink.schema.json +15 -0
- portable_memory-0.1.2/Schemas/fact.schema.json +28 -0
- portable_memory-0.1.2/Schemas/factLink.schema.json +15 -0
- portable_memory-0.1.2/Schemas/log.schema.json +16 -0
- portable_memory-0.1.2/Schemas/manifest.schema.json +35 -0
- portable_memory-0.1.2/Schemas/preference.schema.json +13 -0
- portable_memory-0.1.2/Schemas/procedure.schema.json +19 -0
- portable_memory-0.1.2/Schemas/resource.schema.json +19 -0
- portable_memory-0.1.2/Schemas/secretRef.schema.json +19 -0
- portable_memory-0.1.2/Schemas/tombstone.schema.json +34 -0
- portable_memory-0.1.2/Spec/portable-memory-spec.md +498 -0
- portable_memory-0.1.2/portable_memory/__init__.py +78 -0
- portable_memory-0.1.2/portable_memory/_codec.py +189 -0
- portable_memory-0.1.2/portable_memory/adapters/__init__.py +15 -0
- portable_memory-0.1.2/portable_memory/adapters/claude.py +162 -0
- portable_memory-0.1.2/portable_memory/adapters/mem0.py +247 -0
- portable_memory-0.1.2/portable_memory/adapters/openai.py +227 -0
- portable_memory-0.1.2/portable_memory/exporter.py +360 -0
- portable_memory-0.1.2/portable_memory/format.py +135 -0
- portable_memory-0.1.2/portable_memory/hashing.py +28 -0
- portable_memory-0.1.2/portable_memory/importer.py +420 -0
- portable_memory-0.1.2/portable_memory/interop.py +51 -0
- portable_memory-0.1.2/portable_memory/records.py +199 -0
- portable_memory-0.1.2/portable_memory/signing.py +182 -0
- portable_memory-0.1.2/portable_memory/store.py +249 -0
- portable_memory-0.1.2/portable_memory/tombstone.py +70 -0
- portable_memory-0.1.2/portable_memory/validator.py +204 -0
- portable_memory-0.1.2/pyproject.toml +48 -0
- portable_memory-0.1.2/tests/conftest.py +174 -0
- portable_memory-0.1.2/tests/test_adapter_mem0_verification.py +99 -0
- portable_memory-0.1.2/tests/test_adapters_openai_claude.py +154 -0
- portable_memory-0.1.2/tests/test_canonical_parity.py +122 -0
- portable_memory-0.1.2/tests/test_coverage_expansion.py +320 -0
- portable_memory-0.1.2/tests/test_interop_fixture.py +89 -0
- portable_memory-0.1.2/tests/test_portable_memory.py +505 -0
- portable_memory-0.1.2/tests/test_vectors.py +48 -0
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
name: Adapter or implementation
|
|
2
|
+
description: Propose or track an adapter (e.g. ChatGPT/Claude/Letta export) or an implementation in another language.
|
|
3
|
+
labels: ["adapter", "interop"]
|
|
4
|
+
body:
|
|
5
|
+
- type: input
|
|
6
|
+
id: target
|
|
7
|
+
attributes:
|
|
8
|
+
label: Source / language
|
|
9
|
+
description: e.g. "ChatGPT export → .mem", or "Go implementation of the reader".
|
|
10
|
+
validations:
|
|
11
|
+
required: true
|
|
12
|
+
- type: textarea
|
|
13
|
+
id: mapping
|
|
14
|
+
attributes:
|
|
15
|
+
label: Mapping / scope
|
|
16
|
+
description: How does it map to the model (episode/core/entity/…)? What rides in `ext`/passthrough? What conformance level do you target?
|
|
17
|
+
validations:
|
|
18
|
+
required: true
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
name: Bug report
|
|
2
|
+
description: Something in the SDK or a bundle doesn't behave as specified.
|
|
3
|
+
labels: ["bug"]
|
|
4
|
+
body:
|
|
5
|
+
- type: textarea
|
|
6
|
+
id: what-happened
|
|
7
|
+
attributes:
|
|
8
|
+
label: What happened
|
|
9
|
+
description: What did you do, what did you expect, and what happened instead?
|
|
10
|
+
validations:
|
|
11
|
+
required: true
|
|
12
|
+
- type: textarea
|
|
13
|
+
id: repro
|
|
14
|
+
attributes:
|
|
15
|
+
label: Reproduction
|
|
16
|
+
description: Steps, code, or a minimal `.mem` bundle that reproduces it.
|
|
17
|
+
validations:
|
|
18
|
+
required: true
|
|
19
|
+
- type: input
|
|
20
|
+
id: version
|
|
21
|
+
attributes:
|
|
22
|
+
label: SDK version / commit
|
|
23
|
+
description: e.g. "portable-memory 0.1.0" or a git commit SHA.
|
|
24
|
+
validations:
|
|
25
|
+
required: true
|
|
26
|
+
- type: input
|
|
27
|
+
id: python
|
|
28
|
+
attributes:
|
|
29
|
+
label: Python version
|
|
30
|
+
description: Output of `python --version` (e.g. "3.12.4").
|
|
31
|
+
validations:
|
|
32
|
+
required: true
|
|
33
|
+
- type: input
|
|
34
|
+
id: os
|
|
35
|
+
attributes:
|
|
36
|
+
label: Operating system
|
|
37
|
+
description: e.g. "Ubuntu 24.04", "macOS 15.1", "Windows 11".
|
|
38
|
+
validations:
|
|
39
|
+
required: true
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
name: Spec / format change
|
|
2
|
+
description: Propose a change to the on-disk format, schemas, or a normative requirement.
|
|
3
|
+
labels: ["spec"]
|
|
4
|
+
body:
|
|
5
|
+
- type: markdown
|
|
6
|
+
attributes:
|
|
7
|
+
value: |
|
|
8
|
+
Format-affecting changes are discussed **before** implementation (see
|
|
9
|
+
CONTRIBUTING and GOVERNANCE). Please describe the change and its compatibility
|
|
10
|
+
impact.
|
|
11
|
+
- type: textarea
|
|
12
|
+
id: proposal
|
|
13
|
+
attributes:
|
|
14
|
+
label: Proposed change
|
|
15
|
+
description: What should change in the spec / schemas, and why?
|
|
16
|
+
validations:
|
|
17
|
+
required: true
|
|
18
|
+
- type: dropdown
|
|
19
|
+
id: compat
|
|
20
|
+
attributes:
|
|
21
|
+
label: Compatibility impact
|
|
22
|
+
options:
|
|
23
|
+
- Backward-compatible addition (minor format bump)
|
|
24
|
+
- Breaking change (major format bump)
|
|
25
|
+
- Clarification only (no wire change)
|
|
26
|
+
validations:
|
|
27
|
+
required: true
|
|
28
|
+
- type: textarea
|
|
29
|
+
id: impls
|
|
30
|
+
attributes:
|
|
31
|
+
label: Affected implementations / adapters
|
|
32
|
+
description: Which implementations or adapters would need to change? (e.g. this Python SDK, the Swift SDK, an adapter.)
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
<!-- Thanks for contributing to Portable Memory! -->
|
|
2
|
+
|
|
3
|
+
## What & why
|
|
4
|
+
|
|
5
|
+
<!-- What does this change and why? Link any issue. -->
|
|
6
|
+
|
|
7
|
+
## Type of change
|
|
8
|
+
|
|
9
|
+
- [ ] SDK change (no wire-format impact)
|
|
10
|
+
- [ ] **Spec / format change** (discussed in an issue first — see CONTRIBUTING)
|
|
11
|
+
- [ ] New adapter / other-language implementation
|
|
12
|
+
- [ ] Docs / conformance / fixtures
|
|
13
|
+
|
|
14
|
+
## Checklist
|
|
15
|
+
|
|
16
|
+
- [ ] `pytest -q` passes (run `pip install -e .[dev]` first)
|
|
17
|
+
- [ ] Added/updated tests for the change
|
|
18
|
+
- [ ] If the wire format changed: the spec, `Schemas/`, the Python DTOs, and the sample
|
|
19
|
+
fixture are all in sync, and the `format` version bump follows GOVERNANCE
|
|
20
|
+
- [ ] If the wire format changed: kept parity with the Swift reference SDK
|
|
21
|
+
(bundles stay byte-identical across both implementations)
|
|
22
|
+
- [ ] Updated `CHANGELOG.md` under **Unreleased**
|
|
23
|
+
- [ ] Commits are signed off (`git commit -s`, DCO)
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
name: CI
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main]
|
|
6
|
+
pull_request:
|
|
7
|
+
|
|
8
|
+
concurrency:
|
|
9
|
+
group: ci-${{ github.ref }}
|
|
10
|
+
cancel-in-progress: true
|
|
11
|
+
|
|
12
|
+
jobs:
|
|
13
|
+
test:
|
|
14
|
+
name: ${{ matrix.os }} (Python ${{ matrix.python-version }})
|
|
15
|
+
runs-on: ${{ matrix.os }}
|
|
16
|
+
strategy:
|
|
17
|
+
fail-fast: false
|
|
18
|
+
matrix:
|
|
19
|
+
os: [ubuntu-latest, macos-latest]
|
|
20
|
+
python-version: ["3.10", "3.11", "3.12", "3.13"]
|
|
21
|
+
steps:
|
|
22
|
+
- uses: actions/checkout@v4
|
|
23
|
+
- uses: actions/setup-python@v5
|
|
24
|
+
with:
|
|
25
|
+
python-version: ${{ matrix.python-version }}
|
|
26
|
+
- name: Install (editable, with dev extras)
|
|
27
|
+
run: pip install -e .[dev]
|
|
28
|
+
- name: Test
|
|
29
|
+
run: pytest -q
|
|
30
|
+
|
|
31
|
+
fixture-integrity:
|
|
32
|
+
name: Conformance fixture checksums
|
|
33
|
+
runs-on: ubuntu-latest
|
|
34
|
+
steps:
|
|
35
|
+
- uses: actions/checkout@v4
|
|
36
|
+
- name: Verify sample.mem CHECKSUMS
|
|
37
|
+
run: cd Conformance/fixtures/sample.mem && sha256sum -c CHECKSUMS
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
name: Publish to PyPI
|
|
2
|
+
|
|
3
|
+
# Publishes the package when a GitHub release is published (tag == pyproject version).
|
|
4
|
+
# Uses PyPI Trusted Publishing (OIDC) — no API token is stored anywhere. One-time setup on
|
|
5
|
+
# pypi.org: project "portable-memory" → Publishing → add a GitHub publisher with
|
|
6
|
+
# owner "MacPaw", repository "portable-memory", workflow "publish.yml", environment "pypi".
|
|
7
|
+
# (For the very first release, register it as a *pending* publisher under your account.)
|
|
8
|
+
|
|
9
|
+
on:
|
|
10
|
+
release:
|
|
11
|
+
types: [published]
|
|
12
|
+
workflow_dispatch:
|
|
13
|
+
|
|
14
|
+
permissions:
|
|
15
|
+
contents: read
|
|
16
|
+
|
|
17
|
+
jobs:
|
|
18
|
+
build:
|
|
19
|
+
runs-on: ubuntu-latest
|
|
20
|
+
steps:
|
|
21
|
+
- uses: actions/checkout@v4
|
|
22
|
+
|
|
23
|
+
- uses: actions/setup-python@v5
|
|
24
|
+
with:
|
|
25
|
+
python-version: "3.12"
|
|
26
|
+
|
|
27
|
+
- name: Build sdist and wheel
|
|
28
|
+
run: |
|
|
29
|
+
python -m pip install --upgrade pip build twine
|
|
30
|
+
python -m build
|
|
31
|
+
python -m twine check dist/*
|
|
32
|
+
|
|
33
|
+
- name: Verify the package version matches the release tag
|
|
34
|
+
if: github.event_name == 'release'
|
|
35
|
+
run: |
|
|
36
|
+
TAG="${GITHUB_REF_NAME}"
|
|
37
|
+
VER="$(python -c 'import tomllib; print(tomllib.load(open("pyproject.toml", "rb"))["project"]["version"])')"
|
|
38
|
+
if [ "$TAG" != "$VER" ]; then
|
|
39
|
+
echo "::error::release tag '$TAG' does not match pyproject version '$VER'"
|
|
40
|
+
exit 1
|
|
41
|
+
fi
|
|
42
|
+
|
|
43
|
+
- name: Smoke-test the built wheel
|
|
44
|
+
run: |
|
|
45
|
+
python -m venv /tmp/smoke && /tmp/smoke/bin/pip install -q dist/*.whl
|
|
46
|
+
/tmp/smoke/bin/python -c "import portable_memory, portable_memory.adapters; print('portable_memory', portable_memory.__version__ if hasattr(portable_memory, '__version__') else 'ok')"
|
|
47
|
+
|
|
48
|
+
- uses: actions/upload-artifact@v4
|
|
49
|
+
with:
|
|
50
|
+
name: dist
|
|
51
|
+
path: dist/
|
|
52
|
+
|
|
53
|
+
publish:
|
|
54
|
+
needs: build
|
|
55
|
+
runs-on: ubuntu-latest
|
|
56
|
+
environment: pypi
|
|
57
|
+
permissions:
|
|
58
|
+
id-token: write # OIDC token for Trusted Publishing
|
|
59
|
+
steps:
|
|
60
|
+
- uses: actions/download-artifact@v4
|
|
61
|
+
with:
|
|
62
|
+
name: dist
|
|
63
|
+
path: dist/
|
|
64
|
+
|
|
65
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/). The SDK package version is
|
|
5
|
+
independent of the on-disk **format** version (`format` in the manifest).
|
|
6
|
+
|
|
7
|
+
## [Unreleased]
|
|
8
|
+
|
|
9
|
+
## [0.1.2] - 2026-09-13
|
|
10
|
+
|
|
11
|
+
First release published to PyPI: `pip install portable-memory`.
|
|
12
|
+
|
|
13
|
+
### Added
|
|
14
|
+
|
|
15
|
+
- **Paper citation** — README callout and *Citation* section (BibTeX), `CITATION.cff` for
|
|
16
|
+
GitHub's "Cite this repository", and a paper link in the spec header (the spec stays
|
|
17
|
+
byte-identical with the Swift repository). Paper:
|
|
18
|
+
<https://research.macpaw.com/publications/portable-memory>.
|
|
19
|
+
- **PyPI packaging** — project URLs (paper, changelog, issues), richer keywords and
|
|
20
|
+
classifiers, and a Trusted-Publishing release workflow
|
|
21
|
+
(`.github/workflows/publish.yml`) that builds, checks, and publishes the package when
|
|
22
|
+
a GitHub release is published — no long-lived API token.
|
|
23
|
+
|
|
24
|
+
### Fixed
|
|
25
|
+
|
|
26
|
+
- **Package version alignment** — `pyproject.toml` still declared `0.1.0` while the
|
|
27
|
+
repository was tagged `0.1.1`. The package version now tracks the git tag, and the
|
|
28
|
+
release workflow refuses to publish when they disagree.
|
|
29
|
+
|
|
30
|
+
## [0.1.1] - 2026-07-03
|
|
31
|
+
|
|
32
|
+
First public release of the Python reference SDK, reaching parity with the Portable Memory
|
|
33
|
+
format (`format` **1.0.0**) and byte-for-byte interoperability with the
|
|
34
|
+
[Swift reference SDK](https://github.com/MacPaw/portable-memory-swift). Versioned `0.1.1`
|
|
35
|
+
to match the Swift SDK's release of the same day.
|
|
36
|
+
|
|
37
|
+
### Added
|
|
38
|
+
|
|
39
|
+
- **Python SDK** — `BundleExporter` / `BundleImporter` / `BundleValidator`, the
|
|
40
|
+
`PortableMemoryStore` seam an adopter subclasses, and dataclass DTOs for all record
|
|
41
|
+
kinds. Fully **synchronous** — local file I/O is done inline, no `async`/`await`.
|
|
42
|
+
- **Canonical JSON** — the serialization/hashing helpers in `portable_memory/_codec.py`
|
|
43
|
+
produce output byte-for-byte identical to the Swift SDK's `MemCodec` (spec §1.1), so
|
|
44
|
+
checksums and signatures match across implementations. Whole-valued floats serialize
|
|
45
|
+
as integers, timestamps are whole-second UTC `Z`, NaN/Infinity are rejected, and
|
|
46
|
+
scalar fields are type-checked on decode.
|
|
47
|
+
- **Pure-stdlib core** — the core depends only on the standard library (`json`, `hashlib`,
|
|
48
|
+
`dataclasses`, `datetime`). Signing is an optional `[signing]` extra (`cryptography`),
|
|
49
|
+
imported lazily so bundles can be produced and read without it.
|
|
50
|
+
- **Deletion propagation (L2)** — portable tombstones with proof-of-reach, applied before
|
|
51
|
+
additions; no-resurrection enforced for **every** kind (not just episodes).
|
|
52
|
+
- **Ed25519 signing (L3)** — detached bundle signatures (`manifest.sig`) and tombstone
|
|
53
|
+
signatures, verified against caller-supplied trusted keys.
|
|
54
|
+
- **Cross-vendor losslessness** — foreign episode fields via `ext` and foreign kinds via
|
|
55
|
+
verbatim passthrough; a mem0 adapter (`portable_memory/adapters/mem0.py`, verified
|
|
56
|
+
against mem0's documented export shapes), an OpenAI adapter for the ChatGPT data
|
|
57
|
+
export (`adapters/openai.py` — `conversations.json` + a saved-memories fallback), and
|
|
58
|
+
a Claude adapter for Claude memory files (`adapters/claude.py` — `MEMORY.md` + topic
|
|
59
|
+
files with frontmatter).
|
|
60
|
+
- **JSON Schemas** for every record kind, the manifest, tombstones, and the audit log,
|
|
61
|
+
validated against samples in the test suite.
|
|
62
|
+
- **Conformance kit** — a sample `.mem` fixture and a signed fixture shared with the
|
|
63
|
+
Swift SDK, plus golden canonical-JSON vectors
|
|
64
|
+
(`Conformance/vectors/canonical-json.json`) that every implementation must reproduce.
|
|
65
|
+
- **Untrusted-input hardening** — path-traversal + symlink-escape rejection, unlisted-file
|
|
66
|
+
rejection, and a per-file size bound (`MemLimits`).
|
|
67
|
+
|
|
68
|
+
[Unreleased]: https://github.com/MacPaw/portable-memory/compare/0.1.2...HEAD
|
|
69
|
+
[0.1.2]: https://github.com/MacPaw/portable-memory/compare/0.1.1...0.1.2
|
|
70
|
+
[0.1.1]: https://github.com/MacPaw/portable-memory/releases/tag/0.1.1
|
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
cff-version: 1.2.0
|
|
2
|
+
message: "If you use Portable Memory in your work, please cite the paper below."
|
|
3
|
+
title: "Portable Memory — Python reference SDK"
|
|
4
|
+
type: software
|
|
5
|
+
license: MIT
|
|
6
|
+
repository-code: "https://github.com/MacPaw/portable-memory"
|
|
7
|
+
url: "https://research.macpaw.com/publications/portable-memory"
|
|
8
|
+
authors:
|
|
9
|
+
- family-names: Kryvoblotskyi
|
|
10
|
+
given-names: Sergii
|
|
11
|
+
- family-names: Stulova
|
|
12
|
+
given-names: Nataliia
|
|
13
|
+
- family-names: Hamolia
|
|
14
|
+
given-names: Vladyslav
|
|
15
|
+
preferred-citation:
|
|
16
|
+
type: generic
|
|
17
|
+
title: "Memory Belongs to the User: Portable Memory, an Open Standard Proposal for Cross-Vendor AI Memory"
|
|
18
|
+
authors:
|
|
19
|
+
- family-names: Kryvoblotskyi
|
|
20
|
+
given-names: Sergii
|
|
21
|
+
- family-names: Stulova
|
|
22
|
+
given-names: Nataliia
|
|
23
|
+
- family-names: Hamolia
|
|
24
|
+
given-names: Vladyslav
|
|
25
|
+
year: 2026
|
|
26
|
+
month: 7
|
|
27
|
+
url: "https://research.macpaw.com/publications/portable-memory"
|
|
28
|
+
notes: "Standards proposal — preprint for community review. MacPaw Research."
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# Contributor Covenant Code of Conduct
|
|
2
|
+
|
|
3
|
+
## Our Pledge
|
|
4
|
+
|
|
5
|
+
We as members, contributors, and leaders pledge to make participation in our community a
|
|
6
|
+
harassment-free experience for everyone, regardless of age, body size, visible or
|
|
7
|
+
invisible disability, ethnicity, sex characteristics, gender identity and expression,
|
|
8
|
+
level of experience, education, socio-economic status, nationality, personal appearance,
|
|
9
|
+
race, religion, or sexual identity and orientation.
|
|
10
|
+
|
|
11
|
+
We pledge to act and interact in ways that contribute to an open, welcoming, diverse,
|
|
12
|
+
inclusive, and healthy community.
|
|
13
|
+
|
|
14
|
+
## Our Standards
|
|
15
|
+
|
|
16
|
+
Examples of behavior that contributes to a positive environment include demonstrating
|
|
17
|
+
empathy and kindness, respecting differing opinions and experiences, giving and
|
|
18
|
+
gracefully accepting constructive feedback, and focusing on what is best for the
|
|
19
|
+
community.
|
|
20
|
+
|
|
21
|
+
Unacceptable behavior includes sexualized language or imagery, trolling or derogatory
|
|
22
|
+
comments, public or private harassment, publishing others' private information without
|
|
23
|
+
permission, and other conduct which could reasonably be considered inappropriate in a
|
|
24
|
+
professional setting.
|
|
25
|
+
|
|
26
|
+
## Enforcement Responsibilities
|
|
27
|
+
|
|
28
|
+
Community leaders are responsible for clarifying and enforcing these standards and will
|
|
29
|
+
take appropriate and fair corrective action in response to any behavior they deem
|
|
30
|
+
inappropriate, threatening, offensive, or harmful.
|
|
31
|
+
|
|
32
|
+
## Scope
|
|
33
|
+
|
|
34
|
+
This Code of Conduct applies within all community spaces and when an individual is
|
|
35
|
+
officially representing the community in public spaces.
|
|
36
|
+
|
|
37
|
+
## Enforcement
|
|
38
|
+
|
|
39
|
+
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to
|
|
40
|
+
the maintainers responsible for enforcement at **tech-research@macpaw.com**. All complaints
|
|
41
|
+
will be reviewed and investigated promptly and fairly. Community leaders are obligated to
|
|
42
|
+
respect the privacy and security of the reporter of any incident.
|
|
43
|
+
|
|
44
|
+
## Attribution
|
|
45
|
+
|
|
46
|
+
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.1,
|
|
47
|
+
available at
|
|
48
|
+
https://www.contributor-covenant.org/version/2/1/code_of_conduct.html.
|
|
49
|
+
|
|
50
|
+
[homepage]: https://www.contributor-covenant.org
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# Contributing to Portable Memory
|
|
2
|
+
|
|
3
|
+
Thanks for helping build an open, vendor-neutral memory format. Contributions of every
|
|
4
|
+
kind are welcome — SDK code, new adapters, other-language implementations, spec
|
|
5
|
+
clarifications, and conformance fixtures.
|
|
6
|
+
|
|
7
|
+
## Ways to contribute
|
|
8
|
+
|
|
9
|
+
- **Implement the spec** in another language. The format is defined by
|
|
10
|
+
[`Spec/portable-memory-spec.md`](Spec/portable-memory-spec.md) and the language-neutral
|
|
11
|
+
[`Schemas/`](Schemas); you don't need this Python SDK to interoperate.
|
|
12
|
+
- **Write an adapter** that maps another vendor's export into the model (see
|
|
13
|
+
`portable_memory/adapters/mem0.py` for the pattern — map what's modeled, keep the rest
|
|
14
|
+
in `ext`/`metadata` so a later export stays lossless).
|
|
15
|
+
- **Improve the SDK** — bug fixes, robustness, docs.
|
|
16
|
+
- **Propose a spec change** (see below).
|
|
17
|
+
|
|
18
|
+
## Building & testing
|
|
19
|
+
|
|
20
|
+
```sh
|
|
21
|
+
pip install -e .[dev]
|
|
22
|
+
pytest
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
The `[dev]` extra pulls in `pytest`, `jsonschema` (for validating `Schemas/` against
|
|
26
|
+
samples), and `cryptography` (so signing paths are exercised). Please add a test with any
|
|
27
|
+
behavior change; the reference in-memory store in the test suite shows the shape an adopter
|
|
28
|
+
implements by subclassing `PortableMemoryStore`.
|
|
29
|
+
|
|
30
|
+
## SDK changes vs spec changes
|
|
31
|
+
|
|
32
|
+
- **SDK change** (code/tests/docs, no wire-format impact): open a PR.
|
|
33
|
+
- **Spec change** (anything that alters the on-disk format, schemas, or a normative
|
|
34
|
+
requirement): open an issue using the **Spec change** template first, so the design can
|
|
35
|
+
be discussed before implementation. Format-affecting changes follow semver on the
|
|
36
|
+
manifest `format` field and are decided per [`GOVERNANCE.md`](GOVERNANCE.md).
|
|
37
|
+
|
|
38
|
+
Keep the spec, the JSON Schemas, the Python DTOs, and the sample fixture **in sync** — a PR
|
|
39
|
+
that changes one usually needs to touch the others. `pytest` validates the shipped fixture;
|
|
40
|
+
please also validate `Schemas/` against your samples. Because there are now two reference
|
|
41
|
+
SDKs (Python here and [Swift](https://github.com/MacPaw/portable-memory-swift)), a
|
|
42
|
+
format-affecting change must land in both before the format version is bumped — see
|
|
43
|
+
[`GOVERNANCE.md`](GOVERNANCE.md).
|
|
44
|
+
|
|
45
|
+
## Conventions
|
|
46
|
+
|
|
47
|
+
- Match the surrounding style; keep the SDK dependency-light. The core is **pure standard
|
|
48
|
+
library** (`json`, `hashlib`, `dataclasses`, `datetime`); the only optional dependency is
|
|
49
|
+
`cryptography`, behind the `[signing]` extra and imported lazily. Don't add runtime
|
|
50
|
+
dependencies to the core.
|
|
51
|
+
- The SDK is fully **synchronous** — local file I/O is done inline, no `async`/`await`.
|
|
52
|
+
(The Swift SDK uses `async` for actor isolation; the Python side has no such requirement.)
|
|
53
|
+
- Canonical JSON rules are normative (spec §1.1) — don't introduce serialization that
|
|
54
|
+
diverges from the canonical helpers in `portable_memory/_codec.py`. Serialize and hash
|
|
55
|
+
through those helpers so bytes match the Swift SDK exactly.
|
|
56
|
+
- Update [`CHANGELOG.md`](CHANGELOG.md) under **Unreleased**.
|
|
57
|
+
|
|
58
|
+
## Developer Certificate of Origin (DCO)
|
|
59
|
+
|
|
60
|
+
We use the [DCO](https://developercertificate.org/) rather than a CLA. Sign off each
|
|
61
|
+
commit (`git commit -s`), certifying you have the right to submit it under the project's
|
|
62
|
+
MIT license:
|
|
63
|
+
|
|
64
|
+
```
|
|
65
|
+
Signed-off-by: Your Name <you@example.com>
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
By contributing, you agree your contributions are licensed under the repository's
|
|
69
|
+
[MIT license](LICENSE).
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Conformance kit
|
|
2
|
+
|
|
3
|
+
How to claim a Portable Memory conformance level, and how to check a bundle.
|
|
4
|
+
|
|
5
|
+
## Levels
|
|
6
|
+
|
|
7
|
+
| Level | Requirement | How to demonstrate |
|
|
8
|
+
|---|---|---|
|
|
9
|
+
| **L0** | Read / Export | `BundleValidator().validate(bundle=...)` returns `ok` on your exported bundle. |
|
|
10
|
+
| **L1** | Import / Merge | Export → import into a fresh store reproduces counts/content; a second import is a no-op (idempotent). |
|
|
11
|
+
| **L2** | **Deletion propagation** (badge) | Delete an item, then prove it is unreachable via **every** route your engine supports + a tombstone was written (the probe below). |
|
|
12
|
+
| **L3** | Governed | Full audit trail + Evidence Pack export + signed tombstones. |
|
|
13
|
+
|
|
14
|
+
## Validate a bundle (L0)
|
|
15
|
+
|
|
16
|
+
```python
|
|
17
|
+
from portable_memory import BundleValidator
|
|
18
|
+
|
|
19
|
+
result = BundleValidator().validate(bundle="path/to/bundle.mem")
|
|
20
|
+
assert result.ok, "\n".join(result.issues)
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
The validator checks: the manifest parses, every listed file matches its `sha256` and
|
|
24
|
+
byte count, no unlisted/injected files are present, and every known-kind stream
|
|
25
|
+
decodes. Unknown (vendor) kinds are intentionally opaque and pass through.
|
|
26
|
+
|
|
27
|
+
Language-neutral validation: the `Schemas/` JSON Schemas cover `manifest.json`, every
|
|
28
|
+
`items/<kind>.jsonl` record kind, and the `audit/` records (`tombstone`, `log`), so a
|
|
29
|
+
non-Python implementer can validate a bundle without the SDK. Unknown (vendor) kinds have
|
|
30
|
+
no schema by design — they are opaque passthrough.
|
|
31
|
+
|
|
32
|
+
## The deletion-propagation probe (L2 — the gate)
|
|
33
|
+
|
|
34
|
+
This is the guarantee the badge rests on. For each of a handful of distinctive
|
|
35
|
+
"needle" memories:
|
|
36
|
+
|
|
37
|
+
1. Add it; confirm it is retrievable.
|
|
38
|
+
2. Delete it.
|
|
39
|
+
3. Assert it is unreachable via **every** route your engine exposes — at minimum:
|
|
40
|
+
dense vector search, lexical/full-text, any k-hop graph, any embedding/derived
|
|
41
|
+
cache, the bi-temporal `as_of` historical view, and any synced replica.
|
|
42
|
+
4. Assert a portable tombstone was written for it (positive proof, not mere absence).
|
|
43
|
+
|
|
44
|
+
A conformant `delete` reaches every artifact derived from the content; a conformant
|
|
45
|
+
`import` applies tombstones **before** additions, so a bundle that still carries the
|
|
46
|
+
stale rows can never resurrect deleted content.
|
|
47
|
+
|
|
48
|
+
The reference SDK demonstrates the offline, bundle-level part of this probe in
|
|
49
|
+
`tests/` — the tombstone-first / no-resurrection cases: tombstones are applied before
|
|
50
|
+
additions, so a stale bundle re-import can never resurrect a deleted item of **any**
|
|
51
|
+
kind. The live, all-routes assertion (dense / lexical / graph / cache / replica) is
|
|
52
|
+
completed by an adopter against its own retrieval stack.
|
|
53
|
+
|
|
54
|
+
## Fixtures
|
|
55
|
+
|
|
56
|
+
`fixtures/sample.mem/` is a small, valid bundle exercising episodes, an entity, an
|
|
57
|
+
edge, an episode carrying a foreign `ext` field, a vendor-specific passthrough kind,
|
|
58
|
+
and a tombstone (with proof-of-reach). Use it to test your importer and validator.
|
|
59
|
+
|
|
60
|
+
## Vectors
|
|
61
|
+
|
|
62
|
+
`vectors/` holds the language-neutral conformance oracle — `canonical-json.json`
|
|
63
|
+
(`input → canonical bytes → sha256`) and a `signed.mem` fixture with its test keypair.
|
|
64
|
+
Reproduce every vector and you are byte-interoperable with the reference SDKs. See
|
|
65
|
+
[`vectors/README.md`](vectors/README.md).
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
1c7a4081b2f8601c37633442837f62acc9d7afbc39de68f5024f77d092a40b8d audit/tombstones.jsonl
|
|
2
|
+
a46150e36da3bed28dead6afc6c14d39c6737758389c5ceb4a2638825bb77869 items/edge.jsonl
|
|
3
|
+
7473a09c9fec9ec42f7a1497314b7aa1477d16a827332d7f257605b2490cd44b items/entity.jsonl
|
|
4
|
+
67ef60b8686b8d5cfdde69700fc54ddb513caa97480c81228dcac11a6a737de5 items/episode.jsonl
|
|
5
|
+
83654ab56d668010b6d8957ec8f3888c6df14d1ff031b62f20908af1e44e1243 items/vendorThing.jsonl
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"actor":"user","deletedAt":"2024-03-09T16:00:00Z","derived":{"chunkIDs":[],"edgeIDs":[],"embeddingCacheKeys":["abc"],"entityIDs":[],"episodeLinkCount":0,"factIDs":["fact_x"],"sentenceIDs":[]},"id":"tomb_0001","op":"delete","reason":"user erasure","targetID":"ep_legacy","targetKind":"episode"}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"confidence":0.9,"dstEntityID":"ent_pv","edgeType":"launches_on","evidenceEpisodeIDs":["ep_0001"],"id":"edge_0001","ingestionTime":"2023-11-14T22:13:20Z","srcEntityID":"ent_pv","tValidFrom":"2023-11-14T22:13:20Z"}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"aliases":[],"canonicalName":"Project Vermilion","id":"ent_pv","sensitivity":"low","summary":"","type":"project","updatedAt":"2023-11-14T22:13:20Z"}
|
|
@@ -0,0 +1,2 @@
|
|
|
1
|
+
{"accessCount":0,"actors":[],"categories":[],"confidence":0.7,"details":"Project Vermilion launches on Pi Day 2027.","eventTime":"2023-11-14T22:13:20Z","extractionState":"done","id":"ep_0001","importance":0.5,"ingestionTime":"2023-11-14T22:13:20Z","lifecycleState":"HOT","mentionTime":"2023-11-14T22:13:20Z","metadata":{},"pinned":false,"sensitivity":"low","sourceType":"note","summary":"Project Vermilion launches on Pi Day 2027.","vaultRefs":[]}
|
|
2
|
+
{"accessCount":0,"actors":[],"categories":[],"confidence":0.7,"details":"The Reykjavik lab is led by Dr. Sabine Holt.","eventTime":"2023-11-14T22:13:20Z","extractionState":"done","id":"ep_0002","importance":0.5,"ingestionTime":"2023-11-14T22:13:20Z","lifecycleState":"HOT","mentionTime":"2023-11-14T22:13:20Z","metadata":{},"pinned":false,"sensitivity":"low","sourceType":"note","summary":"The Reykjavik lab is led by Dr. Sabine Holt.","vaultRefs":[],"vendorScore":0.92}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"id":"vt_1","blob":"opaque"}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"capabilities":["bitemporal","tombstones","redaction","evidence-pack","ext","passthrough"],"conformanceLevel":"L2","counts":{"edge":1,"entity":1,"episode":2,"tombstone":1,"vendorThing":1},"createdAt":"2026-06-30T20:23:59Z","embeddingDim":8,"embeddingModel":"test-embed","embeddingsIncluded":false,"exportMode":"full","files":[{"bytes":295,"path":"audit/tombstones.jsonl","sha256":"1c7a4081b2f8601c37633442837f62acc9d7afbc39de68f5024f77d092a40b8d"},{"bytes":215,"path":"items/edge.jsonl","sha256":"a46150e36da3bed28dead6afc6c14d39c6737758389c5ceb4a2638825bb77869"},{"bytes":150,"path":"items/entity.jsonl","sha256":"7473a09c9fec9ec42f7a1497314b7aa1477d16a827332d7f257605b2490cd44b"},{"bytes":919,"path":"items/episode.jsonl","sha256":"67ef60b8686b8d5cfdde69700fc54ddb513caa97480c81228dcac11a6a737de5"},{"bytes":30,"path":"items/vendorThing.jsonl","sha256":"83654ab56d668010b6d8957ec8f3888c6df14d1ff031b62f20908af1e44e1243"}],"format":"1.0.0","generator":"test/1.0","schemaVersion":1}
|
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
# Conformance vectors
|
|
2
|
+
|
|
3
|
+
Language-neutral test data that pins the parts of the format most likely to drift
|
|
4
|
+
between implementations: **Canonical JSON** serialization (spec §1.1) and **signature**
|
|
5
|
+
verification (§1.3). If your implementation reproduces these, it is byte-interoperable
|
|
6
|
+
with the reference SDKs.
|
|
7
|
+
|
|
8
|
+
These files are **byte-identical** in the Swift and Python repositories.
|
|
9
|
+
|
|
10
|
+
## `canonical-json.json`
|
|
11
|
+
|
|
12
|
+
An array of `vectors`, each with:
|
|
13
|
+
|
|
14
|
+
| field | meaning |
|
|
15
|
+
|---|---|
|
|
16
|
+
| `name` | what the case pins |
|
|
17
|
+
| `input` | a JSON value |
|
|
18
|
+
| `canonical` | the exact Canonical JSON serialization of `input` (spec §1.1) |
|
|
19
|
+
| `sha256` | `sha256( utf8(canonical) )`, lowercase hex |
|
|
20
|
+
|
|
21
|
+
**Conformance:** for every vector, `canonicalize(input)` MUST equal `canonical`
|
|
22
|
+
byte-for-byte, and its SHA-256 MUST equal `sha256`. A ~10-line test in any language does
|
|
23
|
+
this; see `tests/` in either SDK for the reference loaders.
|
|
24
|
+
|
|
25
|
+
The cases cover whole-valued floats (`1.0` → `1`), shortest-round-trip fractions,
|
|
26
|
+
negative zero, exact integers through `UInt64.max` (**integer fields beyond 2^53 require
|
|
27
|
+
a bigint-aware JSON parser** — `JSON.parse` in JS silently rounds them), raw non-ASCII,
|
|
28
|
+
short vs `\uXXXX` escapes, `/` unescaped, recursive key sorting, and array-order
|
|
29
|
+
preservation.
|
|
30
|
+
|
|
31
|
+
## `signed.mem` + `signing-test-key.json`
|
|
32
|
+
|
|
33
|
+
A complete, signed bundle plus the **test keypair** used to sign it (a fixed 32-byte seed
|
|
34
|
+
`00 01 … 1f` — obviously not a real secret). Use it to prove signature interop:
|
|
35
|
+
|
|
36
|
+
- **Verify:** load `publicKeyHex`, verify `manifest.sig` against `manifest.json` → MUST
|
|
37
|
+
pass; flip one byte of `manifest.json` → MUST fail.
|
|
38
|
+
- **Sign + verify:** load `privateKeyHex`, sign the manifest bytes, verify your own
|
|
39
|
+
signature with the public key → MUST pass.
|
|
40
|
+
|
|
41
|
+
> **Signatures are verify-interoperable, not byte-reproducible.** Ed25519 signature bytes
|
|
42
|
+
> may differ between implementations (swift-crypto randomizes the nonce; Python's
|
|
43
|
+
> `cryptography` is deterministic) — both are valid. So `manifest.sig` is **excluded** from
|
|
44
|
+
> the byte-identity guarantee of §1.1; the data files, `manifest.json`, and `CHECKSUMS`
|
|
45
|
+
> are covered, the signature is not.
|
|
46
|
+
|
|
47
|
+
## Known cross-implementation residuals (out of scope for v1)
|
|
48
|
+
|
|
49
|
+
The reference SDKs are **not** guaranteed byte-identical for these, so no vector asserts
|
|
50
|
+
them. Keep them ASCII / in-range where cross-impl byte-identity matters:
|
|
51
|
+
|
|
52
|
+
- **Object keys outside the BMP** (e.g. emoji as a JSON *key*): key ordering differs
|
|
53
|
+
(UTF-16 code-unit vs code-point). Native record keys are all ASCII; only exotic foreign
|
|
54
|
+
`ext` keys are affected.
|
|
55
|
+
- **Integral numeric magnitudes ≥ 1e16** and **integers beyond `UInt64.max`**: exponent
|
|
56
|
+
form and precision handling diverge. Real memory values (scores, counts) never reach
|
|
57
|
+
this range.
|
|
58
|
+
- **Record/line ordering for non-ASCII ids**: within-file ordering differs for non-ASCII
|
|
59
|
+
identifiers; use ASCII, kind-prefixed ids.
|
|
60
|
+
|
|
61
|
+
## Regenerating
|
|
62
|
+
|
|
63
|
+
`generate.py` (Python repo) rebuilds every file deterministically from the reference
|
|
64
|
+
encoder and the fixed test seed. Re-run it after any change to the canonicalization rules,
|
|
65
|
+
then confirm both SDK test suites stay green.
|