@cairnkeep/cli 1.1.0 → 1.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +40 -0
- package/CONTRIBUTING.md +6 -1
- package/README.md +42 -0
- package/docs/domain-knowledge.md +14 -10
- package/docs/operating.md +2 -0
- package/docs/privacy-and-data-flow.md +49 -0
- package/docs/releasing.md +33 -0
- package/docs/storage.md +24 -0
- package/examples/anythingllm/README.md +66 -0
- package/examples/anythingllm/anythingllm-projects.example.json +30 -0
- package/examples/anythingllm/sync_to_anythingllm.py +596 -0
- package/mcp-memory-server/dist/index.js +3 -4
- package/package.json +5 -3
- package/scripts/probe-memory-server.mjs +17 -5
- package/scripts/smoke-mcp-boot.sh +6 -13
- package/scripts/test-package-install.sh +4 -0
- package/templates/env.example.template +6 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable user-facing changes are documented here. This project follows
|
|
4
|
+
[Semantic Versioning](https://semver.org/).
|
|
5
|
+
|
|
6
|
+
## [1.1.1] - 2026-07-14
|
|
7
|
+
|
|
8
|
+
### Fixed
|
|
9
|
+
|
|
10
|
+
- Ship and resolve the default document-RAG sync helper from the npm package,
|
|
11
|
+
with user-owned XDG config/state paths and legacy-path compatibility.
|
|
12
|
+
- Make repository CI install locked root dependencies before running checks.
|
|
13
|
+
- Make the clean macOS bootstrap test create its target directory explicitly.
|
|
14
|
+
|
|
15
|
+
### Changed
|
|
16
|
+
|
|
17
|
+
- Add release-to-npm automation with provenance, tarball, and SBOM artifacts.
|
|
18
|
+
- Document supported clients, platforms, storage placement, and optional data
|
|
19
|
+
flows more explicitly.
|
|
20
|
+
- Add community contribution, support, and conduct templates.
|
|
21
|
+
|
|
22
|
+
## [1.1.0] - 2026-07-13
|
|
23
|
+
|
|
24
|
+
### Added
|
|
25
|
+
|
|
26
|
+
- Add authenticated remote HTTP memory with stable per-project session routing.
|
|
27
|
+
- Add explicit client routing headers for scopes and document-RAG workspaces.
|
|
28
|
+
- Document local, remote, export, backup, and bearer-token deployment models.
|
|
29
|
+
|
|
30
|
+
## [1.0.5] - 2026-07-13
|
|
31
|
+
|
|
32
|
+
### Fixed
|
|
33
|
+
|
|
34
|
+
- Make the npm tarball install self-contained and verify it on clean systems.
|
|
35
|
+
- Preserve executable permissions and Bash 3.2 portability on macOS.
|
|
36
|
+
- Add backup-first uninstall and SQLite-safe memory export/import guidance.
|
|
37
|
+
|
|
38
|
+
[1.1.1]: https://github.com/cairnkeep/cairnkeep/compare/v1.1.0...v1.1.1
|
|
39
|
+
[1.1.0]: https://github.com/cairnkeep/cairnkeep/compare/v1.0.5...v1.1.0
|
|
40
|
+
[1.0.5]: https://github.com/cairnkeep/cairnkeep/compare/v1.0.4...v1.0.5
|
package/CONTRIBUTING.md
CHANGED
|
@@ -15,7 +15,9 @@ Contributions are welcome through GitHub issues and pull requests.
|
|
|
15
15
|
|
|
16
16
|
## Development setup
|
|
17
17
|
|
|
18
|
-
The memory server requires Node.js 18 or newer. Node.js
|
|
18
|
+
The memory server requires Node.js 18 or newer. Node.js 22 or 24 is recommended
|
|
19
|
+
for development; current CI uses Node.js 24 and retains a clean Node.js 18
|
|
20
|
+
compatibility boot check.
|
|
19
21
|
|
|
20
22
|
```bash
|
|
21
23
|
git clone https://github.com/cairnkeep/cairnkeep.git
|
|
@@ -47,3 +49,6 @@ endpoints. They are not part of the default offline test suite.
|
|
|
47
49
|
|
|
48
50
|
By contributing, you agree that your contribution is licensed under the
|
|
49
51
|
Apache License 2.0 found in [LICENSE](LICENSE).
|
|
52
|
+
|
|
53
|
+
Release publishing is maintainer-only and documented in
|
|
54
|
+
[docs/releasing.md](docs/releasing.md).
|
package/README.md
CHANGED
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# Cairnkeep
|
|
2
2
|
|
|
3
|
+
[](https://github.com/cairnkeep/cairnkeep/actions/workflows/ci.yml)
|
|
4
|
+
[](https://www.npmjs.com/package/@cairnkeep/cli)
|
|
5
|
+
[](LICENSE)
|
|
6
|
+
|
|
3
7
|
> A durable, harness-agnostic **memory + context layer** for coding agents.
|
|
4
8
|
|
|
5
9
|
A *cairn* is a stack of stones left as a trail marker for whoever follows; a
|
|
@@ -20,6 +24,23 @@ a thin routing seam (`route_check`), both of which delegate to
|
|
|
20
24
|
[token-miser](https://github.com/cairnkeep/token-miser), a public
|
|
21
25
|
cairnkeep-org sibling project.
|
|
22
26
|
|
|
27
|
+
## Compatibility
|
|
28
|
+
|
|
29
|
+
Node.js 22 or 24 is recommended. Node.js 18 and 20 remain compatible for the
|
|
30
|
+
1.x line but are end-of-life upstream; the minimum will move in the next major
|
|
31
|
+
release.
|
|
32
|
+
|
|
33
|
+
| Client or platform | Support level |
|
|
34
|
+
|---|---|
|
|
35
|
+
| Claude Code on Linux/macOS | Memory server plus commands, agents, hooks, and launchers |
|
|
36
|
+
| OpenCode on Linux/macOS | Memory server plus commands, plugins, hooks, and launchers |
|
|
37
|
+
| Codex CLI | Memory MCP server; no Cairnkeep operating-layer assets |
|
|
38
|
+
| Other MCP clients | Memory and optional domain-knowledge MCP tools |
|
|
39
|
+
| Native Windows | Not supported by the Bash-based installer; use WSL (not yet CI-verified) |
|
|
40
|
+
|
|
41
|
+
Linux, macOS, Bash 3.2 portability, and a clean Node 18 compatibility install
|
|
42
|
+
are exercised in CI.
|
|
43
|
+
|
|
23
44
|
## Components
|
|
24
45
|
|
|
25
46
|
- **`mcp-memory-server/`** — an MCP server exposing durable, scoped memory
|
|
@@ -85,6 +106,19 @@ cd /path/to/project && cairn doctor
|
|
|
85
106
|
./.ai/start-claude.sh
|
|
86
107
|
```
|
|
87
108
|
|
|
109
|
+
After setup, the basic workflow is intentionally small:
|
|
110
|
+
|
|
111
|
+
```text
|
|
112
|
+
> /remember Use transactional migrations for schema changes
|
|
113
|
+
Stored as patterns/transactional-migrations.
|
|
114
|
+
|
|
115
|
+
> /recall transactional migrations
|
|
116
|
+
patterns/transactional-migrations: Use transactional migrations for schema changes
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
The exact command rendering depends on the client. Any MCP client can call
|
|
120
|
+
`memory_write` and `memory_search` directly.
|
|
121
|
+
|
|
88
122
|
Prefer working from a clone? Build the server with `cd mcp-memory-server && npm
|
|
89
123
|
install && npm run build`, then use `scripts/sync-claude-assets.sh` and
|
|
90
124
|
`bin/cairn` in place of the installed `cairn`.
|
|
@@ -139,8 +173,14 @@ search):
|
|
|
139
173
|
| `CAIRN_EXPLORE_REPO_ROOT` | Default repo root for `context_explore` when no per-call `repo_root` is given |
|
|
140
174
|
| `CAIRN_EXTRA_SETTINGS` | Optional settings/config file the launcher layers onto the harness (wrapper seam) |
|
|
141
175
|
| `CAIRN_ANYTHINGLLM_SYNC_SCRIPT` | Override path to the domain-knowledge sync script (when the integration lives outside the repo) |
|
|
176
|
+
| `CAIRN_ANYTHINGLLM_PROJECTS_FILE` | Override path to the bundled sync script's project configuration |
|
|
177
|
+
| `CAIRN_ANYTHINGLLM_STATE_FILE` | Override path to the bundled sync script's incremental state |
|
|
142
178
|
|
|
143
179
|
Without an API key, search degrades gracefully to substring matching.
|
|
180
|
+
There is no Cairnkeep telemetry. Optional extraction, embeddings, document RAG,
|
|
181
|
+
remote memory, and delegated exploration can send content to endpoints you
|
|
182
|
+
configure. Review [Privacy and data flow](docs/privacy-and-data-flow.md) before
|
|
183
|
+
enabling them.
|
|
144
184
|
|
|
145
185
|
## More
|
|
146
186
|
|
|
@@ -148,7 +188,9 @@ Without an API key, search degrades gracefully to substring matching.
|
|
|
148
188
|
- **Building a private overlay** (wrap cairnkeep for your org/provider) — [docs/building-an-overlay.md](docs/building-an-overlay.md)
|
|
149
189
|
- **Full operating guide** — [docs/operating.md](docs/operating.md)
|
|
150
190
|
- **Memory storage and deployment** — [docs/storage.md](docs/storage.md)
|
|
191
|
+
- **Privacy and data flow** — [docs/privacy-and-data-flow.md](docs/privacy-and-data-flow.md)
|
|
151
192
|
- **Git providers** — [docs/git-providers.md](docs/git-providers.md)
|
|
193
|
+
- **Support** — [SUPPORT.md](SUPPORT.md)
|
|
152
194
|
- **Contributing** — [CONTRIBUTING.md](CONTRIBUTING.md)
|
|
153
195
|
- **Security policy** — [SECURITY.md](SECURITY.md)
|
|
154
196
|
|
package/docs/domain-knowledge.md
CHANGED
|
@@ -28,7 +28,9 @@ work exactly the same without them.
|
|
|
28
28
|
|---|---|
|
|
29
29
|
| `ANYTHINGLLM_API_KEY` | **Required** for the domain-knowledge tools. Unset → the tools error at call time (everything else works). |
|
|
30
30
|
| `ANYTHINGLLM_BASE_URL` | AnythingLLM base URL. Default `http://localhost:3001`. |
|
|
31
|
-
| `CAIRN_ANYTHINGLLM_SYNC_SCRIPT` |
|
|
31
|
+
| `CAIRN_ANYTHINGLLM_SYNC_SCRIPT` | Optional path to a custom document-sync script. Unset -> `domain_knowledge_sync` uses the bundled deployment-neutral example. |
|
|
32
|
+
| `CAIRN_ANYTHINGLLM_PROJECTS_FILE` | Optional config path for the bundled sync script. Default `${XDG_CONFIG_HOME:-~/.config}/cairnkeep/anythingllm-projects.json`. |
|
|
33
|
+
| `CAIRN_ANYTHINGLLM_STATE_FILE` | Optional state path for the bundled sync script. Default `${XDG_STATE_HOME:-~/.local/state}/cairnkeep/anythingllm-sync.json`. |
|
|
32
34
|
|
|
33
35
|
The workspaces the tools target come from the project's **memory config** — the
|
|
34
36
|
first of `.agent/memory.json`, `.opencode/memory.json`, `.claude/memory.json`, or
|
|
@@ -73,10 +75,10 @@ returns a grounded answer.
|
|
|
73
75
|
|
|
74
76
|
## Setup (sync / embedding)
|
|
75
77
|
|
|
76
|
-
`domain_knowledge_sync` shells out to a document-upload/embed script.
|
|
77
|
-
|
|
78
|
-
deployment-specific,
|
|
79
|
-
at
|
|
78
|
+
`domain_knowledge_sync` shells out to a document-upload/embed script. Cairnkeep
|
|
79
|
+
ships a deployment-neutral example that reads its project mapping from a config
|
|
80
|
+
file. For deployment-specific authentication, checkout rules, or document
|
|
81
|
+
selection, keep a customized script in your overlay and point Cairnkeep at it:
|
|
80
82
|
|
|
81
83
|
```bash
|
|
82
84
|
CAIRN_ANYTHINGLLM_SYNC_SCRIPT=/abs/path/to/your/sync_to_anythingllm.py
|
|
@@ -86,11 +88,13 @@ The script receives the workspace slug and is expected to upload + embed the
|
|
|
86
88
|
project's documents into that AnythingLLM workspace. See
|
|
87
89
|
[building-an-overlay.md](building-an-overlay.md) for the wrapper pattern.
|
|
88
90
|
|
|
89
|
-
|
|
90
|
-
[`examples/anythingllm/`](../examples/anythingllm/): an incremental
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
91
|
+
The bundled default lives in
|
|
92
|
+
[`examples/anythingllm/`](../examples/anythingllm/): an incremental,
|
|
93
|
+
SHA-256-tracked multi-project sync script plus a config schema. Put a customized
|
|
94
|
+
copy of the example JSON at the default user config path, or set
|
|
95
|
+
`CAIRN_ANYTHINGLLM_PROJECTS_FILE`. Existing custom scripts with configuration
|
|
96
|
+
and state beside the script retain that legacy behavior. To customize the
|
|
97
|
+
implementation itself, set `CAIRN_ANYTHINGLLM_SYNC_SCRIPT` to the adapted copy.
|
|
94
98
|
|
|
95
99
|
## Not using RAG?
|
|
96
100
|
|
package/docs/operating.md
CHANGED
|
@@ -156,6 +156,8 @@ vendor or host.
|
|
|
156
156
|
| `ANYTHINGLLM_API_KEY` | Required to enable the optional `domain_knowledge_*` RAG tools (unset → those tools error at call time; nothing else affected). See [domain-knowledge.md](domain-knowledge.md) |
|
|
157
157
|
| `ANYTHINGLLM_BASE_URL` | AnythingLLM base URL for `domain_knowledge_*` (default `http://localhost:3001`) |
|
|
158
158
|
| `CAIRN_ANYTHINGLLM_SYNC_SCRIPT` | Override path to the `domain_knowledge_sync` document-sync script (unset → in-repo default) |
|
|
159
|
+
| `CAIRN_ANYTHINGLLM_PROJECTS_FILE` | Override path to the bundled sync script's project configuration |
|
|
160
|
+
| `CAIRN_ANYTHINGLLM_STATE_FILE` | Override path to the bundled sync script's incremental state |
|
|
159
161
|
|
|
160
162
|
### Domain knowledge (RAG via AnythingLLM, opt-in)
|
|
161
163
|
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Privacy and data flow
|
|
2
|
+
|
|
3
|
+
Cairnkeep has no telemetry, analytics collector, hosted control plane, or
|
|
4
|
+
automatic remote-service discovery. A default local stdio installation stores
|
|
5
|
+
memory in SQLite on the machine running the MCP server and makes no model or RAG
|
|
6
|
+
request unless the corresponding endpoint and credential are configured.
|
|
7
|
+
|
|
8
|
+
## Data flows
|
|
9
|
+
|
|
10
|
+
| Feature | Data that can leave the server machine | Destination |
|
|
11
|
+
|---|---|---|
|
|
12
|
+
| Local substring search | None | Local SQLite only |
|
|
13
|
+
| Embedding-ranked search | Stored memory values that are not cached, plus the search query | `CAIRN_MEMORY_EMBEDDING_URL` or `CAIRN_LLM_API_URL` |
|
|
14
|
+
| `memory_extract` | The transcript supplied to the tool | `CAIRN_LLM_API_URL` |
|
|
15
|
+
| `domain_knowledge_query` | Workspace slug and query | `ANYTHINGLLM_BASE_URL` |
|
|
16
|
+
| `domain_knowledge_sync` | Files selected by the sync configuration | `ANYTHINGLLM_BASE_URL` |
|
|
17
|
+
| `route_check` | A health request, with no memory or prompt content | `CAIRN_ROUTE_ENDPOINT` |
|
|
18
|
+
| `context_explore` | Repository path and query are passed to the configured local executable | `CAIRN_EXPLORE_BINARY`; any further data flow is controlled by that tool |
|
|
19
|
+
| Remote HTTP memory | MCP requests and responses, including memory content | The explicitly registered Cairnkeep HTTP server |
|
|
20
|
+
|
|
21
|
+
Model endpoints may be local or remote. Cairnkeep cannot determine a provider's
|
|
22
|
+
retention, training, or logging policy; verify it before sending confidential
|
|
23
|
+
material. Disabling `CAIRN_LLM_API_KEY`, `ANYTHINGLLM_API_KEY`, remote HTTP
|
|
24
|
+
registration, and delegated tools keeps the core memory workflow local.
|
|
25
|
+
|
|
26
|
+
## Data at rest
|
|
27
|
+
|
|
28
|
+
Memories are stored in SQLite databases. They are not encrypted by Cairnkeep at
|
|
29
|
+
the application layer. Use operating-system disk encryption, restrictive file
|
|
30
|
+
permissions, encrypted backups, and host access controls appropriate to the
|
|
31
|
+
sensitivity of the material. SQLite `-wal` and `-shm` sidecars and exported
|
|
32
|
+
archives can contain sensitive content too.
|
|
33
|
+
|
|
34
|
+
Project-scoped and named/global database locations are documented in
|
|
35
|
+
[Memory storage and deployment](storage.md). A remote client stores memory on
|
|
36
|
+
the remote server host; changing a storage environment variable on the client
|
|
37
|
+
does not relocate that server's databases.
|
|
38
|
+
|
|
39
|
+
## Credentials and transport
|
|
40
|
+
|
|
41
|
+
Keep API keys and bearer tokens out of repositories and command output. Load
|
|
42
|
+
them from a secret manager or protected environment file. For remote HTTP mode,
|
|
43
|
+
use TLS or an encrypted private network and keep the raw listener on loopback.
|
|
44
|
+
One HTTP bearer token grants access to the entire server; Cairnkeep does not
|
|
45
|
+
provide tenant isolation or per-scope authorization.
|
|
46
|
+
|
|
47
|
+
Before sharing diagnostics, remove credentials, private endpoints, database
|
|
48
|
+
files, memory values, local paths, and project names. Report vulnerabilities
|
|
49
|
+
through the private channel in [SECURITY.md](../SECURITY.md).
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Releasing
|
|
2
|
+
|
|
3
|
+
Publishing is driven by a GitHub Release. Maintainers do not publish the package
|
|
4
|
+
from a workstation.
|
|
5
|
+
|
|
6
|
+
## Release contract
|
|
7
|
+
|
|
8
|
+
1. Update the root `package.json` version and `CHANGELOG.md` on a pull request.
|
|
9
|
+
2. Run `npm ci`, `npm run check:public`, and
|
|
10
|
+
`npm --prefix mcp-memory-server test`.
|
|
11
|
+
3. Merge only after every required CI check passes.
|
|
12
|
+
4. Publish a GitHub Release whose tag is exactly `v<package.json version>`.
|
|
13
|
+
5. The `Publish release` workflow checks out that immutable tag, repeats the
|
|
14
|
+
tests, and publishes `@cairnkeep/cli` with npm provenance.
|
|
15
|
+
6. The workflow attaches the npm tarball, a CycloneDX SBOM, and SHA-256 checksums
|
|
16
|
+
to the GitHub Release.
|
|
17
|
+
|
|
18
|
+
Stable versions must use a stable GitHub Release and publish to npm's `latest`
|
|
19
|
+
tag. SemVer prerelease versions must use a GitHub prerelease and publish to
|
|
20
|
+
`next`. A mismatch fails before publication.
|
|
21
|
+
|
|
22
|
+
The workflow is safe to rerun: it skips `npm publish` when that exact version is
|
|
23
|
+
already present, then regenerates and replaces the release artifacts.
|
|
24
|
+
|
|
25
|
+
## Repository configuration
|
|
26
|
+
|
|
27
|
+
The repository must provide an `NPM_TOKEN` Actions secret authorized to publish
|
|
28
|
+
`@cairnkeep/cli`. Keep that credential out of local files and rotate it according
|
|
29
|
+
to the npm account's security policy. The workflow grants only `contents: write`
|
|
30
|
+
and `id-token: write`; the latter is required for npm provenance.
|
|
31
|
+
|
|
32
|
+
Treat a published version as immutable. If a release is wrong, fix it in a new
|
|
33
|
+
version rather than moving its tag or replacing the npm package.
|
package/docs/storage.md
CHANGED
|
@@ -129,8 +129,18 @@ claude mcp remove cairn-memory -s user 2>/dev/null || true
|
|
|
129
129
|
claude mcp add --transport http -s user \
|
|
130
130
|
cairn-memory "$CAIRN_MEMORY_REMOTE_URL" \
|
|
131
131
|
--header "Authorization: Bearer $CAIRN_MEMORY_HTTP_TOKEN"
|
|
132
|
+
|
|
133
|
+
codex mcp remove cairn-memory 2>/dev/null || true
|
|
134
|
+
codex mcp add cairn-memory \
|
|
135
|
+
--url "$CAIRN_MEMORY_REMOTE_URL" \
|
|
136
|
+
--bearer-token-env-var CAIRN_MEMORY_HTTP_TOKEN
|
|
132
137
|
```
|
|
133
138
|
|
|
139
|
+
Codex reads the bearer token from the named environment variable when it
|
|
140
|
+
starts. Do not run `codex mcp login cairn-memory`: that command starts an OAuth
|
|
141
|
+
flow, while Cairnkeep intentionally uses a static bearer token and advertises
|
|
142
|
+
no OAuth authorization endpoint.
|
|
143
|
+
|
|
134
144
|
OpenCode is configured separately in its per-user `opencode.json`. It supports
|
|
135
145
|
environment references in remote MCP headers, so the token does not need to be
|
|
136
146
|
written literally into the JSON:
|
|
@@ -161,3 +171,17 @@ expands `${VAR}` references in project `.mcp.json` URL and header values;
|
|
|
161
171
|
OpenCode uses `{env:VAR}` references. A private overlay should generate and
|
|
162
172
|
merge these files so secrets remain in the process environment rather than the
|
|
163
173
|
repository.
|
|
174
|
+
|
|
175
|
+
For Codex, put the routing configuration in the trusted project's private
|
|
176
|
+
`.codex/config.toml` and exclude it from version control:
|
|
177
|
+
|
|
178
|
+
```toml
|
|
179
|
+
[mcp_servers.cairn-memory]
|
|
180
|
+
url = "https://memory.example.com/mcp"
|
|
181
|
+
bearer_token_env_var = "CAIRN_MEMORY_HTTP_TOKEN"
|
|
182
|
+
|
|
183
|
+
[mcp_servers.cairn-memory.http_headers]
|
|
184
|
+
"X-Cairn-Project" = "example-project"
|
|
185
|
+
"X-Cairn-Scopes" = "identity,personal,project"
|
|
186
|
+
"X-Cairn-AnythingLLM-Workspaces" = "engineering-patterns"
|
|
187
|
+
```
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
# AnythingLLM sync — reference example
|
|
2
|
+
|
|
3
|
+
The bundled reference implementation of the document-sync script that
|
|
4
|
+
Cairnkeep's `domain_knowledge_sync` tool shells out to. It is a usable,
|
|
5
|
+
deployment-neutral starting point; copy and adapt it when endpoint,
|
|
6
|
+
authentication, or include/exclude rules are deployment-specific.
|
|
7
|
+
|
|
8
|
+
See [`../../docs/domain-knowledge.md`](../../docs/domain-knowledge.md) for the
|
|
9
|
+
full RAG picture.
|
|
10
|
+
|
|
11
|
+
## What it does
|
|
12
|
+
|
|
13
|
+
Incrementally syncs each configured project's docs into its own AnythingLLM
|
|
14
|
+
workspace: creates the workspace, uploads changed files (tracked by sha256),
|
|
15
|
+
embeds them, and removes docs that disappeared. Supports `--dry-run`, `--full`,
|
|
16
|
+
`--replace`, `--rebuild-state`, `--list`, and `--project <slug>`.
|
|
17
|
+
|
|
18
|
+
## Setup
|
|
19
|
+
|
|
20
|
+
1. Requires Python 3 and `requests` (`pip install requests`).
|
|
21
|
+
2. Copy the config to the default user config location and fill in your
|
|
22
|
+
projects:
|
|
23
|
+
```bash
|
|
24
|
+
mkdir -p "${XDG_CONFIG_HOME:-$HOME/.config}/cairnkeep"
|
|
25
|
+
cp anythingllm-projects.example.json \
|
|
26
|
+
"${XDG_CONFIG_HOME:-$HOME/.config}/cairnkeep/anythingllm-projects.json"
|
|
27
|
+
# edit: set each project's absolute path, slug, and include/exclude globs
|
|
28
|
+
```
|
|
29
|
+
3. Point at your AnythingLLM instance:
|
|
30
|
+
```bash
|
|
31
|
+
export ANYTHINGLLM_BASE_URL=http://localhost:3001 # default
|
|
32
|
+
export ANYTHINGLLM_API_KEY=<your AnythingLLM API key>
|
|
33
|
+
```
|
|
34
|
+
4. First upload, then keep it in sync:
|
|
35
|
+
```bash
|
|
36
|
+
python3 sync_to_anythingllm.py --full --project my-project
|
|
37
|
+
python3 sync_to_anythingllm.py # incremental, all projects
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
## Wiring into cairnkeep
|
|
41
|
+
|
|
42
|
+
The installed server uses this script by default. To use an adapted copy, set:
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
CAIRN_ANYTHINGLLM_SYNC_SCRIPT=/abs/path/to/sync_to_anythingllm.py
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Override the user-owned configuration or state locations when needed:
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
CAIRN_ANYTHINGLLM_PROJECTS_FILE=/abs/path/to/anythingllm-projects.json
|
|
52
|
+
CAIRN_ANYTHINGLLM_STATE_FILE=/abs/path/to/anythingllm-sync.json
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
For backward compatibility, an existing `anythingllm-projects.json` beside a
|
|
56
|
+
customized script and its `.anythingllm-sync.json` state are still used when the
|
|
57
|
+
override variables are unset.
|
|
58
|
+
|
|
59
|
+
## Files
|
|
60
|
+
|
|
61
|
+
| File | Committed? | Notes |
|
|
62
|
+
|---|---|---|
|
|
63
|
+
| `sync_to_anythingllm.py` | yes | the script |
|
|
64
|
+
| `anythingllm-projects.example.json` | yes | schema template |
|
|
65
|
+
| `anythingllm-projects.json` | **no** | User-owned config; defaults under the XDG config directory |
|
|
66
|
+
| `anythingllm-sync.json` | **no** | Machine-local SHA-256 state; defaults under the XDG state directory |
|
|
@@ -0,0 +1,30 @@
|
|
|
1
|
+
{
|
|
2
|
+
"projects": [
|
|
3
|
+
{
|
|
4
|
+
"slug": "my-project",
|
|
5
|
+
"display_name": "My Project Documentation",
|
|
6
|
+
"path": "/abs/path/to/my-project",
|
|
7
|
+
"patterns": [
|
|
8
|
+
"README.md",
|
|
9
|
+
"docs/**/*.md"
|
|
10
|
+
],
|
|
11
|
+
"exclude_patterns": [
|
|
12
|
+
"CHANGELOG.md",
|
|
13
|
+
"LICENSE.md"
|
|
14
|
+
]
|
|
15
|
+
},
|
|
16
|
+
{
|
|
17
|
+
"slug": "another-project",
|
|
18
|
+
"display_name": "Another Project Docs",
|
|
19
|
+
"path": "/abs/path/to/another-project",
|
|
20
|
+
"patterns": [
|
|
21
|
+
"AGENTS.md",
|
|
22
|
+
"docs/**/*.md",
|
|
23
|
+
".planning/wiki/**/*.md"
|
|
24
|
+
],
|
|
25
|
+
"exclude_patterns": [
|
|
26
|
+
".planning/wiki/tmp/**"
|
|
27
|
+
]
|
|
28
|
+
}
|
|
29
|
+
]
|
|
30
|
+
}
|
|
@@ -0,0 +1,596 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""
|
|
3
|
+
Multi-project incremental sync into AnythingLLM.
|
|
4
|
+
|
|
5
|
+
Each project repo gets its own AnythingLLM workspace. Projects are configured
|
|
6
|
+
in the file selected by CAIRN_ANYTHINGLLM_PROJECTS_FILE, the legacy file next to
|
|
7
|
+
this script when present, or the XDG user config directory.
|
|
8
|
+
|
|
9
|
+
This is the bundled reference implementation wired to Cairnkeep's
|
|
10
|
+
`domain_knowledge_sync` tool. See ../../docs/domain-knowledge.md.
|
|
11
|
+
|
|
12
|
+
Usage:
|
|
13
|
+
# Sync all configured projects (normal workflow after git pull):
|
|
14
|
+
python3 sync_to_anythingllm.py
|
|
15
|
+
|
|
16
|
+
# Only sync one project:
|
|
17
|
+
python3 sync_to_anythingllm.py --project my-project
|
|
18
|
+
|
|
19
|
+
# Preview without making changes:
|
|
20
|
+
python3 sync_to_anythingllm.py --dry-run
|
|
21
|
+
|
|
22
|
+
# First-time upload for a new project:
|
|
23
|
+
python3 sync_to_anythingllm.py --full --project my-project
|
|
24
|
+
|
|
25
|
+
# Replace stale embedded workspace docs, then upload current configured docs:
|
|
26
|
+
python3 sync_to_anythingllm.py --replace --project my-project
|
|
27
|
+
|
|
28
|
+
# Rebuild state from existing store (recovery, no re-upload):
|
|
29
|
+
python3 sync_to_anythingllm.py --rebuild-state --project my-project
|
|
30
|
+
|
|
31
|
+
# List all configured projects and file counts:
|
|
32
|
+
python3 sync_to_anythingllm.py --list
|
|
33
|
+
|
|
34
|
+
Config: ${XDG_CONFIG_HOME:-~/.config}/cairnkeep/anythingllm-projects.json
|
|
35
|
+
State: ${XDG_STATE_HOME:-~/.local/state}/cairnkeep/anythingllm-sync.json
|
|
36
|
+
|
|
37
|
+
State file format:
|
|
38
|
+
{
|
|
39
|
+
"project-a": { "docs/foo.md": {"location": "custom-documents/foo-uuid.json", "sha256": "..."} },
|
|
40
|
+
"project-b": { ... },
|
|
41
|
+
...
|
|
42
|
+
}
|
|
43
|
+
"""
|
|
44
|
+
|
|
45
|
+
import argparse
|
|
46
|
+
import fnmatch
|
|
47
|
+
import hashlib
|
|
48
|
+
import json
|
|
49
|
+
import os
|
|
50
|
+
import sys
|
|
51
|
+
import time
|
|
52
|
+
from pathlib import Path
|
|
53
|
+
|
|
54
|
+
import requests
|
|
55
|
+
|
|
56
|
+
# ---------------------------------------------------------------------------
|
|
57
|
+
# Configuration
|
|
58
|
+
# ---------------------------------------------------------------------------
|
|
59
|
+
|
|
60
|
+
BASE_URL = f"{os.environ.get('ANYTHINGLLM_BASE_URL', 'http://localhost:3001').rstrip('/')}/api/v1"
|
|
61
|
+
API_KEY = os.environ.get("ANYTHINGLLM_API_KEY", "")
|
|
62
|
+
|
|
63
|
+
SCRIPT_DIR = Path(__file__).parent
|
|
64
|
+
LEGACY_CONFIG_FILE = SCRIPT_DIR / "anythingllm-projects.json"
|
|
65
|
+
CONFIG_HOME = Path(os.environ.get("XDG_CONFIG_HOME", Path.home() / ".config"))
|
|
66
|
+
STATE_HOME = Path(os.environ.get("XDG_STATE_HOME", Path.home() / ".local" / "state"))
|
|
67
|
+
|
|
68
|
+
if os.environ.get("CAIRN_ANYTHINGLLM_PROJECTS_FILE"):
|
|
69
|
+
CONFIG_FILE = Path(os.environ["CAIRN_ANYTHINGLLM_PROJECTS_FILE"]).expanduser()
|
|
70
|
+
elif LEGACY_CONFIG_FILE.exists():
|
|
71
|
+
CONFIG_FILE = LEGACY_CONFIG_FILE
|
|
72
|
+
else:
|
|
73
|
+
CONFIG_FILE = CONFIG_HOME / "cairnkeep" / "anythingllm-projects.json"
|
|
74
|
+
|
|
75
|
+
if os.environ.get("CAIRN_ANYTHINGLLM_STATE_FILE"):
|
|
76
|
+
STATE_FILE = Path(os.environ["CAIRN_ANYTHINGLLM_STATE_FILE"]).expanduser()
|
|
77
|
+
elif CONFIG_FILE == LEGACY_CONFIG_FILE:
|
|
78
|
+
STATE_FILE = SCRIPT_DIR / ".anythingllm-sync.json"
|
|
79
|
+
else:
|
|
80
|
+
STATE_FILE = STATE_HOME / "cairnkeep" / "anythingllm-sync.json"
|
|
81
|
+
|
|
82
|
+
HEADERS = {"Authorization": f"Bearer {API_KEY}"}
|
|
83
|
+
JSON_HEADERS = {**HEADERS, "Content-Type": "application/json"}
|
|
84
|
+
|
|
85
|
+
EMBED_BATCH_SIZE = 5
|
|
86
|
+
EMBED_TIMEOUT = 600 # seconds per embed batch
|
|
87
|
+
|
|
88
|
+
WORKSPACE_SYSTEM_PROMPT = (
|
|
89
|
+
"You are an assistant for the {display_name} documentation. "
|
|
90
|
+
"Answer the user question using ONLY the context documents provided below. "
|
|
91
|
+
"Do not rely on prior knowledge or make assumptions beyond what is explicitly "
|
|
92
|
+
"stated in the documents.\n\n"
|
|
93
|
+
"Rules:\n"
|
|
94
|
+
"- If the answer is clearly present in the context, provide a precise and structured answer.\n"
|
|
95
|
+
"- If the context does not contain enough information to answer confidently, respond: "
|
|
96
|
+
'"This information is not available in the provided documentation."\n'
|
|
97
|
+
"- Never fabricate component names, port numbers, version numbers, or procedures "
|
|
98
|
+
"not in the context.\n"
|
|
99
|
+
"- Cite which document or section your answer comes from when possible."
|
|
100
|
+
)
|
|
101
|
+
|
|
102
|
+
|
|
103
|
+
def require_api_key() -> None:
|
|
104
|
+
if API_KEY:
|
|
105
|
+
return
|
|
106
|
+
print("Error: ANYTHINGLLM_API_KEY is not set.")
|
|
107
|
+
sys.exit(1)
|
|
108
|
+
|
|
109
|
+
# ---------------------------------------------------------------------------
|
|
110
|
+
# Config loading
|
|
111
|
+
# ---------------------------------------------------------------------------
|
|
112
|
+
|
|
113
|
+
def load_config() -> list[dict]:
|
|
114
|
+
if not CONFIG_FILE.exists():
|
|
115
|
+
print(f"Error: Config file not found: {CONFIG_FILE}")
|
|
116
|
+
sys.exit(1)
|
|
117
|
+
data = json.loads(CONFIG_FILE.read_text())
|
|
118
|
+
return data["projects"]
|
|
119
|
+
|
|
120
|
+
|
|
121
|
+
# ---------------------------------------------------------------------------
|
|
122
|
+
# State file helpers
|
|
123
|
+
# ---------------------------------------------------------------------------
|
|
124
|
+
|
|
125
|
+
def load_state() -> dict:
|
|
126
|
+
"""
|
|
127
|
+
Load full state: {project_slug: {relative_path: {location, sha256}}}
|
|
128
|
+
|
|
129
|
+
Automatically migrates a legacy flat (single-project) state file into the
|
|
130
|
+
nested multi-project format, keyed under the "default" slug.
|
|
131
|
+
"""
|
|
132
|
+
if not STATE_FILE.exists():
|
|
133
|
+
return {}
|
|
134
|
+
raw = json.loads(STATE_FILE.read_text())
|
|
135
|
+
if not raw:
|
|
136
|
+
return {}
|
|
137
|
+
# Detect old flat format: top-level values are {location, sha256} dicts
|
|
138
|
+
first_val = next(iter(raw.values()))
|
|
139
|
+
if isinstance(first_val, dict) and "location" in first_val:
|
|
140
|
+
print("[INFO] Migrating state file from flat to multi-project format...")
|
|
141
|
+
return {"default": raw}
|
|
142
|
+
return raw
|
|
143
|
+
|
|
144
|
+
|
|
145
|
+
def save_state(state: dict) -> None:
|
|
146
|
+
STATE_FILE.parent.mkdir(parents=True, exist_ok=True)
|
|
147
|
+
STATE_FILE.write_text(json.dumps(state, indent=2))
|
|
148
|
+
|
|
149
|
+
|
|
150
|
+
def file_sha256(path: Path) -> str:
|
|
151
|
+
h = hashlib.sha256()
|
|
152
|
+
h.update(path.read_bytes())
|
|
153
|
+
return h.hexdigest()
|
|
154
|
+
|
|
155
|
+
|
|
156
|
+
# ---------------------------------------------------------------------------
|
|
157
|
+
# File collection
|
|
158
|
+
# ---------------------------------------------------------------------------
|
|
159
|
+
|
|
160
|
+
def is_excluded(rel: str, excludes: list[str]) -> bool:
|
|
161
|
+
"""Return True if the relative path matches any exclude pattern."""
|
|
162
|
+
name = os.path.basename(rel)
|
|
163
|
+
for exc in excludes:
|
|
164
|
+
if fnmatch.fnmatch(rel, exc):
|
|
165
|
+
return True
|
|
166
|
+
if fnmatch.fnmatch(name, exc):
|
|
167
|
+
return True
|
|
168
|
+
return False
|
|
169
|
+
|
|
170
|
+
|
|
171
|
+
def collect_files(project: dict) -> dict[str, Path]:
|
|
172
|
+
"""
|
|
173
|
+
Return {relative_path_str: absolute_Path} for all files matching the
|
|
174
|
+
project's include patterns minus any exclude patterns. Paths are relative
|
|
175
|
+
to the project root.
|
|
176
|
+
"""
|
|
177
|
+
root = Path(project["path"])
|
|
178
|
+
patterns = project.get("patterns", ["docs/**/*.md"])
|
|
179
|
+
excludes = project.get("exclude_patterns", [])
|
|
180
|
+
|
|
181
|
+
result: dict[str, Path] = {}
|
|
182
|
+
for pattern in patterns:
|
|
183
|
+
for p in root.glob(pattern):
|
|
184
|
+
if not p.is_file():
|
|
185
|
+
continue
|
|
186
|
+
rel = str(p.relative_to(root))
|
|
187
|
+
if not is_excluded(rel, excludes):
|
|
188
|
+
result[rel] = p
|
|
189
|
+
return result
|
|
190
|
+
|
|
191
|
+
|
|
192
|
+
# ---------------------------------------------------------------------------
|
|
193
|
+
# AnythingLLM API helpers
|
|
194
|
+
# ---------------------------------------------------------------------------
|
|
195
|
+
|
|
196
|
+
def ensure_workspace(slug: str, display_name: str) -> str:
|
|
197
|
+
"""
|
|
198
|
+
Create the workspace if it doesn't exist, then apply standard RAG settings.
|
|
199
|
+
Returns the actual workspace slug to use for all subsequent API calls,
|
|
200
|
+
or empty string on failure.
|
|
201
|
+
|
|
202
|
+
Uses the project slug as the workspace name so AnythingLLM generates
|
|
203
|
+
a matching slug (e.g. name='my-project' -> slug='my-project').
|
|
204
|
+
"""
|
|
205
|
+
resp = requests.get(f"{BASE_URL}/workspaces", headers=HEADERS, timeout=10)
|
|
206
|
+
if resp.status_code == 200:
|
|
207
|
+
for ws in resp.json().get("workspaces", []):
|
|
208
|
+
if ws["slug"] == slug:
|
|
209
|
+
update_workspace_settings(slug, display_name)
|
|
210
|
+
return slug
|
|
211
|
+
|
|
212
|
+
# Create -- use slug as name so AnythingLLM generates the exact slug we want
|
|
213
|
+
resp = requests.post(
|
|
214
|
+
f"{BASE_URL}/workspace/new",
|
|
215
|
+
headers=JSON_HEADERS,
|
|
216
|
+
data=json.dumps({"name": slug}),
|
|
217
|
+
timeout=10,
|
|
218
|
+
)
|
|
219
|
+
if resp.status_code != 200:
|
|
220
|
+
print(f" [WARN] Could not create workspace '{slug}': {resp.status_code} {resp.text[:200]}")
|
|
221
|
+
return ""
|
|
222
|
+
|
|
223
|
+
actual_slug = resp.json().get("workspace", {}).get("slug", slug)
|
|
224
|
+
|
|
225
|
+
update_workspace_settings(actual_slug, display_name)
|
|
226
|
+
print(f" Created workspace: {actual_slug}")
|
|
227
|
+
return actual_slug
|
|
228
|
+
|
|
229
|
+
|
|
230
|
+
def update_workspace_settings(slug: str, display_name: str) -> None:
|
|
231
|
+
"""Apply standard RAG settings and project-specific system prompt."""
|
|
232
|
+
requests.post(
|
|
233
|
+
f"{BASE_URL}/workspace/{slug}/update",
|
|
234
|
+
headers=JSON_HEADERS,
|
|
235
|
+
data=json.dumps({
|
|
236
|
+
"openAiTemp": 0,
|
|
237
|
+
"topN": 6,
|
|
238
|
+
"similarityThreshold": 0.20,
|
|
239
|
+
"openAiPrompt": WORKSPACE_SYSTEM_PROMPT.format(display_name=display_name),
|
|
240
|
+
"queryRefusalResponse": f"This information is not available in the provided {display_name} documentation.",
|
|
241
|
+
"chatMode": "automatic",
|
|
242
|
+
}),
|
|
243
|
+
timeout=10,
|
|
244
|
+
)
|
|
245
|
+
|
|
246
|
+
|
|
247
|
+
def upload_file(filepath: Path) -> str | None:
|
|
248
|
+
"""Upload a single markdown file. Returns the location string or None."""
|
|
249
|
+
with open(filepath, "rb") as f:
|
|
250
|
+
resp = requests.post(
|
|
251
|
+
f"{BASE_URL}/document/upload",
|
|
252
|
+
headers=HEADERS,
|
|
253
|
+
files={"file": (filepath.name, f, "text/markdown")},
|
|
254
|
+
timeout=60,
|
|
255
|
+
)
|
|
256
|
+
if resp.status_code == 200:
|
|
257
|
+
docs = resp.json().get("documents", [])
|
|
258
|
+
if docs:
|
|
259
|
+
return docs[0].get("location")
|
|
260
|
+
print(f" [WARN] Upload failed for {filepath.name}: {resp.status_code} {resp.text[:200]}")
|
|
261
|
+
return None
|
|
262
|
+
|
|
263
|
+
|
|
264
|
+
def embed_locations(slug: str, locations: list[str]) -> bool:
|
|
265
|
+
"""Embed document locations into a workspace in batches."""
|
|
266
|
+
for i in range(0, len(locations), EMBED_BATCH_SIZE):
|
|
267
|
+
batch = locations[i: i + EMBED_BATCH_SIZE]
|
|
268
|
+
resp = requests.post(
|
|
269
|
+
f"{BASE_URL}/workspace/{slug}/update-embeddings",
|
|
270
|
+
headers=JSON_HEADERS,
|
|
271
|
+
data=json.dumps({"adds": batch, "deletes": []}),
|
|
272
|
+
timeout=EMBED_TIMEOUT,
|
|
273
|
+
)
|
|
274
|
+
if resp.status_code != 200:
|
|
275
|
+
print(f" [WARN] Embed failed: {resp.status_code} {resp.text[:200]}")
|
|
276
|
+
return False
|
|
277
|
+
time.sleep(1)
|
|
278
|
+
return True
|
|
279
|
+
|
|
280
|
+
|
|
281
|
+
def unembed_locations(slug: str, locations: list[str]) -> bool:
|
|
282
|
+
"""Remove document locations from a workspace (un-embed only, keeps store)."""
|
|
283
|
+
if not locations:
|
|
284
|
+
return True
|
|
285
|
+
resp = requests.post(
|
|
286
|
+
f"{BASE_URL}/workspace/{slug}/update-embeddings",
|
|
287
|
+
headers=JSON_HEADERS,
|
|
288
|
+
data=json.dumps({"adds": [], "deletes": locations}),
|
|
289
|
+
timeout=60,
|
|
290
|
+
)
|
|
291
|
+
return resp.status_code == 200
|
|
292
|
+
|
|
293
|
+
|
|
294
|
+
def get_workspace_locations(slug: str) -> list[str]:
|
|
295
|
+
"""Return document locations currently embedded in the workspace."""
|
|
296
|
+
resp = requests.get(f"{BASE_URL}/workspace/{slug}", headers=HEADERS, timeout=30)
|
|
297
|
+
if resp.status_code != 200:
|
|
298
|
+
print(f" [WARN] Could not inspect workspace '{slug}': {resp.status_code} {resp.text[:200]}")
|
|
299
|
+
return []
|
|
300
|
+
|
|
301
|
+
data = resp.json()
|
|
302
|
+
workspaces = data.get("workspace", [])
|
|
303
|
+
workspace = workspaces[0] if workspaces else {}
|
|
304
|
+
return [
|
|
305
|
+
doc.get("docpath", "")
|
|
306
|
+
for doc in workspace.get("documents", [])
|
|
307
|
+
if doc.get("docpath")
|
|
308
|
+
]
|
|
309
|
+
|
|
310
|
+
|
|
311
|
+
def delete_from_store(locations: list[str]) -> bool:
|
|
312
|
+
"""Permanently delete documents from the global document store."""
|
|
313
|
+
if not locations:
|
|
314
|
+
return True
|
|
315
|
+
resp = requests.delete(
|
|
316
|
+
f"{BASE_URL}/system/remove-documents",
|
|
317
|
+
headers=JSON_HEADERS,
|
|
318
|
+
data=json.dumps({"names": locations}),
|
|
319
|
+
timeout=60,
|
|
320
|
+
)
|
|
321
|
+
return resp.status_code == 200
|
|
322
|
+
|
|
323
|
+
|
|
324
|
+
# ---------------------------------------------------------------------------
|
|
325
|
+
# Diff logic
|
|
326
|
+
# ---------------------------------------------------------------------------
|
|
327
|
+
|
|
328
|
+
def compute_diff(
|
|
329
|
+
project_state: dict, files: dict[str, Path]
|
|
330
|
+
) -> tuple[list[str], list[str], list[str]]:
|
|
331
|
+
"""Return (added, modified, deleted) lists of relative path strings."""
|
|
332
|
+
added = [rel for rel in files if rel not in project_state]
|
|
333
|
+
deleted = [rel for rel in project_state if rel not in files]
|
|
334
|
+
modified = [
|
|
335
|
+
rel for rel, path in files.items()
|
|
336
|
+
if rel in project_state and file_sha256(path) != project_state[rel].get("sha256")
|
|
337
|
+
]
|
|
338
|
+
return added, modified, deleted
|
|
339
|
+
|
|
340
|
+
|
|
341
|
+
# ---------------------------------------------------------------------------
|
|
342
|
+
# Per-project operations
|
|
343
|
+
# ---------------------------------------------------------------------------
|
|
344
|
+
|
|
345
|
+
def full_sync_project(project: dict, state: dict, dry_run: bool) -> None:
|
|
346
|
+
slug = project["slug"]
|
|
347
|
+
display_name = project.get("display_name", slug)
|
|
348
|
+
files = collect_files(project)
|
|
349
|
+
|
|
350
|
+
print(f"\n=== Full sync: {display_name} ({len(files)} files) ===")
|
|
351
|
+
|
|
352
|
+
if dry_run:
|
|
353
|
+
for rel in sorted(files):
|
|
354
|
+
print(f" would upload: {rel}")
|
|
355
|
+
return
|
|
356
|
+
|
|
357
|
+
actual_slug = ensure_workspace(slug, display_name)
|
|
358
|
+
if not actual_slug:
|
|
359
|
+
return
|
|
360
|
+
|
|
361
|
+
project_state: dict = {}
|
|
362
|
+
locations: list[str] = []
|
|
363
|
+
|
|
364
|
+
print(" Uploading...")
|
|
365
|
+
for i, (rel, filepath) in enumerate(sorted(files.items()), 1):
|
|
366
|
+
print(f" [{i}/{len(files)}] {rel}")
|
|
367
|
+
loc = upload_file(filepath)
|
|
368
|
+
if loc:
|
|
369
|
+
project_state[rel] = {"location": loc, "sha256": file_sha256(filepath)}
|
|
370
|
+
locations.append(loc)
|
|
371
|
+
time.sleep(0.1)
|
|
372
|
+
|
|
373
|
+
print(f" Embedding {len(locations)} docs...")
|
|
374
|
+
ok = embed_locations(actual_slug, locations)
|
|
375
|
+
if not ok:
|
|
376
|
+
print(" [WARN] Some embed batches failed. State saved; retry with --full to re-embed.")
|
|
377
|
+
|
|
378
|
+
state[slug] = project_state
|
|
379
|
+
save_state(state)
|
|
380
|
+
print(f" Done. {len(locations)} docs uploaded to workspace '{actual_slug}'")
|
|
381
|
+
|
|
382
|
+
|
|
383
|
+
def replace_sync_project(project: dict, state: dict, dry_run: bool) -> None:
|
|
384
|
+
"""Remove current workspace docs, then full-sync configured project docs."""
|
|
385
|
+
slug = project["slug"]
|
|
386
|
+
display_name = project.get("display_name", slug)
|
|
387
|
+
files = collect_files(project)
|
|
388
|
+
|
|
389
|
+
print(f"\n=== Replace sync: {display_name} ({len(files)} files) ===")
|
|
390
|
+
|
|
391
|
+
if dry_run:
|
|
392
|
+
print(f" would remove embedded docs from workspace: {slug}")
|
|
393
|
+
for rel in sorted(files):
|
|
394
|
+
print(f" would upload: {rel}")
|
|
395
|
+
return
|
|
396
|
+
|
|
397
|
+
actual_slug = ensure_workspace(slug, display_name)
|
|
398
|
+
if not actual_slug:
|
|
399
|
+
return
|
|
400
|
+
|
|
401
|
+
existing_locations = get_workspace_locations(actual_slug)
|
|
402
|
+
if existing_locations:
|
|
403
|
+
print(f" Removing {len(existing_locations)} existing embedded doc(s)...")
|
|
404
|
+
unembed_locations(actual_slug, existing_locations)
|
|
405
|
+
delete_from_store(existing_locations)
|
|
406
|
+
|
|
407
|
+
state[slug] = {}
|
|
408
|
+
save_state(state)
|
|
409
|
+
full_sync_project(project, state, dry_run=False)
|
|
410
|
+
|
|
411
|
+
|
|
412
|
+
def incremental_sync_project(project: dict, state: dict, dry_run: bool) -> None:
|
|
413
|
+
slug = project["slug"]
|
|
414
|
+
display_name = project.get("display_name", slug)
|
|
415
|
+
actual_slug = slug if dry_run else (ensure_workspace(slug, display_name) or slug)
|
|
416
|
+
project_state = state.get(slug, {})
|
|
417
|
+
|
|
418
|
+
if not project_state:
|
|
419
|
+
print(
|
|
420
|
+
f"\n[{display_name}] No sync state found. "
|
|
421
|
+
f"Run --full --project {slug} for initial upload."
|
|
422
|
+
)
|
|
423
|
+
return
|
|
424
|
+
|
|
425
|
+
files = collect_files(project)
|
|
426
|
+
added, modified, deleted = compute_diff(project_state, files)
|
|
427
|
+
|
|
428
|
+
if not added and not modified and not deleted:
|
|
429
|
+
print(f"[{display_name}] Up to date.")
|
|
430
|
+
return
|
|
431
|
+
|
|
432
|
+
print(f"\n=== [{display_name}] Changes: +{len(added)} ~{len(modified)} -{len(deleted)} ===")
|
|
433
|
+
|
|
434
|
+
if dry_run:
|
|
435
|
+
for rel in sorted(added):
|
|
436
|
+
print(f" + {rel}")
|
|
437
|
+
for rel in sorted(modified):
|
|
438
|
+
print(f" ~ {rel}")
|
|
439
|
+
for rel in sorted(deleted):
|
|
440
|
+
print(f" - {rel}")
|
|
441
|
+
return
|
|
442
|
+
|
|
443
|
+
if deleted:
|
|
444
|
+
old_locs = [project_state[rel]["location"] for rel in deleted if "location" in project_state[rel]]
|
|
445
|
+
unembed_locations(actual_slug, old_locs)
|
|
446
|
+
delete_from_store(old_locs)
|
|
447
|
+
for rel in deleted:
|
|
448
|
+
del project_state[rel]
|
|
449
|
+
print(f" [DEL] {rel}")
|
|
450
|
+
state[slug] = project_state
|
|
451
|
+
save_state(state)
|
|
452
|
+
|
|
453
|
+
if modified:
|
|
454
|
+
old_locs = [project_state[rel]["location"] for rel in modified if "location" in project_state[rel]]
|
|
455
|
+
unembed_locations(actual_slug, old_locs)
|
|
456
|
+
delete_from_store(old_locs)
|
|
457
|
+
new_locs: list[str] = []
|
|
458
|
+
for rel in modified:
|
|
459
|
+
filepath = files[rel]
|
|
460
|
+
print(f" [MOD] {rel}")
|
|
461
|
+
loc = upload_file(filepath)
|
|
462
|
+
if loc:
|
|
463
|
+
project_state[rel] = {"location": loc, "sha256": file_sha256(filepath)}
|
|
464
|
+
new_locs.append(loc)
|
|
465
|
+
time.sleep(0.1)
|
|
466
|
+
if new_locs:
|
|
467
|
+
embed_locations(actual_slug, new_locs)
|
|
468
|
+
state[slug] = project_state
|
|
469
|
+
save_state(state)
|
|
470
|
+
|
|
471
|
+
if added:
|
|
472
|
+
new_locs = []
|
|
473
|
+
for rel in sorted(added):
|
|
474
|
+
filepath = files[rel]
|
|
475
|
+
print(f" [ADD] {rel}")
|
|
476
|
+
loc = upload_file(filepath)
|
|
477
|
+
if loc:
|
|
478
|
+
project_state[rel] = {"location": loc, "sha256": file_sha256(filepath)}
|
|
479
|
+
new_locs.append(loc)
|
|
480
|
+
time.sleep(0.1)
|
|
481
|
+
if new_locs:
|
|
482
|
+
embed_locations(actual_slug, new_locs)
|
|
483
|
+
state[slug] = project_state
|
|
484
|
+
save_state(state)
|
|
485
|
+
|
|
486
|
+
print(f" Done.")
|
|
487
|
+
|
|
488
|
+
|
|
489
|
+
def rebuild_state_project(project: dict, state: dict) -> None:
|
|
490
|
+
"""
|
|
491
|
+
Rebuild state by matching files against documents already in the store.
|
|
492
|
+
Useful for recovery when the state file is lost after docs are already uploaded.
|
|
493
|
+
Note: matches by filename (title); may be ambiguous if two projects share filenames.
|
|
494
|
+
"""
|
|
495
|
+
slug = project["slug"]
|
|
496
|
+
display_name = project.get("display_name", slug)
|
|
497
|
+
|
|
498
|
+
resp = requests.get(f"{BASE_URL}/documents", headers=HEADERS, timeout=30)
|
|
499
|
+
resp.raise_for_status()
|
|
500
|
+
|
|
501
|
+
store_docs: dict[str, dict] = {}
|
|
502
|
+
for folder in resp.json().get("localFiles", {}).get("items", []):
|
|
503
|
+
folder_name = folder.get("name", "")
|
|
504
|
+
for item in folder.get("items", []):
|
|
505
|
+
title = item.get("title", "")
|
|
506
|
+
loc = f"{folder_name}/{item['name']}"
|
|
507
|
+
if title not in store_docs or item.get("published", "") > store_docs[title]["published"]:
|
|
508
|
+
store_docs[title] = {"location": loc, "published": item.get("published", "")}
|
|
509
|
+
|
|
510
|
+
files = collect_files(project)
|
|
511
|
+
project_state: dict = {}
|
|
512
|
+
matched = 0
|
|
513
|
+
unmatched: list[str] = []
|
|
514
|
+
|
|
515
|
+
for rel, filepath in files.items():
|
|
516
|
+
title = filepath.name
|
|
517
|
+
if title in store_docs:
|
|
518
|
+
project_state[rel] = {
|
|
519
|
+
"location": store_docs[title]["location"],
|
|
520
|
+
"sha256": file_sha256(filepath),
|
|
521
|
+
}
|
|
522
|
+
matched += 1
|
|
523
|
+
else:
|
|
524
|
+
unmatched.append(rel)
|
|
525
|
+
|
|
526
|
+
state[slug] = project_state
|
|
527
|
+
save_state(state)
|
|
528
|
+
print(f"[{display_name}] Rebuilt: {matched} matched, {len(unmatched)} unmatched")
|
|
529
|
+
if unmatched:
|
|
530
|
+
for u in unmatched:
|
|
531
|
+
print(f" unmatched: {u}")
|
|
532
|
+
|
|
533
|
+
|
|
534
|
+
# ---------------------------------------------------------------------------
|
|
535
|
+
# Entry point
|
|
536
|
+
# ---------------------------------------------------------------------------
|
|
537
|
+
|
|
538
|
+
def main() -> None:
|
|
539
|
+
parser = argparse.ArgumentParser(
|
|
540
|
+
description="Multi-project AnythingLLM sync",
|
|
541
|
+
formatter_class=argparse.RawDescriptionHelpFormatter,
|
|
542
|
+
epilog=(
|
|
543
|
+
"examples:\n"
|
|
544
|
+
" sync all projects after git pull:\n"
|
|
545
|
+
" python3 sync_to_anythingllm.py\n\n"
|
|
546
|
+
" first-time upload for a new project:\n"
|
|
547
|
+
" python3 sync_to_anythingllm.py --full --project my-project\n\n"
|
|
548
|
+
" preview what would change:\n"
|
|
549
|
+
" python3 sync_to_anythingllm.py --dry-run\n"
|
|
550
|
+
),
|
|
551
|
+
)
|
|
552
|
+
parser.add_argument(
|
|
553
|
+
"--project", metavar="SLUG",
|
|
554
|
+
help="Only operate on this project (slug from anythingllm-projects.json).",
|
|
555
|
+
)
|
|
556
|
+
group = parser.add_mutually_exclusive_group()
|
|
557
|
+
group.add_argument("--full", action="store_true", help="Full re-upload and re-embed")
|
|
558
|
+
group.add_argument("--replace", action="store_true", help="Remove currently embedded workspace docs, then full re-upload and re-embed")
|
|
559
|
+
group.add_argument("--rebuild-state", action="store_true", help="Rebuild state from existing store (no upload)")
|
|
560
|
+
group.add_argument("--list", action="store_true", help="List configured projects and file counts")
|
|
561
|
+
parser.add_argument("--dry-run", action="store_true", help="Show what would change without doing it")
|
|
562
|
+
args = parser.parse_args()
|
|
563
|
+
|
|
564
|
+
projects = load_config()
|
|
565
|
+
slug_map = {p["slug"]: p for p in projects}
|
|
566
|
+
|
|
567
|
+
if args.list:
|
|
568
|
+
print(f"{'SLUG':<25} {'FILES':>6} PATH")
|
|
569
|
+
print("-" * 70)
|
|
570
|
+
for p in projects:
|
|
571
|
+
files = collect_files(p)
|
|
572
|
+
print(f"{p['slug']:<25} {len(files):>6} {p['path']}")
|
|
573
|
+
return
|
|
574
|
+
|
|
575
|
+
require_api_key()
|
|
576
|
+
|
|
577
|
+
if args.project and args.project not in slug_map:
|
|
578
|
+
print(f"Error: Unknown project '{args.project}'. Known: {', '.join(slug_map)}")
|
|
579
|
+
sys.exit(1)
|
|
580
|
+
|
|
581
|
+
target_projects = [slug_map[args.project]] if args.project else projects
|
|
582
|
+
state = load_state()
|
|
583
|
+
|
|
584
|
+
for project in target_projects:
|
|
585
|
+
if args.rebuild_state:
|
|
586
|
+
rebuild_state_project(project, state)
|
|
587
|
+
elif args.replace:
|
|
588
|
+
replace_sync_project(project, state, dry_run=args.dry_run)
|
|
589
|
+
elif args.full:
|
|
590
|
+
full_sync_project(project, state, dry_run=args.dry_run)
|
|
591
|
+
else:
|
|
592
|
+
incremental_sync_project(project, state, dry_run=args.dry_run)
|
|
593
|
+
|
|
594
|
+
|
|
595
|
+
if __name__ == "__main__":
|
|
596
|
+
main()
|
|
@@ -17,12 +17,11 @@ class ClientContextError extends Error {
|
|
|
17
17
|
}
|
|
18
18
|
const moduleDir = dirname(fileURLToPath(import.meta.url));
|
|
19
19
|
const infraRoot = resolve(moduleDir, "..", "..");
|
|
20
|
-
// Path to the
|
|
21
|
-
//
|
|
22
|
-
// (the integration scripts often live in the wrapper, not the core).
|
|
20
|
+
// Path to the bundled deployment-neutral sync script. An overlay can replace it
|
|
21
|
+
// with a deployment-specific implementation.
|
|
23
22
|
const anythingllmSyncScript = process.env.CAIRN_ANYTHINGLLM_SYNC_SCRIPT
|
|
24
23
|
? resolve(expandHome(process.env.CAIRN_ANYTHINGLLM_SYNC_SCRIPT))
|
|
25
|
-
: join(infraRoot, "anythingllm", "sync_to_anythingllm.py");
|
|
24
|
+
: join(infraRoot, "examples", "anythingllm", "sync_to_anythingllm.py");
|
|
26
25
|
const HISTORY_NAMESPACE = "__history__";
|
|
27
26
|
const PROJECT_ID_PATTERN = /^[a-z0-9][a-z0-9-]{0,63}$/;
|
|
28
27
|
const WORKSPACE_PATTERN = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/;
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@cairnkeep/cli",
|
|
3
|
-
"version": "1.1.
|
|
3
|
+
"version": "1.1.1",
|
|
4
4
|
"description": "A durable, harness-agnostic memory + context layer for coding agents (Claude Code, OpenCode).",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"type": "module",
|
|
@@ -16,7 +16,9 @@
|
|
|
16
16
|
"templates/",
|
|
17
17
|
"claude/",
|
|
18
18
|
"opencode/",
|
|
19
|
+
"examples/anythingllm/",
|
|
19
20
|
"docs/",
|
|
21
|
+
"CHANGELOG.md",
|
|
20
22
|
"CONTRIBUTING.md",
|
|
21
23
|
"SECURITY.md",
|
|
22
24
|
"mcp-memory-server/dist/",
|
|
@@ -36,8 +38,8 @@
|
|
|
36
38
|
"fix:perms": "git ls-files -s | awk '$1==\"100755\"{print $4}' | xargs chmod 755",
|
|
37
39
|
"prepack": "npm run build:server && npm run fix:perms",
|
|
38
40
|
"test": "for test in scripts/test-*.sh; do \"$test\" || exit 1; done",
|
|
39
|
-
"check:public": "scripts/verify-no-private-references.sh && scripts/verify-docs-parity.sh && npm test",
|
|
40
|
-
"prepublishOnly": "npm run check:public && npm
|
|
41
|
+
"check:public": "scripts/verify-no-private-references.sh && scripts/verify-docs-parity.sh && npm run build:server && npm test",
|
|
42
|
+
"prepublishOnly": "npm run check:public && npm --prefix mcp-memory-server test --global=false --dry-run=false"
|
|
41
43
|
},
|
|
42
44
|
"repository": {
|
|
43
45
|
"type": "git",
|
|
@@ -1,13 +1,21 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
import { spawn } from "node:child_process";
|
|
3
3
|
|
|
4
|
-
const
|
|
5
|
-
if (
|
|
6
|
-
console.error("usage: probe-memory-server.mjs PATH_TO_SERVER");
|
|
4
|
+
const probeArgs = process.argv.slice(2);
|
|
5
|
+
if (probeArgs.length === 0) {
|
|
6
|
+
console.error("usage: probe-memory-server.mjs PATH_TO_SERVER | --command COMMAND [ARGS...]");
|
|
7
7
|
process.exit(2);
|
|
8
8
|
}
|
|
9
9
|
|
|
10
|
-
const
|
|
10
|
+
const commandMode = probeArgs[0] === "--command";
|
|
11
|
+
const command = commandMode ? probeArgs[1] : process.execPath;
|
|
12
|
+
const commandArgs = commandMode ? probeArgs.slice(2) : [probeArgs[0]];
|
|
13
|
+
if (!command) {
|
|
14
|
+
console.error("probe-memory-server.mjs: --command requires a command");
|
|
15
|
+
process.exit(2);
|
|
16
|
+
}
|
|
17
|
+
|
|
18
|
+
const child = spawn(command, commandArgs, {
|
|
11
19
|
env: process.env,
|
|
12
20
|
stdio: ["pipe", "pipe", "pipe"],
|
|
13
21
|
});
|
|
@@ -29,7 +37,11 @@ const response = new Promise((resolve, reject) => {
|
|
|
29
37
|
if (!line.trim()) continue;
|
|
30
38
|
try {
|
|
31
39
|
const message = JSON.parse(line);
|
|
32
|
-
if (
|
|
40
|
+
if (
|
|
41
|
+
message.id === 1 &&
|
|
42
|
+
message.result?.serverInfo?.name === "cairn-memory" &&
|
|
43
|
+
message.result?.capabilities?.tools
|
|
44
|
+
) {
|
|
33
45
|
resolve();
|
|
34
46
|
return;
|
|
35
47
|
}
|
|
@@ -5,35 +5,28 @@
|
|
|
5
5
|
# transport works. This is the core product path, and nothing else tests it
|
|
6
6
|
# after packaging/installation.
|
|
7
7
|
#
|
|
8
|
-
# Prefers `cairn memory-server` (a global install), else falls back to
|
|
9
|
-
#
|
|
8
|
+
# Prefers `cairn memory-server` (a global install), else falls back to the
|
|
9
|
+
# repository build. The Node probe owns the timeout so this works on macOS,
|
|
10
|
+
# which does not ship GNU `timeout`.
|
|
10
11
|
set -uo pipefail
|
|
11
12
|
|
|
12
13
|
ROOT_DIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)
|
|
13
14
|
|
|
14
15
|
if command -v cairn >/dev/null 2>&1; then
|
|
15
|
-
|
|
16
|
+
probe=(node "$ROOT_DIR/scripts/probe-memory-server.mjs" --command cairn memory-server)
|
|
16
17
|
elif [[ -f "$ROOT_DIR/mcp-memory-server/dist/index.js" ]]; then
|
|
17
|
-
|
|
18
|
+
probe=(node "$ROOT_DIR/scripts/probe-memory-server.mjs" "$ROOT_DIR/mcp-memory-server/dist/index.js")
|
|
18
19
|
else
|
|
19
20
|
echo "smoke-mcp-boot: FAIL (no cairn on PATH and no dist build to run)"
|
|
20
21
|
exit 1
|
|
21
22
|
fi
|
|
22
23
|
|
|
23
|
-
req='{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"ci","version":"0"}}}'
|
|
24
|
-
|
|
25
24
|
tmp=$(mktemp -d)
|
|
26
25
|
trap 'rm -rf "$tmp"' EXIT
|
|
27
26
|
|
|
28
|
-
|
|
29
|
-
# lands there; `head -1` closes stdin after the first reply, so the server sees
|
|
30
|
-
# EOF and exits cleanly.
|
|
31
|
-
out=$(printf '%s\n' "$req" | HOME="$tmp" timeout 15 "${server[@]}" 2>/dev/null | head -1)
|
|
32
|
-
|
|
33
|
-
if grep -q '"result"' <<<"$out" && grep -q '"tools"' <<<"$out"; then
|
|
27
|
+
if HOME="$tmp" "${probe[@]}"; then
|
|
34
28
|
echo "smoke-mcp-boot: OK (server booted, initialize returned tools capability)"
|
|
35
29
|
exit 0
|
|
36
30
|
fi
|
|
37
31
|
echo "smoke-mcp-boot: FAIL (no valid initialize response)"
|
|
38
|
-
echo " got: ${out:0:200}"
|
|
39
32
|
exit 1
|
|
@@ -29,4 +29,8 @@ bootstrap_output=$(cairn bootstrap "$tmp/project")
|
|
|
29
29
|
|
|
30
30
|
(cd "$tmp/project" && cairn doctor) >/dev/null || fail "installed package failed cairn doctor"
|
|
31
31
|
|
|
32
|
+
installed_root="$tmp/prefix/lib/node_modules/@cairnkeep/cli"
|
|
33
|
+
[[ -f "$installed_root/examples/anythingllm/sync_to_anythingllm.py" ]] || \
|
|
34
|
+
fail "npm tarball omitted the default AnythingLLM sync script"
|
|
35
|
+
|
|
32
36
|
echo "PASS: npm tarball installs a self-contained CLI and MCP server"
|
|
@@ -27,3 +27,9 @@
|
|
|
27
27
|
# Optional: comma-separated allowed Host headers for DNS-rebinding protection
|
|
28
28
|
# (default: the bind host + localhost on the chosen port).
|
|
29
29
|
# CAIRN_MEMORY_HTTP_ALLOWED_HOSTS=
|
|
30
|
+
#
|
|
31
|
+
# Optional document-RAG sync. API credentials and endpoint may be local or remote.
|
|
32
|
+
# ANYTHINGLLM_BASE_URL=http://localhost:3001
|
|
33
|
+
# ANYTHINGLLM_API_KEY=
|
|
34
|
+
# CAIRN_ANYTHINGLLM_PROJECTS_FILE=~/.config/cairnkeep/anythingllm-projects.json
|
|
35
|
+
# CAIRN_ANYTHINGLLM_STATE_FILE=~/.local/state/cairnkeep/anythingllm-sync.json
|