@remnic/import-weclone 9.6.23 → 9.6.25
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +71 -181
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -1,12 +1,13 @@
|
|
|
1
1
|
# @remnic/import-weclone
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
(
|
|
5
|
-
|
|
6
|
-
tool usage.
|
|
3
|
+
Bootstrap a [Remnic](https://github.com/joshuaswarren/remnic) memory store from
|
|
4
|
+
[WeClone](https://github.com/xming521/weclone)-preprocessed chat exports
|
|
5
|
+
(Telegram, WhatsApp, Discord, Slack) instead of waiting for memory to accumulate
|
|
6
|
+
through daily AI-tool usage.
|
|
7
7
|
|
|
8
|
-
|
|
9
|
-
|
|
8
|
+
This is an **optional companion** package, installed separately. Importing it
|
|
9
|
+
registers the WeClone adapter with Remnic's core bulk-import registry as a side
|
|
10
|
+
effect — no explicit setup needed.
|
|
10
11
|
|
|
11
12
|
## Install
|
|
12
13
|
|
|
@@ -14,137 +15,88 @@ layer for AI agents.
|
|
|
14
15
|
npm install @remnic/import-weclone
|
|
15
16
|
```
|
|
16
17
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
`ensureWecloneImportAdapterRegistered()`.
|
|
22
|
-
|
|
23
|
-
## Why WeClone?
|
|
24
|
-
|
|
25
|
-
WeClone already handles the hard parts of chat ingestion:
|
|
26
|
-
|
|
27
|
-
- Platform-specific export parsing (Telegram JSON, WhatsApp, Discord, Slack)
|
|
28
|
-
- PII detection and redaction (Microsoft Presidio)
|
|
29
|
-
- Message deduplication and basic cleanup
|
|
30
|
-
|
|
31
|
-
Rather than duplicate that pipeline, this package consumes WeClone's
|
|
32
|
-
preprocessed JSON directly and maps it into the bulk-import contract defined by
|
|
33
|
-
`@remnic/core`.
|
|
34
|
-
|
|
35
|
-
## Pipeline
|
|
36
|
-
|
|
37
|
-
```
|
|
38
|
-
WeClone preprocessed JSON
|
|
39
|
-
|
|
|
40
|
-
v
|
|
41
|
-
+---------------------------------+
|
|
42
|
-
| parseWeCloneExport | parser.ts
|
|
43
|
-
| - platform resolution |
|
|
44
|
-
| - role inference (self/bot) |
|
|
45
|
-
| - schema validation |
|
|
46
|
-
+---------------+-----------------+
|
|
47
|
-
| BulkImportSource
|
|
48
|
-
v
|
|
49
|
-
+---------------------------------+
|
|
50
|
-
| groupIntoThreads | threader.ts
|
|
51
|
-
| - sort by timestamp |
|
|
52
|
-
| - split on >30 min time gaps |
|
|
53
|
-
| - merge via reply chains |
|
|
54
|
-
+---------------+-----------------+
|
|
55
|
-
| ThreadGroup[]
|
|
56
|
-
v
|
|
57
|
-
+---------------------------------+
|
|
58
|
-
| mapParticipants | participant.ts
|
|
59
|
-
| - count messages per sender |
|
|
60
|
-
| - classify self/frequent/ |
|
|
61
|
-
| occasional |
|
|
62
|
-
+---------------+-----------------+
|
|
63
|
-
| ParticipantEntity[]
|
|
64
|
-
v
|
|
65
|
-
+---------------------------------+
|
|
66
|
-
| chunkThreads | chunker.ts
|
|
67
|
-
| - split long threads with |
|
|
68
|
-
| overlap for context |
|
|
69
|
-
+---------------+-----------------+
|
|
70
|
-
| ImportTurn[][]
|
|
71
|
-
v
|
|
72
|
-
+---------------------------------+
|
|
73
|
-
| runBulkImportPipeline | @remnic/core
|
|
74
|
-
| - batch extraction |
|
|
75
|
-
| - dedup against existing |
|
|
76
|
-
| - store with trustLevel=import |
|
|
77
|
-
+---------------------------------+
|
|
78
|
-
```
|
|
18
|
+
WeClone already handles the hard parts of chat ingestion — platform-specific
|
|
19
|
+
export parsing, PII detection and redaction, and deduplication. Rather than
|
|
20
|
+
duplicate that, this package consumes WeClone's preprocessed JSON and maps it
|
|
21
|
+
into the bulk-import contract defined by `@remnic/core`.
|
|
79
22
|
|
|
80
23
|
## CLI usage
|
|
81
24
|
|
|
82
|
-
|
|
83
|
-
`@remnic/core
|
|
25
|
+
WeClone import runs through the orchestrator-backed `bulk-import` command in
|
|
26
|
+
`@remnic/core`, exposed under the `openclaw engram` command namespace (Remnic's
|
|
27
|
+
hosted CLI surface):
|
|
84
28
|
|
|
85
29
|
```bash
|
|
86
|
-
# Dry-run: parse, validate, and report counts without persisting
|
|
30
|
+
# Dry-run: parse, validate, and report counts without persisting.
|
|
87
31
|
openclaw engram bulk-import \
|
|
88
32
|
--source weclone \
|
|
89
33
|
--file ./preprocessed_telegram.json \
|
|
90
34
|
--platform telegram \
|
|
91
35
|
--dry-run
|
|
92
36
|
|
|
93
|
-
# Persist: run extraction over the export and write memories to disk
|
|
37
|
+
# Persist: run extraction over the export and write memories to disk.
|
|
94
38
|
openclaw engram bulk-import \
|
|
95
39
|
--source weclone \
|
|
96
40
|
--file ./preprocessed_telegram.json \
|
|
97
41
|
--platform telegram
|
|
42
|
+
```
|
|
98
43
|
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
44
|
+
Persistence flows through the Remnic orchestrator's extraction pipeline, so each
|
|
45
|
+
batch is extracted the same way an organic conversation would be. Memories land
|
|
46
|
+
under the orchestrator's default-namespace root (`memoryDir/facts/` by default)
|
|
47
|
+
tagged `trustLevel: "import"`. Use `--dry-run` to validate an export before
|
|
48
|
+
committing to the extraction cost.
|
|
49
|
+
|
|
50
|
+
> The generic `remnic import --adapter <name>` command covers the ChatGPT,
|
|
51
|
+
> Claude, Gemini, Mem0, and Supermemory importers. WeClone uses the core
|
|
52
|
+
> bulk-import source registry instead, which is why it runs through
|
|
53
|
+
> `bulk-import` rather than `remnic import`.
|
|
54
|
+
|
|
55
|
+
## Supported platforms
|
|
56
|
+
|
|
57
|
+
| Platform | `--platform` value |
|
|
58
|
+
|-----------|--------------------|
|
|
59
|
+
| Telegram | `telegram` |
|
|
60
|
+
| WhatsApp | `whatsapp` |
|
|
61
|
+
| Discord | `discord` |
|
|
62
|
+
| Slack | `slack` |
|
|
63
|
+
|
|
64
|
+
The parser defaults to `telegram` when no platform is given. Unknown platforms
|
|
65
|
+
are rejected.
|
|
66
|
+
|
|
67
|
+
## Input schema
|
|
68
|
+
|
|
69
|
+
The parser accepts either a wrapper object with a `messages` array or a raw array
|
|
70
|
+
of messages. Required per-message fields: `sender`, `text`, `timestamp`
|
|
71
|
+
(ISO-8601); optional: `message_id`, `reply_to_id`.
|
|
72
|
+
|
|
73
|
+
```json
|
|
74
|
+
{
|
|
75
|
+
"platform": "telegram",
|
|
76
|
+
"messages": [
|
|
77
|
+
{ "sender": "Alice", "text": "hello", "timestamp": "2025-01-10T08:00:00.000Z" }
|
|
78
|
+
]
|
|
79
|
+
}
|
|
105
80
|
```
|
|
106
81
|
|
|
107
|
-
|
|
108
|
-
Each batch is buffered and extracted the same way an organic conversation
|
|
109
|
-
would be; memories land under the orchestrator's default-namespace root
|
|
110
|
-
(`memoryDir/facts/` by default). Use `--dry-run` to validate an export
|
|
111
|
-
before committing to the extraction cost. Per-invocation namespace
|
|
112
|
-
override for bulk-import writes is not yet wired — see the note in
|
|
113
|
-
[`docs/import-export.md`](../../docs/import-export.md).
|
|
82
|
+
## Programmatic use
|
|
114
83
|
|
|
115
|
-
|
|
84
|
+
The adapter registers automatically on import; look it up from the core registry,
|
|
85
|
+
or drive the pipeline stages directly:
|
|
116
86
|
|
|
117
87
|
```ts
|
|
118
88
|
import { readFileSync } from "node:fs";
|
|
119
|
-
import {
|
|
120
|
-
|
|
121
|
-
groupIntoThreads,
|
|
122
|
-
mapParticipants,
|
|
123
|
-
chunkThreads,
|
|
124
|
-
// Side-effect import also registers the bulk-import adapter; see `index.ts`.
|
|
125
|
-
wecloneImportAdapter,
|
|
126
|
-
} from "@remnic/import-weclone";
|
|
127
|
-
import {
|
|
128
|
-
getBulkImportSource,
|
|
129
|
-
runBulkImportPipeline,
|
|
130
|
-
} from "@remnic/core";
|
|
131
|
-
|
|
132
|
-
// The adapter is registered automatically on import. Look it up:
|
|
133
|
-
const adapter = getBulkImportSource("weclone");
|
|
134
|
-
|
|
135
|
-
// 1) Parse a WeClone-preprocessed export.
|
|
136
|
-
const raw = JSON.parse(readFileSync("./export.json", "utf8"));
|
|
137
|
-
const source = parseWeCloneExport(raw, { platform: "telegram" });
|
|
89
|
+
import { parseWeCloneExport } from "@remnic/import-weclone";
|
|
90
|
+
import { getBulkImportSource, runBulkImportPipeline } from "@remnic/core";
|
|
138
91
|
|
|
139
|
-
|
|
140
|
-
const
|
|
141
|
-
|
|
142
|
-
|
|
92
|
+
const adapter = getBulkImportSource("weclone");
|
|
93
|
+
const source = parseWeCloneExport(
|
|
94
|
+
JSON.parse(readFileSync("./export.json", "utf8")),
|
|
95
|
+
{ platform: "telegram" },
|
|
96
|
+
);
|
|
143
97
|
|
|
144
|
-
//
|
|
145
|
-
//
|
|
146
|
-
// callback that wraps `orchestrator.ingestBulkImportBatch` (see
|
|
147
|
-
// `openclaw engram bulk-import`).
|
|
98
|
+
// dryRun never calls the batch callback; real persistence supplies an
|
|
99
|
+
// ingest callback wired to the orchestrator (see `openclaw engram bulk-import`).
|
|
148
100
|
const result = await runBulkImportPipeline(
|
|
149
101
|
source,
|
|
150
102
|
{ batchSize: 20, dryRun: true, dedup: true, trustLevel: "import" },
|
|
@@ -152,76 +104,14 @@ const result = await runBulkImportPipeline(
|
|
|
152
104
|
);
|
|
153
105
|
```
|
|
154
106
|
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
| Platform | `--platform` value |
|
|
158
|
-
|-----------|--------------------|
|
|
159
|
-
| Telegram | `telegram` |
|
|
160
|
-
| WhatsApp | `whatsapp` |
|
|
161
|
-
| Discord | `discord` |
|
|
162
|
-
| Slack | `slack` |
|
|
163
|
-
|
|
164
|
-
The parser defaults to `telegram` when neither `options.platform` nor an
|
|
165
|
-
export-level `platform` field is provided. Unknown platforms are rejected.
|
|
107
|
+
Tests that call `clearBulkImportSources()` can re-register the adapter via
|
|
108
|
+
`ensureWecloneImportAdapterRegistered()`.
|
|
166
109
|
|
|
167
|
-
##
|
|
110
|
+
## Further reading
|
|
168
111
|
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
1. A wrapper object:
|
|
172
|
-
|
|
173
|
-
```json
|
|
174
|
-
{
|
|
175
|
-
"platform": "telegram",
|
|
176
|
-
"export_date": "2025-01-10T00:00:00.000Z",
|
|
177
|
-
"messages": [
|
|
178
|
-
{
|
|
179
|
-
"sender": "Alice",
|
|
180
|
-
"text": "hello",
|
|
181
|
-
"timestamp": "2025-01-10T08:00:00.000Z",
|
|
182
|
-
"message_id": "m-001",
|
|
183
|
-
"reply_to_id": "m-000"
|
|
184
|
-
}
|
|
185
|
-
]
|
|
186
|
-
}
|
|
187
|
-
```
|
|
188
|
-
|
|
189
|
-
2. A raw array of messages (platform defaults to `telegram`).
|
|
190
|
-
|
|
191
|
-
Required per-message fields: `sender`, `text`, `timestamp` (ISO-8601). Optional:
|
|
192
|
-
`message_id`, `reply_to_id`.
|
|
193
|
-
|
|
194
|
-
## Role inference
|
|
195
|
-
|
|
196
|
-
Each `sender` is mapped to one of the `ImportTurn` roles:
|
|
197
|
-
|
|
198
|
-
- `user` - the "self" sender (first non-bot sender encountered, or override
|
|
199
|
-
via `selfSender`).
|
|
200
|
-
- `assistant` - any sender matching a bot heuristic (`bot`, `assistant`, `ai`,
|
|
201
|
-
`chatgpt`, `gpt`, `claude`, `copilot`, `llama`) or listed in
|
|
202
|
-
`assistantSenders`.
|
|
203
|
-
- `other` - everyone else.
|
|
204
|
-
|
|
205
|
-
The heuristic uses word-boundary matching so human names that happen to contain
|
|
206
|
-
substrings like `ai` (e.g. `Aidan`, `Caitlin`) are not mis-classified.
|
|
207
|
-
|
|
208
|
-
## Design notes
|
|
209
|
-
|
|
210
|
-
- **Imported memories are tagged with `trustLevel: "import"`** in the
|
|
211
|
-
pipeline options so downstream ranking/dedup can reason about their
|
|
212
|
-
provenance. The confidence-weighted ranking discount described in the
|
|
213
|
-
original issue is tracked as a follow-up — today the flag is plumbed but
|
|
214
|
-
not consumed by the recall ranker.
|
|
215
|
-
- **Threads are conversation boundaries, not days.** The default 30-minute
|
|
216
|
-
gap with reply-chain merging produces coherent extraction batches without
|
|
217
|
-
relying on calendar boundaries.
|
|
218
|
-
- **Entity bootstrapping is best-effort.** `mapParticipants` emits
|
|
219
|
-
lightweight `ParticipantEntity` records; the core entity graph is populated
|
|
220
|
-
as the pipeline processes chunks.
|
|
221
|
-
- **Idempotent re-imports.** The pipeline's dedup pass (when wired)
|
|
222
|
-
fingerprint-matches against existing memories, so re-running the importer
|
|
223
|
-
after appending new messages is safe.
|
|
112
|
+
- Import/export guide: [docs/import-export.md](https://github.com/joshuaswarren/remnic/blob/main/docs/import-export.md)
|
|
113
|
+
- Monorepo: [github.com/joshuaswarren/remnic](https://github.com/joshuaswarren/remnic)
|
|
224
114
|
|
|
225
115
|
## License
|
|
226
116
|
|
|
227
|
-
MIT
|
|
117
|
+
MIT
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@remnic/import-weclone",
|
|
3
|
-
"version": "9.6.
|
|
3
|
+
"version": "9.6.25",
|
|
4
4
|
"description": "Import WeClone-preprocessed chat exports to bootstrap Remnic memory",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "dist/index.js",
|
|
@@ -19,13 +19,13 @@
|
|
|
19
19
|
"provenance": true
|
|
20
20
|
},
|
|
21
21
|
"peerDependencies": {
|
|
22
|
-
"@remnic/core": "^9.6.
|
|
22
|
+
"@remnic/core": "^9.6.25"
|
|
23
23
|
},
|
|
24
24
|
"devDependencies": {
|
|
25
25
|
"tsup": "^8.0.0",
|
|
26
26
|
"typescript": "^5.7.0",
|
|
27
27
|
"tsx": "^4.0.0",
|
|
28
|
-
"@remnic/core": "9.6.
|
|
28
|
+
"@remnic/core": "9.6.25"
|
|
29
29
|
},
|
|
30
30
|
"license": "MIT",
|
|
31
31
|
"repository": {
|