decant-core 1.6.0 → 1.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +109 -81
- package/package.json +10 -3
package/README.md
CHANGED
|
@@ -1,116 +1,144 @@
|
|
|
1
1
|
# decant-core
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**A shared extraction layer for the modern web — including AI conversations.**
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
[](https://www.npmjs.com/package/decant-core)
|
|
6
|
+
[](LICENSE)
|
|
7
|
+
[](https://github.com/Covai-Labs/decant-core/stargazers)
|
|
6
8
|
|
|
7
|
-
|
|
9
|
+
Every AI chat exporter ends up solving the same problem: extracting conversations from ChatGPT, Claude, Gemini, Perplexity, DeepSeek and other constantly changing AI interfaces.
|
|
10
|
+
|
|
11
|
+
And every time one of those platforms changes its UI, seriously re-renders a message, or ships a new feature, **somebody's parser breaks**.
|
|
8
12
|
|
|
9
|
-
|
|
10
|
-
| ------------------- | ------------------------- |
|
|
11
|
-
| ChatGPT | `ChatGPTParser` |
|
|
12
|
-
| Claude | `ClaudeParser` |
|
|
13
|
-
| Copilot | `CopilotParser` |
|
|
14
|
-
| DeepSeek | `DeepSeekParser` |
|
|
15
|
-
| Gemini | `GeminiParser` |
|
|
16
|
-
| Gemini Cloud Assist | `GeminiCloudAssistParser` |
|
|
17
|
-
| Google AI Studio | `GoogleAIStudioParser` |
|
|
18
|
-
| Google Search AI | `GoogleSearchAIParser` |
|
|
19
|
-
| Lumo | `LumoParser` |
|
|
20
|
-
| Meta AI | `MetaParser` |
|
|
21
|
-
| Mistral | `MistralParser` |
|
|
22
|
-
| NotebookLM | `NotebookLMParser` |
|
|
23
|
-
| Perplexity | `PerplexityParser` |
|
|
24
|
-
| Qwen | `QwenParser` |
|
|
25
|
-
| Z AI | `ZAiParser` |
|
|
26
|
-
|
|
27
|
-
## Installation
|
|
13
|
+
`decant-core` provides reusable parsers, platform detection, and web article extraction so developers don't have to build and maintain the same fragile parsing layer over and over again.
|
|
28
14
|
|
|
29
15
|
```bash
|
|
30
16
|
npm install decant-core
|
|
31
17
|
```
|
|
32
18
|
|
|
33
|
-
|
|
19
|
+
Use it as the parsing layer underneath your own:
|
|
34
20
|
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
21
|
+
- chat exporters
|
|
22
|
+
- browser extensions
|
|
23
|
+
- web clippers
|
|
24
|
+
- research & data-extraction tools
|
|
25
|
+
- content archivers
|
|
26
|
+
- knowledge-management and PKM applications
|
|
27
|
+
|
|
28
|
+
**Fix platform parsing once, and let the ecosystem benefit from the fix.**
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## Why decant-core?
|
|
33
|
+
|
|
34
|
+
AI platforms don't expose stable public APIs for reading conversation history. No matter what you build, to extract a ChatGPT thread you need to walk the DOM, read internal RPC payloads, or traverse React component trees — and redo it when the frontend changes.
|
|
35
|
+
|
|
36
|
+
Maintaining that per-platform logic in every exporter is wasteful and fragile. `decant-core` centralizes it:
|
|
37
|
+
|
|
38
|
+
- ✅ **17 AI chat platform parsers** with normalized output — you get structured messages, models, metadata and Markdown, not DOM soup.
|
|
39
|
+
- ✅ **Web article extraction** — Mozilla Readability, Defuddle, and Article-Extractor run in parallel and arbitrate by content-quality scoring.
|
|
40
|
+
- ✅ **Detection utilities** — tell an "AI chat page" apart from a "regular web page" before you decide which parser to run.
|
|
41
|
+
- ✅ **Math & Markdown handling** — LaTeX normalization plus GFM tables/code fencing that survive round-trips into Obsidian, Logseq and Notion.
|
|
42
|
+
|
|
43
|
+
The payoff is maintenance: **when a platform changes, the fix happens once, in one place**, instead of being independently reimplemented across dozens of projects.
|
|
38
44
|
|
|
39
|
-
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## Quickstart
|
|
48
|
+
|
|
49
|
+
### 1. Extract an AI conversation
|
|
40
50
|
|
|
41
51
|
```js
|
|
42
|
-
import { detectPlatform,
|
|
52
|
+
import { detectPlatform, isAiChatUrl } from "decant-core";
|
|
43
53
|
|
|
44
|
-
// Check if a URL is an AI chat platform
|
|
45
54
|
if (isAiChatUrl(window.location.href)) {
|
|
46
|
-
const
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
}
|
|
55
|
+
const match = detectPlatform(window.location.href);
|
|
56
|
+
|
|
57
|
+
if (match?.parser && match.parser.isAvailable(window.location.href)) {
|
|
58
|
+
const result = await match.parser.parse();
|
|
59
|
+
// result.title
|
|
60
|
+
// result.messages -> [{ role: 'User' | 'Assistant', content, ... }]
|
|
61
|
+
// result.model
|
|
62
|
+
// result.metadata -> platform-specific extras
|
|
55
63
|
}
|
|
56
64
|
}
|
|
57
65
|
```
|
|
58
66
|
|
|
59
|
-
###
|
|
67
|
+
### 2. Extract a regular web article
|
|
60
68
|
|
|
61
69
|
```js
|
|
62
|
-
|
|
63
|
-
|
|
70
|
+
import { extractArticle } from "decant-core";
|
|
71
|
+
|
|
72
|
+
const article = await extractArticle(document /* or an HTML string */, {
|
|
73
|
+
url: window.location.href,
|
|
74
|
+
});
|
|
75
|
+
|
|
76
|
+
// article.title, article.author, article.published
|
|
77
|
+
// article.markdown -> clean, ready-to-use Markdown
|
|
78
|
+
// article.content -> the body without the title prefix
|
|
79
|
+
// article.engine -> 'readability' | 'defuddle' | 'raw'
|
|
80
|
+
```
|
|
64
81
|
|
|
65
|
-
|
|
82
|
+
### 3. Detection
|
|
83
|
+
|
|
84
|
+
```js
|
|
66
85
|
import { detectPlatform, isAiChatUrl, AI_CHAT_DOMAINS } from "decant-core";
|
|
67
86
|
|
|
87
|
+
isAiChatUrl("https://chatgpt.com/c/abc-123"); // -> true
|
|
88
|
+
const detected = detectPlatform(url); // -> { type: 'ai-chat', platform: 'ChatGPT', parser }
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
### Subpath imports
|
|
92
|
+
|
|
93
|
+
```js
|
|
94
|
+
// Individual parsers (tree-shake the rest)
|
|
95
|
+
import { ChatGPTParser } from "decant-core/ai/chatgpt";
|
|
96
|
+
import { ClaudeParser } from "decant-core/ai/claude";
|
|
97
|
+
import { GeminiParser } from "decant-core/ai/gemini";
|
|
98
|
+
|
|
99
|
+
// Web & article extraction
|
|
100
|
+
import { extractArticle, ArticleParser, scoreContent } from "decant-core";
|
|
101
|
+
|
|
102
|
+
// Detection
|
|
103
|
+
import { detectPlatform, isAiChatUrl, parsers } from "decant-core";
|
|
104
|
+
|
|
68
105
|
// Utilities
|
|
69
106
|
import { convertToMarkdown, cleanMarkdown } from "decant-core";
|
|
107
|
+
import { normalizeLatexMath } from "decant-core";
|
|
70
108
|
```
|
|
71
109
|
|
|
72
|
-
|
|
110
|
+
---
|
|
73
111
|
|
|
74
|
-
|
|
75
|
-
parser-core/
|
|
76
|
-
ai/ # Parser classes
|
|
77
|
-
base.js # Base parser class
|
|
78
|
-
chatgpt.js # ChatGPT parser + linearize
|
|
79
|
-
chatgpt_helper.js # Injected helper for ChatGPT scroll collection
|
|
80
|
-
chatgpt_scroll_collector.js # Scroll/dedup logic for ChatGPT
|
|
81
|
-
claude.js # Claude parser (DOM + API)
|
|
82
|
-
claude_react_reader.js # Injected helper for Claude React tree
|
|
83
|
-
copilot.js # Copilot parser (multi-domain)
|
|
84
|
-
deepseek.js # DeepSeek parser (DOM + API)
|
|
85
|
-
gemini.js # Gemini parser
|
|
86
|
-
gemini_cloud_assist.js # Gemini Cloud Assist parser
|
|
87
|
-
google_ai_studio.js # Google AI Studio parser
|
|
88
|
-
google_search_ai.js # Google Search AI / SGE parser
|
|
89
|
-
index.js # Barrel export
|
|
90
|
-
lumo.js # Lumo parser
|
|
91
|
-
meta.js # Meta AI parser
|
|
92
|
-
mistral.js # Mistral parser
|
|
93
|
-
notebooklm.js # NotebookLM parser
|
|
94
|
-
perplexity.js # Perplexity parser
|
|
95
|
-
qwen.js # Qwen parser
|
|
96
|
-
z_ai.js # Z AI parser
|
|
97
|
-
detection/ # Platform detection
|
|
98
|
-
detect-platform.js # detectPlatform(), isAiChatUrl(), parsers[]
|
|
99
|
-
domains.js # AI_CHAT_DOMAINS, URL_PATTERNS
|
|
100
|
-
lib/ # Vendored libraries
|
|
101
|
-
turndown.js # Turndown HTML→Markdown converter
|
|
102
|
-
utils/ # Utilities
|
|
103
|
-
html-to-markdown.js # AI-specific Turndown rules
|
|
104
|
-
```
|
|
112
|
+
## Supported Platforms
|
|
105
113
|
|
|
106
|
-
|
|
114
|
+
17 AI chat platform parsers plus generic web article extraction:
|
|
107
115
|
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
116
|
+
**ChatGPT · Claude · Google Gemini · Microsoft Copilot · Perplexity · DeepSeek · Qwen · Meta AI · Mistral (Le Chat) · Proton Lumo · Z.ai · Google AI Studio · NotebookLM · Google Search AI · Gemini Cloud Assist · Joyland · Chub**
|
|
117
|
+
|
|
118
|
+
All parsers extend the base [`ChatParser`](ai/base.js) interface — a consistent `isAvailable(url)` +
|
|
119
|
+
normalized `parse()` contract. For the extraction-strategy breakdown and maintenance model, see
|
|
120
|
+
[SUPPORTED_PLATFORMS.md](SUPPORTED_PLATFORMS.md).
|
|
121
|
+
|
|
122
|
+
---
|
|
113
123
|
|
|
114
124
|
## License
|
|
115
125
|
|
|
116
|
-
|
|
126
|
+
`decant-core` is licensed under the **GNU Affero General Public License v3.0 (AGPL-3.0-only)**.
|
|
127
|
+
|
|
128
|
+
That choice is deliberate. AI platforms change constantly, and parser fixes belong in a shared commons so the whole ecosystem benefits — not siloed in a proprietary fork. If you use `decant-core`, network-based deployments that serve modified versions must also offer the corresponding source. Please review [`LICENSE`](LICENSE) before incorporating it into your project.
|
|
129
|
+
|
|
130
|
+
---
|
|
131
|
+
|
|
132
|
+
## Used by
|
|
133
|
+
|
|
134
|
+
- [AI Chat Exporter](https://github.com/Covai-Labs/ai-chat-exporter) — export, archive and transfer AI conversations between platforms.
|
|
135
|
+
- [Decant](https://github.com/Covai-Labs/decant) — the distraction-free web clipper and research batcher.
|
|
136
|
+
|
|
137
|
+
These products are demonstrations of the library, not its purpose. Yours can be next — see [CONTRIBUTING.md](CONTRIBUTING.md).
|
|
138
|
+
|
|
139
|
+
---
|
|
140
|
+
|
|
141
|
+
## Development
|
|
142
|
+
|
|
143
|
+
Building, testing, and extending the library is covered in [DEVELOPMENT.md](DEVELOPMENT.md);
|
|
144
|
+
platform contributions follow the parser pattern and CLA in [CONTRIBUTING.md](CONTRIBUTING.md).
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "decant-core",
|
|
3
|
-
"version": "1.
|
|
4
|
-
"description": "
|
|
3
|
+
"version": "1.7.0",
|
|
4
|
+
"description": "A shared web extraction layer for AI conversations and regular web pages.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"engines": {
|
|
7
7
|
"node": ">=26.0.0"
|
|
@@ -47,7 +47,14 @@
|
|
|
47
47
|
"gemini",
|
|
48
48
|
"deepseek",
|
|
49
49
|
"copilot",
|
|
50
|
-
"
|
|
50
|
+
"perplexity",
|
|
51
|
+
"web-clipper",
|
|
52
|
+
"extraction",
|
|
53
|
+
"markdown",
|
|
54
|
+
"readability",
|
|
55
|
+
"turndown",
|
|
56
|
+
"browser-extension",
|
|
57
|
+
"scraping"
|
|
51
58
|
],
|
|
52
59
|
"author": "deadrat-in",
|
|
53
60
|
"license": "AGPL-3.0-only",
|