decant-core 1.11.0 → 1.12.0

package/README.md

@@ -16,7 +16,7 @@ And every time one of those platforms changes its UI, seriously re-renders a mes
 
 Existing chat exporters often suffer from two major flaws: they break whenever platform DOMs update, and many route user conversations through third-party servers.
 
-`decant-core` was created to solve both at the foundation. Originally built to power local-first extensions like [AI Chat Exporter](https://ai-chat-exporter.covai.org/) and [Decant](https://decant.covai.org/), it decouples fragile platform parsing from presentation. By sharing this engine under AGPL-3.0, any browser extension, web clipper, archiver, or research tool can rely on a maintained, local-first extraction layer instead of reverse-engineering AI platforms in isolation.
+`decant-core` was created to solve both at the foundation. Originally built to power local-first extensions like [AI Chat Exporter](https://ace.covai.org/) and [Decant](https://decant.covai.org/), it decouples fragile platform parsing from presentation. By sharing this engine under AGPL-3.0, any browser extension, web clipper, archiver, or research tool can rely on a maintained, local-first extraction layer instead of reverse-engineering AI platforms in isolation.
 
 ```bash
 npm install decant-core
@@ -119,28 +119,28 @@ import { normalizeLatexMath } from "decant-core";
 
 19 AI chat platform parsers plus generic web article extraction:
 
-| Platform                            | Parser                    | Extraction strategy                                   |
-| :---------------------------------- | :------------------------ | :---------------------------------------------------- |
-| **ChatGPT**                         | `ChatGPTParser`           | DOM + internal API                                    |
-| **Claude**                          | `ClaudeParser`            | DOM + internal API + React fiber                      |
-| **Google Gemini**                   | `GeminiParser`            | DOM + batchexecute RPC                                |
-| **Microsoft Copilot**               | `CopilotParser`           | DOM (multi-domain)                                    |
-| **Perplexity**                      | `PerplexityParser`        | Internal API + DOM fallback                           |
-| **DeepSeek**                        | `DeepSeekParser`          | DOM + internal API (`fragments[]`)                    |
-| **Qwen**                            | `QwenParser`              | DOM                                                   |
-| **Meta AI**                         | `MetaParser`              | Internal API (GraphQL) + DOM fallback                 |
-| **Mistral / Le Chat**               | `MistralParser`           | DOM                                                   |
-| **Proton Lumo**                     | `LumoParser`              | DOM (API is E2E-encrypted, not readable)              |
-| **Z.ai**                            | `ZAiParser`               | Internal API (chat + batch) + DOM fallback            |
-| **Grok**                            | `GrokParser`              | Internal API (response-node + load) + DOM             |
-| **Google AI Studio**                | `GoogleAIStudioParser`    | DOM                                                   |
-| **NotebookLM**                      | `NotebookLMParser`        | DOM                                                   |
-| **Google Search AI (AI Overviews)** | `GoogleSearchAIParser`    | DOM                                                   |
-| **Gemini Cloud Assist**             | `GeminiCloudAssistParser` | DOM                                                   |
-| **Joyland**                         | `JoylandParser`           | DOM                                                   |
-| **Chub**                            | `ChubParser`              | DOM                                                   |
-| **Duck.ai (DuckDuckGo AI)**         | `DuckAIParser`            | DOM (client-side privacy, no server chat history API) |
-| **Generic Web Article**             | `ArticleParser`           | Readability + Defuddle + Article-Extractor            |
+| Platform                            | Parser                    | Extraction strategy                        |
+| :---------------------------------- | :------------------------ | :----------------------------------------- |
+| **ChatGPT**                         | `ChatGPTParser`           | DOM + internal API                         |
+| **Claude**                          | `ClaudeParser`            | DOM + internal API + React fiber           |
+| **Google Gemini**                   | `GeminiParser`            | DOM + batchexecute RPC                     |
+| **Microsoft Copilot**               | `CopilotParser`           | DOM                                        |
+| **Perplexity**                      | `PerplexityParser`        | Internal API + DOM                         |
+| **DeepSeek**                        | `DeepSeekParser`          | DOM + internal API                         |
+| **Qwen**                            | `QwenParser`              | DOM                                        |
+| **Meta AI**                         | `MetaParser`              | Internal API + DOM                         |
+| **Mistral / Le Chat**               | `MistralParser`           | DOM                                        |
+| **Proton Lumo**                     | `LumoParser`              | DOM                                        |
+| **Z.ai**                            | `ZAiParser`               | Internal API + DOM                         |
+| **Grok**                            | `GrokParser`              | Internal API + DOM                         |
+| **Google AI Studio**                | `GoogleAIStudioParser`    | DOM                                        |
+| **NotebookLM**                      | `NotebookLMParser`        | DOM                                        |
+| **Google Search AI (AI Overviews)** | `GoogleSearchAIParser`    | DOM                                        |
+| **Gemini Cloud Assist**             | `GeminiCloudAssistParser` | DOM                                        |
+| **Joyland**                         | `JoylandParser`           | DOM                                        |
+| **Chub**                            | `ChubParser`              | DOM                                        |
+| **Duck.ai (DuckDuckGo AI)**         | `DuckAIParser`            | DOM                                        |
+| **Generic Web Article**             | `ArticleParser`           | Readability + Defuddle + Article-Extractor |
 
 All parsers extend the base [`ChatParser`](ai/base.js) interface — a consistent `isAvailable(url)` +
 normalized `parse()` contract. For the full extraction-strategy breakdown and maintenance model, see

package/ai/chatgpt.js

@@ -831,6 +831,7 @@ export class ChatGPTParser extends ChatParser {
     const messages = [];
     for (const msg of apiMessages) {
       let content = "";
+      let thinking = "";
       for (const seg of msg.segments) {
         if (seg.type === "text") {
           content +=
@@ -843,7 +844,7 @@ export class ChatGPTParser extends ChatParser {
             msg.imageGroupMap,
           );
           if (thoughtText) {
-            content += `<details><summary>Thought Process</summary>\n\n${thoughtText}\n\n</details>\n\n`;
+            thinking += (thinking ? "\n\n" : "") + thoughtText;
           }
         } else if (seg.type === "image") {
           const src = images[seg.fileId];
@@ -852,12 +853,22 @@ export class ChatGPTParser extends ChatParser {
           }
         }
       }
-      content = content.trim();
-      if (content) {
+      let fullContent = "";
+      if (thinking) {
+        fullContent += `<think>\n${thinking}\n</think>\n\n`;
+      }
+      if (content.trim()) {
+        fullContent += content.trim();
+      }
+      fullContent = fullContent.trim();
+      if (fullContent) {
         const msgObj = {
           role: msg.role,
-          content: content,
+          content: fullContent,
         };
+        if (thinking) {
+          msgObj.thinking = thinking;
+        }
         if (msg.timestamp) {
           msgObj.timestamp = msg.timestamp;
         }
@@ -876,8 +887,10 @@ export class ChatGPTParser extends ChatParser {
       Link: currentUrl,
       Model:
         convoData?.model_slug ||
-        document.querySelector('[data-testid="model-selector-dropdown"]')
-          ?.innerText ||
+        (typeof document !== "undefined" && document.querySelector
+          ? document.querySelector('[data-testid="model-selector-dropdown"]')
+              ?.innerText
+          : null) ||
         "ChatGPT",
       Method: method,
     };

package/ai/claude.js

@@ -402,12 +402,30 @@ export class ClaudeParser extends ChatParser {
             const role = message.sender === "human" ? "User" : "Claude";
 
             let contentStr = "";
+            let thinkingStr = "";
 
             // Construct content
             if (message.content && Array.isArray(message.content)) {
               for (const block of message.content) {
-                if (block.type === "thinking" && block.thinking) {
-                  contentStr += `> **Thinking Process:**\n> \n> ${block.thinking.replace(/\n/g, "\n> ")}\n\n`;
+                if (block.type === "thinking") {
+                  let thoughtText = "";
+                  if (
+                    typeof block.thinking === "string" &&
+                    block.thinking.trim()
+                  ) {
+                    thoughtText = block.thinking.trim();
+                  } else if (Array.isArray(block.summaries)) {
+                    thoughtText = block.summaries
+                      .map((s) =>
+                        typeof s === "string" ? s : s?.summary || "",
+                      )
+                      .map((s) => s.trim())
+                      .filter(Boolean)
+                      .join("\n");
+                  }
+                  if (thoughtText) {
+                    thinkingStr += (thinkingStr ? "\n\n" : "") + thoughtText;
+                  }
                 } else if (block.type === "text" && block.text) {
                   const cleanText = block.text
                     .replace(/<antArtifact[^>]*>[\s\S]*?<\/antArtifact>/g, "")
@@ -516,9 +534,17 @@ export class ClaudeParser extends ChatParser {
               }
             }
 
+            if (thinkingStr) {
+              contentStr = `<think>\n${thinkingStr}\n</think>\n\n` + contentStr;
+            }
+
             contentStr = contentStr.trim();
             if (contentStr) {
-              messages.push({ role, content: contentStr });
+              const msgObj = { role, content: contentStr };
+              if (thinkingStr) {
+                msgObj.thinking = thinkingStr;
+              }
+              messages.push(msgObj);
             }
 
             // Extract and push artifacts

package/ai/deepseek.js

@@ -98,7 +98,29 @@ async function fetchDeepSeekConversation(sessionId, token) {
       const isUser = msgNode.role === "USER" || msgNode.role === "user";
       const role = isUser ? "User" : "DeepSeek";
       const content = extractDeepSeekMessageContent(msgNode);
-      return { role, content: content.trim() };
+      let thinking = "";
+      if (!isUser && Array.isArray(msgNode.fragments)) {
+        const thinkFragments = msgNode.fragments.filter(
+          (f) => f && f.type === "THINK" && typeof f.content === "string",
+        );
+        thinking = thinkFragments
+          .map((f) => f.content.trim())
+          .filter(Boolean)
+          .join("\n\n");
+      }
+      let fullContent = "";
+      if (thinking) {
+        fullContent += `<think>\n${thinking}\n</think>\n\n`;
+      }
+      if (content.trim()) {
+        fullContent += content.trim();
+      }
+      fullContent = fullContent.trim();
+      const msg = { role, content: fullContent };
+      if (thinking) {
+        msg.thinking = thinking;
+      }
+      return msg;
     })
     .filter((msg) => msg.content.length > 0);
 }
@@ -182,15 +204,40 @@ export class DeepSeekParser extends ChatParser {
 
     outerElements.forEach((el) => {
       let role = "Unknown";
+      let thinking = "";
       if (el.matches(userSelector)) {
         role = "User";
       } else if (el.matches(assistantSelector)) {
         role = "DeepSeek";
+        const messageContainer =
+          (el.closest && el.closest(".ds-message")) || el.parentElement;
+        if (messageContainer) {
+          const thinkContainers =
+            messageContainer.querySelectorAll(".ds-think-content");
+          if (thinkContainers.length > 0) {
+            thinking = Array.from(thinkContainers)
+              .map((tc) => convertToMarkdown(tc).trim())
+              .filter(Boolean)
+              .join("\n\n");
+          }
+        }
       }
 
       const text = convertToMarkdown(el);
+      let fullContent = "";
+      if (thinking) {
+        fullContent += `<think>\n${thinking}\n</think>\n\n`;
+      }
       if (text.trim()) {
-        messages.push({ role, content: text.trim() });
+        fullContent += text.trim();
+      }
+      fullContent = fullContent.trim();
+      if (fullContent) {
+        const msg = { role, content: fullContent };
+        if (thinking) {
+          msg.thinking = thinking;
+        }
+        messages.push(msg);
       }
     });
 
@@ -202,9 +249,37 @@ export class DeepSeekParser extends ChatParser {
       messageRows.forEach((row) => {
         const isUser = row.classList.contains("ds-user-message");
         const role = isUser ? "User" : "DeepSeek";
-        const text = convertToMarkdown(row);
+        const rowClone = row.cloneNode(true);
+        let thinking = "";
+        if (!isUser) {
+          const thinkContainers =
+            rowClone.querySelectorAll(".ds-think-content");
+          if (thinkContainers.length > 0) {
+            thinking = Array.from(thinkContainers)
+              .map((tc) => {
+                const md = convertToMarkdown(tc).trim();
+                tc.remove();
+                return md;
+              })
+              .filter(Boolean)
+              .join("\n\n");
+          }
+        }
+        const text = convertToMarkdown(rowClone);
+        let fullContent = "";
+        if (thinking) {
+          fullContent += `<think>\n${thinking}\n</think>\n\n`;
+        }
         if (text.trim()) {
-          messages.push({ role, content: text.trim() });
+          fullContent += text.trim();
+        }
+        fullContent = fullContent.trim();
+        if (fullContent) {
+          const msg = { role, content: fullContent };
+          if (thinking) {
+            msg.thinking = thinking;
+          }
+          messages.push(msg);
         }
       });
     }

package/ai/z_ai.js

@@ -60,13 +60,12 @@ export function formatZaiMessage(entry) {
     .filter(Boolean)
     .join("\n\n");
   if (reasoning) {
-    const quoted = reasoning
-      .split("\n")
-      .map((line) => `> ${line}`)
-      .join("\n");
-    content += `\n\n> 🧠 Thinking\n${quoted}`;
+    content = `<think>\n${reasoning}\n</think>\n\n${content}`;
   }
   const msg = { role, content };
+  if (reasoning) {
+    msg.thinking = reasoning;
+  }
   if (entry.timestamp) {
     try {
       msg.timestamp = new Date(entry.timestamp * 1000).toISOString();

package/data/platforms.json

@@ -0,0 +1,162 @@
+[
+  {
+    "id": "chatgpt",
+    "platform": "ChatGPT",
+    "parser": "ChatGPTParser",
+    "module": "decant-core/ai/chatgpt",
+    "strategy": "DOM + internal API",
+    "notes": "Fallback and RPC-assisted extraction; scroll/dedup helper for long threads."
+  },
+  {
+    "id": "claude",
+    "platform": "Claude",
+    "parser": "ClaudeParser",
+    "module": "decant-core/ai/claude",
+    "strategy": "DOM + internal API + React fiber",
+    "notes": "Internal API first with DOM fallback; reads the React tree for artifacts and structured blocks."
+  },
+  {
+    "id": "gemini",
+    "platform": "Google Gemini",
+    "parser": "GeminiParser",
+    "module": "decant-core/ai/gemini",
+    "strategy": "DOM + batchexecute RPC",
+    "notes": "Uses batchexecute RPC pagination with resilient DOM fallback."
+  },
+  {
+    "id": "copilot",
+    "platform": "Microsoft Copilot",
+    "parser": "CopilotParser",
+    "module": "decant-core/ai/copilot",
+    "strategy": "DOM",
+    "notes": "Multi-domain (bing + copilot) DOM extraction."
+  },
+  {
+    "id": "perplexity",
+    "platform": "Perplexity",
+    "parser": "PerplexityParser",
+    "module": "decant-core/ai/perplexity",
+    "strategy": "Internal API + DOM",
+    "notes": "Internal API first, DOM fallback for source citations and answers."
+  },
+  {
+    "id": "deepseek",
+    "platform": "DeepSeek",
+    "parser": "DeepSeekParser",
+    "module": "decant-core/ai/deepseek",
+    "strategy": "DOM + internal API",
+    "notes": "API-assisted parsing (fragments[]) with DOM fallback."
+  },
+  {
+    "id": "qwen",
+    "platform": "Qwen",
+    "parser": "QwenParser",
+    "module": "decant-core/ai/qwen",
+    "strategy": "DOM",
+    "notes": "Structured DOM extraction incl. file attachments."
+  },
+  {
+    "id": "meta",
+    "platform": "Meta AI",
+    "parser": "MetaParser",
+    "module": "decant-core/ai/meta",
+    "strategy": "Internal API + DOM",
+    "notes": "Internal GraphQL API (pinned + auto-resolved doc_ids) with DOM fallback."
+  },
+  {
+    "id": "mistral",
+    "platform": "Mistral / Le Chat",
+    "parser": "MistralParser",
+    "module": "decant-core/ai/mistral",
+    "strategy": "DOM",
+    "notes": "DOM extraction."
+  },
+  {
+    "id": "lumo",
+    "platform": "Proton Lumo",
+    "parser": "LumoParser",
+    "module": "decant-core/ai/lumo",
+    "strategy": "DOM",
+    "notes": "DOM only — API responses are E2E-encrypted and not readable."
+  },
+  {
+    "id": "z-ai",
+    "platform": "Z.ai",
+    "parser": "ZAiParser",
+    "module": "decant-core/ai/z_ai",
+    "strategy": "Internal API + DOM",
+    "notes": "Internal API (chat skeleton + batched bodies) with DOM fallback."
+  },
+  {
+    "id": "grok",
+    "platform": "Grok",
+    "parser": "GrokParser",
+    "module": "decant-core/ai/grok",
+    "strategy": "Internal API + DOM",
+    "notes": "Internal API (response-node ordering + load-responses bodies) with DOM fallback."
+  },
+  {
+    "id": "google-ai-studio",
+    "platform": "Google AI Studio",
+    "parser": "GoogleAIStudioParser",
+    "module": "decant-core/ai/google_ai_studio",
+    "strategy": "DOM",
+    "notes": "DOM extraction for aistudio.google.com sessions."
+  },
+  {
+    "id": "notebooklm",
+    "platform": "NotebookLM",
+    "parser": "NotebookLMParser",
+    "module": "decant-core/ai/notebooklm",
+    "strategy": "DOM",
+    "notes": "DOM extraction incl. notes and citations."
+  },
+  {
+    "id": "google-search-ai",
+    "platform": "Google Search AI (AI Overviews)",
+    "parser": "GoogleSearchAIParser",
+    "module": "decant-core/ai/google_search_ai",
+    "strategy": "DOM",
+    "notes": "DOM extraction for SGE overviews."
+  },
+  {
+    "id": "gemini-cloud-assist",
+    "platform": "Gemini Cloud Assist",
+    "parser": "GeminiCloudAssistParser",
+    "module": "decant-core/ai/gemini_cloud_assist",
+    "strategy": "DOM",
+    "notes": "DOM extraction for console.cloud.google.com assistants."
+  },
+  {
+    "id": "joyland",
+    "platform": "Joyland",
+    "parser": "JoylandParser",
+    "module": "decant-core/ai/joyland",
+    "strategy": "DOM",
+    "notes": "DOM extraction for character chat platforms."
+  },
+  {
+    "id": "chub",
+    "platform": "Chub",
+    "parser": "ChubParser",
+    "module": "decant-core/ai/chub",
+    "strategy": "DOM",
+    "notes": "DOM extraction."
+  },
+  {
+    "id": "duck-ai",
+    "platform": "Duck.ai (DuckDuckGo AI)",
+    "parser": "DuckAIParser",
+    "module": "decant-core/ai/duck_ai",
+    "strategy": "DOM",
+    "notes": "DOM (client-side privacy, no server chat history API)."
+  },
+  {
+    "id": "article",
+    "platform": "Generic Web Article",
+    "parser": "ArticleParser",
+    "module": "decant-core/article",
+    "strategy": "Readability + Defuddle + Article-Extractor",
+    "notes": "Runs three extractors concurrently and arbitrates by content-quality scoring."
+  }
+]

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "decant-core",
-  "version": "1.11.0",
+  "version": "1.12.0",
   "description": "A shared web extraction layer for AI conversations and regular web pages.",
   "type": "module",
   "engines": {
@@ -13,7 +13,8 @@
     "./web": "./web/index.js",
     "./web/*": "./web/*.js",
     "./article": "./web/article.js",
-    "./detection/*": "./detection/*.js"
+    "./detection/*": "./detection/*.js",
+    "./platforms": "./data/platforms.json"
   },
   "files": [
     "ai",
@@ -21,6 +22,7 @@
     "detection",
     "lib",
     "utils",
+    "data",
     "LICENSE",
     "README.md"
   ],
@@ -36,7 +38,9 @@
     "test": "node --test tests/*.test.js",
     "lint": "eslint .",
     "format": "prettier --write .",
-    "format:check": "prettier --check ."
+    "format:check": "prettier --check .",
+    "sync:platforms": "node scripts/sync-platforms.js",
+    "sync:check": "node scripts/sync-platforms.js --check"
   },
   "keywords": [
     "ai",