decant-core 1.5.0 → 1.7.0

package/README.md

@@ -1,116 +1,144 @@
 # decant-core
 
-Shared AI chat platform parsers and detection logic for [Covai](https://github.com/Covai-Labs) browser extensions.
+**A shared extraction layer for the modern web — including AI conversations.**
 
-This package powers the AI chat extraction features in both [AI Chat Exporter](https://github.com/Covai-Labs/ai-chat-exporter) and [Decant](https://github.com/Covai-Labs/decant).
+[![npm version](https://img.shields.io/npm/v/decant-core?logo=npm&logoColor=white&label=npm&color=cb3837)](https://www.npmjs.com/package/decant-core)
+[![License: AGPL-3.0](https://img.shields.io/badge/License-AGPL--3.0-red.svg)](LICENSE)
+[![GitHub](https://img.shields.io/github/stars/Covai-Labs/decant-core?logo=github&logoColor=white&color=yellow&label=Stars)](https://github.com/Covai-Labs/decant-core/stargazers)
 
-## Supported Platforms
+Every AI chat exporter ends up solving the same problem: extracting conversations from ChatGPT, Claude, Gemini, Perplexity, DeepSeek and other constantly changing AI interfaces.
+
+And every time one of those platforms changes its UI, seriously re-renders a message, or ships a new feature, **somebody's parser breaks**.
 
-| Platform            | Parser                    |
-| ------------------- | ------------------------- |
-| ChatGPT             | `ChatGPTParser`           |
-| Claude              | `ClaudeParser`            |
-| Copilot             | `CopilotParser`           |
-| DeepSeek            | `DeepSeekParser`          |
-| Gemini              | `GeminiParser`            |
-| Gemini Cloud Assist | `GeminiCloudAssistParser` |
-| Google AI Studio    | `GoogleAIStudioParser`    |
-| Google Search AI    | `GoogleSearchAIParser`    |
-| Lumo                | `LumoParser`              |
-| Meta AI             | `MetaParser`              |
-| Mistral             | `MistralParser`           |
-| NotebookLM          | `NotebookLMParser`        |
-| Perplexity          | `PerplexityParser`        |
-| Qwen                | `QwenParser`              |
-| Z AI                | `ZAiParser`               |
-
-## Installation
+`decant-core` provides reusable parsers, platform detection, and web article extraction so developers don't have to build and maintain the same fragile parsing layer over and over again.
 
 ```bash
 npm install decant-core
 ```
 
-For local development:
+Use it as the parsing layer underneath your own:
 
-```bash
-npm install decant-core@file:../parser-core
-```
+- chat exporters
+- browser extensions
+- web clippers
+- research & data-extraction tools
+- content archivers
+- knowledge-management and PKM applications
+
+**Fix platform parsing once, and let the ecosystem benefit from the fix.**
+
+---
+
+## Why decant-core?
+
+AI platforms don't expose stable public APIs for reading conversation history. No matter what you build, to extract a ChatGPT thread you need to walk the DOM, read internal RPC payloads, or traverse React component trees — and redo it when the frontend changes.
+
+Maintaining that per-platform logic in every exporter is wasteful and fragile. `decant-core` centralizes it:
+
+- ✅ **17 AI chat platform parsers** with normalized output — you get structured messages, models, metadata and Markdown, not DOM soup.
+- ✅ **Web article extraction** — Mozilla Readability, Defuddle, and Article-Extractor run in parallel and arbitrate by content-quality scoring.
+- ✅ **Detection utilities** — tell an "AI chat page" apart from a "regular web page" before you decide which parser to run.
+- ✅ **Math & Markdown handling** — LaTeX normalization plus GFM tables/code fencing that survive round-trips into Obsidian, Logseq and Notion.
+
+The payoff is maintenance: **when a platform changes, the fix happens once, in one place**, instead of being independently reimplemented across dozens of projects.
 
-## Usage
+---
+
+## Quickstart
+
+### 1. Extract an AI conversation
 
 ```js
-import { detectPlatform, parsers, isAiChatUrl } from "decant-core";
+import { detectPlatform, isAiChatUrl } from "decant-core";
 
-// Check if a URL is an AI chat platform
 if (isAiChatUrl(window.location.href)) {
-  const platform = detectPlatform(window.location.href);
-  const ParserClass = parsers.find((p) => p.platform === platform);
-
-  if (ParserClass) {
-    const parser = new ParserClass();
-    if (parser.canParse(window.location.href)) {
-      const result = parser.parse();
-      // result.title, result.messages, result.model, etc.
-    }
+  const match = detectPlatform(window.location.href);
+
+  if (match?.parser && match.parser.isAvailable(window.location.href)) {
+    const result = await match.parser.parse();
+    // result.title
+    // result.messages  -> [{ role: 'User' | 'Assistant', content, ... }]
+    // result.model
+    // result.metadata  -> platform-specific extras
   }
 }
 ```
 
-### Subpath imports
+### 2. Extract a regular web article
 
 ```js
-// Individual parser
-import { ChatGPTParser } from "decant-core";
+import { extractArticle } from "decant-core";
+
+const article = await extractArticle(document /* or an HTML string */, {
+  url: window.location.href,
+});
+
+// article.title, article.author, article.published
+// article.markdown       -> clean, ready-to-use Markdown
+// article.content        -> the body without the title prefix
+// article.engine         -> 'readability' | 'defuddle' | 'raw'
+```
 
-// Detection helpers
+### 3. Detection
+
+```js
 import { detectPlatform, isAiChatUrl, AI_CHAT_DOMAINS } from "decant-core";
 
+isAiChatUrl("https://chatgpt.com/c/abc-123"); // -> true
+const detected = detectPlatform(url); // -> { type: 'ai-chat', platform: 'ChatGPT', parser }
+```
+
+### Subpath imports
+
+```js
+// Individual parsers (tree-shake the rest)
+import { ChatGPTParser } from "decant-core/ai/chatgpt";
+import { ClaudeParser } from "decant-core/ai/claude";
+import { GeminiParser } from "decant-core/ai/gemini";
+
+// Web & article extraction
+import { extractArticle, ArticleParser, scoreContent } from "decant-core";
+
+// Detection
+import { detectPlatform, isAiChatUrl, parsers } from "decant-core";
+
 // Utilities
 import { convertToMarkdown, cleanMarkdown } from "decant-core";
+import { normalizeLatexMath } from "decant-core";
 ```
 
-## Project Structure
+---
 
-```
-parser-core/
-  ai/                          # Parser classes
-    base.js                    # Base parser class
-    chatgpt.js                 # ChatGPT parser + linearize
-    chatgpt_helper.js          # Injected helper for ChatGPT scroll collection
-    chatgpt_scroll_collector.js # Scroll/dedup logic for ChatGPT
-    claude.js                  # Claude parser (DOM + API)
-    claude_react_reader.js     # Injected helper for Claude React tree
-    copilot.js                 # Copilot parser (multi-domain)
-    deepseek.js                # DeepSeek parser (DOM + API)
-    gemini.js                  # Gemini parser
-    gemini_cloud_assist.js     # Gemini Cloud Assist parser
-    google_ai_studio.js        # Google AI Studio parser
-    google_search_ai.js        # Google Search AI / SGE parser
-    index.js                   # Barrel export
-    lumo.js                    # Lumo parser
-    meta.js                    # Meta AI parser
-    mistral.js                 # Mistral parser
-    notebooklm.js              # NotebookLM parser
-    perplexity.js              # Perplexity parser
-    qwen.js                    # Qwen parser
-    z_ai.js                    # Z AI parser
-  detection/                   # Platform detection
-    detect-platform.js         # detectPlatform(), isAiChatUrl(), parsers[]
-    domains.js                 # AI_CHAT_DOMAINS, URL_PATTERNS
-  lib/                         # Vendored libraries
-    turndown.js                # Turndown HTML→Markdown converter
-  utils/                       # Utilities
-    html-to-markdown.js        # AI-specific Turndown rules
-```
+## Supported Platforms
 
-## Development
+17 AI chat platform parsers plus generic web article extraction:
 
-```bash
-npm install
-npm run lint
-npm run format:check
-```
+**ChatGPT · Claude · Google Gemini · Microsoft Copilot · Perplexity · DeepSeek · Qwen · Meta AI · Mistral (Le Chat) · Proton Lumo · Z.ai · Google AI Studio · NotebookLM · Google Search AI · Gemini Cloud Assist · Joyland · Chub**
+
+All parsers extend the base [`ChatParser`](ai/base.js) interface — a consistent `isAvailable(url)` +
+normalized `parse()` contract. For the extraction-strategy breakdown and maintenance model, see
+[SUPPORTED_PLATFORMS.md](SUPPORTED_PLATFORMS.md).
+
+---
 
 ## License
 
-[AGPL-3.0](LICENSE)
+`decant-core` is licensed under the **GNU Affero General Public License v3.0 (AGPL-3.0-only)**.
+
+That choice is deliberate. AI platforms change constantly, and parser fixes belong in a shared commons so the whole ecosystem benefits — not siloed in a proprietary fork. If you use `decant-core`, network-based deployments that serve modified versions must also offer the corresponding source. Please review [`LICENSE`](LICENSE) before incorporating it into your project.
+
+---
+
+## Used by
+
+- [AI Chat Exporter](https://github.com/Covai-Labs/ai-chat-exporter) — export, archive and transfer AI conversations between platforms.
+- [Decant](https://github.com/Covai-Labs/decant) — the distraction-free web clipper and research batcher.
+
+These products are demonstrations of the library, not its purpose. Yours can be next — see [CONTRIBUTING.md](CONTRIBUTING.md).
+
+---
+
+## Development
+
+Building, testing, and extending the library is covered in [DEVELOPMENT.md](DEVELOPMENT.md);
+platform contributions follow the parser pattern and CLA in [CONTRIBUTING.md](CONTRIBUTING.md).

package/ai/gemini.js

@@ -1,5 +1,6 @@
 import { ChatParser } from "./base.js";
 import { convertToMarkdown } from "../utils/html-to-markdown.js";
+import { normalizeLatexMath } from "../utils/latex-math.js";
 
 const GEMINI_RPC_ID = "hNvQHb";
 const DEFAULT_BARD_PATH = "/_/BardChatUi";
@@ -94,23 +95,7 @@ export class GeminiParser extends ChatParser {
 
     const mode = options.parserMode || "auto";
 
-    // 1. Prefer DOM extraction first when on a live page with conversation containers
-    if (typeof document !== "undefined" && document.querySelector) {
-      const hasDomMessages = document.querySelector(
-        ".conversation-container, user-query, model-response, deep-research-immersive-panel",
-      );
-      if (hasDomMessages && mode !== "api") {
-        const domResult = this.parseFromDom(currentUrl, options);
-        if (domResult && domResult.messages && domResult.messages.length > 0) {
-          console.log(
-            `[Gemini Parser] Successfully parsed ${domResult.messages.length} messages from DOM`,
-          );
-          return domResult;
-        }
-      }
-    }
-
-    // 2. Attempt API / RPC extraction if DOM parsing didn't find messages or mode is API
+    // 1. Attempt API / RPC extraction first when not explicitly in 'dom' mode
     if (mode !== "dom" && typeof fetch === "function") {
       try {
         const convoId = this.getConversationId(currentUrl);
@@ -149,7 +134,7 @@ export class GeminiParser extends ChatParser {
       }
     }
 
-    // Fall back to robust DOM parsing
+    // 2. Fall back to robust DOM parsing
     return this.parseFromDom(currentUrl, options);
   }
 
@@ -175,14 +160,17 @@ export class GeminiParser extends ChatParser {
       `&_reqid=${encodeURIComponent(reqId)}` +
       `&rt=c`;
 
+    const formattedConvoId = convoId.startsWith("c_")
+      ? convoId
+      : `c_${convoId}`;
     const allItems = [];
     let cursor = null;
     let pageCount = 0;
 
-    while (pageCount < 50) {
+    while (pageCount < 250) {
       pageCount++;
       const payloadArg = JSON.stringify([
-        `c_${convoId}`,
+        formattedConvoId,
         100,
         cursor,
         1,
@@ -227,11 +215,11 @@ export class GeminiParser extends ChatParser {
       const continueCursor = payload[1] || null;
 
       if (items.length > 0) {
-        // Items are in reverse chronological order from API
+        // Items are in reverse chronological order from API; reverse to maintain oldest-first order
         allItems.unshift(...items.slice().reverse());
       }
 
-      if (!continueCursor || items.length < 100) {
+      if (!continueCursor) {
         break;
       }
       cursor = continueCursor;
@@ -241,7 +229,158 @@ export class GeminiParser extends ChatParser {
       return null;
     }
 
+    return this.formatApiResult(allItems, currentUrl, options);
+  }
+
+  enrichMessagesWithDomAttachments(messages) {
+    try {
+      if (typeof document === "undefined" || !document.querySelectorAll) {
+        return messages;
+      }
+      const parentContainers = document.querySelectorAll(
+        ".conversation-container",
+      );
+      const domTurns = [];
+
+      if (parentContainers.length > 0) {
+        parentContainers.forEach((container, idx) => {
+          const rawId = (container.id || "").replace(/^r_/, "");
+          const fileElements = container.querySelectorAll(
+            "user-query-file-preview, .file-preview, .attachment-preview, [data-testid='file-preview']",
+          );
+          const fileNames = [];
+          fileElements.forEach((fe) => {
+            const name = (fe.textContent || "").trim();
+            if (name && !fileNames.includes(name)) fileNames.push(name);
+          });
+
+          const userQuery = container.querySelector("user-query") || container;
+          const clone = userQuery.cloneNode(true);
+          clone
+            .querySelectorAll(
+              "user-query-file-preview, .file-preview, .attachment-preview, button, .edit-button, model-response",
+            )
+            .forEach((el) => el.remove());
+          const queryText = (
+            clone.innerText !== undefined
+              ? clone.innerText
+              : clone.textContent || ""
+          )
+            .replace(/^You said\s*/i, "")
+            .trim();
+
+          if (fileNames.length > 0) {
+            domTurns.push({
+              turnId: rawId,
+              queryText,
+              domIndex: idx,
+              totalDom: parentContainers.length,
+              fileNames,
+            });
+          }
+        });
+      } else {
+        const userQueries = document.querySelectorAll("user-query");
+        userQueries.forEach((uq, idx) => {
+          const fileElements = uq.querySelectorAll(
+            "user-query-file-preview, .file-preview, .attachment-preview, [data-testid='file-preview']",
+          );
+          const fileNames = [];
+          fileElements.forEach((fe) => {
+            const name = (fe.textContent || "").trim();
+            if (name && !fileNames.includes(name)) fileNames.push(name);
+          });
+
+          const clone = uq.cloneNode(true);
+          clone
+            .querySelectorAll(
+              "user-query-file-preview, .file-preview, .attachment-preview, button, .edit-button",
+            )
+            .forEach((el) => el.remove());
+          const queryText = (
+            clone.innerText !== undefined
+              ? clone.innerText
+              : clone.textContent || ""
+          )
+            .replace(/^You said\s*/i, "")
+            .trim();
+
+          if (fileNames.length > 0) {
+            domTurns.push({
+              domIndex: idx,
+              queryText,
+              totalDom: userQueries.length,
+              fileNames,
+            });
+          }
+        });
+      }
+
+      if (domTurns.length === 0) return messages;
+
+      const userMessages = messages.filter((m) => m.role === "User");
+
+      for (const domTurn of domTurns) {
+        let matchedMsg = null;
+
+        // 1. Match by exact turnId if present
+        if (domTurn.turnId) {
+          matchedMsg = userMessages.find((m) => m.turnId === domTurn.turnId);
+        }
+
+        // 2. Fallback: match by queryText if present and non-empty
+        if (!matchedMsg && domTurn.queryText) {
+          matchedMsg = userMessages.find(
+            (m) =>
+              m.content &&
+              (m.content.includes(domTurn.queryText) ||
+                domTurn.queryText.includes(m.content)),
+          );
+        }
+
+        // 3. Fallback: match by trailing turn ordinal (mounted DOM elements align with latest turns)
+        if (!matchedMsg && typeof domTurn.domIndex === "number") {
+          const offset = userMessages.length - domTurn.totalDom;
+          const targetIndex =
+            offset >= 0 ? offset + domTurn.domIndex : domTurn.domIndex;
+          matchedMsg = userMessages[targetIndex];
+        }
+
+        if (matchedMsg) {
+          const attachBlock =
+            `\n\n**Attachments:**\n` +
+            domTurn.fileNames.map((fn) => `- ${fn}`).join("\n");
+          if (!matchedMsg.content.includes("**Attachments:**")) {
+            matchedMsg.content = matchedMsg.content
+              ? `${matchedMsg.content}${attachBlock}`
+              : `**Attachments:**\n` +
+                domTurn.fileNames.map((fn) => `- ${fn}`).join("\n");
+          }
+        }
+      }
+    } catch (e) {
+      console.warn(
+        "[Gemini Parser] Failed to enrich messages with DOM attachments:",
+        e,
+      );
+    } finally {
+      // Clean internal turnId tracking before returning messages
+      for (const m of messages) {
+        delete m.turnId;
+      }
+    }
+
+    return messages;
+  }
+
+  formatApiResult(allItems, currentUrl, options = {}) {
+    if (!Array.isArray(allItems) || allItems.length === 0) {
+      return null;
+    }
+
     const messages = this.convertApiItemsToMessages(allItems, options);
+    this.enrichMessagesWithDomAttachments(messages);
+
     let title = this.extractTitleFromPage();
     if (!title || title === "Gemini Conversation") {
       const firstUserMsg = messages.find((m) => m.role === "User");
@@ -296,17 +435,28 @@ export class GeminiParser extends ChatParser {
     return null;
   }
 
+  extractTurnId(item) {
+    try {
+      const raw = item[0]?.[1] || item[1]?.[1] || "";
+      return String(raw).replace(/^r_/, "");
+    } catch {
+      return "";
+    }
+  }
+
   convertApiItemsToMessages(items, options = {}) {
     const messages = [];
 
     for (const item of items) {
       if (!Array.isArray(item)) continue;
 
+      const turnId = this.extractTurnId(item);
       const userText = this.findUserTextInApiItem(item);
       if (userText) {
         messages.push({
           role: "User",
           content: userText.trim(),
+          turnId,
         });
       }
 
@@ -314,7 +464,8 @@ export class GeminiParser extends ChatParser {
       if (modelText) {
         messages.push({
           role: "Model",
-          content: modelText.trim(),
+          content: normalizeLatexMath(modelText.trim()),
+          turnId,
         });
       }
     }
@@ -324,6 +475,7 @@ export class GeminiParser extends ChatParser {
 
   findUserTextInApiItem(item) {
     try {
+      if (typeof item[2]?.[0]?.[0] === "string") return item[2][0][0];
       if (typeof item[2]?.[0] === "string") return item[2][0];
       if (typeof item[1]?.[0] === "string" && !Array.isArray(item[1][0]))
         return item[1][0];
@@ -336,6 +488,25 @@ export class GeminiParser extends ChatParser {
 
   findModelTextInApiItem(item) {
     try {
+      // 1. Candidate responses in item[3]
+      if (Array.isArray(item[3])) {
+        const candidates = Array.isArray(item[3][0]) ? item[3][0] : item[3];
+        for (const cand of candidates) {
+          if (!Array.isArray(cand)) continue;
+          // Shape: ["rc_...", ["markdown text", ...], ...]
+          if (Array.isArray(cand[1]) && typeof cand[1][0] === "string") {
+            return cand[1][0];
+          }
+          if (typeof cand[1] === "string") {
+            return cand[1];
+          }
+          if (typeof cand[0] === "string" && cand[0].length > 50) {
+            return cand[0];
+          }
+        }
+      }
+
+      // 2. Fallback candidate in item[1]
       if (Array.isArray(item[1])) {
         const candidate = item[1][0];
         if (typeof candidate === "string") return candidate;

package/package.json

@@ -1,7 +1,7 @@
 {
   "name": "decant-core",
-  "version": "1.5.0",
-  "description": "Shared AI chat platform parsers and detection for Covai browser extensions.",
+  "version": "1.7.0",
+  "description": "A shared web extraction layer for AI conversations and regular web pages.",
   "type": "module",
   "engines": {
     "node": ">=26.0.0"
@@ -47,7 +47,14 @@
     "gemini",
     "deepseek",
     "copilot",
-    "browser-extension"
+    "perplexity",
+    "web-clipper",
+    "extraction",
+    "markdown",
+    "readability",
+    "turndown",
+    "browser-extension",
+    "scraping"
   ],
   "author": "deadrat-in",
   "license": "AGPL-3.0-only",