markitai 0.3.0__tar.gz → 0.3.1__tar.gz

{markitai-0.3.0 → markitai-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: markitai
-Version: 0.3.0
+Version: 0.3.1
 Summary: Document to Markdown converter with LLM enhancement
 Project-URL: Homepage, https://markitai.ynewtime.com
 Project-URL: Documentation, https://markitai.ynewtime.com/guide/getting-started
@@ -27,6 +27,7 @@ Requires-Dist: instructor>=1.14.0
 Requires-Dist: litellm>=1.80.16
 Requires-Dist: loguru>=0.7.3
 Requires-Dist: markitdown[all]>=0.1.4
+Requires-Dist: opencv-python>=4.8.0
 Requires-Dist: pillow>=12.1.0
 Requires-Dist: pydantic>=2.10.0
 Requires-Dist: pymupdf4llm>=0.2.9
@@ -39,19 +40,20 @@ Description-Content-Type: text/markdown
 
 # Markitai
 
-开箱即用的 Markdown 转换器，原生支持 LLM 增强。
+Opinionated Markdown converter with native LLM enhancement support.
 
-## 特性
+## Features
 
-- **多格式支持** - DOCX/DOC, PPTX/PPT, XLSX/XLS, PDF, TXT, MD, JPG/PNG/WebP, URLs
-- **LLM 增强** - 格式清洗、元数据生成、图片分析
-- **批量处理** - 并发转换、断点恢复、进度显示
-- **OCR 识别** - 扫描版 PDF 和图片文字提取
-- **URL 转换** - 直接转换网页，支持 SPA 浏览器渲染
+- **Multi-format Support** - DOCX/DOC, PPTX/PPT, XLSX/XLS, PDF, TXT, MD, JPG/PNG/WebP, URLs
+- **LLM Enhancement** - Format cleaning, metadata generation, image analysis
+- **Batch Processing** - Concurrent conversion, resume capability, progress display
+- **OCR Recognition** - Text extraction from scanned PDFs and images
+- **URL Conversion** - Direct webpage conversion with SPA browser rendering support
+- **Smart Caching** - LLM result caching, SPA domain learning, auto-proxy detection
 
-## 安装
+## Installation
 
-### 一键安装（推荐）
+### One-Click Setup (Recommended)
 
 ```bash
 # Linux/macOS
@@ -61,98 +63,98 @@ curl -fsSL https://raw.githubusercontent.com/Ynewtime/markitai/main/scripts/setu
 irm https://raw.githubusercontent.com/Ynewtime/markitai/main/scripts/setup.ps1 | iex
 ```
 
-### 手动安装
+### Manual Installation
 
 ```bash
-# 需要 Python 3.11+
+# Requires Python 3.11+
 uv tool install markitai
 
-# 或使用 pip
+# Or using pip
 pip install --user markitai
 ```
 
-## 快速开始
+## Quick Start
 
 ```bash
-# 基础转换
+# Basic conversion
 markitai document.docx
 
-# URL 转换
+# URL conversion
 markitai https://example.com/article
 
-# LLM 增强
+# LLM enhancement
 markitai document.docx --llm
 
-# 使用预设
+# Using presets
 markitai document.pdf --preset rich      # LLM + alt + desc + screenshot
 markitai document.pdf --preset standard  # LLM + alt + desc
-markitai document.pdf --preset minimal   # 仅基础转换
+markitai document.pdf --preset minimal   # Basic conversion only
 
-# 批量处理
+# Batch processing
 markitai ./docs -o ./output
 
-# 断点恢复
+# Resume interrupted job
 markitai ./docs -o ./output --resume
 
-# URL 批量处理（自动识别 .urls 文件）
+# Batch URL processing (auto-detect .urls files)
 markitai urls.urls -o ./output
 ```
 
-## 输出结构
+## Output Structure
 
 ```
 output/
-├── document.docx.md        # 基础 Markdown
-├── document.docx.llm.md    # LLM 优化版
+├── document.docx.md        # Basic Markdown
+├── document.docx.llm.md    # LLM-enhanced version
 ├── assets/
 │   ├── document.docx.0001.jpg
-│   └── images.json         # 图片描述
-├── screenshots/            # 页面截图（--screenshot 时）
+│   └── images.json         # Image descriptions
+├── screenshots/            # Page screenshots (with --screenshot)
 │   └── example_com.full.jpg
 ```
 
-## 配置
+## Configuration
 
-优先级：命令行 > 环境变量 > 配置文件 > 默认值
+Priority: CLI arguments > Environment variables > Config file > Defaults
 
 ```bash
-# 查看配置
+# View configuration
 markitai config list
 
-# 初始化配置文件
+# Initialize config file
 markitai config init -o .
 
-# 查看缓存状态
+# View cache status
 markitai cache stats
 
-# 清理缓存
+# Clear cache
 markitai cache clear
 ```
 
-配置文件路径：`./markitai.json` 或 `~/.markitai/config.json`
+Config file location: `./markitai.json` or `~/.markitai/config.json`
 
-## 环境变量
+## Environment Variables
 
-| 变量 | 说明 |
-|------|------|
+| Variable | Description |
+|----------|-------------|
 | `OPENAI_API_KEY` | OpenAI API Key |
 | `GEMINI_API_KEY` | Google Gemini API Key |
 | `DEEPSEEK_API_KEY` | DeepSeek API Key |
 | `ANTHROPIC_API_KEY` | Anthropic API Key |
-| `JINA_API_KEY` | Jina Reader API Key（URL 转换） |
+| `JINA_API_KEY` | Jina Reader API Key (URL conversion) |
 
-## 依赖
+## Dependencies
 
-- [pymupdf4llm](https://github.com/pymupdf/RAG) - PDF 转换
-- [markitdown](https://github.com/microsoft/markitdown) - Office 文档和 URL 转换
-- [LiteLLM](https://github.com/BerriAI/litellm) - LLM 网关
-- [RapidOCR](https://github.com/RapidAI/RapidOCR) - OCR 识别
+- [pymupdf4llm](https://github.com/pymupdf/RAG) - PDF conversion
+- [markitdown](https://github.com/microsoft/markitdown) - Office documents and URL conversion
+- [LiteLLM](https://github.com/BerriAI/litellm) - LLM gateway
+- [RapidOCR](https://github.com/RapidAI/RapidOCR) - OCR recognition
 
-## 文档
+## Documentation
 
-- [快速开始](https://ynewtime.github.io/markitai/guide/getting-started)
-- [配置说明](https://ynewtime.github.io/markitai/guide/configuration)
-- [CLI 命令参考](https://ynewtime.github.io/markitai/guide/cli)
+- [Getting Started](https://markitai.ynewtime.com/guide/getting-started)
+- [Configuration](https://markitai.ynewtime.com/guide/configuration)
+- [CLI Reference](https://markitai.ynewtime.com/guide/cli)
 
 ## License
 

markitai-0.3.1/README.md

@@ -0,0 +1,121 @@
+# Markitai
+
+Opinionated Markdown converter with native LLM enhancement support.
+
+## Features
+
+- **Multi-format Support** - DOCX/DOC, PPTX/PPT, XLSX/XLS, PDF, TXT, MD, JPG/PNG/WebP, URLs
+- **LLM Enhancement** - Format cleaning, metadata generation, image analysis
+- **Batch Processing** - Concurrent conversion, resume capability, progress display
+- **OCR Recognition** - Text extraction from scanned PDFs and images
+- **URL Conversion** - Direct webpage conversion with SPA browser rendering support
+- **Smart Caching** - LLM result caching, SPA domain learning, auto-proxy detection
+
+## Installation
+
+### One-Click Setup (Recommended)
+
+```bash
+# Linux/macOS
+curl -fsSL https://raw.githubusercontent.com/Ynewtime/markitai/main/scripts/setup.sh | sh
+
+# Windows (PowerShell)
+irm https://raw.githubusercontent.com/Ynewtime/markitai/main/scripts/setup.ps1 | iex
+```
+
+### Manual Installation
+
+```bash
+# Requires Python 3.11+
+uv tool install markitai
+
+# Or using pip
+pip install --user markitai
+```
+
+## Quick Start
+
+```bash
+# Basic conversion
+markitai document.docx
+
+# URL conversion
+markitai https://example.com/article
+
+# LLM enhancement
+markitai document.docx --llm
+
+# Using presets
+markitai document.pdf --preset rich      # LLM + alt + desc + screenshot
+markitai document.pdf --preset standard  # LLM + alt + desc
+markitai document.pdf --preset minimal   # Basic conversion only
+
+# Batch processing
+markitai ./docs -o ./output
+
+# Resume interrupted job
+markitai ./docs -o ./output --resume
+
+# Batch URL processing (auto-detect .urls files)
+markitai urls.urls -o ./output
+```
+
+## Output Structure
+
+```
+output/
+├── document.docx.md        # Basic Markdown
+├── document.docx.llm.md    # LLM-enhanced version
+├── assets/
+│   ├── document.docx.0001.jpg
+│   └── images.json         # Image descriptions
+├── screenshots/            # Page screenshots (with --screenshot)
+│   └── example_com.full.jpg
+```
+
+## Configuration
+
+Priority: CLI arguments > Environment variables > Config file > Defaults
+
+```bash
+# View configuration
+markitai config list
+
+# Initialize config file
+markitai config init -o .
+
+# View cache status
+markitai cache stats
+
+# Clear cache
+markitai cache clear
+```
+
+Config file location: `./markitai.json` or `~/.markitai/config.json`
+
+## Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `OPENAI_API_KEY` | OpenAI API Key |
+| `GEMINI_API_KEY` | Google Gemini API Key |
+| `DEEPSEEK_API_KEY` | DeepSeek API Key |
+| `ANTHROPIC_API_KEY` | Anthropic API Key |
+| `JINA_API_KEY` | Jina Reader API Key (URL conversion) |
+
+## Dependencies
+
+- [pymupdf4llm](https://github.com/pymupdf/RAG) - PDF conversion
+- [markitdown](https://github.com/microsoft/markitdown) - Office documents and URL conversion
+- [LiteLLM](https://github.com/BerriAI/litellm) - LLM gateway
+- [RapidOCR](https://github.com/RapidAI/RapidOCR) - OCR recognition
+
+## Documentation
+
+- [Getting Started](https://markitai.ynewtime.com/guide/getting-started)
+- [Configuration](https://markitai.ynewtime.com/guide/configuration)
+- [CLI Reference](https://markitai.ynewtime.com/guide/cli)
+
+## License
+
+MIT

{markitai-0.3.0 → markitai-0.3.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "markitai"
-version = "0.3.0"
+version = "0.3.1"
 description = "Document to Markdown converter with LLM enhancement"
 license = "MIT"
 readme = "README.md"
@@ -32,6 +32,7 @@ dependencies = [
     "loguru>=0.7.3",
     "rich>=14.2.0",
     "Pillow>=12.1.0",
+    "opencv-python>=4.8.0",
     "aiofiles>=25.1.0",
     "pydantic>=2.10.0",
     "python-dotenv>=1.2.1",

{markitai-0.3.0 → markitai-0.3.1}/src/markitai/__init__.py

@@ -1,3 +1,3 @@
 """Markitai - Document to Markdown converter with LLM enhancement."""
 
-__version__ = "0.3.0"
+__version__ = "0.3.1"

{markitai-0.3.0 → markitai-0.3.1}/src/markitai/batch.py

@@ -1037,6 +1037,15 @@ class BatchProcessor:
             self.save_state(force=True)
             return self.state
 
+        # Preheat OCR engine if OCR is enabled to eliminate cold start delay
+        if options and options.get("ocr_enabled"):
+            try:
+                from markitai.ocr import OCRProcessor
+
+                OCRProcessor.preheat()
+            except ImportError:
+                logger.debug("OCR preheat skipped: RapidOCR not installed")
+
         # Create semaphore for concurrency control
         semaphore = asyncio.Semaphore(self.config.concurrency)
 

{markitai-0.3.0 → markitai-0.3.1}/src/markitai/cli.py

@@ -978,9 +978,16 @@ def app(
             # Cleanup shared resources
             await close_shared_clients()  # Close httpx.AsyncClient for Jina
             shutdown_converter_executor()  # Shutdown ThreadPoolExecutor
-            # Note: FetchCache cleanup happens automatically when process exits
-            # as SQLite handles connection cleanup. For explicit cleanup, the
-            # global _fetch_cache.close() could be called, but it's not critical.
+
+            # Close LiteLLM's aiohttp sessions to prevent "Unclosed connection" warning
+            try:
+                from litellm.llms.custom_httpx.async_client_cleanup import (
+                    close_litellm_async_clients,
+                )
+
+                await close_litellm_async_clients()
+            except Exception:
+                pass  # Ignore cleanup errors
 
     asyncio.run(run_workflow_with_cleanup())
 
@@ -1344,13 +1351,18 @@ def cache_stats(as_json: bool, verbose: bool, limit: int, scope: str) -> None:
     default="project",
     help="Which cache to clear (default: project).",
 )
+@click.option(
+    "--include-spa-domains",
+    is_flag=True,
+    help="Also clear learned SPA domains.",
+)
 @click.option(
     "--yes",
     "-y",
     is_flag=True,
     help="Skip confirmation prompt.",
 )
-def cache_clear(scope: str, yes: bool) -> None:
+def cache_clear(scope: str, include_spa_domains: bool, yes: bool) -> None:
     """Clear cache entries."""
     from markitai.constants import (
         DEFAULT_CACHE_DB_FILENAME,
@@ -1368,11 +1380,14 @@ def cache_clear(scope: str, yes: bool) -> None:
             "global": "global cache (~/.markitai)",
             "all": "ALL caches (project + global)",
         }
-        if not click.confirm(f"Clear {scope_desc[scope]}?"):
+        desc = scope_desc[scope]
+        if include_spa_domains:
+            desc += " + learned SPA domains"
+        if not click.confirm(f"Clear {desc}?"):
             console.print("[yellow]Aborted[/yellow]")
             return
 
-    result = {"project": 0, "global": 0}
+    result = {"project": 0, "global": 0, "spa_domains": 0}
 
     # Clear project cache
     if scope in ("project", "all"):
@@ -1400,18 +1415,105 @@ def cache_clear(scope: str, yes: bool) -> None:
             except Exception as e:
                 console.print(f"[red]Failed to clear global cache:[/red] {e}")
 
+    # Clear SPA domains if requested
+    if include_spa_domains:
+        from markitai.fetch import get_spa_domain_cache
+
+        try:
+            spa_cache = get_spa_domain_cache()
+            result["spa_domains"] = spa_cache.clear()
+        except Exception as e:
+            console.print(f"[red]Failed to clear SPA domains:[/red] {e}")
+
     # Report results
     total = result["project"] + result["global"]
-    if total > 0:
+    if total > 0 or result["spa_domains"] > 0:
         console.print(f"[green]Cleared {total} cache entries[/green]")
         if result["project"] > 0:
             console.print(f"  Project: {result['project']}")
         if result["global"] > 0:
             console.print(f"  Global: {result['global']}")
+        if result["spa_domains"] > 0:
+            console.print(f"  SPA domains: {result['spa_domains']}")
     else:
         console.print("[dim]No cache entries to clear[/dim]")
 
 
+@cache.command("spa-domains")
+@click.option(
+    "--json",
+    "as_json",
+    is_flag=True,
+    help="Output as JSON.",
+)
+@click.option(
+    "--clear",
+    is_flag=True,
+    help="Clear all learned SPA domains.",
+)
+def cache_spa_domains(as_json: bool, clear: bool) -> None:
+    """View or manage learned SPA domains.
+
+    Shows domains that were automatically detected as requiring browser
+    rendering (JavaScript-heavy sites). These domains will use browser
+    strategy directly on future requests, avoiding wasted static fetch attempts.
+    """
+    from rich.table import Table
+
+    from markitai.fetch import get_spa_domain_cache
+
+    spa_cache = get_spa_domain_cache()
+
+    if clear:
+        count = spa_cache.clear()
+        if as_json:
+            console.print(json.dumps({"cleared": count}))
+        else:
+            console.print(f"[green]Cleared {count} learned SPA domains[/green]")
+        return
+
+    domains = spa_cache.list_domains()
+
+    if as_json:
+        console.print(json.dumps(domains, indent=2, ensure_ascii=False), soft_wrap=True)
+        return
+
+    if not domains:
+        console.print("[dim]No learned SPA domains yet[/dim]")
+        console.print(
+            "\n[dim]Domains are learned automatically when static fetch "
+            "detects JavaScript requirement.[/dim]"
+        )
+        return
+
+    console.print(f"[bold]Learned SPA Domains[/bold] ({len(domains)} total)\n")
+
+    table = Table()
+    table.add_column("Domain", style="cyan")
+    table.add_column("Hits", justify="right")
+    table.add_column("Learned At", style="dim")
+    table.add_column("Last Hit", style="dim")
+    table.add_column("Status")
+
+    for d in domains:
+        status = "[red]Expired[/red]" if d.get("expired") else "[green]Active[/green]"
+        learned_at = d.get("learned_at", "")[:10] if d.get("learned_at") else "-"
+        last_hit = d.get("last_hit", "")[:10] if d.get("last_hit") else "-"
+        table.add_row(
+            d["domain"],
+            str(d.get("hits", 0)),
+            learned_at,
+            last_hit,
+            status,
+        )
+
+    console.print(table)
+    console.print(
+        "\n[dim]Tip: Use --clear to reset learned domains, "
+        "or configure fallback_patterns in config file for permanent rules.[/dim]"
+    )
+
+
 # =============================================================================
 # Check dependencies command
 # =============================================================================
@@ -3097,7 +3199,7 @@ def _check_agent_browser_for_urls(cfg: Any, console: Any) -> None:
         from rich.panel import Panel
 
         warning_text = (
-            f"[yellow]⚠ {message}[/yellow]\n\n"
+            f"[yellow]{message}[/yellow]\n\n"
             "[dim]URL processing will fall back to static fetch strategy.\n"
             "For JavaScript-rendered pages (Twitter/X, etc.), browser support is recommended.\n\n"
             "To install browser support:[/dim]\n"

{markitai-0.3.0 → markitai-0.3.1}/src/markitai/config.py

@@ -203,17 +203,40 @@ class ScreenshotConfig(BaseModel):
 
 
 class PromptsConfig(BaseModel):
-    """Prompts configuration."""
+    """Prompts configuration for custom prompt overrides.
+
+    Each prompt has a system and user variant:
+    - system: Contains instructions and context
+    - user: Contains the actual request with content placeholders
+    """
 
     dir: str = DEFAULT_PROMPTS_DIR
-    cleaner: str | None = None
-    frontmatter: str | None = None
-    image_caption: str | None = None
-    image_description: str | None = None
-    image_analysis: str | None = None  # Combined caption + description
-    page_content: str | None = None  # Page content extraction
-    document_enhance: str | None = None  # Document enhancement with vision
-    url_enhance: str | None = None  # URL/web page content enhancement
+    # Cleaner prompts
+    cleaner_system: str | None = None
+    cleaner_user: str | None = None
+    # Frontmatter prompts
+    frontmatter_system: str | None = None
+    frontmatter_user: str | None = None
+    # Image prompts
+    image_caption_system: str | None = None
+    image_caption_user: str | None = None
+    image_description_system: str | None = None
+    image_description_user: str | None = None
+    image_analysis_system: str | None = None
+    image_analysis_user: str | None = None
+    # Page content prompts
+    page_content_system: str | None = None
+    page_content_user: str | None = None
+    # Document prompts
+    document_enhance_system: str | None = None
+    document_enhance_user: str | None = None
+    document_enhance_complete_system: str | None = None
+    document_enhance_complete_user: str | None = None
+    document_process_system: str | None = None
+    document_process_user: str | None = None
+    # URL prompts
+    url_enhance_system: str | None = None
+    url_enhance_user: str | None = None
 
 
 class BatchConfig(BaseModel):