scientific-writer 2.2.1__py3-none-any.whl → 2.2.3__py3-none-any.whl

scientific_writer/.claude/skills/markitdown/QUICK_REFERENCE.md

@@ -0,0 +1,309 @@
+# MarkItDown Quick Reference
+
+## Installation
+
+```bash
+# All features
+pip install 'markitdown[all]'
+
+# Specific formats
+pip install 'markitdown[pdf,docx,pptx,xlsx]'
+```
+
+## Basic Usage
+
+```python
+from markitdown import MarkItDown
+
+md = MarkItDown()
+result = md.convert("file.pdf")
+print(result.text_content)
+```
+
+## Command Line
+
+```bash
+# Simple conversion
+markitdown input.pdf > output.md
+markitdown input.pdf -o output.md
+
+# With plugins
+markitdown --use-plugins file.pdf -o output.md
+```
+
+## Common Tasks
+
+### Convert PDF
+```python
+md = MarkItDown()
+result = md.convert("paper.pdf")
+```
+
+### Convert with AI
+```python
+from openai import OpenAI
+
+# Use OpenRouter for multiple model access
+client = OpenAI(
+    api_key="your-openrouter-api-key",
+    base_url="https://openrouter.ai/api/v1"
+)
+
+md = MarkItDown(
+    llm_client=client,
+    llm_model="anthropic/claude-sonnet-4.5"  # recommended for vision
+)
+result = md.convert("slides.pptx")
+```
+
+### Batch Convert
+```bash
+python scripts/batch_convert.py input/ output/ --extensions .pdf .docx
+```
+
+### Literature Conversion
+```bash
+python scripts/convert_literature.py papers/ markdown/ --create-index
+```
+
+## Supported Formats
+
+| Format | Extension | Notes |
+|--------|-----------|-------|
+| PDF | `.pdf` | Full text + OCR |
+| Word | `.docx` | Tables, formatting |
+| PowerPoint | `.pptx` | Slides + notes |
+| Excel | `.xlsx`, `.xls` | Tables |
+| Images | `.jpg`, `.png`, `.gif`, `.webp` | EXIF + OCR |
+| Audio | `.wav`, `.mp3` | Transcription |
+| HTML | `.html`, `.htm` | Clean conversion |
+| Data | `.csv`, `.json`, `.xml` | Structured |
+| Archives | `.zip` | Iterates contents |
+| E-books | `.epub` | Full text |
+| YouTube | URLs | Transcripts |
+
+## Optional Dependencies
+
+```bash
+[all]                  # All features
+[pdf]                  # PDF support
+[docx]                 # Word documents
+[pptx]                 # PowerPoint
+[xlsx]                 # Excel
+[xls]                  # Old Excel
+[outlook]              # Outlook messages
+[az-doc-intel]         # Azure Document Intelligence
+[audio-transcription]  # Audio files
+[youtube-transcription] # YouTube videos
+```
+
+## AI-Enhanced Conversion
+
+### Scientific Papers
+```python
+from openai import OpenAI
+
+# Initialize OpenRouter client
+client = OpenAI(
+    api_key="your-openrouter-api-key",
+    base_url="https://openrouter.ai/api/v1"
+)
+
+md = MarkItDown(
+    llm_client=client,
+    llm_model="anthropic/claude-sonnet-4.5",  # recommended for scientific vision
+    llm_prompt="Describe scientific figures with technical precision"
+)
+result = md.convert("paper.pdf")
+```
+
+### Custom Prompts
+```python
+prompt = """
+Analyze this data visualization. Describe:
+- Type of chart/graph
+- Key trends and patterns
+- Notable data points
+"""
+
+md = MarkItDown(
+    llm_client=client,
+    llm_model="anthropic/claude-sonnet-4.5",
+    llm_prompt=prompt
+)
+```
+
+### Available Models via OpenRouter
+- `anthropic/claude-sonnet-4.5` - **Claude Sonnet 4.5 (recommended for scientific vision)**
+- `anthropic/claude-3.5-sonnet` - Claude 3.5 Sonnet (vision)
+- `openai/gpt-4o` - GPT-4 Omni (vision)
+- `openai/gpt-4-vision` - GPT-4 Vision
+- `google/gemini-pro-vision` - Gemini Pro Vision
+
+See https://openrouter.ai/models for full list
+
+## Azure Document Intelligence
+
+```python
+md = MarkItDown(docintel_endpoint="https://YOUR-ENDPOINT.cognitiveservices.azure.com/")
+result = md.convert("complex_layout.pdf")
+```
+
+## Batch Processing
+
+### Python
+```python
+from markitdown import MarkItDown
+from pathlib import Path
+
+md = MarkItDown()
+
+for file in Path("input/").glob("*.pdf"):
+    result = md.convert(str(file))
+    output = Path("output") / f"{file.stem}.md"
+    output.write_text(result.text_content)
+```
+
+### Script
+```bash
+# Parallel conversion
+python scripts/batch_convert.py input/ output/ --workers 8
+
+# Recursive
+python scripts/batch_convert.py input/ output/ -r
+```
+
+## Error Handling
+
+```python
+try:
+    result = md.convert("file.pdf")
+except FileNotFoundError:
+    print("File not found")
+except Exception as e:
+    print(f"Error: {e}")
+```
+
+## Streaming
+
+```python
+with open("large_file.pdf", "rb") as f:
+    result = md.convert_stream(f, file_extension=".pdf")
+```
+
+## Common Prompts
+
+### Scientific
+```
+Analyze this scientific figure. Describe:
+- Type of visualization
+- Key data points and trends
+- Axes, labels, and legends
+- Scientific significance
+```
+
+### Medical
+```
+Describe this medical image. Include:
+- Type of imaging (X-ray, MRI, CT, etc.)
+- Anatomical structures visible
+- Notable findings
+- Clinical relevance
+```
+
+### Data Visualization
+```
+Analyze this data visualization:
+- Chart type
+- Variables and axes
+- Data ranges
+- Key patterns and outliers
+```
+
+## Performance Tips
+
+1. **Reuse instance**: Create once, use many times
+2. **Parallel processing**: Use ThreadPoolExecutor for multiple files
+3. **Stream large files**: Use `convert_stream()` for big files
+4. **Choose right format**: Install only needed dependencies
+
+## Environment Variables
+
+```bash
+# OpenRouter for AI-enhanced conversions
+export OPENROUTER_API_KEY="sk-or-v1-..."
+
+# Azure Document Intelligence (optional)
+export AZURE_DOCUMENT_INTELLIGENCE_KEY="key..."
+export AZURE_DOCUMENT_INTELLIGENCE_ENDPOINT="https://..."
+```
+
+## Scripts Quick Reference
+
+### batch_convert.py
+```bash
+python scripts/batch_convert.py INPUT OUTPUT [OPTIONS]
+
+Options:
+  --extensions .pdf .docx    File types to convert
+  --recursive, -r            Search subdirectories
+  --workers 4                Parallel workers
+  --verbose, -v              Detailed output
+  --plugins, -p              Enable plugins
+```
+
+### convert_with_ai.py
+```bash
+python scripts/convert_with_ai.py INPUT OUTPUT [OPTIONS]
+
+Options:
+  --api-key KEY              OpenRouter API key
+  --model MODEL              Model name (default: anthropic/claude-sonnet-4.5)
+  --prompt-type TYPE         Preset prompt (scientific, medical, etc.)
+  --custom-prompt TEXT       Custom prompt
+  --list-prompts             Show available prompts
+```
+
+### convert_literature.py
+```bash
+python scripts/convert_literature.py INPUT OUTPUT [OPTIONS]
+
+Options:
+  --organize-by-year, -y     Organize by year
+  --create-index, -i         Create index file
+  --recursive, -r            Search subdirectories
+```
+
+## Troubleshooting
+
+### Missing Dependencies
+```bash
+pip install 'markitdown[pdf]'  # Install PDF support
+```
+
+### Binary File Error
+```python
+# Wrong
+with open("file.pdf", "r") as f:
+
+# Correct
+with open("file.pdf", "rb") as f:  # Binary mode
+```
+
+### OCR Not Working
+```bash
+# macOS
+brew install tesseract
+
+# Ubuntu
+sudo apt-get install tesseract-ocr
+```
+
+## More Information
+
+- **Full Documentation**: See `SKILL.md`
+- **API Reference**: See `references/api_reference.md`
+- **Format Details**: See `references/file_formats.md`
+- **Examples**: See `assets/example_usage.md`
+- **GitHub**: https://github.com/microsoft/markitdown
+

scientific_writer/.claude/skills/markitdown/README.md

@@ -0,0 +1,184 @@
+# MarkItDown Skill
+
+This skill provides comprehensive support for converting various file formats to Markdown using Microsoft's MarkItDown tool.
+
+## Overview
+
+MarkItDown is a Python tool that converts files and office documents to Markdown format. This skill includes:
+
+- Complete API documentation
+- Format-specific conversion guides
+- Utility scripts for batch processing
+- AI-enhanced conversion examples
+- Integration with scientific workflows
+
+## Contents
+
+### Main Skill File
+- **SKILL.md** - Complete guide to using MarkItDown with quick start, examples, and best practices
+
+### References
+- **api_reference.md** - Detailed API documentation, class references, and method signatures
+- **file_formats.md** - Format-specific details for all supported file types
+
+### Scripts
+- **batch_convert.py** - Batch convert multiple files with parallel processing
+- **convert_with_ai.py** - AI-enhanced conversion with custom prompts
+- **convert_literature.py** - Scientific literature conversion with metadata extraction
+
+### Assets
+- **example_usage.md** - Practical examples for common use cases
+
+## Installation
+
+```bash
+# Install with all features
+pip install 'markitdown[all]'
+
+# Or install specific features
+pip install 'markitdown[pdf,docx,pptx,xlsx]'
+```
+
+## Quick Start
+
+```python
+from markitdown import MarkItDown
+
+md = MarkItDown()
+result = md.convert("document.pdf")
+print(result.text_content)
+```
+
+## Supported Formats
+
+- **Documents**: PDF, DOCX, PPTX, XLSX, EPUB
+- **Images**: JPEG, PNG, GIF, WebP (with OCR)
+- **Audio**: WAV, MP3 (with transcription)
+- **Web**: HTML, YouTube URLs
+- **Data**: CSV, JSON, XML
+- **Archives**: ZIP files
+
+## Key Features
+
+### 1. AI-Enhanced Conversions
+Use AI models via OpenRouter to generate detailed image descriptions:
+
+```python
+from openai import OpenAI
+
+# OpenRouter provides access to 100+ AI models
+client = OpenAI(
+    api_key="your-openrouter-api-key",
+    base_url="https://openrouter.ai/api/v1"
+)
+
+md = MarkItDown(
+    llm_client=client,
+    llm_model="anthropic/claude-sonnet-4.5"  # recommended for vision
+)
+result = md.convert("presentation.pptx")
+```
+
+### 2. Batch Processing
+Convert multiple files efficiently:
+
+```bash
+python scripts/batch_convert.py papers/ output/ --extensions .pdf .docx
+```
+
+### 3. Scientific Literature
+Convert and organize research papers:
+
+```bash
+python scripts/convert_literature.py papers/ output/ --organize-by-year --create-index
+```
+
+### 4. Azure Document Intelligence
+Enhanced PDF conversion with Microsoft Document Intelligence:
+
+```python
+md = MarkItDown(docintel_endpoint="https://YOUR-ENDPOINT.cognitiveservices.azure.com/")
+result = md.convert("complex_document.pdf")
+```
+
+## Use Cases
+
+### Literature Review
+Convert research papers to Markdown for easier analysis and note-taking.
+
+### Data Extraction
+Extract tables from Excel files into Markdown format.
+
+### Presentation Processing
+Convert PowerPoint slides with AI-generated descriptions.
+
+### Document Analysis
+Process documents for LLM consumption with token-efficient Markdown.
+
+### YouTube Transcripts
+Fetch and convert YouTube video transcriptions.
+
+## Scripts Usage
+
+### Batch Convert
+```bash
+# Convert all PDFs in a directory
+python scripts/batch_convert.py input_dir/ output_dir/ --extensions .pdf
+
+# Recursive with multiple formats
+python scripts/batch_convert.py docs/ markdown/ --extensions .pdf .docx .pptx -r
+```
+
+### AI-Enhanced Conversion
+```bash
+# Convert with AI descriptions via OpenRouter
+export OPENROUTER_API_KEY="sk-or-v1-..."
+python scripts/convert_with_ai.py paper.pdf output.md --prompt-type scientific
+
+# Use different models
+python scripts/convert_with_ai.py image.png output.md --model anthropic/claude-sonnet-4.5
+
+# Use custom prompt
+python scripts/convert_with_ai.py image.png output.md --custom-prompt "Describe this diagram"
+```
+
+### Literature Conversion
+```bash
+# Convert papers with metadata extraction
+python scripts/convert_literature.py papers/ markdown/ --organize-by-year --create-index
+```
+
+## Integration with Scientific Writer
+
+This skill integrates seamlessly with the Scientific Writer CLI for:
+- Converting source materials for paper writing
+- Processing literature for reviews
+- Extracting data from various document formats
+- Preparing documents for LLM analysis
+
+## Resources
+
+- **MarkItDown GitHub**: https://github.com/microsoft/markitdown
+- **PyPI**: https://pypi.org/project/markitdown/
+- **OpenRouter**: https://openrouter.ai (AI model access)
+- **OpenRouter API Keys**: https://openrouter.ai/keys
+- **OpenRouter Models**: https://openrouter.ai/models
+- **License**: MIT
+
+## Requirements
+
+- Python 3.10+
+- Optional dependencies based on formats needed
+- OpenRouter API key (for AI-enhanced conversions) - Get at https://openrouter.ai/keys
+- Azure subscription (optional, for Document Intelligence)
+
+## Examples
+
+See `assets/example_usage.md` for comprehensive examples covering:
+- Basic conversions
+- Scientific workflows
+- AI-enhanced processing
+- Batch operations
+- Error handling
+- Integration patterns
+