content-core 1.2.2__tar.gz → 1.3.0__tar.gz

{content_core-1.2.2 → content_core-1.3.0}/.gitignore

@@ -25,4 +25,6 @@ WIP/
 CLAUDE.md
 
 node_modules/
-**/notebooks/private
+**/notebooks/private
+
+.claude/

{content_core-1.2.2 → content_core-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: content-core
-Version: 1.2.2
+Version: 1.3.0
 Summary: Extract what matters from any media source. Available as Python Library, macOS Service, CLI and MCP Server
 Author-email: LUIS NOVO <lfnovo@gmail.com>
 License-File: LICENSE
@@ -11,6 +11,7 @@ Requires-Dist: asciidoc>=10.2.1
 Requires-Dist: bs4>=0.0.2
 Requires-Dist: dicttoxml>=1.7.16
 Requires-Dist: esperanto>=1.2.0
+Requires-Dist: fastmcp>=0.5.0
 Requires-Dist: firecrawl-py>=2.7.0
 Requires-Dist: jinja2>=3.1.6
 Requires-Dist: langdetect>=1.0.9
@@ -32,8 +33,6 @@ Requires-Dist: validators>=0.34.0
 Requires-Dist: youtube-transcript-api>=1.0.3
 Provides-Extra: docling
 Requires-Dist: docling>=2.34.0; extra == 'docling'
-Provides-Extra: mcp
-Requires-Dist: fastmcp>=0.5.0; extra == 'mcp'
 Description-Content-Type: text/markdown
 
 # Content Core
@@ -118,11 +117,11 @@ pip install content-core
 # With enhanced document processing (adds Docling)
 pip install content-core[docling]
 
-# With MCP server support
-pip install content-core[mcp]
+# With MCP server support (now included by default)
+pip install content-core
 
-# Full installation
-pip install content-core[docling,mcp]
+# Full installation (with enhanced document processing)
+pip install content-core[docling]
 ```
 
 Alternatively, if you’re developing locally:
@@ -268,11 +267,11 @@ Content Core includes a Model Context Protocol (MCP) server that enables seamles
 ### Quick Setup with Claude Desktop
 
 ```bash
-# Install with MCP support
-pip install content-core[mcp]
+# Install Content Core (MCP server included)
+pip install content-core
 
 # Or use directly with uvx (no installation required)
-uvx --from "content-core[mcp]" content-core-mcp
+uvx --from "content-core" content-core-mcp
 ```
 
 Add to your `claude_desktop_config.json`:
@@ -283,7 +282,7 @@ Add to your `claude_desktop_config.json`:
       "command": "uvx",
       "args": [
         "--from",
-        "content-core[mcp]",
+        "content-core",
         "content-core-mcp"
       ]
     }

{content_core-1.2.2 → content_core-1.3.0}/README.md

@@ -80,11 +80,11 @@ pip install content-core
 # With enhanced document processing (adds Docling)
 pip install content-core[docling]
 
-# With MCP server support
-pip install content-core[mcp]
+# With MCP server support (now included by default)
+pip install content-core
 
-# Full installation
-pip install content-core[docling,mcp]
+# Full installation (with enhanced document processing)
+pip install content-core[docling]
 ```
 
 Alternatively, if you’re developing locally:
@@ -230,11 +230,11 @@ Content Core includes a Model Context Protocol (MCP) server that enables seamles
 ### Quick Setup with Claude Desktop
 
 ```bash
-# Install with MCP support
-pip install content-core[mcp]
+# Install Content Core (MCP server included)
+pip install content-core
 
 # Or use directly with uvx (no installation required)
-uvx --from "content-core[mcp]" content-core-mcp
+uvx --from "content-core" content-core-mcp
 ```
 
 Add to your `claude_desktop_config.json`:
@@ -245,7 +245,7 @@ Add to your `claude_desktop_config.json`:
       "command": "uvx",
       "args": [
         "--from",
-        "content-core[mcp]",
+        "content-core",
         "content-core-mcp"
       ]
     }

{content_core-1.2.2 → content_core-1.3.0}/docs/mcp.md

@@ -20,8 +20,8 @@ The [Model Context Protocol (MCP)](https://modelcontextprotocol.io/) is an open
 ### Option 1: Install with pip (Recommended for local development)
 
 ```bash
-# Install Content Core with MCP support
-pip install content-core[mcp]
+# Install Content Core (MCP server included by default)
+pip install content-core
 
 # The content-core-mcp command becomes available
 content-core-mcp
@@ -31,7 +31,7 @@ content-core-mcp
 
 ```bash
 # Run MCP server directly without installation
-uvx --from "content-core[mcp]" content-core-mcp
+uvx --from "content-core" content-core-mcp
 
 # Also works for CLI tools
 uvx --from "content-core" ccore https://example.com
@@ -58,7 +58,7 @@ Add Content Core to your Claude Desktop configuration file:
       "command": "uvx",
       "args": [
         "--from",
-        "content-core[mcp]",
+        "content-core",
         "content-core-mcp"
       ]
     }
@@ -102,7 +102,7 @@ For optimal functionality, you'll need to configure API keys. Here's what each k
       "command": "uvx",
       "args": [
         "--from",
-        "content-core[mcp]",
+        "content-core",
         "content-core-mcp"
       ],
       "env": {
@@ -342,13 +342,13 @@ export PROMPT_PATH="/path/to/your/custom/prompts"
 content-core-mcp
 
 # Or with uvx
-uvx --from "content-core[mcp]" content-core-mcp
+uvx --from "content-core" content-core-mcp
 ```
 
 **Missing dependencies:**
 ```bash
-# Reinstall with MCP dependencies
-pip install --force-reinstall content-core[mcp]
+# Reinstall Content Core
+pip install --force-reinstall content-core
 ```
 
 **Audio/video extraction failing:**

content_core-1.3.0/examples/main.py

@@ -0,0 +1,241 @@
+#!/usr/bin/env python3
+"""
+MarkDowny - Convert files and URLs to Markdown using Microsoft's MarkItDown library.
+
+This script processes all files in the input_content/ directory and URLs from urls.txt,
+converting them to Markdown format and saving the results to separate files.
+"""
+
+import os
+import sys
+from pathlib import Path
+from typing import List, Tuple
+from urllib.parse import urlparse
+
+from loguru import logger
+from markitdown import MarkItDown
+
+
+def setup_logging() -> None:
+    """Configure logging with loguru."""
+    logger.remove()
+    logger.add(
+        sys.stderr,
+        format="<green>{time:YYYY-MM-DD HH:mm:ss}</green> | <level>{level: <8}</level> | <cyan>{name}</cyan>:<cyan>{function}</cyan>:<cyan>{line}</cyan> - <level>{message}</level>",
+        level="INFO"
+    )
+    logger.add(
+        "processing.log",
+        format="{time:YYYY-MM-DD HH:mm:ss} | {level: <8} | {name}:{function}:{line} - {message}",
+        level="DEBUG",
+        rotation="10 MB"
+    )
+
+
+def create_output_directory(output_dir: Path) -> None:
+    """Create output directory if it doesn't exist."""
+    output_dir.mkdir(exist_ok=True)
+    logger.info(f"Output directory: {output_dir}")
+
+
+def sanitize_filename(filename: str) -> str:
+    """Sanitize filename for safe file system usage."""
+    # Remove or replace problematic characters
+    invalid_chars = '<>:"/\\|?*'
+    for char in invalid_chars:
+        filename = filename.replace(char, '_')
+    return filename.strip()
+
+
+def process_file(md_converter: MarkItDown, file_path: Path, output_dir: Path) -> bool:
+    """
+    Process a single file and convert it to Markdown.
+    
+    Args:
+        md_converter: MarkItDown instance
+        file_path: Path to the input file
+        output_dir: Directory to save the output
+        
+    Returns:
+        bool: True if successful, False otherwise
+    """
+    try:
+        logger.info(f"Processing file: {file_path.name}")
+        
+        # Convert file to markdown
+        result = md_converter.convert(str(file_path))
+        
+        # Create output filename
+        base_name = file_path.name  # Use full filename with extension
+        safe_name = sanitize_filename(base_name.replace('.', '_'))
+        output_filename = f"{safe_name}_converted.md"
+        output_path = output_dir / output_filename
+        
+        # Create markdown content with metadata
+        content = f"""# Converted from: {file_path.name}
+
+**Source File:** {file_path.name}  
+**Source Path:** {file_path}  
+**Conversion Date:** {result.title if hasattr(result, 'title') else 'N/A'}  
+**File Size:** {file_path.stat().st_size} bytes  
+
+---
+
+{result.text_content}
+"""
+        
+        # Write to output file
+        with open(output_path, 'w', encoding='utf-8') as f:
+            f.write(content)
+            
+        logger.success(f"Successfully converted {file_path.name} -> {output_filename}")
+        return True
+        
+    except Exception as e:
+        logger.error(f"Failed to process {file_path.name}: {str(e)}")
+        return False
+
+
+def process_url(md_converter: MarkItDown, url: str, output_dir: Path, index: int) -> bool:
+    """
+    Process a single URL and convert it to Markdown.
+    
+    Args:
+        md_converter: MarkItDown instance
+        url: URL to process
+        output_dir: Directory to save the output
+        index: Index for naming the output file
+        
+    Returns:
+        bool: True if successful, False otherwise
+    """
+    try:
+        logger.info(f"Processing URL: {url}")
+        
+        # Convert URL to markdown
+        result = md_converter.convert(url)
+        
+        # Create output filename based on URL
+        parsed_url = urlparse(url)
+        domain = parsed_url.netloc.replace('www.', '')
+        safe_domain = sanitize_filename(domain)
+        output_filename = f"url_{index:02d}_{safe_domain}_converted.md"
+        output_path = output_dir / output_filename
+        
+        # Create markdown content with metadata
+        content = f"""# Converted from URL: {url}
+
+**Source URL:** {url}  
+**Domain:** {parsed_url.netloc}  
+**Conversion Date:** {result.title if hasattr(result, 'title') else 'N/A'}  
+
+---
+
+{result.text_content}
+"""
+        
+        # Write to output file
+        with open(output_path, 'w', encoding='utf-8') as f:
+            f.write(content)
+            
+        logger.success(f"Successfully converted {url} -> {output_filename}")
+        return True
+        
+    except Exception as e:
+        logger.error(f"Failed to process URL {url}: {str(e)}")
+        return False
+
+
+def load_urls(urls_file: Path) -> List[str]:
+    """Load URLs from the urls.txt file."""
+    try:
+        with open(urls_file, 'r', encoding='utf-8') as f:
+            urls = [line.strip() for line in f if line.strip() and not line.strip().startswith('#')]
+        logger.info(f"Loaded {len(urls)} URLs from {urls_file}")
+        return urls
+    except Exception as e:
+        logger.error(f"Failed to load URLs from {urls_file}: {str(e)}")
+        return []
+
+
+def get_input_files(input_dir: Path) -> List[Path]:
+    """Get all files from the input directory, excluding audio/video files."""
+    try:
+        # Skip audio/video files for now
+        skip_extensions = [] #{'.mp3', '.mp4', '.wav', '.avi', '.mov', '.mkv', '.m4a'}
+        files = [f for f in input_dir.iterdir() 
+                if f.is_file() and f.suffix.lower() not in skip_extensions]
+        logger.info(f"Found {len(files)} files in {input_dir} (excluding audio/video)")
+        return files
+    except Exception as e:
+        logger.error(f"Failed to read files from {input_dir}: {str(e)}")
+        return []
+
+
+def main():
+    """Main function to orchestrate the conversion process."""
+    setup_logging()
+    logger.info("Starting MarkDowny processing...")
+    
+    # Setup paths
+    project_root = Path(__file__).parent
+    input_dir = project_root / "input_content"
+    urls_file = project_root / "urls.txt"
+    output_dir = project_root / "output"
+    
+    # Create output directory
+    create_output_directory(output_dir)
+    
+    # Initialize MarkItDown
+    md_converter = MarkItDown()
+    
+    # Process files
+    files_processed = 0
+    files_failed = 0
+    
+    if input_dir.exists():
+        input_files = get_input_files(input_dir)
+        logger.info(f"Processing {len(input_files)} files...")
+        
+        for file_path in input_files:
+            if process_file(md_converter, file_path, output_dir):
+                files_processed += 1
+            else:
+                files_failed += 1
+    else:
+        logger.warning(f"Input directory {input_dir} does not exist")
+    
+    # Process URLs
+    urls_processed = 0
+    urls_failed = 0
+    
+    if urls_file.exists():
+        urls = load_urls(urls_file)
+        logger.info(f"Processing {len(urls)} URLs...")
+        
+        for index, url in enumerate(urls, 1):
+            if process_url(md_converter, url, output_dir, index):
+                urls_processed += 1
+            else:
+                urls_failed += 1
+    else:
+        logger.warning(f"URLs file {urls_file} does not exist")
+    
+    # Summary
+    logger.info("=" * 50)
+    logger.info("PROCESSING SUMMARY")
+    logger.info("=" * 50)
+    logger.info(f"Files: {files_processed} successful, {files_failed} failed")
+    logger.info(f"URLs: {urls_processed} successful, {urls_failed} failed")
+    logger.info(f"Total: {files_processed + urls_processed} successful, {files_failed + urls_failed} failed")
+    logger.info(f"Output directory: {output_dir}")
+    
+    if files_failed + urls_failed > 0:
+        logger.warning("Some items failed to process. Check the logs for details.")
+        sys.exit(1)
+    else:
+        logger.success("All items processed successfully!")
+
+
+if __name__ == "__main__":
+    main()

{content_core-1.2.2 → content_core-1.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "content-core"
-version = "1.2.2"
+version = "1.3.0"
 description = "Extract what matters from any media source. Available as Python Library, macOS Service, CLI and MCP Server"
 readme = "README.md"
 homepage = "https://github.com/lfnovo/content-core"
@@ -34,10 +34,10 @@ dependencies = [
     "asciidoc>=10.2.1",
     "python-magic-bin==0.4.14; sys_platform == 'win32'",
     "pytubefix>=9.1.1",
+    "fastmcp>=0.5.0",
 ]
 
 [project.optional-dependencies]
-mcp = ["fastmcp>=0.5.0"]
 docling = ["docling>=2.34.0"]
 
 [project.scripts]

{content_core-1.2.2 → content_core-1.3.0}/uv.lock

@@ -419,7 +419,7 @@ wheels = [
 
 [[package]]
 name = "content-core"
-version = "1.2.2"
+version = "1.3.0"
 source = { editable = "." }
 dependencies = [
     { name = "ai-prompter" },
@@ -428,6 +428,7 @@ dependencies = [
     { name = "bs4" },
     { name = "dicttoxml" },
     { name = "esperanto" },
+    { name = "fastmcp" },
     { name = "firecrawl-py" },
     { name = "jinja2" },
     { name = "langdetect" },
@@ -453,9 +454,6 @@ dependencies = [
 docling = [
     { name = "docling" },
 ]
-mcp = [
-    { name = "fastmcp" },
-]
 
 [package.dev-dependencies]
 dev = [
@@ -476,7 +474,7 @@ requires-dist = [
     { name = "dicttoxml", specifier = ">=1.7.16" },
     { name = "docling", marker = "extra == 'docling'", specifier = ">=2.34.0" },
     { name = "esperanto", specifier = ">=1.2.0" },
-    { name = "fastmcp", marker = "extra == 'mcp'", specifier = ">=0.5.0" },
+    { name = "fastmcp", specifier = ">=0.5.0" },
     { name = "firecrawl-py", specifier = ">=2.7.0" },
     { name = "jinja2", specifier = ">=3.1.6" },
     { name = "langdetect", specifier = ">=1.0.9" },
@@ -497,7 +495,7 @@ requires-dist = [
     { name = "validators", specifier = ">=0.34.0" },
     { name = "youtube-transcript-api", specifier = ">=1.0.3" },
 ]
-provides-extras = ["mcp", "docling"]
+provides-extras = ["docling"]
 
 [package.metadata.requires-dev]
 dev = [

content_core-1.2.2/.claude/commands/pr-review.md

@@ -1,6 +0,0 @@
-
-When I ask you to do a PR, you will commit the necessary files and open up a pull request. As soon as you open the pull request, an automatic review process will start. The review process will provide feedback on the code quality and best practices, potential bugs or issues, performance considerations, security concerns, and test coverage.
-
-You should query the PR after opening it to get the commends from the review tool. Then, you will assess the comments and propose me which of them we need to address. 
-
-Then, we'll do the changes, commit them and add a comment back to the PR so that the review tool undertands what we changed and what we ignored. 

content_core-1.2.2/mcp.md

@@ -1,248 +0,0 @@
-# MCP Server Implementation Plan for Content Core
-
-## Overview
-Implement a FastMCP server that exposes Content Core's extraction functionality through the Model Context Protocol (MCP). The server will provide a single tool `extract_content` that accepts either a URL or file path and returns extracted content in JSON format using the 'auto' engine.
-
-## Architecture
-
-### 1. Dependencies
-```toml
-# Add to pyproject.toml as optional dependency
-[project.optional-dependencies]
-mcp = ["fastmcp>=0.5.0"]
-
-# Add script entry point for uvx
-[project.scripts]
-content-core-mcp = "content_core.mcp.server:main"
-```
-
-This allows users to install with MCP support using:
-```bash
-pip install content-core[mcp]
-# or with uv
-uv pip install content-core[mcp]
-```
-
-### 2. Server Structure
-```
-src/content_core/
-├── mcp/
-│   ├── __init__.py
-│   └── server.py       # Main MCP server implementation
-```
-
-### 3. Implementation Details
-
-#### Server Setup (`server.py`)
-```python
-from fastmcp import FastMCP
-from typing import Optional, Dict, Any
-import content_core as cc
-
-# Initialize MCP server
-mcp = FastMCP("Content Core MCP Server")
-```
-
-#### Tool Definition
-The `extract_content` tool will:
-- Accept either `url` or `file_path` as optional parameters
-- Validate that exactly one is provided
-- Return extracted content in JSON format
-- Use the 'auto' engine by default
-
-```python
-@mcp.tool
-async def extract_content(
-    url: Optional[str] = None,
-    file_path: Optional[str] = None
-) -> Dict[str, Any]:
-    """
-    Extract content from a URL or file using Content Core's auto engine.
-    
-    Args:
-        url: Optional URL to extract content from
-        file_path: Optional file path to extract content from
-        
-    Returns:
-        JSON object containing extracted content and metadata
-        
-    Raises:
-        ValueError: If neither or both url and file_path are provided
-    """
-    # Implementation details below
-```
-
-#### Input Validation
-- Ensure exactly one input source is provided
-- Validate URL format if URL is provided
-- Validate file existence if file_path is provided
-
-#### Integration with Content Core
-```python
-# Build extraction request
-extraction_request = {}
-if url:
-    extraction_request["url"] = url
-elif file_path:
-    extraction_request["file_path"] = file_path
-
-# Use Content Core's extract_content with auto engine
-result = await cc.extract_content(extraction_request)
-```
-
-#### Return Format
-The tool will return a JSON structure containing:
-```json
-{
-    "source_type": "url" | "file",
-    "source": "<url or file_path>",
-    "content": "<extracted content>",
-    "metadata": {
-        "engine_used": "<actual engine used by auto>",
-        "content_type": "<detected content type>",
-        "extraction_time": "<ISO timestamp>",
-        // Additional metadata from Content Core
-    },
-    "success": true,
-    "error": null  // or error message if extraction failed
-}
-```
-
-#### Error Handling
-- Wrap extraction in try/except block
-- Return structured error response on failure
-- Log errors using Context if needed
-- Handle specific Content Core exceptions
-
-### 4. Running the Server
-
-#### Entry Point (`main` function)
-```python
-def main():
-    """Entry point for the MCP server."""
-    import sys
-    # Run with STDIO transport for MCP compatibility
-    mcp.run()
-
-if __name__ == "__main__":
-    main()
-```
-
-#### Usage with uvx
-Users can run the server directly with uvx (no installation required):
-```bash
-# Run the MCP server
-uvx --from "content-core[mcp]" content-core-mcp
-```
-
-#### Claude Desktop Configuration
-Add to `claude_desktop_config.json`:
-```json
-{
-  "mcpServers": {
-    "content-core": {
-      "command": "uvx",
-      "args": [
-        "--from",
-        "content-core[mcp]",
-        "content-core-mcp"
-      ]
-    }
-  }
-}
-```
-
-#### Alternative: Local Development
-```bash
-# Install with MCP support
-uv pip install -e ".[mcp]"
-
-# Run directly
-content-core-mcp
-```
-
-### 5. Testing Strategy
-
-#### Unit Tests
-- Test input validation logic
-- Mock Content Core extraction calls
-- Verify JSON response structure
-
-#### Integration Tests
-- Test with real URLs
-- Test with various file types
-- Verify auto engine selection works correctly
-
-### 6. Documentation
-
-#### Usage Example
-```python
-# Client code example
-from fastmcp import Client
-
-client = Client()
-result = await client.call_tool(
-    "extract_content",
-    {"url": "https://example.com/article"}
-)
-```
-
-### 7. Configuration
-
-#### Environment Variables
-- Support existing Content Core environment variables
-- Allow MCP-specific configuration if needed
-
-#### Config File Support
-- Use existing `cc_config.yaml` if present
-- Allow override via MCP server initialization
-
-### 8. Future Enhancements (Not in initial implementation)
-- Add streaming support for large files
-- Support for multiple URLs/files in one request
-- Add content cleaning and summarization tools
-- Support custom extraction engines per request
-- Add resource endpoints for browsing extracted content
-
-### 9. Implementation Steps
-
-1. Create `mcp/` directory structure
-2. Implement basic server with extract_content tool
-3. Add input validation and error handling
-4. Integrate with Content Core extraction
-5. Format JSON response properly
-6. Add comprehensive logging
-7. Write unit tests
-8. Write integration tests
-9. Add documentation and examples
-10. Test with various MCP clients
-
-### 10. Key Considerations
-
-- **Async First**: Use async/await throughout since Content Core is async
-- **Error Messages**: Provide clear, actionable error messages
-- **Performance**: Consider caching for repeated requests
-- **Security**: Validate file paths to prevent directory traversal
-- **Compatibility**: Ensure works with all Content Core extraction engines
-
-### 11. Publishing and Distribution
-
-#### PyPI Package
-The MCP server will be included as an optional extra in the main content-core package:
-- Users install with `pip install content-core[mcp]`
-- The `content-core-mcp` command becomes available after installation
-- Works seamlessly with `uvx` for zero-install usage
-
-#### Benefits of uvx approach:
-1. **No installation required**: Users can run directly with `uvx`
-2. **Automatic updates**: Always uses the latest published version
-3. **Isolation**: Runs in isolated environment, avoiding dependency conflicts
-4. **Simple configuration**: Just add to `claude_desktop_config.json`
-
-#### Example MCP server listing entry:
-```yaml
-name: content-core
-description: Extract content from URLs and files using Content Core
-commands:
-  - uvx --from "content-core[mcp]" content-core-mcp
-```

content_core-1.2.2/test.py

@@ -1,16 +0,0 @@
-import asyncio
-
-from crawl4ai import *
-
-
-async def main():
-    async with AsyncWebCrawler() as crawler:
-        result = await crawler.arun(
-            url="https://www.nbcnews.com/business",
-        )
-        print(result.markdown)
-        print(result.title)
-
-
-if __name__ == "__main__":
-    asyncio.run(main())