content-core 0.7.2__tar.gz → 0.8.1__tar.gz

{content_core-0.7.2 → content_core-0.8.1}/PKG-INFO

@@ -1,15 +1,18 @@
 Metadata-Version: 2.4
 Name: content-core
-Version: 0.7.2
+Version: 0.8.1
 Summary: Extract what matters from any media source
 Author-email: LUIS NOVO <lfnovo@gmail.com>
 License-File: LICENSE
 Requires-Python: >=3.10
 Requires-Dist: ai-prompter>=0.2.3
 Requires-Dist: aiohttp>=3.11
+Requires-Dist: asciidoc>=10.2.1
 Requires-Dist: bs4>=0.0.2
 Requires-Dist: dicttoxml>=1.7.16
+Requires-Dist: docling>=2.34.0
 Requires-Dist: esperanto[openai]>=1.2.0
+Requires-Dist: firecrawl-py>=2.7.0
 Requires-Dist: jinja2>=3.1.6
 Requires-Dist: langdetect>=1.0.9
 Requires-Dist: langgraph>=0.3.29
@@ -17,18 +20,15 @@ Requires-Dist: loguru>=0.7.3
 Requires-Dist: moviepy>=2.1.2
 Requires-Dist: openpyxl>=3.1.5
 Requires-Dist: pandas>=2.2.3
+Requires-Dist: pillow>=10.4.0
 Requires-Dist: pymupdf>=1.25.5
 Requires-Dist: python-docx>=1.1.2
 Requires-Dist: python-dotenv>=1.1.0
 Requires-Dist: python-magic>=0.4.27
 Requires-Dist: python-pptx>=1.0.2
+Requires-Dist: readability-lxml>=0.8.4.1
 Requires-Dist: validators>=0.34.0
 Requires-Dist: youtube-transcript-api>=1.0.3
-Provides-Extra: docling
-Requires-Dist: asciidoc; extra == 'docling'
-Requires-Dist: docling; extra == 'docling'
-Requires-Dist: pandas; extra == 'docling'
-Requires-Dist: pillow; extra == 'docling'
 Description-Content-Type: text/markdown
 
 # Content Core
@@ -39,6 +39,8 @@ Description-Content-Type: text/markdown
 
 ## Overview
 
+> **Note:** As of v0.8, the default extraction engine is `'auto'`. Content Core will automatically select the best extraction method based on your environment and available API keys, with a smart fallback order for both URLs and files. For files/documents, `'auto'` now tries Docling first, then falls back to simple extraction. You can override the engine if needed, but `'auto'` is recommended for most users.
+
 The primary goal of Content Core is to simplify the process of ingesting content from diverse origins. Whether you have raw text, a URL pointing to an article, or a local file like a video or markdown document, Content Core aims to extract the meaningful content for further use.
 
 ## Key Features
@@ -48,6 +50,10 @@ The primary goal of Content Core is to simplify the process of ingesting content
     *   Web URLs (using robust extraction methods).
     *   Local files (including automatic transcription for video/audio files and parsing for text-based formats).
 *   **Intelligent Processing:** Applies appropriate extraction techniques based on the source type. See the [Processors Documentation](./docs/processors.md) for detailed information on how different content types are handled.
+*   **Smart Engine Selection:** By default, Content Core uses the `'auto'` engine, which:
+    * For URLs: Uses Firecrawl if `FIRECRAWL_API_KEY` is set, else tries Jina. Jina might fail because of rate limits, which can be fixed by adding `JINA_API_KEY`. If Jina failes, BeautifulSoup is used as a fallback.
+    * For files: Tries Docling extraction first (for robust document parsing), then falls back to simple extraction if needed.
+    * You can override this by specifying an engine, but `'auto'` is recommended for most users.
 *   **Content Cleaning (Optional):** Likely integrates with LLMs (via `prompter.py` and Jinja templates) to refine and clean the extracted content.
 *   **Asynchronous:** Built with `asyncio` for efficient I/O operations.
 
@@ -60,8 +66,6 @@ Install Content Core using `pip`:
 ```bash
 # Install the package (without Docling)
 pip install content-core
-# Install with Docling support
-pip install content-core[docling]
 ```
 
 Alternatively, if you’re developing locally:
@@ -218,15 +222,15 @@ async def main():
     text_data = await extract_content({"content": "This is my sample text content."})
     print(text_data)
 
-    # Extract from a URL
+    # Extract from a URL (uses 'auto' engine by default)
     url_data = await extract_content({"url": "https://www.example.com"})
     print(url_data)
 
-    # Extract from a local video file (gets transcript)
+    # Extract from a local video file (gets transcript, engine='auto' by default)
     video_data = await extract_content({"file_path": "path/to/your/video.mp4"})
     print(video_data)
 
-    # Extract from a local markdown file
+    # Extract from a local markdown file (engine='auto' by default)
     md_data = await extract_content({"file_path": "path/to/your/document.md"})
     print(md_data)
 
@@ -248,15 +252,11 @@ if __name__ == "__main__":
 
 Content Core supports an optional Docling-based extraction engine for rich document formats (PDF, DOCX, PPTX, XLSX, Markdown, AsciiDoc, HTML, CSV, Images).
 
-### Installation
-
-```bash
-# Install with Docling support
-pip install content-core[docling]
-```
 
 ### Enabling Docling
 
+Docling is not the default engine when parsing documents. If you don't want to use it, you need to set engine to "simple". 
+
 #### Via configuration file
 
 In your `cc_config.yaml` or custom config, set:

{content_core-0.7.2 → content_core-0.8.1}/README.md

@@ -6,6 +6,8 @@
 
 ## Overview
 
+> **Note:** As of v0.8, the default extraction engine is `'auto'`. Content Core will automatically select the best extraction method based on your environment and available API keys, with a smart fallback order for both URLs and files. For files/documents, `'auto'` now tries Docling first, then falls back to simple extraction. You can override the engine if needed, but `'auto'` is recommended for most users.
+
 The primary goal of Content Core is to simplify the process of ingesting content from diverse origins. Whether you have raw text, a URL pointing to an article, or a local file like a video or markdown document, Content Core aims to extract the meaningful content for further use.
 
 ## Key Features
@@ -15,6 +17,10 @@ The primary goal of Content Core is to simplify the process of ingesting content
     *   Web URLs (using robust extraction methods).
     *   Local files (including automatic transcription for video/audio files and parsing for text-based formats).
 *   **Intelligent Processing:** Applies appropriate extraction techniques based on the source type. See the [Processors Documentation](./docs/processors.md) for detailed information on how different content types are handled.
+*   **Smart Engine Selection:** By default, Content Core uses the `'auto'` engine, which:
+    * For URLs: Uses Firecrawl if `FIRECRAWL_API_KEY` is set, else tries Jina. Jina might fail because of rate limits, which can be fixed by adding `JINA_API_KEY`. If Jina failes, BeautifulSoup is used as a fallback.
+    * For files: Tries Docling extraction first (for robust document parsing), then falls back to simple extraction if needed.
+    * You can override this by specifying an engine, but `'auto'` is recommended for most users.
 *   **Content Cleaning (Optional):** Likely integrates with LLMs (via `prompter.py` and Jinja templates) to refine and clean the extracted content.
 *   **Asynchronous:** Built with `asyncio` for efficient I/O operations.
 
@@ -27,8 +33,6 @@ Install Content Core using `pip`:
 ```bash
 # Install the package (without Docling)
 pip install content-core
-# Install with Docling support
-pip install content-core[docling]
 ```
 
 Alternatively, if you’re developing locally:
@@ -185,15 +189,15 @@ async def main():
     text_data = await extract_content({"content": "This is my sample text content."})
     print(text_data)
 
-    # Extract from a URL
+    # Extract from a URL (uses 'auto' engine by default)
     url_data = await extract_content({"url": "https://www.example.com"})
     print(url_data)
 
-    # Extract from a local video file (gets transcript)
+    # Extract from a local video file (gets transcript, engine='auto' by default)
     video_data = await extract_content({"file_path": "path/to/your/video.mp4"})
     print(video_data)
 
-    # Extract from a local markdown file
+    # Extract from a local markdown file (engine='auto' by default)
     md_data = await extract_content({"file_path": "path/to/your/document.md"})
     print(md_data)
 
@@ -215,15 +219,11 @@ if __name__ == "__main__":
 
 Content Core supports an optional Docling-based extraction engine for rich document formats (PDF, DOCX, PPTX, XLSX, Markdown, AsciiDoc, HTML, CSV, Images).
 
-### Installation
-
-```bash
-# Install with Docling support
-pip install content-core[docling]
-```
 
 ### Enabling Docling
 
+Docling is not the default engine when parsing documents. If you don't want to use it, you need to set engine to "simple". 
+
 #### Via configuration file
 
 In your `cc_config.yaml` or custom config, set:

{content_core-0.7.2 → content_core-0.8.1}/docs/processors.md

@@ -1,5 +1,7 @@
 # Content Core Processors
 
+**Note:** As of vNEXT, the default extraction engine is now `'auto'`. This means Content Core will automatically select the best extraction method based on your environment and available API keys, with a smart fallback order for both URLs and files. For files/documents, `'auto'` now tries Docling first, then falls back to simple extraction. See details below.
+
 This document provides an overview of the content processors available in Content Core. These processors are responsible for extracting and handling content from various sources and file types.
 
 ## Overview
@@ -19,6 +21,11 @@ Content Core uses a modular approach to process content from different sources.
 - **Supported Input**: URLs (web pages).
 - **Returned Data**: Extracted text content from the web page, often in a cleaned format.
 - **Location**: `src/content_core/processors/url.py`
+- **Default Engine (`auto`) Logic**:
+    - If `FIRECRAWL_API_KEY` is set, uses Firecrawl for extraction.
+    - Else it tries Jina until it fails because of rate limits (unless `JINA_API_KEY` is set).
+    - Else, falls back to BeautifulSoup-based extraction.
+    - You can explicitly specify an engine (`'firecrawl'`, `'jina'`, `'simple'`, etc.), but `'auto'` is now the default and recommended for most users.
 
 ### 3. **File Processor**
 - **Purpose**: Processes local files of various types, extracting content based on file format.
@@ -40,10 +47,14 @@ Content Core uses a modular approach to process content from different sources.
 - **Supported Input**: PDF, DOCX, XLSX, PPTX, Markdown, AsciiDoc, HTML, CSV, Images (PNG, JPEG, TIFF, BMP).
 - **Returned Data**: Content converted to configured format (markdown, html, json).
 - **Location**: `src/content_core/processors/docling.py`
+- **Default Engine (`auto`) Logic for Files/Documents**:
+    - Tries the `'docling'` extraction method first (robust document parsing for supported types).
+    - If `'docling'` fails or is not supported, automatically falls back to simple extraction (fast, lightweight for supported types).
+    - You can explicitly specify `'docling'`, `'simple'`, or `'legacy'` as the engine, but `'auto'` is now the default and recommended for most users.
 - **Configuration**: Activate the Docling engine in `cc_config.yaml` or custom config:
   ```yaml
   extraction:
-    engine: docling       # 'legacy' (default) or 'docling'
+    engine: docling       # 'auto' (default), 'docling', or 'simple'
     docling:
       output_format: markdown  # markdown | html | json
   ```

{content_core-0.7.2 → content_core-0.8.1}/docs/usage.md

@@ -1,5 +1,7 @@
 # Using the Content Core Library
 
+> **Note:** As of vNEXT, the default extraction engine is `'auto'`. Content Core will automatically select the best extraction method based on your environment and available API keys, with a smart fallback order for both URLs and files. For files/documents, `'auto'` now tries Docling first, then falls back to simple extraction. You can override the engine if needed, but `'auto'` is recommended for most users.
+
 This documentation explains how to configure and use the **Content Core** library in your projects. The library allows customization of AI model settings through a YAML file and environment variables.
 
 ## Environment Variable for Configuration
@@ -76,20 +78,28 @@ To simplify setup, we suggest copying the provided sample files:
 
 This will allow you to quickly start with customized settings without needing to create the files from scratch.
 
-### Docling Engine
+### Extraction Engine Selection
+
+By default, Content Core uses the `'auto'` engine for all extraction tasks. The logic is as follows:
+- **For URLs**: Uses Firecrawl if `FIRECRAWL_API_KEY` is set, else Jina if `JINA_API_KEY` is set, else falls back to BeautifulSoup.
+- **For files**: Tries Docling extraction first (for robust document parsing), then falls back to simple extraction if needed.
+
+You can override this behavior by specifying an engine in your config or function call, but `'auto'` is recommended for most users.
+
+#### Docling Engine
 
-Content Core supports an optional Docling engine for advanced document parsing. To enable:
+Content Core supports an optional Docling engine for advanced document parsing. To enable Docling explicitly:
 
-#### In YAML config
+##### In YAML config
 Add under the `extraction` section:
 ```yaml
 extraction:
-  engine: docling        # legacy (default) or docling
+  engine: docling        # auto (default), docling, or simple
   docling:
     output_format: html  # markdown | html | json
 ```
 
-#### Programmatically in Python
+##### Programmatically in Python
 ```python
 from content_core.config import set_extraction_engine, set_docling_output_format
 

{content_core-0.7.2 → content_core-0.8.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "content-core"
-version = "0.7.2"
+version = "0.8.1"
 description = "Extract what matters from any media source"
 readme = "README.md"
 homepage = "https://github.com/lfnovo/content-core"
@@ -28,11 +28,13 @@ dependencies = [
     "validators>=0.34.0",
     "ai-prompter>=0.2.3",
     "moviepy>=2.1.2",
+    "readability-lxml>=0.8.4.1",
+    "firecrawl-py>=2.7.0",
+    "docling>=2.34.0",
+    "pillow>=10.4.0",
+    "asciidoc>=10.2.1",
 ]
 
-[project.optional-dependencies]
-docling = ["docling", "Pillow", "pandas", "asciidoc"]
-
 [project.scripts]
 ccore = "content_core:ccore"
 cclean = "content_core:cclean"

{content_core-0.7.2 → content_core-0.8.1}/src/content_core/common/state.py

@@ -2,6 +2,9 @@ from typing import Optional
 
 from pydantic import BaseModel, Field
 
+from content_core.common.types import Engine
+from content_core.common.types import Engine
+
 
 class ProcessSourceState(BaseModel):
     file_path: Optional[str] = ""
@@ -13,8 +16,9 @@ class ProcessSourceState(BaseModel):
     identified_provider: Optional[str] = ""
     metadata: Optional[dict] = Field(default_factory=lambda: {})
     content: Optional[str] = ""
-    engine: Optional[str] = Field(
-        default=None, description="Override extraction engine: 'legacy' or 'docling'"
+    engine: Optional[Engine] = Field(
+        default=None,
+        description="Override extraction engine: 'auto', 'simple', 'legacy', 'firecrawl', 'jina', or 'docling'",
     )
     output_format: Optional[str] = Field(
         default=None,

content_core-0.8.1/src/content_core/common/types.py

@@ -0,0 +1,21 @@
+from typing import Literal
+import warnings
+
+Engine = Literal[
+    "auto",
+    "simple",
+    "legacy",
+    "firecrawl",
+    "jina",
+    "docling",
+]
+
+DEPRECATED_ENGINES = {"legacy": "simple"}
+
+def warn_if_deprecated_engine(engine: str):
+    if engine in DEPRECATED_ENGINES:
+        warnings.warn(
+            f"Engine '{engine}' is deprecated and will be removed in a future release. Use '{DEPRECATED_ENGINES[engine]}' instead.",
+            DeprecationWarning,
+            stacklevel=2,
+        )

{content_core-0.7.2 → content_core-0.8.1}/src/content_core/content/extraction/graph.py

@@ -2,6 +2,7 @@ import os
 import tempfile
 from typing import Any, Dict, Optional
 from urllib.parse import urlparse
+from content_core.common.types import warn_if_deprecated_engine
 
 import aiohttp
 import magic
@@ -114,14 +115,28 @@ async def download_remote_file(state: ProcessSourceState) -> Dict[str, Any]:
     return {"file_path": tmp, "identified_type": mime}
 
 
+
 async def file_type_router_docling(state: ProcessSourceState) -> str:
     """
-    Route to Docling if enabled and supported; otherwise use legacy file type edge.
+    Route to Docling if enabled and supported; otherwise use simple file type edge.
+    Supports 'auto', 'docling', 'simple', and 'legacy' (deprecated, alias for simple).
+    'auto' tries simple first, then falls back to docling if simple fails.
     """
-    # allow per-execution override of engine via state.engine
-    engine = state.engine or CONFIG.get("extraction", {}).get("engine", "legacy")
+    engine = state.engine or CONFIG.get("extraction", {}).get("engine", "auto")
+    warn_if_deprecated_engine(engine)
+    if engine == "auto":
+        # Try docling first; if it fails or is not supported, fallback to simple
+        if state.identified_type in DOCLING_SUPPORTED:
+            try:
+                return "extract_docling"
+            except Exception as e:
+                logger.warning(f"Docling extraction failed in 'auto' mode, falling back to simple: {e}")
+        # Fallback to simple
+        return await file_type_edge(state)
+
     if engine == "docling" and state.identified_type in DOCLING_SUPPORTED:
         return "extract_docling"
+    # For 'simple' and 'legacy', use the default file type edge
     return await file_type_edge(state)
 
 

{content_core-0.7.2 → content_core-0.8.1}/src/content_core/processors/audio.py

@@ -1,9 +1,10 @@
 import asyncio
+import math
 import os
 import tempfile
-import math
 import traceback
 from functools import partial
+
 from moviepy import AudioFileClip
 
 from content_core.common import ProcessSourceState
@@ -64,7 +65,9 @@ async def split_audio(input_file, segment_length_minutes=15, output_prefix=None)
     )
 
 
-def extract_audio(input_file: str, output_file: str, start_time: float = None, end_time: float = None) -> None:
+def extract_audio(
+    input_file: str, output_file: str, start_time: float = None, end_time: float = None
+) -> None:
     """
     Extract audio from a video or audio file and save it as an MP3 file.
     If start_time and end_time are provided, only that segment of audio is extracted.
@@ -78,17 +81,17 @@ def extract_audio(input_file: str, output_file: str, start_time: float = None, e
     try:
         # Load the file as an AudioFileClip
         audio_clip = AudioFileClip(input_file)
-        
-        # If start_time and end_time are provided, trim the audio
+
+        # If start_time and/or end_time are provided, trim the audio using subclipped
         if start_time is not None and end_time is not None:
-            audio_clip = audio_clip.cutout(0, start_time).cutout(end_time - start_time, audio_clip.duration)
+            audio_clip = audio_clip.subclipped(start_time, end_time)
         elif start_time is not None:
-            audio_clip = audio_clip.cutout(0, start_time)
+            audio_clip = audio_clip.subclipped(start_time)
         elif end_time is not None:
-            audio_clip = audio_clip.cutout(end_time, audio_clip.duration)
+            audio_clip = audio_clip.subclipped(0, end_time)
 
         # Export the audio as MP3
-        audio_clip.write_audiofile(output_file, codec='mp3')
+        audio_clip.write_audiofile(output_file, codec="mp3")
         audio_clip.close()
     except Exception as e:
         logger.error(f"Error extracting audio: {str(e)}")
@@ -117,7 +120,9 @@ async def extract_audio_data(data: ProcessSourceState):
         output_files = []
 
         if duration_s > segment_length_s:
-            logger.info(f"Audio is longer than 10 minutes ({duration_s}s), splitting into {math.ceil(duration_s / segment_length_s)} segments")
+            logger.info(
+                f"Audio is longer than 10 minutes ({duration_s}s), splitting into {math.ceil(duration_s / segment_length_s)} segments"
+            )
             for i in range(math.ceil(duration_s / segment_length_s)):
                 start_time = i * segment_length_s
                 end_time = min((i + 1) * segment_length_s, audio.duration)
@@ -134,15 +139,18 @@ async def extract_audio_data(data: ProcessSourceState):
 
         # Transcribe audio files
         from content_core.models import ModelFactory
+
         speech_to_text_model = ModelFactory.get_model("speech_to_text")
         transcriptions = []
         for audio_file in output_files:
-            transcription = await transcribe_audio_segment(audio_file, speech_to_text_model)
+            transcription = await transcribe_audio_segment(
+                audio_file, speech_to_text_model
+            )
             transcriptions.append(transcription)
 
         return {
             "metadata": {"audio_files": output_files},
-            "content": " ".join(transcriptions)
+            "content": " ".join(transcriptions),
         }
     except Exception as e:
         logger.error(f"Error processing audio: {str(e)}")