content-core 0.1.1__tar.gz → 0.2.0__tar.gz

{content_core-0.1.1 → content_core-0.2.0}/.gitignore

@@ -18,4 +18,6 @@ wheels/
 ai_docs/
 
 todo.md
-WIP/
+WIP/
+
+*.ignore

{content_core-0.1.1 → content_core-0.2.0}/PKG-INFO

@@ -1,11 +1,11 @@
 Metadata-Version: 2.4
 Name: content-core
-Version: 0.1.1
+Version: 0.2.0
 Summary: Extract what matters from any media source
 Author-email: LUIS NOVO <lfnovo@gmail.com>
 License-File: LICENSE
 Requires-Python: >=3.10
-Requires-Dist: aiohttp>=3.11.16
+Requires-Dist: aiohttp>=3.11
 Requires-Dist: bs4>=0.0.2
 Requires-Dist: dicttoxml>=1.7.16
 Requires-Dist: esperanto>=1.2.0
@@ -43,7 +43,7 @@ The primary goal of Content Core is to simplify the process of ingesting content
     *   Direct text strings.
     *   Web URLs (using robust extraction methods).
     *   Local files (including automatic transcription for video/audio files and parsing for text-based formats).
-*   **Intelligent Processing:** Applies appropriate extraction techniques based on the source type.
+*   **Intelligent Processing:** Applies appropriate extraction techniques based on the source type. See the [Processors Documentation](./docs/processors.md) for detailed information on how different content types are handled.
 *   **Content Cleaning (Optional):** Likely integrates with LLMs (via `prompter.py` and Jinja templates) to refine and clean the extracted content.
 *   **Asynchronous:** Built with `asyncio` for efficient I/O operations.
 
@@ -220,6 +220,20 @@ OPENAI_API_KEY=your-key-here
 GOOGLE_API_KEY=your-key-here
 ```
 
+### Custom Prompt Templates
+
+Content Core allows you to define custom prompt templates for content processing. By default, the library uses built-in prompts located in the `prompts` directory. However, you can create your own prompt templates and store them in a dedicated directory. To specify the location of your custom prompts, set the `PROMPT_PATH` environment variable in your `.env` file or system environment.
+
+Example `.env` with custom prompt path:
+
+```plaintext
+OPENAI_API_KEY=your-key-here
+GOOGLE_API_KEY=your-key-here
+PROMPT_PATH=/path/to/your/custom/prompts
+```
+
+When a prompt template is requested, Content Core will first look in the custom directory specified by `PROMPT_PATH` (if set and exists). If the template is not found there, it will fall back to the default built-in prompts. This allows you to override specific prompts while still using the default ones for others.
+
 ## Development
 
 To set up a development environment:

{content_core-0.1.1 → content_core-0.2.0}/README.md

@@ -14,7 +14,7 @@ The primary goal of Content Core is to simplify the process of ingesting content
     *   Direct text strings.
     *   Web URLs (using robust extraction methods).
     *   Local files (including automatic transcription for video/audio files and parsing for text-based formats).
-*   **Intelligent Processing:** Applies appropriate extraction techniques based on the source type.
+*   **Intelligent Processing:** Applies appropriate extraction techniques based on the source type. See the [Processors Documentation](./docs/processors.md) for detailed information on how different content types are handled.
 *   **Content Cleaning (Optional):** Likely integrates with LLMs (via `prompter.py` and Jinja templates) to refine and clean the extracted content.
 *   **Asynchronous:** Built with `asyncio` for efficient I/O operations.
 
@@ -191,6 +191,20 @@ OPENAI_API_KEY=your-key-here
 GOOGLE_API_KEY=your-key-here
 ```
 
+### Custom Prompt Templates
+
+Content Core allows you to define custom prompt templates for content processing. By default, the library uses built-in prompts located in the `prompts` directory. However, you can create your own prompt templates and store them in a dedicated directory. To specify the location of your custom prompts, set the `PROMPT_PATH` environment variable in your `.env` file or system environment.
+
+Example `.env` with custom prompt path:
+
+```plaintext
+OPENAI_API_KEY=your-key-here
+GOOGLE_API_KEY=your-key-here
+PROMPT_PATH=/path/to/your/custom/prompts
+```
+
+When a prompt template is requested, Content Core will first look in the custom directory specified by `PROMPT_PATH` (if set and exists). If the template is not found there, it will fall back to the default built-in prompts. This allows you to override specific prompts while still using the default ones for others.
+
 ## Development
 
 To set up a development environment:

content_core-0.2.0/docs/processors.md

@@ -0,0 +1,53 @@
+# Content Core Processors
+
+This document provides an overview of the content processors available in Content Core. These processors are responsible for extracting and handling content from various sources and file types.
+
+## Overview
+
+Content Core uses a modular approach to process content from different sources. Each processor is designed to handle specific types of input, such as web URLs, local files, or direct text input. Below, you'll find detailed information about each processor, including supported file types, returned data formats, and their purpose.
+
+## Processors
+
+### 1. **Text Processor**
+- **Purpose**: Handles direct text input provided by the user.
+- **Supported Input**: Raw text strings.
+- **Returned Data**: The input text as-is, wrapped in a structured format compatible with Content Core's output schema.
+- **Location**: `src/content_core/processors/text.py`
+
+### 2. **Web Processor**
+- **Purpose**: Extracts content from web URLs, focusing on meaningful text while ignoring boilerplate (ads, navigation, etc.).
+- **Supported Input**: URLs (web pages).
+- **Returned Data**: Extracted text content from the web page, often in a cleaned format.
+- **Location**: `src/content_core/processors/web.py`
+
+### 3. **File Processor**
+- **Purpose**: Processes local files of various types, extracting content based on file format.
+- **Supported Input**: Local files including:
+  - Text-based formats: `.txt`, `.md` (Markdown), `.html`, etc.
+  - Document formats: `.pdf`, `.docx`, etc.
+  - Media files: `.mp4`, `.mp3` (audio/video, via transcription).
+- **Returned Data**: Extracted text content or transcriptions (for media files), structured according to Content Core's schema.
+- **Location**: `src/content_core/processors/file.py`
+
+### 4. **Media Transcription Processor**
+- **Purpose**: Specifically handles transcription of audio and video files using external services or libraries.
+- **Supported Input**: Audio and video files (e.g., `.mp3`, `.mp4`).
+- **Returned Data**: Transcribed text from the media content.
+- **Location**: `src/content_core/processors/transcription.py`
+
+## How Processors Work
+
+Content Core automatically selects the appropriate processor based on the input type:
+- If a URL is provided, the Web Processor is used.
+- If a file path is provided, the File Processor determines the file type and delegates to specialized handlers (like the Media Transcription Processor for audio/video).
+- If raw text is provided, the Text Processor handles it directly.
+
+Each processor returns data in a consistent format, allowing seamless integration with other components of Content Core for further processing (like cleaning or summarization).
+
+## Custom Processors
+
+Developers can extend Content Core by creating custom processors for unsupported file types or specialized extraction needs. To do so, create a new processor module in `src/content_core/processors/` and ensure it adheres to the expected interface for integration with the content extraction pipeline.
+
+## Contributing
+
+If you have suggestions for improving existing processors or adding support for new file types, please contribute to the project by submitting a pull request or opening an issue on the GitHub repository.

{content_core-0.1.1 → content_core-0.2.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "content-core"
-version = "0.1.1"
+version = "0.2.0"
 description = "Extract what matters from any media source"
 readme = "README.md"
 authors = [
@@ -8,7 +8,7 @@ authors = [
 ]
 requires-python = ">=3.10"
 dependencies = [
-    "aiohttp>=3.11.16",
+    "aiohttp>=3.11",
     "bs4>=0.0.2",
     "esperanto>=1.2.0",
     "google-genai>=1.10.0",