content-core 0.1.2__tar.gz → 0.3.0__tar.gz

{content_core-0.1.2 → content_core-0.3.0}/.gitignore

@@ -18,4 +18,6 @@ wheels/
 ai_docs/
 
 todo.md
-WIP/
+WIP/
+
+*.ignore

content_core-0.3.0/.windsurfrules

@@ -0,0 +1,13 @@
+Also use uv as the package manager: uv run, uv sync, uv add.
+
+All documentation (code or readmes) must be in english.
+Whenever I ask you to tag and release, make sure to run `make test` as part of the process. 
+
+The full release process is: 
+- Run `make test` to make sure everything is working
+- Update version on pyproject.toml
+- Run `uv sync` to update the lock file
+- Commit all that's needed
+- Merge to main
+- Tag the release
+- Push to GitHub

{content_core-0.1.2 → content_core-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: content-core
-Version: 0.1.2
+Version: 0.3.0
 Summary: Extract what matters from any media source
 Author-email: LUIS NOVO <lfnovo@gmail.com>
 License-File: LICENSE
@@ -43,7 +43,7 @@ The primary goal of Content Core is to simplify the process of ingesting content
     *   Direct text strings.
     *   Web URLs (using robust extraction methods).
     *   Local files (including automatic transcription for video/audio files and parsing for text-based formats).
-*   **Intelligent Processing:** Applies appropriate extraction techniques based on the source type.
+*   **Intelligent Processing:** Applies appropriate extraction techniques based on the source type. See the [Processors Documentation](./docs/processors.md) for detailed information on how different content types are handled.
 *   **Content Cleaning (Optional):** Likely integrates with LLMs (via `prompter.py` and Jinja templates) to refine and clean the extracted content.
 *   **Asynchronous:** Built with `asyncio` for efficient I/O operations.
 
@@ -161,6 +161,27 @@ csum https://example.com
 csum document.txt
 ```
 
+## Quick Start
+
+You can quickly integrate `content-core` into your Python projects to extract, clean, and summarize content from various sources.
+
+```python
+import content_core as cc
+
+# Extract content from a URL, file, or text
+result = await cc.extract("https://example.com/article")
+
+# Clean messy content
+cleaned_text = await cc.clean("...messy text with [brackets] and extra spaces...")
+
+# Summarize content with optional context
+summary = await cc.summarize_content("long article text", context="explain to a child")
+```
+
+## Documentation
+
+For more information on how to use the Content Core library, including details on AI model configuration and customization, refer to our [Usage Documentation](docs/usage.md).
+
 ## Using with Langchain
 
 For users integrating with the [Langchain](https://python.langchain.com/) framework, `content-core` exposes a set of compatible tools. These tools, located in the `src/content_core/tools` directory, allow you to leverage `content-core` extraction, cleaning, and summarization capabilities directly within your Langchain agents and chains.
@@ -220,6 +241,20 @@ OPENAI_API_KEY=your-key-here
 GOOGLE_API_KEY=your-key-here
 ```
 
+### Custom Prompt Templates
+
+Content Core allows you to define custom prompt templates for content processing. By default, the library uses built-in prompts located in the `prompts` directory. However, you can create your own prompt templates and store them in a dedicated directory. To specify the location of your custom prompts, set the `PROMPT_PATH` environment variable in your `.env` file or system environment.
+
+Example `.env` with custom prompt path:
+
+```plaintext
+OPENAI_API_KEY=your-key-here
+GOOGLE_API_KEY=your-key-here
+PROMPT_PATH=/path/to/your/custom/prompts
+```
+
+When a prompt template is requested, Content Core will first look in the custom directory specified by `PROMPT_PATH` (if set and exists). If the template is not found there, it will fall back to the default built-in prompts. This allows you to override specific prompts while still using the default ones for others.
+
 ## Development
 
 To set up a development environment:

{content_core-0.1.2 → content_core-0.3.0}/README.md

@@ -14,7 +14,7 @@ The primary goal of Content Core is to simplify the process of ingesting content
     *   Direct text strings.
     *   Web URLs (using robust extraction methods).
     *   Local files (including automatic transcription for video/audio files and parsing for text-based formats).
-*   **Intelligent Processing:** Applies appropriate extraction techniques based on the source type.
+*   **Intelligent Processing:** Applies appropriate extraction techniques based on the source type. See the [Processors Documentation](./docs/processors.md) for detailed information on how different content types are handled.
 *   **Content Cleaning (Optional):** Likely integrates with LLMs (via `prompter.py` and Jinja templates) to refine and clean the extracted content.
 *   **Asynchronous:** Built with `asyncio` for efficient I/O operations.
 
@@ -132,6 +132,27 @@ csum https://example.com
 csum document.txt
 ```
 
+## Quick Start
+
+You can quickly integrate `content-core` into your Python projects to extract, clean, and summarize content from various sources.
+
+```python
+import content_core as cc
+
+# Extract content from a URL, file, or text
+result = await cc.extract("https://example.com/article")
+
+# Clean messy content
+cleaned_text = await cc.clean("...messy text with [brackets] and extra spaces...")
+
+# Summarize content with optional context
+summary = await cc.summarize_content("long article text", context="explain to a child")
+```
+
+## Documentation
+
+For more information on how to use the Content Core library, including details on AI model configuration and customization, refer to our [Usage Documentation](docs/usage.md).
+
 ## Using with Langchain
 
 For users integrating with the [Langchain](https://python.langchain.com/) framework, `content-core` exposes a set of compatible tools. These tools, located in the `src/content_core/tools` directory, allow you to leverage `content-core` extraction, cleaning, and summarization capabilities directly within your Langchain agents and chains.
@@ -191,6 +212,20 @@ OPENAI_API_KEY=your-key-here
 GOOGLE_API_KEY=your-key-here
 ```
 
+### Custom Prompt Templates
+
+Content Core allows you to define custom prompt templates for content processing. By default, the library uses built-in prompts located in the `prompts` directory. However, you can create your own prompt templates and store them in a dedicated directory. To specify the location of your custom prompts, set the `PROMPT_PATH` environment variable in your `.env` file or system environment.
+
+Example `.env` with custom prompt path:
+
+```plaintext
+OPENAI_API_KEY=your-key-here
+GOOGLE_API_KEY=your-key-here
+PROMPT_PATH=/path/to/your/custom/prompts
+```
+
+When a prompt template is requested, Content Core will first look in the custom directory specified by `PROMPT_PATH` (if set and exists). If the template is not found there, it will fall back to the default built-in prompts. This allows you to override specific prompts while still using the default ones for others.
+
 ## Development
 
 To set up a development environment:

content_core-0.3.0/docs/processors.md

@@ -0,0 +1,53 @@
+# Content Core Processors
+
+This document provides an overview of the content processors available in Content Core. These processors are responsible for extracting and handling content from various sources and file types.
+
+## Overview
+
+Content Core uses a modular approach to process content from different sources. Each processor is designed to handle specific types of input, such as web URLs, local files, or direct text input. Below, you'll find detailed information about each processor, including supported file types, returned data formats, and their purpose.
+
+## Processors
+
+### 1. **Text Processor**
+- **Purpose**: Handles direct text input provided by the user.
+- **Supported Input**: Raw text strings.
+- **Returned Data**: The input text as-is, wrapped in a structured format compatible with Content Core's output schema.
+- **Location**: `src/content_core/processors/text.py`
+
+### 2. **Web Processor**
+- **Purpose**: Extracts content from web URLs, focusing on meaningful text while ignoring boilerplate (ads, navigation, etc.).
+- **Supported Input**: URLs (web pages).
+- **Returned Data**: Extracted text content from the web page, often in a cleaned format.
+- **Location**: `src/content_core/processors/web.py`
+
+### 3. **File Processor**
+- **Purpose**: Processes local files of various types, extracting content based on file format.
+- **Supported Input**: Local files including:
+  - Text-based formats: `.txt`, `.md` (Markdown), `.html`, etc.
+  - Document formats: `.pdf`, `.docx`, etc.
+  - Media files: `.mp4`, `.mp3` (audio/video, via transcription).
+- **Returned Data**: Extracted text content or transcriptions (for media files), structured according to Content Core's schema.
+- **Location**: `src/content_core/processors/file.py`
+
+### 4. **Media Transcription Processor**
+- **Purpose**: Specifically handles transcription of audio and video files using external services or libraries.
+- **Supported Input**: Audio and video files (e.g., `.mp3`, `.mp4`).
+- **Returned Data**: Transcribed text from the media content.
+- **Location**: `src/content_core/processors/transcription.py`
+
+## How Processors Work
+
+Content Core automatically selects the appropriate processor based on the input type:
+- If a URL is provided, the Web Processor is used.
+- If a file path is provided, the File Processor determines the file type and delegates to specialized handlers (like the Media Transcription Processor for audio/video).
+- If raw text is provided, the Text Processor handles it directly.
+
+Each processor returns data in a consistent format, allowing seamless integration with other components of Content Core for further processing (like cleaning or summarization).
+
+## Custom Processors
+
+Developers can extend Content Core by creating custom processors for unsupported file types or specialized extraction needs. To do so, create a new processor module in `src/content_core/processors/` and ensure it adheres to the expected interface for integration with the content extraction pipeline.
+
+## Contributing
+
+If you have suggestions for improving existing processors or adding support for new file types, please contribute to the project by submitting a pull request or opening an issue on the GitHub repository.

content_core-0.3.0/docs/usage.md

@@ -0,0 +1,81 @@
+# Using the Content Core Library
+
+This documentation explains how to configure and use the **Content Core** library in your projects. The library allows customization of AI model settings through a YAML file and environment variables.
+
+## Environment Variable for Configuration
+
+The library uses the `CCORE_MODEL_CONFIG_PATH` environment variable to locate the custom YAML configuration file. If this variable is not set or the specified file is not found, the library will fall back to internal default settings.
+
+To set the environment variable, add the following line to your `.env` file or set it directly in your environment:
+
+```
+CCORE_MODEL_CONFIG_PATH=/path/to/your/models_config.yaml
+```
+
+## YAML File Schema
+
+The YAML configuration file defines the AI models that the library will use. The structure of the file is as follows:
+
+- **speech_to_text**: Configuration for the speech-to-text model.
+  - **provider**: Model provider (example: `openai`).
+  - **model_name**: Model name (example: `whisper-1`).
+- **default_model**: Configuration for the default language model.
+  - **provider**: Model provider.
+  - **model_name**: Model name.
+  - **config**: Additional parameters like `temperature`, `top_p`, `max_tokens`.
+- **cleanup_model**: Configuration for the content cleanup model.
+  - **provider**: Model provider.
+  - **model_name**: Model name.
+  - **config**: Additional parameters.
+- **summary_model**: Configuration for the summary model.
+  - **provider**: Model provider.
+  - **model_name**: Model name.
+  - **config**: Additional parameters.
+
+### Default YAML File
+
+Here is the content of the default YAML file used by the library:
+
+```yaml
+speech_to_text:
+  provider: openai
+  model_name: whisper-1
+
+default_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0.5
+    top_p: 1
+    max_tokens: 2000
+
+cleanup_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0
+    max_tokens: 8000
+    output_format: json
+
+summary_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0
+    top_p: 1
+    max_tokens: 2000
+```
+
+## Customization
+
+You can customize any aspect of the YAML file to suit your needs. Change the providers, model names, or configuration parameters as desired.
+
+To simplify setup, we suggest copying the provided sample files:
+- Copy `.env.sample` to `.env` and adjust the environment variables, including `CCORE_MODEL_CONFIG_PATH`.
+- Copy `models_config.yaml.sample` to your desired location and modify it as needed.
+
+This will allow you to quickly start with customized settings without needing to create the files from scratch.
+
+## Support
+
+If you have questions or encounter issues while using the library, open an issue in the repository or contact the support team.

{content_core-0.1.2 → content_core-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "content-core"
-version = "0.1.2"
+version = "0.3.0"
 description = "Extract what matters from any media source"
 readme = "README.md"
 authors = [

{content_core-0.1.2 → content_core-0.3.0}/src/content_core/__init__.py

@@ -7,18 +7,21 @@ from xml.etree import ElementTree as ET
 
 from dicttoxml import dicttoxml  # type: ignore
 from dotenv import load_dotenv
-from loguru import logger
 
 from content_core.common import ProcessSourceInput
 from content_core.content.cleanup import cleanup_content
 from content_core.content.extraction import extract_content
 from content_core.content.summary import summarize
+from content_core.logging import configure_logging, logger
+
+# Exposing functions for direct access when importing content_core as cc
+extract = extract_content
+clean = cleanup_content
 
 load_dotenv()
 
-# Configure loguru logger
-logger.remove()  # Remove default handler
-logger.add(sys.stderr, level="INFO")  # Default to INFO level
+# Configure loguru logger using centralized configuration
+configure_logging(debug=False)
 
 
 def parse_content_format(content: str) -> str:
@@ -94,10 +97,9 @@ async def ccore_main():
 
     args = parser.parse_args()
 
-    # Adjust logging level based on debug flag
+    # Adjust logging level based on debug flag using centralized configuration
+    configure_logging(debug=args.debug)
     if args.debug:
-        logger.remove()
-        logger.add(sys.stderr, level="DEBUG")
         logger.debug("Debug logging enabled")
 
     content = get_content(args, parser)
@@ -136,10 +138,9 @@ async def cclean_main():
 
     args = parser.parse_args()
 
-    # Adjust logging level based on debug flag
+    # Adjust logging level based on debug flag using centralized configuration
+    configure_logging(debug=args.debug)
     if args.debug:
-        logger.remove()
-        logger.add(sys.stderr, level="DEBUG")
         logger.debug("Debug logging enabled")
 
     content = get_content(args, parser)
@@ -176,10 +177,9 @@ async def csum_main():
 
     args = parser.parse_args()
 
-    # Adjust logging level based on debug flag
+    # Adjust logging level based on debug flag using centralized configuration
+    configure_logging(debug=args.debug)
     if args.debug:
-        logger.remove()
-        logger.add(sys.stderr, level="DEBUG")
         logger.debug("Debug logging enabled")
 
     content = get_content(args, parser)

content_core-0.3.0/src/content_core/config.py

@@ -0,0 +1,27 @@
+import os
+import pkgutil
+
+import yaml
+from dotenv import load_dotenv
+
+# Load environment variables from .env file
+load_dotenv()
+
+
+def load_config():
+    config_path = os.environ.get("CCORE_MODEL_CONFIG_PATH")
+    if config_path and os.path.exists(config_path):
+        try:
+            with open(config_path, "r") as file:
+                return yaml.safe_load(file)
+        except Exception as e:
+            print(f"Erro ao carregar o arquivo de configuração de {config_path}: {e}")
+            print("Usando configurações padrão internas.")
+
+    default_config_data = pkgutil.get_data("content_core", "models_config.yaml")
+    if default_config_data:
+        return yaml.safe_load(default_config_data)
+    return {}
+
+
+CONFIG = load_config()

content_core-0.3.0/src/content_core/content/__init__.py

@@ -0,0 +1,5 @@
+from .cleanup import cleanup_content
+from .extraction import extract_content
+from .summary import summarize
+
+__all__ = ["extract_content", "cleanup_content", "summarize"]

{content_core-0.1.2 → content_core-0.3.0}/src/content_core/content/cleanup/core.py

@@ -1,11 +1,11 @@
 from functools import partial
 
-from content_core.config import CLEANUP_MODEL
+from content_core.models import ModelFactory
 from content_core.templated_message import TemplatedMessageInput, templated_message
 
 
 async def cleanup_content(content) -> str:
-    templated_summary_fn = partial(templated_message, model=CLEANUP_MODEL)
+    templated_summary_fn = partial(templated_message, model=ModelFactory.get_model('cleanup_model'))
     input = TemplatedMessageInput(
         system_prompt_template="content/cleanup",
         user_prompt_text=content,

{content_core-0.1.2 → content_core-0.3.0}/src/content_core/content/extraction/graph.py

@@ -3,13 +3,13 @@ from typing import Any, Dict, Optional
 
 import magic
 from langgraph.graph import END, START, StateGraph
-from loguru import logger
 
 from content_core.common import (
     ProcessSourceInput,
     ProcessSourceState,
     UnsupportedTypeException,
 )
+from content_core.logging import logger
 from content_core.processors.audio import extract_audio  # type: ignore
 from content_core.processors.office import (
     SUPPORTED_OFFICE_TYPES,

{content_core-0.1.2 → content_core-0.3.0}/src/content_core/content/summary/core.py

@@ -1,11 +1,11 @@
 from functools import partial
 
-from content_core.config import SUMMARY_MODEL
+from content_core.models import ModelFactory
 from content_core.templated_message import TemplatedMessageInput, templated_message
 
 
 async def summarize(content: str, context: str) -> str:
-    templated_message_fn = partial(templated_message, model=SUMMARY_MODEL)
+    templated_message_fn = partial(templated_message, model=ModelFactory.get_model('summary_model'))
     response = await templated_message_fn(
         TemplatedMessageInput(
             user_prompt_template="content/summarize",

content_core-0.3.0/src/content_core/logging.py

@@ -0,0 +1,15 @@
+import sys
+from loguru import logger
+
+def configure_logging(debug=False):
+    """
+    Configure the global logger for the application.
+    
+    Args:
+        debug (bool): If True, set logging level to DEBUG; otherwise, set to INFO.
+    """
+    logger.remove()  # Remove any existing handlers
+    logger.add(sys.stderr, level="DEBUG" if debug else "INFO")
+
+# Initial configuration with default level (INFO)
+configure_logging(debug=False)

content_core-0.3.0/src/content_core/models.py

@@ -0,0 +1,24 @@
+from esperanto import AIFactory
+from esperanto.providers.stt import SpeechToTextModel
+from .config import CONFIG
+
+class ModelFactory:
+    _instances = {}
+
+    @staticmethod
+    def get_model(model_alias):
+        if model_alias not in ModelFactory._instances:
+            config = CONFIG.get(model_alias, {})
+            if not config:
+                raise ValueError(f"Configuração para o modelo {model_alias} não encontrada.")
+
+            provider = config.get('provider')
+            model_name = config.get('model_name')
+            model_config = config.get('config', {})
+
+            if model_alias == 'speech_to_text':
+                ModelFactory._instances[model_alias] = AIFactory.create_speech_to_text(provider, model_name)
+            else:
+                ModelFactory._instances[model_alias] = AIFactory.create_language(provider, model_name, config=model_config)
+
+        return ModelFactory._instances[model_alias]

content_core-0.3.0/src/content_core/models_config.yaml

@@ -0,0 +1,27 @@
+speech_to_text:
+  provider: openai
+  model_name: whisper-1
+
+default_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0.5
+    top_p: 1
+    max_tokens: 2000
+
+cleanup_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0
+    max_tokens: 8000
+    output_format: json
+
+summary_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0
+    top_p: 1
+    max_tokens: 2000