content-core 0.2.0__tar.gz → 0.3.1__tar.gz

content_core-0.3.1/.windsurfrules

@@ -0,0 +1,13 @@
+Also use uv as the package manager: uv run, uv sync, uv add.
+
+All documentation (code or readmes) must be in english.
+Whenever I ask you to tag and release, make sure to run `make test` as part of the process. 
+
+The full release process is: 
+- Run `make test` to make sure everything is working
+- Update version on pyproject.toml
+- Run `uv sync` to update the lock file
+- Commit all that's needed
+- Merge to main
+- Tag the release
+- Push to GitHub

{content_core-0.2.0 → content_core-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: content-core
-Version: 0.2.0
+Version: 0.3.1
 Summary: Extract what matters from any media source
 Author-email: LUIS NOVO <lfnovo@gmail.com>
 License-File: LICENSE
@@ -161,6 +161,27 @@ csum https://example.com
 csum document.txt
 ```
 
+## Quick Start
+
+You can quickly integrate `content-core` into your Python projects to extract, clean, and summarize content from various sources.
+
+```python
+import content_core as cc
+
+# Extract content from a URL, file, or text
+result = await cc.extract("https://example.com/article")
+
+# Clean messy content
+cleaned_text = await cc.clean("...messy text with [brackets] and extra spaces...")
+
+# Summarize content with optional context
+summary = await cc.summarize_content("long article text", context="explain to a child")
+```
+
+## Documentation
+
+For more information on how to use the Content Core library, including details on AI model configuration and customization, refer to our [Usage Documentation](docs/usage.md).
+
 ## Using with Langchain
 
 For users integrating with the [Langchain](https://python.langchain.com/) framework, `content-core` exposes a set of compatible tools. These tools, located in the `src/content_core/tools` directory, allow you to leverage `content-core` extraction, cleaning, and summarization capabilities directly within your Langchain agents and chains.

{content_core-0.2.0 → content_core-0.3.1}/README.md

@@ -132,6 +132,27 @@ csum https://example.com
 csum document.txt
 ```
 
+## Quick Start
+
+You can quickly integrate `content-core` into your Python projects to extract, clean, and summarize content from various sources.
+
+```python
+import content_core as cc
+
+# Extract content from a URL, file, or text
+result = await cc.extract("https://example.com/article")
+
+# Clean messy content
+cleaned_text = await cc.clean("...messy text with [brackets] and extra spaces...")
+
+# Summarize content with optional context
+summary = await cc.summarize_content("long article text", context="explain to a child")
+```
+
+## Documentation
+
+For more information on how to use the Content Core library, including details on AI model configuration and customization, refer to our [Usage Documentation](docs/usage.md).
+
 ## Using with Langchain
 
 For users integrating with the [Langchain](https://python.langchain.com/) framework, `content-core` exposes a set of compatible tools. These tools, located in the `src/content_core/tools` directory, allow you to leverage `content-core` extraction, cleaning, and summarization capabilities directly within your Langchain agents and chains.

content_core-0.3.1/docs/usage.md

@@ -0,0 +1,81 @@
+# Using the Content Core Library
+
+This documentation explains how to configure and use the **Content Core** library in your projects. The library allows customization of AI model settings through a YAML file and environment variables.
+
+## Environment Variable for Configuration
+
+The library uses the `CCORE_MODEL_CONFIG_PATH` environment variable to locate the custom YAML configuration file. If this variable is not set or the specified file is not found, the library will fall back to internal default settings.
+
+To set the environment variable, add the following line to your `.env` file or set it directly in your environment:
+
+```
+CCORE_MODEL_CONFIG_PATH=/path/to/your/models_config.yaml
+```
+
+## YAML File Schema
+
+The YAML configuration file defines the AI models that the library will use. The structure of the file is as follows:
+
+- **speech_to_text**: Configuration for the speech-to-text model.
+  - **provider**: Model provider (example: `openai`).
+  - **model_name**: Model name (example: `whisper-1`).
+- **default_model**: Configuration for the default language model.
+  - **provider**: Model provider.
+  - **model_name**: Model name.
+  - **config**: Additional parameters like `temperature`, `top_p`, `max_tokens`.
+- **cleanup_model**: Configuration for the content cleanup model.
+  - **provider**: Model provider.
+  - **model_name**: Model name.
+  - **config**: Additional parameters.
+- **summary_model**: Configuration for the summary model.
+  - **provider**: Model provider.
+  - **model_name**: Model name.
+  - **config**: Additional parameters.
+
+### Default YAML File
+
+Here is the content of the default YAML file used by the library:
+
+```yaml
+speech_to_text:
+  provider: openai
+  model_name: whisper-1
+
+default_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0.5
+    top_p: 1
+    max_tokens: 2000
+
+cleanup_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0
+    max_tokens: 8000
+    output_format: json
+
+summary_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0
+    top_p: 1
+    max_tokens: 2000
+```
+
+## Customization
+
+You can customize any aspect of the YAML file to suit your needs. Change the providers, model names, or configuration parameters as desired.
+
+To simplify setup, we suggest copying the provided sample files:
+- Copy `.env.sample` to `.env` and adjust the environment variables, including `CCORE_MODEL_CONFIG_PATH`.
+- Copy `models_config.yaml.sample` to your desired location and modify it as needed.
+
+This will allow you to quickly start with customized settings without needing to create the files from scratch.
+
+## Support
+
+If you have questions or encounter issues while using the library, open an issue in the repository or contact the support team.

{content_core-0.2.0 → content_core-0.3.1}/pyproject.toml

@@ -1,8 +1,9 @@
 [project]
 name = "content-core"
-version = "0.2.0"
+version = "0.3.1"
 description = "Extract what matters from any media source"
 readme = "README.md"
+homepage = "https://github.com/lfnovo/content-core"
 authors = [
     { name = "LUIS NOVO", email = "lfnovo@gmail.com" }
 ]

{content_core-0.2.0 → content_core-0.3.1}/src/content_core/__init__.py

@@ -7,18 +7,21 @@ from xml.etree import ElementTree as ET
 
 from dicttoxml import dicttoxml  # type: ignore
 from dotenv import load_dotenv
-from loguru import logger
 
 from content_core.common import ProcessSourceInput
 from content_core.content.cleanup import cleanup_content
 from content_core.content.extraction import extract_content
 from content_core.content.summary import summarize
+from content_core.logging import configure_logging, logger
+
+# Exposing functions for direct access when importing content_core as cc
+extract = extract_content
+clean = cleanup_content
 
 load_dotenv()
 
-# Configure loguru logger
-logger.remove()  # Remove default handler
-logger.add(sys.stderr, level="INFO")  # Default to INFO level
+# Configure loguru logger using centralized configuration
+configure_logging(debug=False)
 
 
 def parse_content_format(content: str) -> str:
@@ -94,10 +97,9 @@ async def ccore_main():
 
     args = parser.parse_args()
 
-    # Adjust logging level based on debug flag
+    # Adjust logging level based on debug flag using centralized configuration
+    configure_logging(debug=args.debug)
     if args.debug:
-        logger.remove()
-        logger.add(sys.stderr, level="DEBUG")
         logger.debug("Debug logging enabled")
 
     content = get_content(args, parser)
@@ -136,10 +138,9 @@ async def cclean_main():
 
     args = parser.parse_args()
 
-    # Adjust logging level based on debug flag
+    # Adjust logging level based on debug flag using centralized configuration
+    configure_logging(debug=args.debug)
     if args.debug:
-        logger.remove()
-        logger.add(sys.stderr, level="DEBUG")
         logger.debug("Debug logging enabled")
 
     content = get_content(args, parser)
@@ -176,10 +177,9 @@ async def csum_main():
 
     args = parser.parse_args()
 
-    # Adjust logging level based on debug flag
+    # Adjust logging level based on debug flag using centralized configuration
+    configure_logging(debug=args.debug)
     if args.debug:
-        logger.remove()
-        logger.add(sys.stderr, level="DEBUG")
         logger.debug("Debug logging enabled")
 
     content = get_content(args, parser)

content_core-0.3.1/src/content_core/config.py

@@ -0,0 +1,27 @@
+import os
+import pkgutil
+
+import yaml
+from dotenv import load_dotenv
+
+# Load environment variables from .env file
+load_dotenv()
+
+
+def load_config():
+    config_path = os.environ.get("CCORE_MODEL_CONFIG_PATH")
+    if config_path and os.path.exists(config_path):
+        try:
+            with open(config_path, "r") as file:
+                return yaml.safe_load(file)
+        except Exception as e:
+            print(f"Erro ao carregar o arquivo de configuração de {config_path}: {e}")
+            print("Usando configurações padrão internas.")
+
+    default_config_data = pkgutil.get_data("content_core", "models_config.yaml")
+    if default_config_data:
+        return yaml.safe_load(default_config_data)
+    return {}
+
+
+CONFIG = load_config()

content_core-0.3.1/src/content_core/content/__init__.py

@@ -0,0 +1,5 @@
+from .cleanup import cleanup_content
+from .extraction import extract_content
+from .summary import summarize
+
+__all__ = ["extract_content", "cleanup_content", "summarize"]

{content_core-0.2.0 → content_core-0.3.1}/src/content_core/content/cleanup/core.py

@@ -1,11 +1,11 @@
 from functools import partial
 
-from content_core.config import CLEANUP_MODEL
+from content_core.models import ModelFactory
 from content_core.templated_message import TemplatedMessageInput, templated_message
 
 
 async def cleanup_content(content) -> str:
-    templated_summary_fn = partial(templated_message, model=CLEANUP_MODEL)
+    templated_summary_fn = partial(templated_message, model=ModelFactory.get_model('cleanup_model'))
     input = TemplatedMessageInput(
         system_prompt_template="content/cleanup",
         user_prompt_text=content,

{content_core-0.2.0 → content_core-0.3.1}/src/content_core/content/extraction/graph.py

@@ -3,13 +3,13 @@ from typing import Any, Dict, Optional
 
 import magic
 from langgraph.graph import END, START, StateGraph
-from loguru import logger
 
 from content_core.common import (
     ProcessSourceInput,
     ProcessSourceState,
     UnsupportedTypeException,
 )
+from content_core.logging import logger
 from content_core.processors.audio import extract_audio  # type: ignore
 from content_core.processors.office import (
     SUPPORTED_OFFICE_TYPES,

{content_core-0.2.0 → content_core-0.3.1}/src/content_core/content/summary/core.py

@@ -1,11 +1,11 @@
 from functools import partial
 
-from content_core.config import SUMMARY_MODEL
+from content_core.models import ModelFactory
 from content_core.templated_message import TemplatedMessageInput, templated_message
 
 
 async def summarize(content: str, context: str) -> str:
-    templated_message_fn = partial(templated_message, model=SUMMARY_MODEL)
+    templated_message_fn = partial(templated_message, model=ModelFactory.get_model('summary_model'))
     response = await templated_message_fn(
         TemplatedMessageInput(
             user_prompt_template="content/summarize",

content_core-0.3.1/src/content_core/logging.py

@@ -0,0 +1,15 @@
+import sys
+from loguru import logger
+
+def configure_logging(debug=False):
+    """
+    Configure the global logger for the application.
+    
+    Args:
+        debug (bool): If True, set logging level to DEBUG; otherwise, set to INFO.
+    """
+    logger.remove()  # Remove any existing handlers
+    logger.add(sys.stderr, level="DEBUG" if debug else "INFO")
+
+# Initial configuration with default level (INFO)
+configure_logging(debug=False)

content_core-0.3.1/src/content_core/models.py

@@ -0,0 +1,24 @@
+from esperanto import AIFactory
+from esperanto.providers.stt import SpeechToTextModel
+from .config import CONFIG
+
+class ModelFactory:
+    _instances = {}
+
+    @staticmethod
+    def get_model(model_alias):
+        if model_alias not in ModelFactory._instances:
+            config = CONFIG.get(model_alias, {})
+            if not config:
+                raise ValueError(f"Configuração para o modelo {model_alias} não encontrada.")
+
+            provider = config.get('provider')
+            model_name = config.get('model_name')
+            model_config = config.get('config', {})
+
+            if model_alias == 'speech_to_text':
+                ModelFactory._instances[model_alias] = AIFactory.create_speech_to_text(provider, model_name)
+            else:
+                ModelFactory._instances[model_alias] = AIFactory.create_language(provider, model_name, config=model_config)
+
+        return ModelFactory._instances[model_alias]

content_core-0.3.1/src/content_core/models_config.yaml

@@ -0,0 +1,27 @@
+speech_to_text:
+  provider: openai
+  model_name: whisper-1
+
+default_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0.5
+    top_p: 1
+    max_tokens: 2000
+
+cleanup_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0
+    max_tokens: 8000
+    output_format: json
+
+summary_model:
+  provider: openai
+  model_name: gpt-4o-mini
+  config:
+    temperature: 0
+    top_p: 1
+    max_tokens: 2000