ostruct-cli 0.6.2__tar.gz → 0.7.1__tar.gz

{ostruct_cli-0.6.2 → ostruct_cli-0.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: ostruct-cli
-Version: 0.6.2
+Version: 0.7.1
 Summary: CLI for OpenAI Structured Output
 Author: Yaniv Golan
 Author-email: yaniv@golan.name
@@ -16,8 +16,9 @@ Requires-Dist: click (>=8.1.7,<9.0.0)
 Requires-Dist: ijson (>=3.2.3,<4.0.0)
 Requires-Dist: jsonschema (>=4.23.0,<5.0.0)
 Requires-Dist: openai (>=1.0.0,<2.0.0)
-Requires-Dist: openai-structured (>=2.0.0,<3.0.0)
+Requires-Dist: openai-structured (>=3.0.0,<4.0.0)
 Requires-Dist: pydantic (>=2.6.3,<3.0.0)
+Requires-Dist: pygments (>=2.15.0,<3.0.0)
 Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
 Requires-Dist: tiktoken (==0.9.0)
 Requires-Dist: tomli (>=2.0.1,<3.0.0) ; python_version < "3.11"
@@ -25,7 +26,9 @@ Requires-Dist: typing-extensions (>=4.9.0,<5.0.0)
 Requires-Dist: werkzeug (>=3.1.3,<4.0.0)
 Description-Content-Type: text/markdown
 
-# ostruct-cli
+![ostruct](src/assets/ostruct-header.png)
+
+<div align="center">
 
 [![PyPI version](https://badge.fury.io/py/ostruct-cli.svg)](https://badge.fury.io/py/ostruct-cli)
 [![Python Versions](https://img.shields.io/pypi/pyversions/ostruct-cli.svg)](https://pypi.org/project/ostruct-cli)
@@ -33,10 +36,66 @@ Description-Content-Type: text/markdown
 [![CI](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml/badge.svg)](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-ostruct tranforms unstructured inputs into structured, usable JSON output using OpenAI APIs.
+**ostruct** tranforms **unstructured** inputs into **structured**, usable **JSON** output using **OpenAI APIs** using dynamic **templates**
+
+</div>
+
+# ostruct-cli
 
 ostruct will process a set of plain text files (data, source code, CSV, etc), input variables, a dynamic prompt template, and a JSON schema specifying the desired output format, and will produce the result in JSON format.
 
+<div align="center">
+
+![How ostruct works](src/assets/ostrict-hl-diagram.png)
+
+</div>
+
+## Why ostruct?
+
+LLMs are powerful, but getting consistent, structured output from them can be challenging. ostruct solves this problem by providing a streamlined approach to transform unstructured data into reliable JSON structures. The motivation behind creating ostruct was to:
+
+- **Bridge the gap** between freeform LLM capabilities and structured data needs in production systems
+- **Simplify integration** of AI into existing workflows and applications that expect consistent data formats
+- **Ensure reliability** and validate output against a defined schema to avoid unexpected formats or missing data
+- **Reduce development time** by providing a standardized way to interact with OpenAI models for structured outputs
+- **Enable non-developers** to leverage AI capabilities through a simple CLI interface with templates
+
+## Real-World Use Cases
+
+ostruct can be used for various scenarios, including:
+
+### Etymology Analysis
+
+```bash
+ostruct run prompts/task.j2 schemas/etymology.json -f input examples/scientific.txt --model gpt-4o
+```
+
+Break down words into their components, showing their origins, meanings, and hierarchical relationships. Useful for linguistics, educational tools, and understanding terminology in specialized fields.
+
+### Automated Code Review
+
+```bash
+ostruct run prompts/task.j2 schemas/code_review.json -p source "examples/security/*.py" --model gpt-4o
+```
+
+Analyze code for security vulnerabilities, style issues, and performance problems, producing structured reports that can be easily integrated into CI/CD pipelines or developer workflows.
+
+### Security Vulnerability Scanning
+
+```bash
+ostruct run prompts/task.j2 schemas/scan_result.json -d examples/intermediate --model gpt-4o
+```
+
+Scan codebases for security vulnerabilities, combining static analysis with AI-powered reasoning to identify potential issues, suggest fixes, and provide detailed explanations.
+
+### Configuration Validation & Analysis
+
+```bash
+ostruct run prompts/task.j2 schemas/validation_result.json -f dev examples/basic/dev.yaml -f prod examples/basic/prod.yaml
+```
+
+Validate configuration files across environments, check for inconsistencies, and provide intelligent feedback on potential issues or improvements in infrastructure setups.
+
 ## Features
 
 - Generate structured JSON output from natural language using OpenAI models and a JSON schema
@@ -44,6 +103,8 @@ ostruct will process a set of plain text files (data, source code, CSV, etc), in
 - Automatic token counting and context window management
 - Streaming support for real-time output
 - Secure handling of sensitive data
+- Model registry management with support for updating to the latest OpenAI models
+- Non-intrusive registry update checks with user notifications
 
 ## Requirements
 
@@ -63,6 +124,16 @@ pip install ostruct-cli
 
 If you plan to contribute to the project, see the [Development Setup](#development-setup) section below for instructions on setting up the development environment with Poetry.
 
+## Environment Variables
+
+ostruct-cli respects the following environment variables:
+
+- `OPENAI_API_KEY`: Your OpenAI API key (required unless provided via command line)
+- `OPENAI_API_BASE`: Custom API base URL (optional)
+- `OPENAI_API_VERSION`: API version to use (optional)
+- `OPENAI_API_TYPE`: API type (e.g., "azure") (optional)
+- `OSTRUCT_DISABLE_UPDATE_CHECKS`: Set to "1", "true", or "yes" to disable automatic registry update checks
+
 ## Shell Completion
 
 ostruct-cli supports shell completion for Bash, Zsh, and Fish shells. To enable it:
@@ -264,3 +335,35 @@ ostruct run template.j2 schema.json --sys-prompt "Override prompt"
 ostruct run template.j2 schema.json --ignore-task-sysprompt
 ```
 
+## Model Registry Management
+
+ostruct-cli maintains a registry of OpenAI models and their capabilities, which includes:
+
+- Context window sizes for each model
+- Maximum output token limits
+- Supported parameters and their constraints
+- Model version information
+
+To ensure you're using the latest models and features, you can update the registry:
+
+```bash
+# Update from the official repository
+ostruct update-registry
+
+# Update from a custom URL
+ostruct update-registry --url https://example.com/models.yml
+
+# Force an update even if the registry is current
+ostruct update-registry --force
+```
+
+This is especially useful when:
+
+- New OpenAI models are released
+- Model capabilities or parameters change
+- You need to work with custom model configurations
+
+The registry file is stored at `~/.openai_structured/config/models.yml` and is automatically referenced when validating model parameters and token limits.
+
+The update command uses HTTP conditional requests (If-Modified-Since headers) to check if the remote registry has changed before downloading, ensuring efficient updates.
+

{ostruct_cli-0.6.2 → ostruct_cli-0.7.1}/README.md

@@ -1,4 +1,6 @@
-# ostruct-cli
+![ostruct](src/assets/ostruct-header.png)
+
+<div align="center">
 
 [![PyPI version](https://badge.fury.io/py/ostruct-cli.svg)](https://badge.fury.io/py/ostruct-cli)
 [![Python Versions](https://img.shields.io/pypi/pyversions/ostruct-cli.svg)](https://pypi.org/project/ostruct-cli)
@@ -6,10 +8,66 @@
 [![CI](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml/badge.svg)](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-ostruct tranforms unstructured inputs into structured, usable JSON output using OpenAI APIs.
+**ostruct** tranforms **unstructured** inputs into **structured**, usable **JSON** output using **OpenAI APIs** using dynamic **templates**
+
+</div>
+
+# ostruct-cli
 
 ostruct will process a set of plain text files (data, source code, CSV, etc), input variables, a dynamic prompt template, and a JSON schema specifying the desired output format, and will produce the result in JSON format.
 
+<div align="center">
+
+![How ostruct works](src/assets/ostrict-hl-diagram.png)
+
+</div>
+
+## Why ostruct?
+
+LLMs are powerful, but getting consistent, structured output from them can be challenging. ostruct solves this problem by providing a streamlined approach to transform unstructured data into reliable JSON structures. The motivation behind creating ostruct was to:
+
+- **Bridge the gap** between freeform LLM capabilities and structured data needs in production systems
+- **Simplify integration** of AI into existing workflows and applications that expect consistent data formats
+- **Ensure reliability** and validate output against a defined schema to avoid unexpected formats or missing data
+- **Reduce development time** by providing a standardized way to interact with OpenAI models for structured outputs
+- **Enable non-developers** to leverage AI capabilities through a simple CLI interface with templates
+
+## Real-World Use Cases
+
+ostruct can be used for various scenarios, including:
+
+### Etymology Analysis
+
+```bash
+ostruct run prompts/task.j2 schemas/etymology.json -f input examples/scientific.txt --model gpt-4o
+```
+
+Break down words into their components, showing their origins, meanings, and hierarchical relationships. Useful for linguistics, educational tools, and understanding terminology in specialized fields.
+
+### Automated Code Review
+
+```bash
+ostruct run prompts/task.j2 schemas/code_review.json -p source "examples/security/*.py" --model gpt-4o
+```
+
+Analyze code for security vulnerabilities, style issues, and performance problems, producing structured reports that can be easily integrated into CI/CD pipelines or developer workflows.
+
+### Security Vulnerability Scanning
+
+```bash
+ostruct run prompts/task.j2 schemas/scan_result.json -d examples/intermediate --model gpt-4o
+```
+
+Scan codebases for security vulnerabilities, combining static analysis with AI-powered reasoning to identify potential issues, suggest fixes, and provide detailed explanations.
+
+### Configuration Validation & Analysis
+
+```bash
+ostruct run prompts/task.j2 schemas/validation_result.json -f dev examples/basic/dev.yaml -f prod examples/basic/prod.yaml
+```
+
+Validate configuration files across environments, check for inconsistencies, and provide intelligent feedback on potential issues or improvements in infrastructure setups.
+
 ## Features
 
 - Generate structured JSON output from natural language using OpenAI models and a JSON schema
@@ -17,6 +75,8 @@ ostruct will process a set of plain text files (data, source code, CSV, etc), in
 - Automatic token counting and context window management
 - Streaming support for real-time output
 - Secure handling of sensitive data
+- Model registry management with support for updating to the latest OpenAI models
+- Non-intrusive registry update checks with user notifications
 
 ## Requirements
 
@@ -36,6 +96,16 @@ pip install ostruct-cli
 
 If you plan to contribute to the project, see the [Development Setup](#development-setup) section below for instructions on setting up the development environment with Poetry.
 
+## Environment Variables
+
+ostruct-cli respects the following environment variables:
+
+- `OPENAI_API_KEY`: Your OpenAI API key (required unless provided via command line)
+- `OPENAI_API_BASE`: Custom API base URL (optional)
+- `OPENAI_API_VERSION`: API version to use (optional)
+- `OPENAI_API_TYPE`: API type (e.g., "azure") (optional)
+- `OSTRUCT_DISABLE_UPDATE_CHECKS`: Set to "1", "true", or "yes" to disable automatic registry update checks
+
 ## Shell Completion
 
 ostruct-cli supports shell completion for Bash, Zsh, and Fish shells. To enable it:
@@ -236,3 +306,35 @@ ostruct run template.j2 schema.json --sys-prompt "Override prompt"
 # Ignore template frontmatter and use default
 ostruct run template.j2 schema.json --ignore-task-sysprompt
 ```
+
+## Model Registry Management
+
+ostruct-cli maintains a registry of OpenAI models and their capabilities, which includes:
+
+- Context window sizes for each model
+- Maximum output token limits
+- Supported parameters and their constraints
+- Model version information
+
+To ensure you're using the latest models and features, you can update the registry:
+
+```bash
+# Update from the official repository
+ostruct update-registry
+
+# Update from a custom URL
+ostruct update-registry --url https://example.com/models.yml
+
+# Force an update even if the registry is current
+ostruct update-registry --force
+```
+
+This is especially useful when:
+
+- New OpenAI models are released
+- Model capabilities or parameters change
+- You need to work with custom model configurations
+
+The registry file is stored at `~/.openai_structured/config/models.yml` and is automatically referenced when validating model parameters and token limits.
+
+The update command uses HTTP conditional requests (If-Modified-Since headers) to check if the remote registry has changed before downloading, ensuring efficient updates.

{ostruct_cli-0.6.2 → ostruct_cli-0.7.1}/pyproject.toml

@@ -4,7 +4,7 @@
 
     [tool.poetry]
     name = "ostruct-cli"
-    version = "0.6.2"
+    version = "0.7.1"
     description = "CLI for OpenAI Structured Output"
     authors = ["Yaniv Golan <yaniv@golan.name>"]
     readme = "README.md"
@@ -24,8 +24,9 @@
     click = "^8.1.7"
     werkzeug = "^3.1.3"
     openai = "^1.0.0"
-    openai-structured = "^2.0.0"
+    openai-structured = "^3.0.0"
     tiktoken = "0.9.0"
+    pygments = "^2.15.0"
 
     [tool.poetry.scripts]
     ostruct = "ostruct.cli.cli:main"

{ostruct_cli-0.6.2 → ostruct_cli-0.7.1}/src/ostruct/cli/__init__.py

@@ -8,6 +8,7 @@ from .cli import (
     validate_variable_mapping,
 )
 from .path_utils import validate_path_mapping
+from .registry_updates import get_update_notification
 
 __all__ = [
     "ExitCode",
@@ -16,4 +17,5 @@ __all__ = [
     "validate_schema_file",
     "validate_task_template",
     "validate_variable_mapping",
+    "get_update_notification",
 ]

{ostruct_cli-0.6.2 → ostruct_cli-0.7.1}/src/ostruct/cli/cli.py

@@ -45,7 +45,10 @@ from openai_structured.errors import (
     OpenAIClientError,
     StreamBufferError,
 )
-from openai_structured.model_registry import ModelRegistry
+from openai_structured.model_registry import (
+    ModelRegistry,
+    RegistryUpdateStatus,
+)
 from pydantic import AnyUrl, BaseModel, EmailStr, Field
 from pydantic.fields import FieldInfo as FieldInfoType
 from pydantic.functional_validators import BeforeValidator
@@ -75,6 +78,7 @@ from .errors import (
 from .file_utils import FileInfoList, collect_files
 from .model_creation import _create_enum_type, create_dynamic_model
 from .path_utils import validate_path_mapping
+from .registry_updates import get_update_notification
 from .security import SecurityManager
 from .serialization import LogSerializer
 from .template_env import create_jinja_env
@@ -1490,7 +1494,14 @@ def cli() -> None:
 
         ostruct run task.j2 schema.json -J config='{"env":"prod"}' -m o3-mini
     """
-    pass
+    # Check for registry updates in a non-intrusive way
+    try:
+        update_message = get_update_notification()
+        if update_message:
+            click.secho(f"Note: {update_message}", fg="blue", err=True)
+    except Exception:
+        # Ensure any errors don't affect normal operation
+        pass
 
 
 @cli.command()
@@ -1560,8 +1571,78 @@ def run(
         raise
 
 
-# Remove the old @create_click_command() decorator and cli function definition
-# Keep all the other functions and code below this point
+@cli.command("update-registry")
+@click.option(
+    "--url",
+    help="URL to fetch the registry from. Defaults to official repository.",
+    default=None,
+)
+@click.option(
+    "--force",
+    is_flag=True,
+    help="Force update even if the registry is already up to date.",
+    default=False,
+)
+def update_registry(url: Optional[str] = None, force: bool = False) -> None:
+    """Update the model registry with the latest model definitions.
+
+    This command fetches the latest model registry from the official repository
+    or a custom URL if provided, and updates the local registry file.
+
+    Example:
+        ostruct update-registry
+        ostruct update-registry --url https://example.com/models.yml
+    """
+    try:
+        registry = ModelRegistry()
+
+        # Show current registry config path
+        config_path = registry._config_path
+        click.echo(f"Current registry file: {config_path}")
+
+        if force:
+            click.echo("Forcing registry update...")
+            success = registry.refresh_from_remote(url)
+            if success:
+                click.echo("✅ Registry successfully updated!")
+            else:
+                click.echo(
+                    "❌ Failed to update registry. See logs for details."
+                )
+            sys.exit(ExitCode.SUCCESS.value)
+
+        if config_path is None or not os.path.exists(config_path):
+            click.echo("Registry file not found. Creating new one...")
+            success = registry.refresh_from_remote(url)
+            if success:
+                click.echo("✅ Registry successfully created!")
+            else:
+                click.echo(
+                    "❌ Failed to create registry. See logs for details."
+                )
+            sys.exit(ExitCode.SUCCESS.value)
+
+        # Use the built-in update checking functionality
+        click.echo("Checking for updates...")
+        update_result = registry.check_for_updates()
+
+        if update_result.status == RegistryUpdateStatus.UPDATE_AVAILABLE:
+            click.echo(
+                f"{click.style('✓', fg='green')} {update_result.message}"
+            )
+            exit_code = ExitCode.SUCCESS
+        elif update_result.status == RegistryUpdateStatus.ALREADY_CURRENT:
+            click.echo(
+                f"{click.style('✓', fg='green')} Registry is up to date"
+            )
+            exit_code = ExitCode.SUCCESS
+        else:
+            click.echo("❓ Unable to determine if updates are available.")
+
+        sys.exit(exit_code)
+    except Exception as e:
+        click.echo(f"❌ Error updating registry: {str(e)}")
+        sys.exit(ExitCode.API_ERROR.value)
 
 
 async def validate_model_params(args: CLIParams) -> Dict[str, Any]:

ostruct_cli-0.7.1/src/ostruct/cli/registry_updates.py

@@ -0,0 +1,162 @@
+"""Registry update checks for ostruct CLI.
+
+This module provides functionality to check for updates to the model registry
+and notify users when updates are available.
+"""
+
+import json
+import logging
+import os
+import time
+from pathlib import Path
+from typing import Optional, Tuple
+
+from openai_structured.model_registry import (
+    ModelRegistry,
+    RegistryUpdateStatus,
+)
+
+logger = logging.getLogger(__name__)
+
+# Constants
+UPDATE_CHECK_ENV_VAR = "OSTRUCT_DISABLE_UPDATE_CHECKS"
+UPDATE_CHECK_INTERVAL_SECONDS = (
+    86400  # Check for updates once per day (24 hours)
+)
+LAST_CHECK_CACHE_FILE = ".ostruct_registry_check"
+
+
+def _get_cache_dir() -> Path:
+    """Get the cache directory for ostruct.
+
+    Returns:
+        Path: Path to the cache directory
+    """
+    # Use XDG_CACHE_HOME if available, otherwise use ~/.cache
+    xdg_cache_home = os.environ.get("XDG_CACHE_HOME")
+    if xdg_cache_home:
+        base_dir = Path(xdg_cache_home)
+    else:
+        base_dir = Path.home() / ".cache"
+
+    cache_dir = base_dir / "ostruct"
+    cache_dir.mkdir(parents=True, exist_ok=True)
+    return cache_dir
+
+
+def _get_last_check_time() -> Optional[float]:
+    """Get the timestamp of the last update check.
+
+    Returns:
+        Optional[float]: Timestamp of the last check, or None if never checked
+    """
+    cache_file = _get_cache_dir() / LAST_CHECK_CACHE_FILE
+
+    if not cache_file.exists():
+        return None
+
+    try:
+        with open(cache_file, "r") as f:
+            data = json.load(f)
+            last_check_time = data.get("last_check_time")
+            return (
+                float(last_check_time) if last_check_time is not None else None
+            )
+    except (json.JSONDecodeError, IOError, OSError):
+        return None
+
+
+def _save_last_check_time() -> None:
+    """Save the current time as the last update check time."""
+    cache_file = _get_cache_dir() / LAST_CHECK_CACHE_FILE
+
+    try:
+        data = {"last_check_time": time.time()}
+        with open(cache_file, "w") as f:
+            json.dump(data, f)
+    except (IOError, OSError) as e:
+        logger.debug(f"Failed to save last check time: {e}")
+
+
+def should_check_for_updates() -> bool:
+    """Determine if we should check for registry updates.
+
+    Returns:
+        bool: True if update checks are enabled, False otherwise
+    """
+    # Allow users to disable update checks via environment variable
+    if os.environ.get(UPDATE_CHECK_ENV_VAR, "").lower() in (
+        "1",
+        "true",
+        "yes",
+    ):
+        logger.debug(
+            "Registry update checks disabled via environment variable"
+        )
+        return False
+
+    # Check if we've checked recently
+    last_check_time = _get_last_check_time()
+    if last_check_time is not None:
+        time_since_last_check = time.time() - last_check_time
+        if time_since_last_check < UPDATE_CHECK_INTERVAL_SECONDS:
+            logger.debug(
+                f"Skipping update check, last check was {time_since_last_check:.1f} seconds ago"
+            )
+            return False
+
+    return True
+
+
+def check_for_registry_updates() -> Tuple[bool, Optional[str]]:
+    """Check if there are updates available for the model registry.
+
+    This function is designed to be non-intrusive and fail gracefully.
+
+    Returns:
+        Tuple[bool, Optional[str]]: (update_available, message)
+            - update_available: True if an update is available
+            - message: A message to display to the user, or None if no update is available
+    """
+    if not should_check_for_updates():
+        return False, None
+
+    try:
+        registry = ModelRegistry()
+        result = registry.check_for_updates()
+
+        # Save the check time regardless of the result
+        _save_last_check_time()
+
+        if result.status == RegistryUpdateStatus.UPDATE_AVAILABLE:
+            return True, (
+                "A new model registry is available. "
+                "This may include support for new models or features. "
+                "The registry will be automatically updated when needed."
+            )
+
+        return False, None
+    except Exception as e:
+        # Ensure any errors don't affect normal operation
+        logger.debug(f"Error checking for registry updates: {e}")
+        return False, None
+
+
+def get_update_notification() -> Optional[str]:
+    """Get a notification message if registry updates are available.
+
+    This function is designed to be called from the CLI to provide
+    a non-intrusive notification to users.
+
+    Returns:
+        Optional[str]: A notification message, or None if no notification is needed
+    """
+    try:
+        update_available, message = check_for_registry_updates()
+        if update_available and message:
+            return message
+        return None
+    except Exception as e:
+        # Ensure any errors don't affect normal operation
+        logger.debug(f"Error getting update notification: {e}")
+        return None