ostruct-cli 0.1.0__tar.gz

ostruct_cli-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Yaniv Golan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ostruct_cli-0.1.0/PKG-INFO

@@ -0,0 +1,182 @@
+Metadata-Version: 2.3
+Name: ostruct-cli
+Version: 0.1.0
+Summary: CLI for OpenAI Structured Output
+Author: Yaniv Golan
+Author-email: yaniv@golan.name
+Requires-Python: >=3.9,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: cachetools (>=5.3.2,<6.0.0)
+Requires-Dist: chardet (>=5.0.0,<6.0.0)
+Requires-Dist: ijson (>=3.2.3,<4.0.0)
+Requires-Dist: jsonschema (>=4.23.0,<5.0.0)
+Requires-Dist: openai-structured (>=1.0.0,<2.0.0)
+Requires-Dist: pydantic (>=2.6.3,<3.0.0)
+Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
+Requires-Dist: tiktoken (>=0.6.0,<0.7.0)
+Requires-Dist: typing-extensions (>=4.9.0,<5.0.0)
+Description-Content-Type: text/markdown
+
+# ostruct-cli
+
+[![PyPI version](https://badge.fury.io/py/ostruct-cli.svg)](https://badge.fury.io/py/ostruct-cli)
+[![Python Versions](https://img.shields.io/pypi/pyversions/ostruct-cli.svg)](https://pypi.org/project/ostruct-cli/)
+[![Documentation Status](https://readthedocs.org/projects/ostruct-cli/badge/?version=latest)](https://ostruct-cli.readthedocs.io/en/latest/?badge=latest)
+[![CI](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml/badge.svg)](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Command-line interface for working with OpenAI models and structured output, powered by the [openai-structured](https://github.com/yaniv-golan/openai-structured) library.
+
+## Features
+
+- Generate structured output from natural language using OpenAI models
+- Rich template system for defining output schemas
+- Automatic token counting and context window management
+- Streaming support for real-time output
+- Caching system for cost optimization
+- Secure handling of sensitive data
+
+## Installation
+
+```bash
+pip install ostruct-cli
+```
+
+## Quick Start
+
+1. Set your OpenAI API key:
+
+```bash
+export OPENAI_API_KEY=your-api-key
+```
+
+2. Create a task template file `task.j2`:
+
+```
+Extract information about the person: {{ stdin }}
+```
+
+3. Create a schema file `schema.json`:
+
+```json
+{
+  "type": "object",
+  "properties": {
+    "name": {
+      "type": "string",
+      "description": "The person's full name"
+    },
+    "age": {
+      "type": "integer",
+      "description": "The person's age"
+    },
+    "occupation": {
+      "type": "string",
+      "description": "The person's job or profession"
+    }
+  },
+  "required": ["name", "age", "occupation"]
+}
+```
+
+4. Run the CLI:
+
+```bash
+echo "John Smith is a 35 year old software engineer" | ostruct --task @task.j2 --schema schema.json
+```
+
+Output:
+
+```json
+{
+  "name": "John Smith",
+  "age": 35,
+  "occupation": "software engineer"
+}
+```
+
+### About Template Files
+
+Template files use the `.j2` extension to indicate they contain Jinja2 template syntax. This convention:
+
+- Enables proper syntax highlighting in most editors
+- Makes it clear the file contains template logic
+- Follows industry standards for Jinja2 templates
+
+While the CLI accepts templates with any extension (when prefixed with `@`), we recommend using `.j2` for better tooling support and clarity.
+
+## Debug Options
+
+- `--show-model-schema`: Display the generated Pydantic model schema
+- `--debug-validation`: Show detailed schema validation debugging
+- `--verbose-schema`: Enable verbose schema debugging output
+- `--debug-openai-stream`: Enable low-level debug output for OpenAI streaming (very verbose)
+- `--progress-level {none,basic,detailed}`: Set progress reporting level (default: basic)
+
+All debug and error logs are written to:
+
+- `~/.ostruct/logs/ostruct.log`: General application logs
+- `~/.ostruct/logs/openai_stream.log`: OpenAI streaming operations logs
+
+For more detailed documentation and examples, visit our [documentation](https://ostruct-cli.readthedocs.io/).
+
+## Development
+
+To contribute or report issues, please visit our [GitHub repository](https://github.com/yaniv-golan/ostruct).
+
+## Development Setup
+
+1. Clone the repository:
+
+```bash
+git clone https://github.com/yanivgolan/ostruct.git
+cd ostruct
+```
+
+2. Install Poetry if you haven't already:
+
+```bash
+curl -sSL https://install.python-poetry.org | python3 -
+```
+
+3. Install dependencies:
+
+```bash
+poetry install
+```
+
+4. Install openai-structured in editable mode:
+
+```bash
+poetry add --editable ../openai-structured  # Adjust path as needed
+```
+
+5. Run tests:
+
+```bash
+poetry run pytest
+```
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
+
+## Migration from openai-structured
+
+If you were previously using the CLI bundled with openai-structured (pre-1.0.0), this is its new home. The migration is straightforward:
+
+1. Update openai-structured to version 1.0.0 or later
+2. Install ostruct-cli
+3. Replace any `openai-structured` CLI commands with `ostruct`
+
+The functionality remains the same, just moved to a dedicated package for better maintenance and focus.
+

ostruct_cli-0.1.0/README.md

@@ -0,0 +1,157 @@
+# ostruct-cli
+
+[![PyPI version](https://badge.fury.io/py/ostruct-cli.svg)](https://badge.fury.io/py/ostruct-cli)
+[![Python Versions](https://img.shields.io/pypi/pyversions/ostruct-cli.svg)](https://pypi.org/project/ostruct-cli/)
+[![Documentation Status](https://readthedocs.org/projects/ostruct-cli/badge/?version=latest)](https://ostruct-cli.readthedocs.io/en/latest/?badge=latest)
+[![CI](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml/badge.svg)](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+Command-line interface for working with OpenAI models and structured output, powered by the [openai-structured](https://github.com/yaniv-golan/openai-structured) library.
+
+## Features
+
+- Generate structured output from natural language using OpenAI models
+- Rich template system for defining output schemas
+- Automatic token counting and context window management
+- Streaming support for real-time output
+- Caching system for cost optimization
+- Secure handling of sensitive data
+
+## Installation
+
+```bash
+pip install ostruct-cli
+```
+
+## Quick Start
+
+1. Set your OpenAI API key:
+
+```bash
+export OPENAI_API_KEY=your-api-key
+```
+
+2. Create a task template file `task.j2`:
+
+```
+Extract information about the person: {{ stdin }}
+```
+
+3. Create a schema file `schema.json`:
+
+```json
+{
+  "type": "object",
+  "properties": {
+    "name": {
+      "type": "string",
+      "description": "The person's full name"
+    },
+    "age": {
+      "type": "integer",
+      "description": "The person's age"
+    },
+    "occupation": {
+      "type": "string",
+      "description": "The person's job or profession"
+    }
+  },
+  "required": ["name", "age", "occupation"]
+}
+```
+
+4. Run the CLI:
+
+```bash
+echo "John Smith is a 35 year old software engineer" | ostruct --task @task.j2 --schema schema.json
+```
+
+Output:
+
+```json
+{
+  "name": "John Smith",
+  "age": 35,
+  "occupation": "software engineer"
+}
+```
+
+### About Template Files
+
+Template files use the `.j2` extension to indicate they contain Jinja2 template syntax. This convention:
+
+- Enables proper syntax highlighting in most editors
+- Makes it clear the file contains template logic
+- Follows industry standards for Jinja2 templates
+
+While the CLI accepts templates with any extension (when prefixed with `@`), we recommend using `.j2` for better tooling support and clarity.
+
+## Debug Options
+
+- `--show-model-schema`: Display the generated Pydantic model schema
+- `--debug-validation`: Show detailed schema validation debugging
+- `--verbose-schema`: Enable verbose schema debugging output
+- `--debug-openai-stream`: Enable low-level debug output for OpenAI streaming (very verbose)
+- `--progress-level {none,basic,detailed}`: Set progress reporting level (default: basic)
+
+All debug and error logs are written to:
+
+- `~/.ostruct/logs/ostruct.log`: General application logs
+- `~/.ostruct/logs/openai_stream.log`: OpenAI streaming operations logs
+
+For more detailed documentation and examples, visit our [documentation](https://ostruct-cli.readthedocs.io/).
+
+## Development
+
+To contribute or report issues, please visit our [GitHub repository](https://github.com/yaniv-golan/ostruct).
+
+## Development Setup
+
+1. Clone the repository:
+
+```bash
+git clone https://github.com/yanivgolan/ostruct.git
+cd ostruct
+```
+
+2. Install Poetry if you haven't already:
+
+```bash
+curl -sSL https://install.python-poetry.org | python3 -
+```
+
+3. Install dependencies:
+
+```bash
+poetry install
+```
+
+4. Install openai-structured in editable mode:
+
+```bash
+poetry add --editable ../openai-structured  # Adjust path as needed
+```
+
+5. Run tests:
+
+```bash
+poetry run pytest
+```
+
+## Contributing
+
+Contributions are welcome! Please feel free to submit a Pull Request.
+
+## License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
+
+## Migration from openai-structured
+
+If you were previously using the CLI bundled with openai-structured (pre-1.0.0), this is its new home. The migration is straightforward:
+
+1. Update openai-structured to version 1.0.0 or later
+2. Install ostruct-cli
+3. Replace any `openai-structured` CLI commands with `ostruct`
+
+The functionality remains the same, just moved to a dedicated package for better maintenance and focus.

ostruct_cli-0.1.0/pyproject.toml

@@ -0,0 +1,110 @@
+    [build-system]
+    requires = ["poetry-core"]
+    build-backend = "poetry.core.masonry.api"
+
+    [tool.poetry]
+    name = "ostruct-cli"
+    version = "0.1.0"
+    description = "CLI for OpenAI Structured Output"
+    authors = ["Yaniv Golan <yaniv@golan.name>"]
+    readme = "README.md"
+    packages = [{include = "ostruct", from = "src"}]
+
+    [tool.poetry.dependencies]
+    python = ">=3.9,<4.0"
+    pydantic = "^2.6.3"
+    jsonschema = "^4.23.0"
+    chardet = "^5.0.0"
+    cachetools = "^5.3.2"
+    ijson = "^3.2.3"
+    typing-extensions = "^4.9.0"
+    tiktoken = "^0.6.0"
+    pyyaml = "^6.0.2"
+    openai-structured = "^1.0.0"
+
+    [tool.poetry.scripts]
+    ostruct = "ostruct.cli.cli:main"
+
+    [tool.poetry.group.dev.dependencies]
+    pytest = "^8.3.4"
+    flake8 = "^6.0"
+    flake8-pyproject = "^1.2.3"
+    black = "24.8.0"
+    isort = "^5.13"
+    mypy = "^1.0"
+    pytest-asyncio = "^0.25.2"
+    pytest-mock = "^3.14.0"
+    build = "^1.2.2.post1"
+    twine = "^6.0.1"
+    python-dotenv = "^1.0.1"
+    types-jsonschema = "^4.23.0.20241208"
+    anyio = {version = "3.7.1", extras = ["trio"]}
+    sphinx = ">=7.0,<8.0"
+    types-pyyaml = "^6.0.12.20241230"
+    types-pygments = "^2.19.0.20250107"
+    types-chardet = "^5.0.4.6"
+    pyfakefs = "^5.7.4"
+    types-cachetools = "^5.5.0.20240820"
+
+    [tool.poetry.group.docs]
+    optional = true
+
+    [tool.poetry.group.docs.dependencies]
+    sphinx = "^7.0"
+    sphinx-rtd-theme = "^1.0"
+    myst-parser = "^2.0.0"
+
+    [tool.flake8]
+    max-line-length = 120
+    extend-ignore = ["E203"]
+
+    [tool.mypy]
+    plugins = ["pydantic.mypy"]
+    strict = true
+    exclude = [
+        "docs/",
+        "examples/"
+    ]
+    packages = ["ostruct", "tests"]
+    python_version = "3.9"
+    warn_unused_configs = true
+    disallow_untyped_defs = true
+    check_untyped_defs = true
+    warn_redundant_casts = true
+    warn_unused_ignores = true
+    warn_return_any = true
+    warn_unreachable = true
+    show_error_codes = true
+    ignore_missing_imports = false
+
+    [tool.black]
+    line-length = 79
+    target-version = ["py39"]
+    include = '\.pyi?$'
+    preview = false
+    required-version = "24.8.0"
+
+    [tool.isort]
+    profile = "black"
+    multi_line_output = 3
+    line_length = 79
+
+    [tool.pytest.ini_options]
+    asyncio_mode = "strict"
+    testpaths = ["tests"]
+    python_files = ["test_*.py"]
+    markers = [
+        "live: mark test as a live test that should use real API key",
+        "asyncio: mark test as requiring async loop",
+    ]
+    asyncio_default_fixture_loop_scope = "function"
+
+    [tool.ruff]
+    target-version = "py39"
+
+    [tool.poetry.group.examples]
+    optional = true
+
+    [tool.poetry.group.examples.dependencies]
+    tenacity = "^8.2.3"
+    asyncio-throttle = "^1.0.2"

ostruct_cli-0.1.0/src/ostruct/cli/__init__.py

@@ -0,0 +1,19 @@
+"""Command-line interface for making structured OpenAI API calls."""
+
+from .cli import (
+    ExitCode,
+    _main,
+    validate_schema_file,
+    validate_task_template,
+    validate_variable_mapping,
+)
+from .path_utils import validate_path_mapping
+
+__all__ = [
+    "ExitCode",
+    "_main",
+    "validate_path_mapping",
+    "validate_schema_file",
+    "validate_task_template",
+    "validate_variable_mapping",
+]

ostruct_cli-0.1.0/src/ostruct/cli/cache_manager.py

@@ -0,0 +1,175 @@
+"""Cache management for file content.
+
+This module provides a thread-safe cache manager for file content
+with LRU eviction and automatic invalidation on file changes.
+"""
+
+import logging
+from dataclasses import dataclass
+from typing import Any, Optional, Tuple
+
+from cachetools import LRUCache
+from cachetools.keys import hashkey
+
+logger = logging.getLogger(__name__)
+
+# Type alias for cache keys
+CacheKey = Tuple[Any, ...]
+
+
+@dataclass(frozen=True)
+class CacheEntry:
+    """Represents a cached file entry.
+
+    Note: This class is immutable (frozen) to ensure thread safety
+    when used as a cache value.
+    """
+
+    content: str
+    encoding: Optional[str]
+    hash_value: Optional[str]
+    mtime_ns: int  # Nanosecond precision mtime
+    size: int  # Actual file size from stat
+
+
+class FileCache:
+    """Thread-safe LRU cache for file content with size limit."""
+
+    def __init__(self, max_size_bytes: int = 50 * 1024 * 1024):  # 50MB default
+        """Initialize cache with maximum size in bytes.
+
+        Args:
+            max_size_bytes: Maximum cache size in bytes
+        """
+        self._max_size = max_size_bytes
+        self._current_size = 0
+        self._cache: LRUCache[CacheKey, CacheEntry] = LRUCache(maxsize=1024)
+        logger.debug(
+            "Initialized FileCache with max_size=%d bytes, maxsize=%d entries",
+            max_size_bytes,
+            1024,
+        )
+
+    def _remove_entry(self, key: CacheKey) -> None:
+        """Remove entry from cache and update size.
+
+        Args:
+            key: Cache key to remove
+        """
+        entry = self._cache.get(key)
+        if entry is not None:
+            self._current_size -= entry.size
+            logger.debug(
+                "Removed cache entry: key=%s, size=%d, new_total_size=%d",
+                key,
+                entry.size,
+                self._current_size,
+            )
+        self._cache.pop(key, None)
+
+    def get(
+        self, path: str, current_mtime_ns: int, current_size: int
+    ) -> Optional[CacheEntry]:
+        """Get cache entry if it exists and is valid.
+
+        Args:
+            path: Absolute path to the file
+            current_mtime_ns: Current modification time in nanoseconds
+            current_size: Current file size in bytes
+
+        Returns:
+            CacheEntry if valid cache exists, None otherwise
+        """
+        key = hashkey(path)
+        entry = self._cache.get(key)
+
+        if entry is None:
+            logger.debug("Cache miss for %s: no entry found", path)
+            return None
+
+        # Check if file has been modified using both mtime and size
+        if entry.mtime_ns != current_mtime_ns or entry.size != current_size:
+            logger.info(
+                "Cache invalidated for %s: mtime_ns=%d->%d (%s), size=%d->%d (%s)",
+                path,
+                entry.mtime_ns,
+                current_mtime_ns,
+                "changed" if entry.mtime_ns != current_mtime_ns else "same",
+                entry.size,
+                current_size,
+                "changed" if entry.size != current_size else "same",
+            )
+            self._remove_entry(key)
+            return None
+
+        logger.debug(
+            "Cache hit for %s: mtime_ns=%d, size=%d",
+            path,
+            entry.mtime_ns,
+            entry.size,
+        )
+        return entry
+
+    def put(
+        self,
+        path: str,
+        content: str,
+        encoding: Optional[str],
+        hash_value: Optional[str],
+        mtime_ns: int,
+        size: int,
+    ) -> None:
+        """Add or update cache entry.
+
+        Args:
+            path: Absolute path to the file
+            content: File content
+            encoding: File encoding
+            hash_value: Content hash
+            mtime_ns: File modification time in nanoseconds
+            size: File size in bytes from stat
+        """
+        if size > self._max_size:
+            logger.warning(
+                "File %s size (%d bytes) exceeds cache max size (%d bytes)",
+                path,
+                size,
+                self._max_size,
+            )
+            return
+
+        key = hashkey(path)
+        self._remove_entry(key)
+
+        entry = CacheEntry(content, encoding, hash_value, mtime_ns, size)
+
+        # Evict entries if needed
+        evicted_count = 0
+        while self._current_size + size > self._max_size and self._cache:
+            evicted_key, evicted = self._cache.popitem()
+            self._current_size -= evicted.size
+            evicted_count += 1
+            logger.debug(
+                "Evicted cache entry: key=%s, size=%d, new_total_size=%d",
+                evicted_key,
+                evicted.size,
+                self._current_size,
+            )
+
+        if evicted_count > 0:
+            logger.info(
+                "Evicted %d entries to make room for %s (size=%d)",
+                evicted_count,
+                path,
+                size,
+            )
+
+        self._cache[key] = entry
+        self._current_size += size
+        logger.debug(
+            "Added cache entry: path=%s, size=%d, total_size=%d/%d",
+            path,
+            size,
+            self._current_size,
+            self._max_size,
+        )