ostruct-cli 0.6.1__tar.gz → 0.7.0__tar.gz

{ostruct_cli-0.6.1 → ostruct_cli-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: ostruct-cli
-Version: 0.6.1
+Version: 0.7.0
 Summary: CLI for OpenAI Structured Output
 Author: Yaniv Golan
 Author-email: yaniv@golan.name
@@ -16,7 +16,7 @@ Requires-Dist: click (>=8.1.7,<9.0.0)
 Requires-Dist: ijson (>=3.2.3,<4.0.0)
 Requires-Dist: jsonschema (>=4.23.0,<5.0.0)
 Requires-Dist: openai (>=1.0.0,<2.0.0)
-Requires-Dist: openai-structured (>=2.0.0,<3.0.0)
+Requires-Dist: openai-structured (>=2.1.0,<3.0.0)
 Requires-Dist: pydantic (>=2.6.3,<3.0.0)
 Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
 Requires-Dist: tiktoken (==0.9.0)
@@ -33,7 +33,9 @@ Description-Content-Type: text/markdown
 [![CI](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml/badge.svg)](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml)
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-Command-line interface for working with OpenAI models and structured output, powered by the [openai-structured](https://github.com/yaniv-golan/openai-structured) library.
+ostruct tranforms unstructured inputs into structured, usable JSON output using OpenAI APIs.
+
+ostruct will process a set of plain text files (data, source code, CSV, etc), input variables, a dynamic prompt template, and a JSON schema specifying the desired output format, and will produce the result in JSON format.
 
 ## Features
 
@@ -42,6 +44,12 @@ Command-line interface for working with OpenAI models and structured output, pow
 - Automatic token counting and context window management
 - Streaming support for real-time output
 - Secure handling of sensitive data
+- Model registry management with support for updating to the latest OpenAI models
+- Non-intrusive registry update checks with user notifications
+
+## Requirements
+
+- Python 3.10 or higher
 
 ## Installation
 
@@ -57,6 +65,16 @@ pip install ostruct-cli
 
 If you plan to contribute to the project, see the [Development Setup](#development-setup) section below for instructions on setting up the development environment with Poetry.
 
+## Environment Variables
+
+ostruct-cli respects the following environment variables:
+
+- `OPENAI_API_KEY`: Your OpenAI API key (required unless provided via command line)
+- `OPENAI_API_BASE`: Custom API base URL (optional)
+- `OPENAI_API_VERSION`: API version to use (optional)
+- `OPENAI_API_TYPE`: API type (e.g., "azure") (optional)
+- `OSTRUCT_DISABLE_UPDATE_CHECKS`: Set to "1", "true", or "yes" to disable automatic registry update checks
+
 ## Shell Completion
 
 ostruct-cli supports shell completion for Bash, Zsh, and Fish shells. To enable it:
@@ -209,196 +227,84 @@ The command will output:
 }
 ```
 
-### Example 3: Processing Multiple Files
+## System Prompt Handling
 
-1. Create a template file `extract_from_profiles.j2`:
+ostruct-cli provides three ways to specify a system prompt, with a clear precedence order:
 
-```jinja
-Extract information about the people from this data:
+1. Command-line option (`--sys-prompt` or `--sys-file`):
 
-{% for profile in profiles %}
-== {{ profile.name }}
+   ```bash
+   # Direct string
+   ostruct run template.j2 schema.json --sys-prompt "You are an expert analyst"
 
-{{ profile.content }}
+   # From file
+   ostruct run template.j2 schema.json --sys-file system_prompt.txt
+   ```
 
-{% endfor %}
-```
+2. Template frontmatter:
 
-2. Use the same schema file `schema.json` as above, but updated for multiple people:
+   ```jinja
+   ---
+   system_prompt: You are an expert analyst
+   ---
+   Extract information from: {{ text }}
+   ```
 
-```json
-{
-  "type": "object",
-  "properties": {
-    "people": {
-      "type": "array",
-      "items": {
-        "type": "object",
-        "properties": {
-          "name": {
-            "type": "string",
-            "description": "The person's full name"
-          },
-          "age": {
-            "type": "integer",
-            "description": "The person's age"
-          },
-          "occupation": {
-            "type": "string",
-            "description": "The person's job or profession"
-          }
-        },
-        "required": ["name", "age", "occupation"],
-        "additionalProperties": false
-      }
-    }
-  },
-  "required": ["people"],
-  "additionalProperties": false
-}
-```
+3. Default system prompt (built into the CLI)
 
-3. Run the CLI:
+### Precedence Rules
 
-```bash
-# Basic usage
-ostruct run extract_from_profiles.j2 schema.json -p profiles "profiles/*.txt"
+When multiple system prompts are provided, they are resolved in this order:
 
-# With advanced options
-ostruct run extract_from_profiles.j2 schema.json \
-  -p profiles "profiles/*.txt" \
-  --model gpt-4o \
-  --sys-prompt "Extract precise information about the person" \
-  --temperature 0.5
-```
+1. Command-line options take highest precedence:
+   - If both `--sys-prompt` and `--sys-file` are provided, `--sys-prompt` wins
+   - Use `--ignore-task-sysprompt` to ignore template frontmatter
 
-The command will output:
+2. Template frontmatter is used if:
+   - No command-line options are provided
+   - `--ignore-task-sysprompt` is not set
 
-```json
-{
-  "people": [
-    {
-      "name": "John Smith",
-      "age": 35,
-      "occupation": "software engineer"
-    },
-    {
-      "name": "Jane Doe",
-      "age": 28,
-      "occupation": "data scientist"
-    }
-  ]
-}
-```
-
-### About Template Files
-
-Template files use the `.j2` extension to indicate they contain Jinja2 template syntax. This convention:
-
-- Enables proper syntax highlighting in most editors
-- Makes it clear the file contains template logic
-- Follows industry standards for Jinja2 templates
+3. Default system prompt is used only if no other prompts are provided
 
-## CLI Options
-
-The CLI revolves around a single subcommand called `run`. Basic usage:
+Example combining multiple sources:
 
 ```bash
-ostruct run <TASK_TEMPLATE> <SCHEMA_FILE> [OPTIONS]
-```
-
-Common options include:
-
-- File & Directory Inputs:
-  - `-f <NAME> <PATH>`: Map a single file to a variable name
-  - `-d <NAME> <DIR>`: Map a directory to a variable name
-  - `-p <NAME> <PATTERN>`: Map files matching a glob pattern to a variable name
-  - `-R, --recursive`: Enable recursive directory/pattern scanning
-
-- Variables:
-  - `-V name=value`: Define a simple string variable
-  - `-J name='{"key":"value"}'`: Define a JSON variable
-
-- Model Parameters:
-  - `-m, --model MODEL`: Select the OpenAI model (supported: gpt-4o, o1, o3-mini)
-  - `--temperature FLOAT`: Set sampling temperature (0.0-2.0)
-  - `--max-output-tokens INT`: Set maximum output tokens
-  - `--top-p FLOAT`: Set top-p sampling parameter (0.0-1.0)
-  - `--frequency-penalty FLOAT`: Adjust frequency penalty (-2.0-2.0)
-  - `--presence-penalty FLOAT`: Adjust presence penalty (-2.0-2.0)
-  - `--reasoning-effort [low|medium|high]`: Control model reasoning effort
-
-- System Prompt:
-  - `--sys-prompt TEXT`: Provide system prompt directly
-  - `--sys-file FILE`: Load system prompt from file
-  - `--ignore-task-sysprompt`: Ignore system prompt in template frontmatter
-
-- API Configuration:
-  - `--api-key KEY`: OpenAI API key (defaults to OPENAI_API_KEY env var)
-  - `--timeout FLOAT`: API timeout in seconds (default: 60.0)
-
-## Debug Options
-
-- `--debug-validation`: Show detailed schema validation debugging
-- `--debug-openai-stream`: Enable low-level debug output for OpenAI streaming
-- `--progress-level {none,basic,detailed}`: Set progress reporting level
-  - `none`: No progress indicators
-  - `basic`: Show key operation steps (default)
-  - `detailed`: Show all steps with additional info
-- `--show-model-schema`: Display the generated Pydantic model schema
-- `--verbose`: Enable verbose logging
-- `--dry-run`: Validate and render template without making API calls
-- `--no-progress`: Disable all progress indicators
-
-All debug and error logs are written to:
+# Command-line prompt will override template frontmatter
+ostruct run template.j2 schema.json --sys-prompt "Override prompt"
 
-- `~/.ostruct/logs/ostruct.log`: General application logs
-- `~/.ostruct/logs/openai_stream.log`: OpenAI streaming operations logs
-
-For more detailed documentation and examples, visit our [documentation](https://ostruct.readthedocs.io/).
-
-## Development
-
-To contribute or report issues, please visit our [GitHub repository](https://github.com/yaniv-golan/ostruct).
-
-## Development Setup
-
-1. Clone the repository:
-
-```bash
-git clone https://github.com/yanivgolan/ostruct.git
-cd ostruct
+# Ignore template frontmatter and use default
+ostruct run template.j2 schema.json --ignore-task-sysprompt
 ```
 
-2. Install Poetry if you haven't already:
+## Model Registry Management
 
-```bash
-curl -sSL https://install.python-poetry.org | python3 -
-```
+ostruct-cli maintains a registry of OpenAI models and their capabilities, which includes:
 
-3. Install dependencies:
+- Context window sizes for each model
+- Maximum output token limits
+- Supported parameters and their constraints
+- Model version information
 
-```bash
-poetry install
-```
-
-4. Install openai-structured in editable mode:
+To ensure you're using the latest models and features, you can update the registry:
 
 ```bash
-poetry add --editable ../openai-structured  # Adjust path as needed
-```
+# Update from the official repository
+ostruct update-registry
 
-5. Run tests:
+# Update from a custom URL
+ostruct update-registry --url https://example.com/models.yml
 
-```bash
-poetry run pytest
+# Force an update even if the registry is current
+ostruct update-registry --force
 ```
 
-## Contributing
+This is especially useful when:
 
-Contributions are welcome! Please feel free to submit a Pull Request.
+- New OpenAI models are released
+- Model capabilities or parameters change
+- You need to work with custom model configurations
 
-## License
+The registry file is stored at `~/.openai_structured/config/models.yml` and is automatically referenced when validating model parameters and token limits.
 
-This project is licensed under the MIT License - see the LICENSE file for details.
+The update command uses HTTP conditional requests (If-Modified-Since headers) to check if the remote registry has changed before downloading, ensuring efficient updates.
 

ostruct_cli-0.7.0/README.md

@@ -0,0 +1,282 @@
+# ostruct-cli
+
+[![PyPI version](https://badge.fury.io/py/ostruct-cli.svg)](https://badge.fury.io/py/ostruct-cli)
+[![Python Versions](https://img.shields.io/pypi/pyversions/ostruct-cli.svg)](https://pypi.org/project/ostruct-cli)
+[![Documentation Status](https://readthedocs.org/projects/ostruct/badge/?version=latest)](https://ostruct.readthedocs.io/en/latest/?badge=latest)
+[![CI](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml/badge.svg)](https://github.com/yaniv-golan/ostruct/actions/workflows/ci.yml)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+ostruct tranforms unstructured inputs into structured, usable JSON output using OpenAI APIs.
+
+ostruct will process a set of plain text files (data, source code, CSV, etc), input variables, a dynamic prompt template, and a JSON schema specifying the desired output format, and will produce the result in JSON format.
+
+## Features
+
+- Generate structured JSON output from natural language using OpenAI models and a JSON schema
+- Rich template system for defining prompts (Jinja2-based)
+- Automatic token counting and context window management
+- Streaming support for real-time output
+- Secure handling of sensitive data
+- Model registry management with support for updating to the latest OpenAI models
+- Non-intrusive registry update checks with user notifications
+
+## Requirements
+
+- Python 3.10 or higher
+
+## Installation
+
+### For Users
+
+To install the latest stable version from PyPI:
+
+```bash
+pip install ostruct-cli
+```
+
+### For Developers
+
+If you plan to contribute to the project, see the [Development Setup](#development-setup) section below for instructions on setting up the development environment with Poetry.
+
+## Environment Variables
+
+ostruct-cli respects the following environment variables:
+
+- `OPENAI_API_KEY`: Your OpenAI API key (required unless provided via command line)
+- `OPENAI_API_BASE`: Custom API base URL (optional)
+- `OPENAI_API_VERSION`: API version to use (optional)
+- `OPENAI_API_TYPE`: API type (e.g., "azure") (optional)
+- `OSTRUCT_DISABLE_UPDATE_CHECKS`: Set to "1", "true", or "yes" to disable automatic registry update checks
+
+## Shell Completion
+
+ostruct-cli supports shell completion for Bash, Zsh, and Fish shells. To enable it:
+
+### Bash
+
+Add this to your `~/.bashrc`:
+
+```bash
+eval "$(_OSTRUCT_COMPLETE=bash_source ostruct)"
+```
+
+### Zsh
+
+Add this to your `~/.zshrc`:
+
+```bash
+eval "$(_OSTRUCT_COMPLETE=zsh_source ostruct)"
+```
+
+### Fish
+
+Add this to your `~/.config/fish/completions/ostruct.fish`:
+
+```fish
+eval (env _OSTRUCT_COMPLETE=fish_source ostruct)
+```
+
+After adding the appropriate line, restart your shell or source the configuration file.
+Shell completion will help you with:
+
+- Command options and their arguments
+- File paths for template and schema files
+- Directory paths for `-d` and `--base-dir` options
+- And more!
+
+## Quick Start
+
+1. Set your OpenAI API key:
+
+```bash
+export OPENAI_API_KEY=your-api-key
+```
+
+### Example 1: Using stdin (Simplest)
+
+1. Create a template file `extract_person.j2`:
+
+```jinja
+Extract information about the person from this text: {{ stdin }}
+```
+
+2. Create a schema file `schema.json`:
+
+```json
+{
+  "type": "object",
+  "properties": {
+    "person": {
+      "type": "object",
+      "properties": {
+        "name": {
+          "type": "string",
+          "description": "The person's full name"
+        },
+        "age": {
+          "type": "integer",
+          "description": "The person's age"
+        },
+        "occupation": {
+          "type": "string",
+          "description": "The person's job or profession"
+        }
+      },
+      "required": ["name", "age", "occupation"],
+      "additionalProperties": false
+    }
+  },
+  "required": ["person"],
+  "additionalProperties": false
+}
+```
+
+3. Run the CLI:
+
+```bash
+# Basic usage
+echo "John Smith is a 35-year-old software engineer" | ostruct run extract_person.j2 schema.json
+
+# For longer text using heredoc
+cat << EOF | ostruct run extract_person.j2 schema.json
+John Smith is a 35-year-old software engineer
+working at Tech Corp. He has been programming
+for over 10 years.
+EOF
+
+# With advanced options
+echo "John Smith is a 35-year-old software engineer" | \
+  ostruct run extract_person.j2 schema.json \
+  --model gpt-4o \
+  --sys-prompt "Extract precise information about the person" \
+  --temperature 0.7
+```
+
+The command will output:
+
+```json
+{
+  "person": {
+    "name": "John Smith",
+    "age": 35,
+    "occupation": "software engineer"
+  }
+}
+```
+
+### Example 2: Processing a Single File
+
+1. Create a template file `extract_from_file.j2`:
+
+```jinja
+Extract information about the person from this text: {{ text.content }}
+```
+
+2. Use the same schema file `schema.json` as above.
+
+3. Run the CLI:
+
+```bash
+# Basic usage
+ostruct run extract_from_file.j2 schema.json -f text input.txt
+
+# With advanced options
+ostruct run extract_from_file.j2 schema.json \
+  -f text input.txt \
+  --model gpt-4o \
+  --max-output-tokens 1000 \
+  --temperature 0.7
+```
+
+The command will output:
+
+```json
+{
+  "person": {
+    "name": "John Smith",
+    "age": 35,
+    "occupation": "software engineer"
+  }
+}
+```
+
+## System Prompt Handling
+
+ostruct-cli provides three ways to specify a system prompt, with a clear precedence order:
+
+1. Command-line option (`--sys-prompt` or `--sys-file`):
+
+   ```bash
+   # Direct string
+   ostruct run template.j2 schema.json --sys-prompt "You are an expert analyst"
+
+   # From file
+   ostruct run template.j2 schema.json --sys-file system_prompt.txt
+   ```
+
+2. Template frontmatter:
+
+   ```jinja
+   ---
+   system_prompt: You are an expert analyst
+   ---
+   Extract information from: {{ text }}
+   ```
+
+3. Default system prompt (built into the CLI)
+
+### Precedence Rules
+
+When multiple system prompts are provided, they are resolved in this order:
+
+1. Command-line options take highest precedence:
+   - If both `--sys-prompt` and `--sys-file` are provided, `--sys-prompt` wins
+   - Use `--ignore-task-sysprompt` to ignore template frontmatter
+
+2. Template frontmatter is used if:
+   - No command-line options are provided
+   - `--ignore-task-sysprompt` is not set
+
+3. Default system prompt is used only if no other prompts are provided
+
+Example combining multiple sources:
+
+```bash
+# Command-line prompt will override template frontmatter
+ostruct run template.j2 schema.json --sys-prompt "Override prompt"
+
+# Ignore template frontmatter and use default
+ostruct run template.j2 schema.json --ignore-task-sysprompt
+```
+
+## Model Registry Management
+
+ostruct-cli maintains a registry of OpenAI models and their capabilities, which includes:
+
+- Context window sizes for each model
+- Maximum output token limits
+- Supported parameters and their constraints
+- Model version information
+
+To ensure you're using the latest models and features, you can update the registry:
+
+```bash
+# Update from the official repository
+ostruct update-registry
+
+# Update from a custom URL
+ostruct update-registry --url https://example.com/models.yml
+
+# Force an update even if the registry is current
+ostruct update-registry --force
+```
+
+This is especially useful when:
+
+- New OpenAI models are released
+- Model capabilities or parameters change
+- You need to work with custom model configurations
+
+The registry file is stored at `~/.openai_structured/config/models.yml` and is automatically referenced when validating model parameters and token limits.
+
+The update command uses HTTP conditional requests (If-Modified-Since headers) to check if the remote registry has changed before downloading, ensuring efficient updates.

{ostruct_cli-0.6.1 → ostruct_cli-0.7.0}/pyproject.toml

@@ -4,7 +4,7 @@
 
     [tool.poetry]
     name = "ostruct-cli"
-    version = "0.6.1"
+    version = "0.7.0"
     description = "CLI for OpenAI Structured Output"
     authors = ["Yaniv Golan <yaniv@golan.name>"]
     readme = "README.md"
@@ -24,7 +24,7 @@
     click = "^8.1.7"
     werkzeug = "^3.1.3"
     openai = "^1.0.0"
-    openai-structured = "^2.0.0"
+    openai-structured = "^2.1.0"
     tiktoken = "0.9.0"
 
     [tool.poetry.scripts]

{ostruct_cli-0.6.1 → ostruct_cli-0.7.0}/src/ostruct/cli/__init__.py

@@ -8,6 +8,7 @@ from .cli import (
     validate_variable_mapping,
 )
 from .path_utils import validate_path_mapping
+from .registry_updates import get_update_notification
 
 __all__ = [
     "ExitCode",
@@ -16,4 +17,5 @@ __all__ = [
     "validate_schema_file",
     "validate_task_template",
     "validate_variable_mapping",
+    "get_update_notification",
 ]