osmosis-ai 0.2.2__tar.gz → 0.2.3__tar.gz

{osmosis_ai-0.2.2 → osmosis_ai-0.2.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: osmosis-ai
-Version: 0.2.2
+Version: 0.2.3
 Summary: A Python library for reward function validation with strict type enforcement.
 Author-email: Osmosis AI <jake@osmosis.ai>
 License: MIT License
@@ -29,9 +29,18 @@ Project-URL: Issues, https://github.com/Osmosis-AI/osmosis-sdk-python/issues
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.6
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
+Requires-Dist: PyYAML<7.0,>=6.0
+Requires-Dist: python-dotenv<2.0.0,>=0.1.0
+Requires-Dist: requests<3.0.0,>=2.0.0
+Requires-Dist: xxhash<4.0.0,>=3.0.0
+Requires-Dist: anthropic<0.50.0,>=0.36.0
+Requires-Dist: openai>=2.0.0
+Requires-Dist: google-genai>=1.0.0
+Requires-Dist: xai-sdk>=1.2.0
+Requires-Dist: tqdm<5.0.0,>=4.0.0
 Dynamic: license-file
 
 # osmosis-ai
@@ -44,6 +53,10 @@ A Python library that provides reward and rubric validation helpers for LLM appl
 pip install osmosis-ai
 ```
 
+Requires Python 3.9 or newer.
+
+This installs the Osmosis CLI and pulls in the required provider SDKs (`openai`, `anthropic`, `google-genai`, `xai-sdk`) along with supporting utilities such as `PyYAML`, `python-dotenv`, `requests`, and `xxhash`.
+
 For development:
 ```bash
 git clone https://github.com/Osmosis-AI/osmosis-sdk-python
@@ -211,6 +224,55 @@ def numeric_tolerance(solution_str: str, ground_truth: str, extra_info: dict = N
 
 - `examples/rubric_functions.py` demonstrates `evaluate_rubric` with OpenAI, Anthropic, Gemini, and xAI using the schema-enforced SDK integrations.
 - `examples/reward_functions.py` keeps local reward helpers that showcase the decorator contract without external calls.
+- `examples/rubric_configs.yaml` bundles two rubric definitions, each with its own provider configuration and extra prompt context.
+- `examples/sample_data.jsonl` contains two conversation payloads mapped to those rubrics so you can trial dataset validation.
+
+```yaml
+# examples/rubric_configs.yaml (excerpt)
+version: 1
+rubrics:
+  - id: support_followup
+    model_info:
+      provider: openai
+      model: gpt-5-mini
+      api_key_env: OPENAI_API_KEY
+```
+
+```jsonl
+{"conversation_id": "ticket-001", "rubric_id": "support_followup", "...": "..."}
+{"conversation_id": "ticket-047", "rubric_id": "policy_grounding", "...": "..."}
+```
+
+## CLI Tools
+
+Installing the SDK also provides a lightweight CLI available as `osmosis` (aliases: `osmosis_ai`, `osmosis-ai`) for inspecting rubric YAML files and JSONL test payloads.
+
+Preview a rubric file and print every configuration discovered, including nested entries:
+
+```bash
+osmosis preview --path path/to/rubric.yaml
+```
+
+Preview a dataset of chat transcripts stored as JSONL:
+
+```bash
+osmosis preview --path path/to/data.jsonl
+```
+
+Evaluate a dataset against a hosted rubric configuration and print the returned scores:
+
+```bash
+osmosis eval --rubric support_followup --data examples/sample_data.jsonl
+```
+
+- Supply the dataset with `-d`/`--data path/to/data.jsonl`; the path is resolved relative to the current working directory.
+- Use `--config path/to/rubric_configs.yaml` when the rubric definitions are not located alongside the dataset.
+- Pass `-n`/`--number` to sample the provider multiple times per record; the CLI prints every run along with aggregate statistics (average, variance, standard deviation, and min/max).
+- Provide `--output path/to/dir` to create the directory (if needed) and emit `rubric_eval_result_<unix_timestamp>.json`, or supply a full file path (any extension) to control the filename; each file captures every run, provider payloads, timestamps, and aggregate statistics for downstream analysis.
+- Skip `--output` to collect results under `~/.cache/osmosis/eval_result/<rubric_id>/rubric_eval_result_<identifier>.json`; the CLI writes this JSON whether the evaluation finishes cleanly or hits provider/runtime errors so you can inspect failures later (only a manual Ctrl+C interrupt leaves no file behind).
+- Dataset rows whose `rubric_id` does not match the requested rubric are skipped automatically.
+
+Both commands validate the file, echo a short summary (`Loaded <n> ...`), and pretty-print the parsed records so you can confirm that new rubrics or test fixtures look correct before committing them. Invalid files raise a descriptive error and exit with a non-zero status code.
 
 ## Running Examples
 

{osmosis_ai-0.2.2 → osmosis_ai-0.2.3}/README.md

@@ -8,6 +8,10 @@ A Python library that provides reward and rubric validation helpers for LLM appl
 pip install osmosis-ai
 ```
 
+Requires Python 3.9 or newer.
+
+This installs the Osmosis CLI and pulls in the required provider SDKs (`openai`, `anthropic`, `google-genai`, `xai-sdk`) along with supporting utilities such as `PyYAML`, `python-dotenv`, `requests`, and `xxhash`.
+
 For development:
 ```bash
 git clone https://github.com/Osmosis-AI/osmosis-sdk-python
@@ -175,6 +179,55 @@ def numeric_tolerance(solution_str: str, ground_truth: str, extra_info: dict = N
 
 - `examples/rubric_functions.py` demonstrates `evaluate_rubric` with OpenAI, Anthropic, Gemini, and xAI using the schema-enforced SDK integrations.
 - `examples/reward_functions.py` keeps local reward helpers that showcase the decorator contract without external calls.
+- `examples/rubric_configs.yaml` bundles two rubric definitions, each with its own provider configuration and extra prompt context.
+- `examples/sample_data.jsonl` contains two conversation payloads mapped to those rubrics so you can trial dataset validation.
+
+```yaml
+# examples/rubric_configs.yaml (excerpt)
+version: 1
+rubrics:
+  - id: support_followup
+    model_info:
+      provider: openai
+      model: gpt-5-mini
+      api_key_env: OPENAI_API_KEY
+```
+
+```jsonl
+{"conversation_id": "ticket-001", "rubric_id": "support_followup", "...": "..."}
+{"conversation_id": "ticket-047", "rubric_id": "policy_grounding", "...": "..."}
+```
+
+## CLI Tools
+
+Installing the SDK also provides a lightweight CLI available as `osmosis` (aliases: `osmosis_ai`, `osmosis-ai`) for inspecting rubric YAML files and JSONL test payloads.
+
+Preview a rubric file and print every configuration discovered, including nested entries:
+
+```bash
+osmosis preview --path path/to/rubric.yaml
+```
+
+Preview a dataset of chat transcripts stored as JSONL:
+
+```bash
+osmosis preview --path path/to/data.jsonl
+```
+
+Evaluate a dataset against a hosted rubric configuration and print the returned scores:
+
+```bash
+osmosis eval --rubric support_followup --data examples/sample_data.jsonl
+```
+
+- Supply the dataset with `-d`/`--data path/to/data.jsonl`; the path is resolved relative to the current working directory.
+- Use `--config path/to/rubric_configs.yaml` when the rubric definitions are not located alongside the dataset.
+- Pass `-n`/`--number` to sample the provider multiple times per record; the CLI prints every run along with aggregate statistics (average, variance, standard deviation, and min/max).
+- Provide `--output path/to/dir` to create the directory (if needed) and emit `rubric_eval_result_<unix_timestamp>.json`, or supply a full file path (any extension) to control the filename; each file captures every run, provider payloads, timestamps, and aggregate statistics for downstream analysis.
+- Skip `--output` to collect results under `~/.cache/osmosis/eval_result/<rubric_id>/rubric_eval_result_<identifier>.json`; the CLI writes this JSON whether the evaluation finishes cleanly or hits provider/runtime errors so you can inspect failures later (only a manual Ctrl+C interrupt leaves no file behind).
+- Dataset rows whose `rubric_id` does not match the requested rubric are skipped automatically.
+
+Both commands validate the file, echo a short summary (`Loaded <n> ...`), and pretty-print the parsed records so you can confirm that new rubrics or test fixtures look correct before committing them. Invalid files raise a descriptive error and exit with a non-zero status code.
 
 ## Running Examples
 

osmosis_ai-0.2.3/osmosis_ai/cli.py

@@ -0,0 +1,50 @@
+from __future__ import annotations
+
+import argparse
+import sys
+from typing import Optional
+
+from .cli_commands import EvalCommand, PreviewCommand
+from .cli_services import CLIError
+
+
+def main(argv: Optional[list[str]] = None) -> int:
+    """Entry point for the osmosis CLI."""
+    parser = _build_parser()
+    args = parser.parse_args(argv)
+
+    handler = getattr(args, "handler", None)
+    if handler is None:
+        parser.print_help()
+        return 1
+
+    try:
+        return handler(args)
+    except CLIError as exc:
+        print(f"Error: {exc}", file=sys.stderr)
+        return 1
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="osmosis", description="Utilities for inspecting Osmosis rubric and test data files."
+    )
+    subparsers = parser.add_subparsers(dest="command")
+
+    preview_parser = subparsers.add_parser(
+        "preview",
+        help="Preview a rubric YAML file or test JSONL file and print its parsed contents.",
+    )
+    PreviewCommand().configure_parser(preview_parser)
+
+    eval_parser = subparsers.add_parser(
+        "eval",
+        help="Evaluate JSONL conversations against a rubric using remote providers.",
+    )
+    EvalCommand().configure_parser(eval_parser)
+
+    return parser
+
+
+if __name__ == "__main__":
+    sys.exit(main())

osmosis_ai-0.2.3/osmosis_ai/cli_commands.py

@@ -0,0 +1,181 @@
+from __future__ import annotations
+
+import argparse
+import time
+from pathlib import Path
+from typing import Any, Callable, Optional
+
+from .cli_services import (
+    CLIError,
+    ParsedItem,
+    BaselineComparator,
+    ConsoleReportRenderer,
+    DatasetLoader,
+    EvaluationSession,
+    EvaluationSessionRequest,
+    JsonReportWriter,
+    RubricEvaluationEngine,
+    RubricSuite,
+    discover_rubric_config_path,
+    load_jsonl_records,
+    load_rubric_configs,
+    load_rubric_suite,
+    render_json_records,
+    render_yaml_items,
+)
+
+
+class PreviewCommand:
+    """Handler for `osmosis preview`."""
+
+    def __init__(
+        self,
+        *,
+        yaml_loader: Callable[[Path], list[ParsedItem]] = load_rubric_configs,
+        json_loader: Callable[[Path], list[dict[str, Any]]] = load_jsonl_records,
+    ):
+        self._yaml_loader = yaml_loader
+        self._json_loader = json_loader
+
+    def configure_parser(self, parser: argparse.ArgumentParser) -> None:
+        parser.set_defaults(handler=self.run)
+        parser.add_argument(
+            "-p",
+            "--path",
+            dest="path",
+            required=True,
+            help="Path to the YAML or JSONL file to inspect.",
+        )
+
+    def run(self, args: argparse.Namespace) -> int:
+        path = Path(args.path).expanduser()
+        if not path.exists():
+            raise CLIError(f"Path '{path}' does not exist.")
+        if path.is_dir():
+            raise CLIError(f"Expected a file path but got directory '{path}'.")
+
+        suffix = path.suffix.lower()
+        if suffix in {".yaml", ".yml"}:
+            items = self._yaml_loader(path)
+            print(f"Loaded {len(items)} rubric config(s) from {path}")
+            print(render_yaml_items(items, label="Rubric config"))
+        elif suffix == ".jsonl":
+            records = self._json_loader(path)
+            print(f"Loaded {len(records)} JSONL record(s) from {path}")
+            print(render_json_records(records))
+        else:
+            raise CLIError(f"Unsupported file extension '{suffix}'. Expected .yaml, .yml, or .jsonl.")
+
+        return 0
+
+
+class EvalCommand:
+    """Handler for `osmosis eval`."""
+
+    def __init__(
+        self,
+        *,
+        session: Optional[EvaluationSession] = None,
+        config_locator: Callable[[Optional[str], Path], Path] = discover_rubric_config_path,
+        suite_loader: Callable[[Path], RubricSuite] = load_rubric_suite,
+        dataset_loader: Optional[DatasetLoader] = None,
+        engine: Optional[RubricEvaluationEngine] = None,
+        renderer: Optional[ConsoleReportRenderer] = None,
+        report_writer: Optional[JsonReportWriter] = None,
+        baseline_comparator: Optional[BaselineComparator] = None,
+    ):
+        self._renderer = renderer or ConsoleReportRenderer()
+        if session is not None:
+            self._session = session
+        else:
+            self._session = EvaluationSession(
+                config_locator=config_locator,
+                suite_loader=suite_loader,
+                dataset_loader=dataset_loader,
+                engine=engine,
+                baseline_comparator=baseline_comparator,
+                report_writer=report_writer,
+                identifier_factory=self._generate_output_identifier,
+            )
+
+    def configure_parser(self, parser: argparse.ArgumentParser) -> None:
+        parser.set_defaults(handler=self.run)
+        parser.add_argument(
+            "-r",
+            "--rubric",
+            dest="rubric_id",
+            required=True,
+            help="Rubric identifier declared in the rubric config file.",
+        )
+        parser.add_argument(
+            "-d",
+            "--data",
+            dest="data_path",
+            required=True,
+            help="Path to the JSONL file containing evaluation records.",
+        )
+        parser.add_argument(
+            "-n",
+            "--number",
+            dest="number",
+            type=int,
+            default=1,
+            help="Run the evaluation multiple times to sample provider variance (default: 1).",
+        )
+        parser.add_argument(
+            "-c",
+            "--config",
+            dest="config_path",
+            help="Path to the rubric config YAML (defaults to searching near the data file).",
+        )
+        parser.add_argument(
+            "-o",
+            "--output",
+            dest="output_path",
+            help="Optional path to write evaluation results as JSON.",
+        )
+        parser.add_argument(
+            "-b",
+            "--baseline",
+            dest="baseline_path",
+            help="Optional path to a prior evaluation JSON to compare against.",
+        )
+
+    def run(self, args: argparse.Namespace) -> int:
+        rubric_id_raw = getattr(args, "rubric_id", "")
+        rubric_id = str(rubric_id_raw).strip()
+        if not rubric_id:
+            raise CLIError("Rubric identifier cannot be empty.")
+
+        data_path = Path(args.data_path).expanduser()
+        config_path_value = getattr(args, "config_path", None)
+        output_path_value = getattr(args, "output_path", None)
+        baseline_path_value = getattr(args, "baseline_path", None)
+
+        number_value = getattr(args, "number", None)
+        number = int(number_value) if number_value is not None else 1
+
+        request = EvaluationSessionRequest(
+            rubric_id=rubric_id,
+            data_path=data_path,
+            number=number,
+            config_path=Path(config_path_value).expanduser() if config_path_value else None,
+            output_path=Path(output_path_value).expanduser() if output_path_value else None,
+            baseline_path=Path(baseline_path_value).expanduser() if baseline_path_value else None,
+        )
+
+        try:
+            result = self._session.execute(request)
+        except KeyboardInterrupt:
+            print("Evaluation cancelled by user.")
+            return 1
+        self._renderer.render(result.report, result.baseline)
+
+        if result.written_path is not None:
+            print(f"Wrote evaluation results to {result.written_path}")
+
+        return 0
+
+    @staticmethod
+    def _generate_output_identifier() -> str:
+        return str(int(time.time()))

osmosis_ai-0.2.3/osmosis_ai/cli_services/__init__.py

@@ -0,0 +1,67 @@
+from __future__ import annotations
+
+from .config import (
+    ParsedItem,
+    RubricConfig,
+    RubricConfigParser,
+    RubricSuite,
+    discover_rubric_config_path,
+    load_rubric_configs,
+    load_rubric_suite,
+    render_yaml_items,
+)
+from .dataset import (
+    ConversationMessage,
+    DatasetLoader,
+    DatasetRecord,
+    load_jsonl_records,
+    render_json_records,
+)
+from .engine import (
+    EvaluationRecordResult,
+    EvaluationReport,
+    EvaluationRun,
+    RubricEvaluationEngine,
+    RubricEvaluator,
+)
+from .errors import CLIError
+from .reporting import (
+    BaselineComparator,
+    BaselineStatistics,
+    ConsoleReportRenderer,
+    JsonReportFormatter,
+    JsonReportWriter,
+    TextReportFormatter,
+)
+from .session import EvaluationSession, EvaluationSessionRequest, EvaluationSessionResult
+
+__all__ = [
+    "BaselineComparator",
+    "BaselineStatistics",
+    "CLIError",
+    "ConsoleReportRenderer",
+    "ConversationMessage",
+    "DatasetLoader",
+    "DatasetRecord",
+    "EvaluationSession",
+    "EvaluationSessionRequest",
+    "EvaluationSessionResult",
+    "EvaluationRecordResult",
+    "EvaluationReport",
+    "EvaluationRun",
+    "JsonReportFormatter",
+    "JsonReportWriter",
+    "ParsedItem",
+    "RubricConfig",
+    "RubricConfigParser",
+    "RubricEvaluationEngine",
+    "RubricEvaluator",
+    "RubricSuite",
+    "TextReportFormatter",
+    "discover_rubric_config_path",
+    "load_jsonl_records",
+    "load_rubric_configs",
+    "load_rubric_suite",
+    "render_json_records",
+    "render_yaml_items",
+]