evalwise 0.1.0__py3-none-any.whl

evalwise-0.1.0.dist-info/METADATA

@@ -0,0 +1,255 @@
+Metadata-Version: 2.5
+Name: evalwise
+Version: 0.1.0
+Summary: Deterministic-first AI evaluation for text, image, audio, and video
+Project-URL: Homepage, https://github.com/shreyaspj20/evalwise
+Project-URL: Documentation, https://github.com/shreyaspj20/evalwise#readme
+Project-URL: Repository, https://github.com/shreyaspj20/evalwise
+Author-email: Shreyas Reddy <shreyaspj20@gmail.com>
+License-Expression: MIT
+Keywords: ai,evaluation,llm,machine-learning,ml,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: click>=8.0
+Requires-Dist: httpx>=0.25
+Requires-Dist: jsonschema>=4.0
+Requires-Dist: langdetect>=1.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: rich>=13.0
+Requires-Dist: textstat>=0.7
+Provides-Extra: all
+Requires-Dist: decord>=0.6; extra == 'all'
+Requires-Dist: librosa>=0.10; extra == 'all'
+Requires-Dist: open-clip-torch>=2.20; extra == 'all'
+Requires-Dist: openai-whisper>=20231117; extra == 'all'
+Requires-Dist: opencv-python>=4.8; extra == 'all'
+Requires-Dist: pillow>=10.0; extra == 'all'
+Requires-Dist: sentence-transformers>=2.0; extra == 'all'
+Requires-Dist: soundfile>=0.12; extra == 'all'
+Requires-Dist: torch>=2.0; extra == 'all'
+Requires-Dist: transformers>=4.35; extra == 'all'
+Requires-Dist: ultralytics>=8.0; extra == 'all'
+Provides-Extra: audio
+Requires-Dist: librosa>=0.10; extra == 'audio'
+Requires-Dist: openai-whisper>=20231117; extra == 'audio'
+Requires-Dist: soundfile>=0.12; extra == 'audio'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.5; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: ruff>=0.1; extra == 'dev'
+Provides-Extra: embeddings
+Requires-Dist: sentence-transformers>=2.0; extra == 'embeddings'
+Provides-Extra: image
+Requires-Dist: open-clip-torch>=2.20; extra == 'image'
+Requires-Dist: pillow>=10.0; extra == 'image'
+Requires-Dist: torch>=2.0; extra == 'image'
+Requires-Dist: transformers>=4.35; extra == 'image'
+Requires-Dist: ultralytics>=8.0; extra == 'image'
+Provides-Extra: nli
+Requires-Dist: torch>=2.0; extra == 'nli'
+Requires-Dist: transformers>=4.35; extra == 'nli'
+Provides-Extra: video
+Requires-Dist: decord>=0.6; extra == 'video'
+Requires-Dist: opencv-python>=4.8; extra == 'video'
+Description-Content-Type: text/markdown
+
+# EvalWise
+
+**Deterministic-first AI evaluation for text, image, audio, and video.**
+
+The eval SDK that doesn't default to "ask another AI if this is good."
+
+```bash
+pip install evalwise
+```
+
+## Why EvalWise?
+
+Most eval tools jump straight to LLM-as-judge. That's:
+- **Expensive** — every eval is another API call
+- **Non-deterministic** — same input, different scores
+- **Ungrounded** — "Score: 4/5" tells you nothing
+
+EvalWise flips the default: **deterministic checks first, LLM-judge only when you have to.**
+
+## Quick Start
+
+```python
+from evalwise import Suite, Assert
+
+suite = Suite("summarization")
+
+@suite.test
+def test_format(response: str):
+    Assert.bullet_count(response, exactly=3)
+    Assert.word_count(response, max=200)
+    Assert.json_valid(response)
+
+@suite.test
+def test_factuality(response: str, source: str):
+    Assert.entails(response, source=source)
+    Assert.no_contradiction(response, source=source)
+
+# Run against a dataset
+results = suite.run(dataset="./golden_set.json")
+results.assert_pass_rate(threshold=0.95)
+```
+
+## CLI
+
+```bash
+# Run eval suite
+evalwise run tests/test_summary.py --dataset golden.json
+
+# Run in CI (exit code 1 on failure)
+evalwise run tests/ --ci --threshold 0.95
+
+# Create sample eval file
+evalwise init
+```
+
+## Text Assertions
+
+| Assertion | What it checks |
+|-----------|----------------|
+| `contains(text, substring)` | Substring present |
+| `not_contains(text, substring)` | Substring absent |
+| `regex(text, pattern)` | Pattern match |
+| `json_valid(text)` | Parseable JSON |
+| `json_schema(text, schema)` | Matches JSON schema |
+| `word_count(text, min, max)` | Word count in range |
+| `bullet_count(text, exactly)` | Bullet point count |
+| `readability(text, min_score)` | Flesch Reading Ease |
+| `language_is(text, "en")` | Correct language |
+| `code_parses(text, "python")` | Valid syntax |
+| `code_runs(text)` | Executes without error |
+| `entails(text, source)` | Follows from source (NLI) |
+| `no_contradiction(text, source)` | No contradictions |
+| `embedding_similarity(text, ref)` | Semantic similarity |
+| `urls_valid(text)` | All URLs return 2xx |
+
+## Image Assertions
+
+Image evals cover prompt alignment, object detection, resolution/aspect ratio, NSFW safety, and similarity. Object detection uses **YOLO** via `ultralytics`.
+
+```python
+from evalwise.image import ImageAssert
+
+# Prompt alignment with CLIP
+ImageAssert.clip_score(image, "a cat on a couch", threshold=0.25)
+
+# Object detection
+ImageAssert.contains_object(image, "cat", confidence=0.5)
+ImageAssert.object_count(image, "person", exactly=2, confidence=0.5)
+
+# Metadata
+ImageAssert.resolution_is(image, width=1024, height=1024)
+ImageAssert.resolution_min(image, width=512, height=512)
+ImageAssert.aspect_ratio(image, ratio=1.0, tolerance=0.1)
+ImageAssert.format_is(image, "PNG")
+
+# Safety
+ImageAssert.nsfw_below(image, threshold=0.1)
+
+# Similarity to a reference image
+ImageAssert.image_similarity(image, reference, threshold=0.8)
+```
+
+### Image generation example
+
+`examples/test_image_generation.py` shows a complete eval suite. The dataset can include per-image thresholds and object lists:
+
+```json
+{
+  "image_path": "./examples/sample_cat.jpeg",
+  "prompt": "a person sitting on a couch with a dog and a cat",
+  "min_width": 200,
+  "min_height": 100,
+  "required_objects": ["dog"],
+  "confidence": 0.25
+}
+```
+
+## Audio Assertions
+
+Audio assertions that use Whisper (`transcription_contains`, `transcription_equals`, `language_is`) require **ffmpeg** to be installed on your system in addition to the `evalwise[audio]` Python dependencies.
+
+```python
+from evalwise.audio import AudioAssert
+
+AudioAssert.transcription_contains(audio, "hello world")
+AudioAssert.transcription_equals(audio, expected_text)
+AudioAssert.language_is(audio, "en")
+AudioAssert.duration_between(audio, min_sec=5, max_sec=30)
+AudioAssert.sample_rate_is(audio, hz=44100)
+AudioAssert.no_silence(audio, max_silence_sec=1.0)
+```
+
+## Installation
+
+```bash
+# Core (text assertions)
+pip install evalwise
+
+# With image support
+pip install evalwise[image]
+
+# With audio support (also requires ffmpeg system binary)
+pip install evalwise[audio]
+
+# On macOS: brew install ffmpeg
+# On Ubuntu: sudo apt install ffmpeg
+# On Windows: winget install Gyan.FFmpeg
+
+# Everything
+pip install evalwise[all]
+```
+
+### What each extra gives you
+
+| Extra | Capabilities enabled | Example tests |
+|---|---|---|
+| *(none)* | 22 of 25 text assertions | `test_basic.py` |
+| `[nli]` | `Assert.entails`, `Assert.no_contradiction` | `test_summarization.py` |
+| `[embeddings]` | `Assert.embedding_similarity` | — |
+| `[image]` | CLIP, YOLO object detection, NSFW, resolution, similarity | `test_image_generation.py` |
+| `[audio]` | Whisper transcription, language, duration, sample rate | `test_audio.py` |
+| `[video]` | Video assertions (extra only, no example yet) | — |
+| `[all]` | All of the above | — |
+
+### System dependencies
+
+Some examples also require system binaries:
+
+| Capability | System binary | Install |
+|---|---|---|
+| Audio transcription | `ffmpeg` | `brew install ffmpeg` / `apt install ffmpeg` |
+
+## Philosophy
+
+1. **Deterministic by default** — Same input, same result. Always.
+2. **Cheap first** — Check format, length, syntax before calling models.
+3. **Grounded scores** — Know exactly why something failed.
+4. **LLM-judge as last resort** — Only for truly subjective criteria.
+
+## Comparison
+
+| | EvalWise | Promptfoo | Braintrust | LangSmith |
+|---|--------|-----------|------------|-----------|
+| Deterministic-first | ✓ | Partial | ✗ | ✗ |
+| Image/Audio evals | ✓ | ✗ | ✗ | ✗ |
+| Python-native | ✓ | YAML | SDK | SDK |
+| No account required | ✓ | ✓ | ✗ | ✗ |
+| OSS | ✓ | ✓ | Partial | ✗ |
+
+## License
+
+MIT

evalwise-0.1.0.dist-info/RECORD

@@ -0,0 +1,18 @@
+evalwise/__init__.py,sha256=se25ej6Q09D6IHa6H3oFgcFa2CLPj8fW2_wFALt7_JE,521
+evalwise/__main__.py,sha256=O-ufHkaXo66k4JJuvtq-LLohfHTegw97iRZp5WIMTII,115
+evalwise/cli.py,sha256=maRRYw68N0r1jaF45bvbX9hioSPHsQUekyxQrpGPGxA,7663
+evalwise/audio/__init__.py,sha256=C1i2dKZKbv6isTDlE9n7dgBnSidEcTSsXVsvyuBQ54o,151
+evalwise/audio/assertions.py,sha256=Be0pgt7TIZsnRWOBdvfBwGOXDd_qGqqgcKo28iAUCec,11254
+evalwise/core/__init__.py,sha256=vREc5gtO5fzZ7hZN77GPE8EXmauteBoW5A6AzGzJiF4,268
+evalwise/core/context.py,sha256=x_rGN3Mq0gMMUmY5p0TppqOUaPonQ8wfSxFlhUDy4Bs,1285
+evalwise/core/dataset.py,sha256=O8G7xq7rxFYFFF8rUZcjIiX5TxJzdT1xiAFYph9oBsQ,3066
+evalwise/core/result.py,sha256=9Zap5uwgSx4ZEpo7JWIlmMisECfJz_RnBiFmXWPBki4,4142
+evalwise/core/suite.py,sha256=ZSch2WvjnsUI_Pkp_G6Bs94lVolIeaiHr8IJxwBrllc,7407
+evalwise/image/__init__.py,sha256=4tyIocTBpdTAzzowzAY85rSVymCKHwtLRXVjY0chm_k,152
+evalwise/image/assertions.py,sha256=M36067ue6qWpVrCh6XXBmpSsMN5va2wQgM3PaV5pBSE,13213
+evalwise/text/__init__.py,sha256=uG65z9-FE7CPaSBWaXT1Jgf8hRApS4Fp3ESa5N283kI,140
+evalwise/text/assertions.py,sha256=8NrF0PHLZR7LDxPheBBUlF24Wr89HebR_4NCdkFK7vo,29208
+evalwise-0.1.0.dist-info/METADATA,sha256=_xmWJqRJOqxSCkSyCiwyGFVn6aliSg9et1xvzW9oasQ,8423
+evalwise-0.1.0.dist-info/WHEEL,sha256=zOwg4jB6zX2kU910N-cMawjivD6tO8NEWvE12je1bVk,87
+evalwise-0.1.0.dist-info/entry_points.txt,sha256=h4a9iwz0QrAkN0jCjU4LW84KtesnIGwe0TgT3ETNGWU,47
+evalwise-0.1.0.dist-info/RECORD,,

evalwise-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.0
+Root-Is-Purelib: true
+Tag: py3-none-any

evalwise-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+evalwise = evalwise.cli:main