llm-behave 0.1.0__tar.gz

llm_behave-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,28 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build tools
+        run: pip install build twine hatchling
+
+      - name: Build package
+        run: python -m build --no-isolation
+
+      - name: Upload to PyPI
+        run: twine upload dist/*
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}

llm_behave-0.1.0/.gitignore

@@ -0,0 +1,54 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+*.egg
+.eggs/
+
+# Virtual environment
+.venv/
+venv/
+env/
+
+# Testing
+.pytest_cache/
+htmlcov/
+coverage.xml
+.coverage
+*.cover
+
+# llm-assert baselines (user-specific, not committed)
+.llm_assert_baselines/
+
+# ML models cache
+.models_cache/
+
+# IDE
+.idea/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Reference docs (not part of the library)
+*.docx
+*.pdf
+rought.txt
+
+# Environment
+.env
+.env.local
+
+# Internal / private files (not for public GitHub)
+CLAUDE.md
+ideas.txt
+PROJECT_LOG.txt
+PLAIN_ENGLISH_GUIDE.txt
+launch/
+.claudeignore
+.vscode/

llm_behave-0.1.0/CHANGELOG.md

@@ -0,0 +1,30 @@
+# Changelog
+
+All notable changes to llm-assert will be documented here.
+
+## [0.1.0] — 2026-03-12
+
+First public release.
+
+### Added
+- `assert_behavior(text)` — fluent assertion interface for LLM outputs
+- `.mentions(concept)` — semantic similarity using sentence-level max comparison
+- `.not_mentions(concept)` — assert a topic is absent
+- `.tone(label)` — detect empathetic / professional / rude / helpful / formal / casual etc.
+- `.intent(description)` — match output intent against a natural language description
+- `.calls_tool(name)` — assert a specific tool was called (supports OpenAI + Anthropic formats)
+- `ConversationTest` — multi-turn conversation harness with full history tracking
+- `.recalls(concept)` — assert response references a concept from earlier in the conversation
+- `.contradicts_turn(n)` — NLI-based contradiction detection across turns
+- `.consistent_tone_across_turns()` — tone consistency check across all assistant turns
+- `DriftTest` — baseline save and compare for CI regression detection
+- `@DriftTest.baseline()` — decorator to save test output as baseline
+- `MockProvider` — built-in mock for testing without real LLM API calls
+- `OpenAIProvider` — adapter for OpenAI chat completions
+- `AnthropicProvider` — adapter for Anthropic messages API
+- `OllamaProvider` — adapter for Ollama local models
+- pytest fixtures: `assert_llm`, `mock_provider`, `conversation`
+- pytest markers: `behavioral`, `drift`
+- Offline ML engine: `all-MiniLM-L6-v2` (80MB, CPU, no internet needed)
+- Batch encoding for tone detection (~4x faster than sequential)
+- Lazy model loading — fast import, model loads on first assertion

llm_behave-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Swanand Potnis
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

llm_behave-0.1.0/PKG-INFO

@@ -0,0 +1,307 @@
+Metadata-Version: 2.4
+Name: llm-behave
+Version: 0.1.0
+Summary: Behavioral testing for LLM applications. pytest plugin with semantic assertions, multi-turn conversation testing, and drift detection. No LLM judge needed.
+Project-URL: Homepage, https://github.com/Swanand33/llm-behave
+Project-URL: Repository, https://github.com/Swanand33/llm-behave
+Project-URL: Issues, https://github.com/Swanand33/llm-behave/issues
+Author-email: Swanand Potnis <swanandpotnis500@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ai,anthropic,behavioral-testing,langchain,llm,openai,pytest,testing
+Classifier: Development Status :: 3 - Alpha
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Testing
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.24
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pytest>=7.0
+Provides-Extra: all
+Requires-Dist: anthropic>=0.20; extra == 'all'
+Requires-Dist: ollama>=0.1; extra == 'all'
+Requires-Dist: openai>=1.0; extra == 'all'
+Requires-Dist: sentence-transformers>=2.2; extra == 'all'
+Requires-Dist: torch>=2.0; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.20; extra == 'anthropic'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.0; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0; extra == 'dev'
+Requires-Dist: pytest-html>=3.0; extra == 'dev'
+Requires-Dist: ruff>=0.1; extra == 'dev'
+Provides-Extra: ollama
+Requires-Dist: ollama>=0.1; extra == 'ollama'
+Provides-Extra: openai
+Requires-Dist: openai>=1.0; extra == 'openai'
+Provides-Extra: semantic
+Requires-Dist: sentence-transformers>=2.2; extra == 'semantic'
+Requires-Dist: torch>=2.0; extra == 'semantic'
+Description-Content-Type: text/markdown
+
+# llm-behave
+
+**Behavioral testing for LLM applications. A pytest plugin.**
+
+No LLM judge. No API cost. Offline models. Works with any provider.
+
+```python
+def test_support_bot():
+    output = my_support_bot("I want a refund for order 1234")
+
+    assert_behavior(output) \
+        .mentions("refund policy") \
+        .tone("empathetic") \
+        .not_mentions("competitor")
+```
+
+---
+
+## Why llm-behave?
+
+Most LLM testing tools either:
+- Use another LLM to judge the output (expensive, slow, circular)
+- Only do exact string matching (misses semantic meaning)
+- Don't support multi-turn conversations at all
+
+**llm-behave** uses small offline transformer models (80MB, runs on CPU) to understand meaning — no API calls, no cost, no internet required during tests.
+
+---
+
+## Install
+
+```bash
+# Core (tool call assertions, structure tests — no ML deps)
+pip install llm-behave
+
+# Full (semantic assertions: mentions, tone, intent, drift)
+pip install llm-behave[semantic]
+```
+
+---
+
+## Features at a glance
+
+| Feature | What it does |
+|---|---|
+| `mentions()` | Semantic similarity — "money back" matches "refund" |
+| `not_mentions()` | Assert a topic is NOT brought up |
+| `tone()` | Detect empathetic / professional / rude / helpful etc. |
+| `intent()` | Does the response intend to help? refuse? apologize? |
+| `calls_tool()` | Assert which tool the LLM called |
+| `ConversationTest` | Multi-turn testing with memory, contradiction detection |
+| `DriftTest` | Save baseline behavior, detect regressions in CI |
+
+---
+
+## Usage
+
+### Basic assertions
+
+```python
+from llm_assert import assert_behavior
+
+output = my_llm("I want a refund")
+
+# Semantic match — not exact string
+assert_behavior(output).mentions("refund policy")
+
+# Tone detection
+assert_behavior(output).tone("empathetic")
+assert_behavior(output).tone("professional", threshold=0.6)
+
+# Intent
+assert_behavior(output).intent("offering to help the customer")
+
+# Negative assertions
+assert_behavior(output).not_mentions("competitor")
+
+# Fluent chaining
+assert_behavior(output) \
+    .mentions("refund") \
+    .tone("empathetic") \
+    .not_mentions("competitor")
+```
+
+### Tool call assertions
+
+```python
+text, tool_calls = my_llm.chat_with_tools(messages, tools=my_tools)
+
+assert_behavior(text, tool_calls) \
+    .calls_tool("lookup_order") \
+    .mentions("order")
+```
+
+### Multi-turn conversation testing
+
+```python
+from llm_assert import ConversationTest, MockProvider
+
+conv = ConversationTest(agent=my_agent)
+
+conv.say("Hi, my name is Swanand")
+conv.say("I placed order #5678 last week")
+response = conv.say("When will it arrive?")
+
+# Does it remember context from earlier turns?
+assert response.recalls("order")
+assert response.recalls("Swanand")
+
+# Is tone consistent across the whole conversation?
+assert response.consistent_tone_across_turns(threshold=0.6)
+```
+
+### Drift detection (for CI)
+
+Catch silent regressions when you update your model or prompts.
+
+```python
+from llm_assert import DriftTest
+
+# First run: save baseline
+@DriftTest.baseline(save_as="support_refund_flow")
+def get_baseline_output():
+    return my_llm("I need a refund")
+
+# Every CI run: compare against baseline
+result = DriftTest.compare("support_refund_flow", current_output)
+assert result.passed, f"Behavior drift detected: {result.details}"
+```
+
+### pytest fixtures (auto-registered)
+
+```python
+# These fixtures are available in any test file automatically
+
+def test_with_mock(mock_provider, assert_llm):
+    provider = mock_provider(responses=["I'll help with your refund right away."])
+    output = provider.chat([{"role": "user", "content": "refund please"}])
+    assert_llm(output).mentions("refund").tone("helpful")
+
+def test_conversation(conversation):
+    conv = conversation(responses=["Hello!", "Sure, I can help with that."])
+    conv.say("Hi")
+    response = conv.say("I need help")
+    assert "help" in response.text.lower()
+```
+
+---
+
+## Providers
+
+Built-in adapters for all major LLM providers:
+
+```python
+from llm_assert.providers.openai_adapter import OpenAIProvider
+from llm_assert.providers.anthropic_adapter import AnthropicProvider
+from llm_assert.providers.ollama_adapter import OllamaProvider
+from llm_assert import MockProvider  # for tests, no API calls
+
+# All providers have the same interface
+provider = OpenAIProvider(model="gpt-4o-mini")
+provider = AnthropicProvider(model="claude-haiku-4-5-20251001")
+provider = OllamaProvider(model="llama3")
+
+output = provider.chat([{"role": "user", "content": "Hello"}])
+text, tool_calls = provider.chat_with_tools(messages, tools=my_tools)
+```
+
+Bring your own provider by subclassing `LLMProvider`:
+
+```python
+from llm_assert.providers.base import LLMProvider
+
+class MyProvider(LLMProvider):
+    def chat(self, messages, **kwargs):
+        ...
+    def chat_with_tools(self, messages, tools, **kwargs):
+        ...
+```
+
+---
+
+## How it works
+
+llm-behave uses [`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) — an 80MB sentence-transformer model that runs fully offline on CPU.
+
+- **`mentions()` / `not_mentions()`** — splits text into sentences, computes max cosine similarity between any sentence and your concept
+- **`tone()`** — batch-encodes input text against example sentences for each tone, returns max similarity
+- **`intent()`** — semantic similarity between output and your intent description
+- **`contradicts_turn()`** — NLI (Natural Language Inference) model detects logical contradictions
+
+Models load **lazily** on first use and are cached for the rest of the test session. Import time stays fast.
+
+---
+
+## Performance
+
+Measured after model warmup (model loads once per test session):
+
+| Assertion | Time |
+|---|---|
+| `mentions()` | ~32ms |
+| `tone()` | ~40ms |
+| `intent()` | ~32ms |
+| 4-assertion chain | ~350ms |
+
+---
+
+## pytest markers
+
+```python
+import pytest
+
+@pytest.mark.behavioral
+def test_refund_flow():
+    ...
+
+@pytest.mark.drift
+def test_no_regression():
+    ...
+```
+
+Run only behavioral tests:
+```bash
+pytest -m behavioral
+pytest -m drift
+```
+
+---
+
+## Full install options
+
+```bash
+pip install llm-behave                          # core only
+pip install llm-behave[semantic]                # + sentence-transformers + torch
+pip install llm-behave[openai]                  # + openai SDK
+pip install llm-behave[anthropic]               # + anthropic SDK
+pip install llm-behave[ollama]                  # + ollama SDK
+pip install llm-behave[all]                     # everything
+```
+
+---
+
+## Requirements
+
+- Python 3.10+
+- pytest 7.0+
+- For semantic assertions: `pip install llm-behave[semantic]`
+
+---
+
+## License
+
+MIT — free to use in personal and commercial projects.
+
+---
+
+## Author
+
+Built by [Swanand Potnis](https://github.com/Swanand33) — Pune, India.

llm_behave-0.1.0/README.md

@@ -0,0 +1,260 @@
+# llm-behave
+
+**Behavioral testing for LLM applications. A pytest plugin.**
+
+No LLM judge. No API cost. Offline models. Works with any provider.
+
+```python
+def test_support_bot():
+    output = my_support_bot("I want a refund for order 1234")
+
+    assert_behavior(output) \
+        .mentions("refund policy") \
+        .tone("empathetic") \
+        .not_mentions("competitor")
+```
+
+---
+
+## Why llm-behave?
+
+Most LLM testing tools either:
+- Use another LLM to judge the output (expensive, slow, circular)
+- Only do exact string matching (misses semantic meaning)
+- Don't support multi-turn conversations at all
+
+**llm-behave** uses small offline transformer models (80MB, runs on CPU) to understand meaning — no API calls, no cost, no internet required during tests.
+
+---
+
+## Install
+
+```bash
+# Core (tool call assertions, structure tests — no ML deps)
+pip install llm-behave
+
+# Full (semantic assertions: mentions, tone, intent, drift)
+pip install llm-behave[semantic]
+```
+
+---
+
+## Features at a glance
+
+| Feature | What it does |
+|---|---|
+| `mentions()` | Semantic similarity — "money back" matches "refund" |
+| `not_mentions()` | Assert a topic is NOT brought up |
+| `tone()` | Detect empathetic / professional / rude / helpful etc. |
+| `intent()` | Does the response intend to help? refuse? apologize? |
+| `calls_tool()` | Assert which tool the LLM called |
+| `ConversationTest` | Multi-turn testing with memory, contradiction detection |
+| `DriftTest` | Save baseline behavior, detect regressions in CI |
+
+---
+
+## Usage
+
+### Basic assertions
+
+```python
+from llm_assert import assert_behavior
+
+output = my_llm("I want a refund")
+
+# Semantic match — not exact string
+assert_behavior(output).mentions("refund policy")
+
+# Tone detection
+assert_behavior(output).tone("empathetic")
+assert_behavior(output).tone("professional", threshold=0.6)
+
+# Intent
+assert_behavior(output).intent("offering to help the customer")
+
+# Negative assertions
+assert_behavior(output).not_mentions("competitor")
+
+# Fluent chaining
+assert_behavior(output) \
+    .mentions("refund") \
+    .tone("empathetic") \
+    .not_mentions("competitor")
+```
+
+### Tool call assertions
+
+```python
+text, tool_calls = my_llm.chat_with_tools(messages, tools=my_tools)
+
+assert_behavior(text, tool_calls) \
+    .calls_tool("lookup_order") \
+    .mentions("order")
+```
+
+### Multi-turn conversation testing
+
+```python
+from llm_assert import ConversationTest, MockProvider
+
+conv = ConversationTest(agent=my_agent)
+
+conv.say("Hi, my name is Swanand")
+conv.say("I placed order #5678 last week")
+response = conv.say("When will it arrive?")
+
+# Does it remember context from earlier turns?
+assert response.recalls("order")
+assert response.recalls("Swanand")
+
+# Is tone consistent across the whole conversation?
+assert response.consistent_tone_across_turns(threshold=0.6)
+```
+
+### Drift detection (for CI)
+
+Catch silent regressions when you update your model or prompts.
+
+```python
+from llm_assert import DriftTest
+
+# First run: save baseline
+@DriftTest.baseline(save_as="support_refund_flow")
+def get_baseline_output():
+    return my_llm("I need a refund")
+
+# Every CI run: compare against baseline
+result = DriftTest.compare("support_refund_flow", current_output)
+assert result.passed, f"Behavior drift detected: {result.details}"
+```
+
+### pytest fixtures (auto-registered)
+
+```python
+# These fixtures are available in any test file automatically
+
+def test_with_mock(mock_provider, assert_llm):
+    provider = mock_provider(responses=["I'll help with your refund right away."])
+    output = provider.chat([{"role": "user", "content": "refund please"}])
+    assert_llm(output).mentions("refund").tone("helpful")
+
+def test_conversation(conversation):
+    conv = conversation(responses=["Hello!", "Sure, I can help with that."])
+    conv.say("Hi")
+    response = conv.say("I need help")
+    assert "help" in response.text.lower()
+```
+
+---
+
+## Providers
+
+Built-in adapters for all major LLM providers:
+
+```python
+from llm_assert.providers.openai_adapter import OpenAIProvider
+from llm_assert.providers.anthropic_adapter import AnthropicProvider
+from llm_assert.providers.ollama_adapter import OllamaProvider
+from llm_assert import MockProvider  # for tests, no API calls
+
+# All providers have the same interface
+provider = OpenAIProvider(model="gpt-4o-mini")
+provider = AnthropicProvider(model="claude-haiku-4-5-20251001")
+provider = OllamaProvider(model="llama3")
+
+output = provider.chat([{"role": "user", "content": "Hello"}])
+text, tool_calls = provider.chat_with_tools(messages, tools=my_tools)
+```
+
+Bring your own provider by subclassing `LLMProvider`:
+
+```python
+from llm_assert.providers.base import LLMProvider
+
+class MyProvider(LLMProvider):
+    def chat(self, messages, **kwargs):
+        ...
+    def chat_with_tools(self, messages, tools, **kwargs):
+        ...
+```
+
+---
+
+## How it works
+
+llm-behave uses [`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) — an 80MB sentence-transformer model that runs fully offline on CPU.
+
+- **`mentions()` / `not_mentions()`** — splits text into sentences, computes max cosine similarity between any sentence and your concept
+- **`tone()`** — batch-encodes input text against example sentences for each tone, returns max similarity
+- **`intent()`** — semantic similarity between output and your intent description
+- **`contradicts_turn()`** — NLI (Natural Language Inference) model detects logical contradictions
+
+Models load **lazily** on first use and are cached for the rest of the test session. Import time stays fast.
+
+---
+
+## Performance
+
+Measured after model warmup (model loads once per test session):
+
+| Assertion | Time |
+|---|---|
+| `mentions()` | ~32ms |
+| `tone()` | ~40ms |
+| `intent()` | ~32ms |
+| 4-assertion chain | ~350ms |
+
+---
+
+## pytest markers
+
+```python
+import pytest
+
+@pytest.mark.behavioral
+def test_refund_flow():
+    ...
+
+@pytest.mark.drift
+def test_no_regression():
+    ...
+```
+
+Run only behavioral tests:
+```bash
+pytest -m behavioral
+pytest -m drift
+```
+
+---
+
+## Full install options
+
+```bash
+pip install llm-behave                          # core only
+pip install llm-behave[semantic]                # + sentence-transformers + torch
+pip install llm-behave[openai]                  # + openai SDK
+pip install llm-behave[anthropic]               # + anthropic SDK
+pip install llm-behave[ollama]                  # + ollama SDK
+pip install llm-behave[all]                     # everything
+```
+
+---
+
+## Requirements
+
+- Python 3.10+
+- pytest 7.0+
+- For semantic assertions: `pip install llm-behave[semantic]`
+
+---
+
+## License
+
+MIT — free to use in personal and commercial projects.
+
+---
+
+## Author
+
+Built by [Swanand Potnis](https://github.com/Swanand33) — Pune, India.