pytest-fakellm 0.1.0__tar.gz

pytest_fakellm-0.1.0/.gitignore

@@ -0,0 +1,9 @@
+.DS_Store
+__pycache__/
+*.py[cod]
+*.egg-info/
+build/
+dist/
+.pytest_cache/
+.venv/
+venv/

pytest_fakellm-0.1.0/PKG-INFO

@@ -0,0 +1,124 @@
+Metadata-Version: 2.4
+Name: pytest-fakellm
+Version: 0.1.0
+Summary: Pytest fixtures for the fakellm mock OpenAI/Anthropic server — spin up, reset, and assert with zero boilerplate.
+Project-URL: Homepage, https://github.com/1dg618/pytest-fakellm
+Project-URL: Repository, https://github.com/1dg618/pytest-fakellm
+Project-URL: Issues, https://github.com/1dg618/pytest-fakellm/issues
+Author-email: Douglas Gregor <1dg618@gmail.com>
+License: MIT
+Keywords: anthropic,fakellm,fixtures,llm,mock,openai,pytest,testing
+Classifier: Development Status :: 4 - Beta
+Classifier: Framework :: Pytest
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Testing :: Mocking
+Requires-Python: >=3.10
+Requires-Dist: fakellm>=0.3.1
+Requires-Dist: httpx>=0.27
+Requires-Dist: pytest>=7.0
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.34; extra == 'anthropic'
+Provides-Extra: dev
+Requires-Dist: anthropic>=0.34; extra == 'dev'
+Requires-Dist: openai>=1.40; extra == 'dev'
+Provides-Extra: openai
+Requires-Dist: openai>=1.40; extra == 'openai'
+Description-Content-Type: text/markdown
+
+# pytest-fakellm
+
+Pytest fixtures for [fakellm](https://github.com/dgregor/fakellm), the mock
+OpenAI/Anthropic server. Spin up a server, get a clean state per test, and
+assert on what your code sent — with zero boilerplate.
+
+```bash
+pip install pytest-fakellm
+```
+
+Once installed, the fixtures are available automatically — no imports, no
+`conftest.py` setup.
+
+## The point
+
+Without the plugin, using fakellm in a test means starting the server, wiring a
+client to its URL, resetting state, and tearing it all down yourself, in every
+test. With the plugin, that becomes:
+
+```python
+def test_agent_handles_search(fakellm):
+    fakellm.set_config_text("""
+    version: 1
+    rules:
+      - name: summarize
+        when: { messages_contain: "research" }
+        respond: { content: "Based on the search, I found what you were looking for." }
+    """)
+    result = run_my_agent(fakellm.openai_client(), prompt="Please research fakellm")
+    assert "found what you were looking for" in result
+    assert fakellm.request_count >= 1
+```
+
+The server starts once per session, state is reset before each test, and
+everything is torn down at the end. You never touch a port number or a
+subprocess.
+
+## Fixtures
+
+| Fixture | What you get |
+|---|---|
+| `fakellm` | A `FakellmServer` handle with fresh conversation state for the test. |
+| `fakellm_openai` | A ready `openai.OpenAI` client pointed at the (reset) server. |
+| `fakellm_anthropic` | A ready `anthropic.Anthropic` client pointed at the (reset) server. |
+
+### `FakellmServer` handle
+
+- `openai_client(**kwargs)` / `anthropic_client(**kwargs)` — clients pointed at the server.
+- `openai_base_url` / `anthropic_base_url` — raw URLs if you build your own client.
+- `set_config_text(yaml)` — write rules inline and reload.
+- `load_rules(path)` — load rules from a file and reload.
+- `reset()` — clear conversation state (done for you between tests).
+- `reload()` — re-read the config from disk.
+- `stats()` / `conversations()` — the admin JSON, for assertions.
+- `request_count` — convenience count of requests seen.
+
+## Configuration
+
+Set a starting config file via the command line:
+
+```bash
+pytest --fakellm-config=tests/fixtures/rules.yaml
+```
+
+or in `pyproject.toml` / `pytest.ini`:
+
+```toml
+[tool.pytest.ini_options]
+fakellm_config = "tests/fixtures/rules.yaml"
+```
+
+If you don't set one, a temporary empty config is created so `set_config_text`
+and `load_rules` work immediately.
+
+`--fakellm-startup-timeout` (default `10.0`) controls how long the fixture waits
+for the server to come up.
+
+## Client extras
+
+`openai_client()` and `anthropic_client()` require the respective SDKs. Install
+what you need:
+
+```bash
+pip install "pytest-fakellm[openai]"      # adds openai
+pip install "pytest-fakellm[anthropic]"   # adds anthropic
+```
+
+## License
+
+MIT

pytest_fakellm-0.1.0/README.md

@@ -0,0 +1,90 @@
+# pytest-fakellm
+
+Pytest fixtures for [fakellm](https://github.com/dgregor/fakellm), the mock
+OpenAI/Anthropic server. Spin up a server, get a clean state per test, and
+assert on what your code sent — with zero boilerplate.
+
+```bash
+pip install pytest-fakellm
+```
+
+Once installed, the fixtures are available automatically — no imports, no
+`conftest.py` setup.
+
+## The point
+
+Without the plugin, using fakellm in a test means starting the server, wiring a
+client to its URL, resetting state, and tearing it all down yourself, in every
+test. With the plugin, that becomes:
+
+```python
+def test_agent_handles_search(fakellm):
+    fakellm.set_config_text("""
+    version: 1
+    rules:
+      - name: summarize
+        when: { messages_contain: "research" }
+        respond: { content: "Based on the search, I found what you were looking for." }
+    """)
+    result = run_my_agent(fakellm.openai_client(), prompt="Please research fakellm")
+    assert "found what you were looking for" in result
+    assert fakellm.request_count >= 1
+```
+
+The server starts once per session, state is reset before each test, and
+everything is torn down at the end. You never touch a port number or a
+subprocess.
+
+## Fixtures
+
+| Fixture | What you get |
+|---|---|
+| `fakellm` | A `FakellmServer` handle with fresh conversation state for the test. |
+| `fakellm_openai` | A ready `openai.OpenAI` client pointed at the (reset) server. |
+| `fakellm_anthropic` | A ready `anthropic.Anthropic` client pointed at the (reset) server. |
+
+### `FakellmServer` handle
+
+- `openai_client(**kwargs)` / `anthropic_client(**kwargs)` — clients pointed at the server.
+- `openai_base_url` / `anthropic_base_url` — raw URLs if you build your own client.
+- `set_config_text(yaml)` — write rules inline and reload.
+- `load_rules(path)` — load rules from a file and reload.
+- `reset()` — clear conversation state (done for you between tests).
+- `reload()` — re-read the config from disk.
+- `stats()` / `conversations()` — the admin JSON, for assertions.
+- `request_count` — convenience count of requests seen.
+
+## Configuration
+
+Set a starting config file via the command line:
+
+```bash
+pytest --fakellm-config=tests/fixtures/rules.yaml
+```
+
+or in `pyproject.toml` / `pytest.ini`:
+
+```toml
+[tool.pytest.ini_options]
+fakellm_config = "tests/fixtures/rules.yaml"
+```
+
+If you don't set one, a temporary empty config is created so `set_config_text`
+and `load_rules` work immediately.
+
+`--fakellm-startup-timeout` (default `10.0`) controls how long the fixture waits
+for the server to come up.
+
+## Client extras
+
+`openai_client()` and `anthropic_client()` require the respective SDKs. Install
+what you need:
+
+```bash
+pip install "pytest-fakellm[openai]"      # adds openai
+pip install "pytest-fakellm[anthropic]"   # adds anthropic
+```
+
+## License
+
+MIT

pytest_fakellm-0.1.0/pyproject.toml

@@ -0,0 +1,56 @@
+[project]
+name = "pytest-fakellm"
+version = "0.1.0"
+description = "Pytest fixtures for the fakellm mock OpenAI/Anthropic server — spin up, reset, and assert with zero boilerplate."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { text = "MIT" }
+authors = [{ name = "Douglas Gregor", email = "1dg618@gmail.com" }]
+keywords = ["pytest", "fakellm", "llm", "testing", "mock", "openai", "anthropic", "fixtures"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Framework :: Pytest",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Software Development :: Testing",
+    "Topic :: Software Development :: Testing :: Mocking",
+]
+dependencies = [
+    "pytest>=7.0",
+    "httpx>=0.27",
+    "fakellm>=0.3.1",
+]
+
+[project.optional-dependencies]
+openai = ["openai>=1.40"]
+anthropic = ["anthropic>=0.34"]
+dev = [
+    "openai>=1.40",
+    "anthropic>=0.34",
+]
+
+[project.urls]
+Homepage = "https://github.com/1dg618/pytest-fakellm"
+Repository = "https://github.com/1dg618/pytest-fakellm"
+Issues = "https://github.com/1dg618/pytest-fakellm/issues"
+
+# This is the line that makes the whole thing work: pytest discovers installed
+# plugins via the "pytest11" entry-point group. Once this package is installed,
+# the fixtures in pytest_fakellm.plugin are available with no imports.
+[project.entry-points.pytest11]
+fakellm = "pytest_fakellm.plugin"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/pytest_fakellm"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

pytest_fakellm-0.1.0/src/pytest_fakellm/__init__.py

@@ -0,0 +1,8 @@
+"""pytest-fakellm: pytest fixtures for the fakellm mock LLM server."""
+
+from __future__ import annotations
+
+from .server import FakellmServer
+
+__all__ = ["FakellmServer"]
+__version__ = "0.1.0"

pytest_fakellm-0.1.0/src/pytest_fakellm/plugin.py

@@ -0,0 +1,130 @@
+"""pytest-fakellm — pytest fixtures for the fakellm mock LLM server.
+
+Installing this package makes the fixtures below available in any test, with no
+imports or registration required (pytest auto-discovers the plugin via the
+``pytest11`` entry point declared in pyproject.toml).
+
+Typical use::
+
+    def test_agent(fakellm):
+        fakellm.set_config_text(MY_RULES_YAML)
+        result = run_my_agent(fakellm.openai_client(), prompt="research fakellm")
+        assert "found what you were looking for" in result
+
+Fixtures
+--------
+fakellm
+    Session-scoped server, automatically reset before each test that uses it.
+fakellm_openai
+    Convenience: a ready ``openai.OpenAI`` client pointed at the server.
+fakellm_anthropic
+    Convenience: a ready ``anthropic.Anthropic`` client pointed at the server.
+
+Configuration
+-------------
+The server's starting config file can be set with the ``--fakellm-config``
+command-line option or the ``fakellm_config`` ini option. If neither is set, a
+temporary empty config is created so that ``set_config_text`` / ``load_rules``
+work out of the box.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any, Iterator
+
+import pytest
+
+from .server import FakellmServer
+
+__all__ = ["FakellmServer", "fakellm", "fakellm_openai", "fakellm_anthropic"]
+
+
+# -- options ---------------------------------------------------------------
+
+
+def pytest_addoption(parser: pytest.Parser) -> None:
+    group = parser.getgroup("fakellm")
+    group.addoption(
+        "--fakellm-config",
+        action="store",
+        default=None,
+        help="Path to a fakellm YAML config to start the server with.",
+    )
+    group.addoption(
+        "--fakellm-startup-timeout",
+        action="store",
+        type=float,
+        default=10.0,
+        help="Seconds to wait for the fakellm server to become ready.",
+    )
+    parser.addini(
+        "fakellm_config",
+        help="Path to a fakellm YAML config to start the server with.",
+        default=None,
+    )
+
+
+def _resolve_config(request: pytest.FixtureRequest) -> str | None:
+    cli = request.config.getoption("--fakellm-config")
+    if cli:
+        return cli
+    ini = request.config.getini("fakellm_config")
+    return ini or None
+
+
+# -- fixtures --------------------------------------------------------------
+
+
+@pytest.fixture(scope="session")
+def fakellm_server(
+    request: pytest.FixtureRequest,
+    tmp_path_factory: pytest.TempPathFactory,
+) -> Iterator[FakellmServer]:
+    """Session-scoped, *unreset* server handle.
+
+    Starts one fakellm process for the whole test session and tears it down at
+    the end. Most tests should use the ``fakellm`` fixture instead, which layers
+    automatic per-test state reset on top of this.
+    """
+    config = _resolve_config(request)
+    if config is None:
+        # Give the server a writable config file so set_config_text/load_rules
+        # have somewhere to write, even when the user didn't supply one.
+        config_path = tmp_path_factory.mktemp("fakellm") / "fakellm.yaml"
+        config_path.write_text("version: 1\nrules: []\n")
+        config = str(config_path)
+
+    timeout = request.config.getoption("--fakellm-startup-timeout")
+    server = FakellmServer(config=config, startup_timeout=timeout)
+    server.start()
+    try:
+        yield server
+    finally:
+        server.stop()
+
+
+@pytest.fixture
+def fakellm(fakellm_server: FakellmServer) -> Iterator[FakellmServer]:
+    """The main fixture: a server handle with fresh state for each test.
+
+    The underlying process is shared across the session (fast), but conversation
+    state is cleared before the test body runs, so tests are isolated from one
+    another regardless of order.
+    """
+    fakellm_server.reset()
+    yield fakellm_server
+    # No teardown reset needed: the next test resets before it runs. Leaving
+    # state intact after a failure can also aid debugging.
+
+
+@pytest.fixture
+def fakellm_openai(fakellm: FakellmServer) -> Any:
+    """A ready-to-use ``openai.OpenAI`` client pointed at the reset server."""
+    return fakellm.openai_client()
+
+
+@pytest.fixture
+def fakellm_anthropic(fakellm: FakellmServer) -> Any:
+    """A ready-to-use ``anthropic.Anthropic`` client pointed at the reset server."""
+    return fakellm.anthropic_client()

pytest_fakellm-0.1.0/src/pytest_fakellm/server.py

@@ -0,0 +1,256 @@
+"""Server lifecycle management and the user-facing control handle.
+
+This module knows how to launch a ``fakellm`` server as a subprocess, wait for
+it to become healthy, and talk to its documented admin endpoints
+(``/_fakellm/reset``, ``/_fakellm/reload``, ``/_fakellm/stats``). It deliberately
+treats fakellm as a black box driven through its public CLI and HTTP surface,
+so the plugin does not couple to fakellm's internals.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import shutil
+import socket
+import subprocess
+import sys
+import time
+from pathlib import Path
+from typing import Any
+
+import httpx
+
+
+def _free_port() -> int:
+    """Ask the OS for an unused TCP port, then release it for fakellm to claim.
+
+    There is a small race between releasing the port and fakellm binding it,
+    but in practice it is negligible for a test fixture and far simpler than
+    handing an open socket to a subprocess.
+    """
+    with contextlib.closing(socket.socket(socket.AF_INET, socket.SOCK_STREAM)) as s:
+        s.bind(("127.0.0.1", 0))
+        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+        return s.getsockname()[1]
+
+
+class FakellmServer:
+    """A running fakellm instance plus helpers for using it from a test.
+
+    Instances are created by the plugin's fixtures; you normally interact with
+    one through the ``fakellm`` fixture rather than constructing it yourself.
+
+    Attributes:
+        host: Host the server is bound to.
+        port: Port the server is bound to.
+        base_url: Root URL of the server, e.g. ``http://127.0.0.1:9999``.
+    """
+
+    def __init__(
+        self,
+        *,
+        config: str | Path | None = None,
+        host: str = "127.0.0.1",
+        port: int | None = None,
+        startup_timeout: float = 10.0,
+        executable: str | None = None,
+    ) -> None:
+        self.host = host
+        self.port = port or _free_port()
+        self._config = Path(config) if config is not None else None
+        self._startup_timeout = startup_timeout
+        # Launch via the module entry point by default so the plugin works even
+        # when the ``fakellm`` console script is not on PATH (e.g. some CI venvs).
+        self._executable = executable
+        self._proc: subprocess.Popen[bytes] | None = None
+        self._client = httpx.Client(base_url=self.base_url, timeout=10.0)
+
+    # -- URLs ---------------------------------------------------------------
+
+    @property
+    def base_url(self) -> str:
+        return f"http://{self.host}:{self.port}"
+
+    @property
+    def openai_base_url(self) -> str:
+        """Base URL to hand to an OpenAI client (note the ``/v1`` suffix)."""
+        return f"{self.base_url}/v1"
+
+    @property
+    def anthropic_base_url(self) -> str:
+        """Base URL to hand to an Anthropic client."""
+        return self.base_url
+
+    # -- lifecycle ----------------------------------------------------------
+
+    def _command(self) -> list[str]:
+        if self._executable:
+            cmd = [self._executable, "serve"]
+        else:
+            script = shutil.which("fakellm")
+            if script:
+                cmd = [script, "serve"]
+            else:
+                cmd = [sys.executable, "-m", "fakellm", "serve"]
+        cmd += ["--port", str(self.port)]
+        if self._config is not None:
+            cmd += ["--config", str(self._config)]
+        return cmd
+
+    def start(self) -> None:
+        """Launch the server subprocess and block until it is accepting requests."""
+        if self._proc is not None:
+            return
+        self._proc = subprocess.Popen(
+            self._command(),
+            stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT,
+        )
+        self._wait_until_ready()
+
+    def _wait_until_ready(self) -> None:
+        deadline = time.monotonic() + self._startup_timeout
+        last_err: Exception | None = None
+        while time.monotonic() < deadline:
+            # If the process died during startup, surface its output immediately
+            # rather than waiting out the full timeout.
+            if self._proc is not None and self._proc.poll() is not None:
+                output = b""
+                if self._proc.stdout is not None:
+                    output = self._proc.stdout.read()
+                raise RuntimeError(
+                    "fakellm server exited during startup "
+                    f"(code {self._proc.returncode}). Command: "
+                    f"{' '.join(self._command())}\n"
+                    f"Output:\n{output.decode(errors='replace')}"
+                )
+            try:
+                resp = self._client.get("/_fakellm/stats")
+                if resp.status_code < 500:
+                    return
+            except httpx.HTTPError as exc:  # not up yet
+                last_err = exc
+            time.sleep(0.05)
+        raise RuntimeError(
+            f"fakellm server did not become ready within {self._startup_timeout}s "
+            f"at {self.base_url}. Last error: {last_err!r}"
+        )
+
+    def stop(self) -> None:
+        """Terminate the server subprocess and close the admin client."""
+        self._client.close()
+        if self._proc is None:
+            return
+        self._proc.terminate()
+        try:
+            self._proc.wait(timeout=5.0)
+        except subprocess.TimeoutExpired:
+            self._proc.kill()
+            self._proc.wait(timeout=5.0)
+        self._proc = None
+
+    # -- admin operations ---------------------------------------------------
+
+    def reset(self) -> None:
+        """Clear all conversation state (POST ``/_fakellm/reset``).
+
+        Stats and request history are preserved, matching fakellm's documented
+        behavior. Called automatically between tests by the ``fakellm`` fixture.
+        """
+        self._client.post("/_fakellm/reset").raise_for_status()
+
+    def reload(self) -> None:
+        """Re-read the YAML config from disk (POST ``/_fakellm/reload``)."""
+        self._client.post("/_fakellm/reload").raise_for_status()
+
+    def load_rules(self, config: str | Path) -> None:
+        """Point the server at a new config file and reload it.
+
+        This rewrites the path the server was started with by writing the given
+        config into place is *not* done here; instead it relies on fakellm's
+        ``--config`` having been set to a path the test controls. For ad-hoc
+        rules use :meth:`set_config_text`.
+        """
+        path = Path(config)
+        if self._config is None:
+            raise RuntimeError(
+                "This server was not started with a config path, so load_rules "
+                "cannot point it at a file. Use the `fakellm_config` fixture to "
+                "set a config path, or call set_config_text()."
+            )
+        # Copy the provided file's contents into the active config path.
+        Path(self._config).write_text(path.read_text())
+        self.reload()
+
+    def set_config_text(self, yaml_text: str) -> None:
+        """Write raw YAML into the active config file and reload.
+
+        Requires the server to have been started with a config path (the
+        ``fakellm_config`` fixture does this for you).
+        """
+        if self._config is None:
+            raise RuntimeError(
+                "This server was not started with a config path. Use the "
+                "`fakellm_config` fixture (or pass config=...) so there is a "
+                "file to write to."
+            )
+        Path(self._config).write_text(yaml_text)
+        self.reload()
+
+    def stats(self) -> dict[str, Any]:
+        """Return the server's stats JSON (GET ``/_fakellm/stats``)."""
+        resp = self._client.get("/_fakellm/stats")
+        resp.raise_for_status()
+        return resp.json()
+
+    def conversations(self) -> dict[str, Any]:
+        """Return per-conversation turn/tool-result info (GET ``/_fakellm/conversations``)."""
+        resp = self._client.get("/_fakellm/conversations")
+        resp.raise_for_status()
+        return resp.json()
+
+    @property
+    def request_count(self) -> int:
+        """Convenience accessor: number of requests seen, from stats."""
+        data = self.stats()
+        # Be tolerant of the exact stats shape; fall back to len(recent).
+        for key in ("request_count", "requests", "total_requests"):
+            if isinstance(data.get(key), int):
+                return data[key]
+        recent = data.get("recent_requests") or data.get("recent") or []
+        return len(recent) if isinstance(recent, list) else 0
+
+    # -- client helpers -----------------------------------------------------
+
+    def openai_client(self, **kwargs: Any) -> Any:
+        """Return an ``openai.OpenAI`` client pointed at this server.
+
+        Requires the ``openai`` package to be installed. Extra kwargs are passed
+        through to the ``OpenAI(...)`` constructor.
+        """
+        try:
+            from openai import OpenAI
+        except ImportError as exc:  # pragma: no cover - depends on user env
+            raise RuntimeError(
+                "openai is not installed. Install it (e.g. `pip install openai`) "
+                "to use fakellm.openai_client()."
+            ) from exc
+        kwargs.setdefault("base_url", self.openai_base_url)
+        kwargs.setdefault("api_key", "not-used")
+        return OpenAI(**kwargs)
+
+    def anthropic_client(self, **kwargs: Any) -> Any:
+        """Return an ``anthropic.Anthropic`` client pointed at this server.
+
+        Requires the ``anthropic`` package to be installed.
+        """
+        try:
+            from anthropic import Anthropic
+        except ImportError as exc:  # pragma: no cover - depends on user env
+            raise RuntimeError(
+                "anthropic is not installed. Install it (e.g. `pip install "
+                "anthropic`) to use fakellm.anthropic_client()."
+            ) from exc
+        kwargs.setdefault("base_url", self.anthropic_base_url)
+        kwargs.setdefault("api_key", "not-used")
+        return Anthropic(**kwargs)

pytest_fakellm-0.1.0/tests/test_plugin_usage.py

@@ -0,0 +1,74 @@
+"""Tests that use the plugin exactly as an end user would.
+
+No imports from pytest_fakellm here on purpose — the `fakellm` fixture must be
+auto-discovered via the entry point, proving the plugin is wired correctly.
+We talk to the server with raw httpx (rather than the openai SDK) so the suite
+needs no extra SDK installs.
+"""
+import httpx
+
+
+RULES = """
+version: 1
+rules:
+  - name: summarize
+    when: { messages_contain: "research" }
+    respond: { content: "I found what you were looking for." }
+"""
+
+
+def _chat(server, text):
+    """Make an OpenAI-style chat call against the server using raw httpx."""
+    resp = httpx.post(
+        f"{server.openai_base_url}/chat/completions",
+        json={"model": "gpt-4", "messages": [{"role": "user", "content": text}]},
+        timeout=10.0,
+    )
+    resp.raise_for_status()
+    return resp.json()["choices"][0]["message"]["content"]
+
+
+def test_fixture_is_autodiscovered(fakellm):
+    # If this argument resolves at all, the entry point worked.
+    assert fakellm.base_url.startswith("http://127.0.0.1:")
+
+
+def test_inline_config_drives_response(fakellm):
+    fakellm.set_config_text(RULES)
+    content = _chat(fakellm, "please research fakellm")
+    assert content == "I found what you were looking for."
+
+
+def test_request_count_increments(fakellm):
+    start = fakellm.request_count
+    _chat(fakellm, "hello")
+    _chat(fakellm, "hello again")
+    assert fakellm.request_count == start + 2
+
+
+def test_state_is_reset_between_tests_part1(fakellm):
+    # Make some requests; the NEXT test should not see this conversation state.
+    _chat(fakellm, "first test traffic")
+    convs = fakellm.conversations()
+    # Stub tracks conversations dict; just assert the call works and is JSON.
+    assert isinstance(convs, dict)
+
+
+def test_state_is_reset_between_tests_part2(fakellm):
+    # Because `fakellm` resets before each test, conversation state is empty
+    # even though the previous test made requests.
+    convs = fakellm.conversations()
+    assert convs == {}
+
+
+def test_raw_base_urls_are_exposed(fakellm):
+    assert fakellm.openai_base_url.endswith("/v1")
+    assert fakellm.anthropic_base_url == fakellm.base_url
+
+
+def test_default_config_returns_something(fakellm):
+    # With no rules set this run, the stub returns its default content.
+    fakellm.set_config_text("version: 1\nrules: []\n")
+    content = _chat(fakellm, "anything")
+    assert isinstance(content, str) and content        # got some response
+    assert "mock response" in content                  # it's the fallback echo