frugon 0.1.0__tar.gz

frugon-0.1.0/.coveragerc-strict

@@ -0,0 +1,9 @@
+[run]
+source = src/frugon
+
+[report]
+include =
+    src/frugon/cost.py
+    src/frugon/pricing.py
+    src/frugon/routing.py
+fail_under = 90

frugon-0.1.0/.gitattributes

@@ -0,0 +1,19 @@
+# Line-ending normalization — the LF invariant.
+#
+# Frugon writes capture.jsonl and the Markdown/HTML reports with an explicit
+# newline="\n" so artifacts are byte-identical on Windows, macOS, and Linux.
+# This file extends that guarantee to the repository itself: every text file is
+# stored LF-only and checked out LF-only, so a contributor on a misconfigured
+# clone (core.autocrlf=true) cannot reintroduce CRLF drift into tracked sources,
+# fixtures, or golden test data.
+* text=auto eol=lf
+
+# Binary assets — never normalize or diff as text.
+*.png  binary
+*.jpg  binary
+*.jpeg binary
+*.gif  binary
+*.ico  binary
+*.gz   binary
+*.woff  binary
+*.woff2 binary

frugon-0.1.0/.github/RELEASE_NOTES_v0.1.0.md

@@ -0,0 +1,77 @@
+# frugon v0.1.0
+
+> **Your data never leaves your machine. Your keys go straight to your own providers. Nothing reaches us.**
+
+Free, local, open-source LLM cost analyzer. Point it at your real call logs and see — on your machine — how much you'd save by switching or routing models.
+
+## Install
+
+```bash
+# One-shot (no install)
+uvx frugon analyze ./logs.jsonl
+
+# Permanent install
+pipx install frugon
+frugon analyze --demo            # bundled sample log, see it work in 5 seconds
+frugon analyze ./logs.jsonl      # your real logs
+```
+
+Cross-platform: macOS, Linux, Windows · Python 3.10 / 3.11 / 3.12 / 3.13.
+
+## What's in this release
+
+**`frugon analyze`** — read OpenAI-compatible JSONL logs and produce a cost analysis. Counts tokens with [tokencost], prices with [LiteLLM's registry], picks a cheaper-than-baseline candidate model, and tells you the dollar saving. Cross-platform, fully local, no LLM calls, no network. Honest savings anchored to [RouteLLM/LMSYS] quality bands — we never inflate the number.
+
+**`frugon capture`** — passive OpenAI-compatible logger. Point your app's base URL at `http://127.0.0.1:8787` for a day; it records every call to a local JSONL file in the canonical shape and forwards the request unchanged to the real upstream. No data goes anywhere but your local file and your existing upstream.
+
+**`frugon pricing update`** — refresh the bundled pricing table from the [LiteLLM model_prices_and_context_window.json] registry. Atomic write, JSON shape validation, weekly GitHub Actions sync.
+
+**`frugon models [QUERY]`** — list the model names frugon can price (the exact names `--candidates` accepts), optionally filtered by a case-insensitive substring. Pure local read — no network.
+
+**`frugon quality update`** — refresh the bundled quality-tier table from the LMArena leaderboard so the Strong / Capable tier labels stay current. Tiers are self-recalibrating percentile bands: Elite (top 10%), Strong (10–30%), Capable (30–60%), Efficient (bottom 40%).
+
+**`frugon analyze --measure`** *(optional `[measure]` extra)* — sample real prompts through candidate models using **your own** API keys. Calls go straight to your providers (OpenAI / Anthropic / etc.) — never to us. Two tiers: side-by-side diffs (human judge) or LLM-as-judge win/loss/tie tallies.
+
+**`frugon analyze --report file.html|file.md`** — shareable single-page report. Self-contained HTML with inline CSS (deep indigo + cyan + silver), or clean Markdown. The viral surface someone shows their boss.
+
+## Realistic savings
+
+Anchored to [RouteLLM] / [LMSYS] research bands:
+
+| Traffic mix | Typical saving |
+|---|---|
+| General mixed traffic | 30–50% |
+| Easy / repetitive (MT-Bench) | up to ~85% |
+| Hard tasks (MMLU) | ~30% |
+
+**Your actual number comes from your logs.** Frugon shows what the math says for your data.
+
+## Privacy guarantees (tested as code, not promised in prose)
+
+- **Cost analysis is fully local.** No LLM, no network, no telemetry.
+- **`capture` never sends data anywhere but your configured upstream.** Asserted at the socket layer in tests — any future regression that introduces a side-channel HTTP client breaks CI.
+- **`--measure` calls only the user's own providers with the user's own keys.** Keys are never logged, never persisted, never sent anywhere but the provider. Asserted by a defense-in-depth fixture patching `socket.socket` / `socket.create_connection` / `socket.getaddrinfo`.
+- **The CLI collects nothing.** Open source — anyone can verify.
+
+## Quality
+
+- 1,960 tests, 95% overall coverage, 97% on the cost-math triad (`cost.py` / `pricing.py` / `routing.py`).
+- CI green on 3 OS × 4 Python (ubuntu / macos / windows × 3.10 / 3.11 / 3.12 / 3.13).
+- Every change is code-reviewed before merge; cost-math changes get an extra dedicated review pass.
+- ruff + mypy clean.
+
+## Keep the savings
+
+This release is the diagnosis. Want it to keep routing automatically and hold the savings? → **https://frugon.rodiun.io**
+
+---
+
+Built by [Rodiun]. MIT licensed.
+
+[tokencost]: https://github.com/AgentOps-AI/tokencost
+[LiteLLM's registry]: https://github.com/BerriAI/litellm
+[LiteLLM model_prices_and_context_window.json]: https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json
+[RouteLLM/LMSYS]: https://github.com/lm-sys/RouteLLM
+[RouteLLM]: https://github.com/lm-sys/RouteLLM
+[LMSYS]: https://lmsys.org/
+[Rodiun]: https://rodiun.io

frugon-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,42 @@
+name: CI
+
+on:
+  push:
+    branches: ["main"]
+  pull_request:
+    branches: ["main"]
+
+jobs:
+  test:
+    name: "Python ${{ matrix.python-version }} / ${{ matrix.os }}"
+    runs-on: "${{ matrix.os }}"
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python-version: ["3.10", "3.11", "3.12", "3.13"]
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2: bump by verifying refs/tags/v5.x.y before changing.
+        with:
+          enable-cache: true
+          python-version: "${{ matrix.python-version }}"
+
+      - name: Install project with dev dependencies
+        run: uv sync --extra dev --extra measure --frozen
+
+      - name: Lint — ruff check
+        run: uv run ruff check .
+
+      - name: Type-check — mypy
+        run: uv run mypy src
+
+      - name: Test — pytest
+        run: uv run pytest
+
+      - name: Test — strict cost coverage
+        run: uv run pytest --cov-config=.coveragerc-strict --cov-fail-under=90

frugon-0.1.0/.github/workflows/pricing-sync.yml

@@ -0,0 +1,70 @@
+name: Pricing sync
+
+on:
+  schedule:
+    # Every Monday at 06:00 UTC
+    - cron: "0 6 * * 1"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  sync:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
+
+      - name: Install project
+        run: uv sync
+
+      - name: Update pricing table
+        run: |
+          uv run python - <<'EOF'
+          from frugon.pricing import fetch_and_update_pricing, _LITELLM_REGISTRY_URL
+          from datetime import date
+          from pathlib import Path
+          result = fetch_and_update_pricing(
+              _LITELLM_REGISTRY_URL,
+              Path("src/frugon/data/pricing.json"),
+              date.today().isoformat(),
+          )
+          print(f"Synced {result['models_synced']} models")
+          EOF
+
+      - name: Check for changes
+        id: diff
+        run: |
+          if git diff --quiet src/frugon/data/pricing.json; then
+            echo "changed=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Open PR if pricing.json changed
+        if: steps.diff.outputs.changed == 'true'
+        uses: peter-evans/create-pull-request@84ae59a2cdc2258d6fa0732dd66352dddae2a412 # v7.0.9
+        with:
+          commit-message: "chore(pricing): sync LiteLLM pricing table"
+          branch: chore/pricing-sync
+          delete-branch: true
+          title: "chore(pricing): weekly pricing table sync"
+          body: |
+            Automated weekly sync of `src/frugon/data/pricing.json` from the
+            [LiteLLM registry](https://github.com/BerriAI/litellm/blob/main/model_prices_and_context_window.json).
+
+            Review the diff to confirm no unexpected price changes before merging.
+          labels: |
+            pricing
+            automated

frugon-0.1.0/.github/workflows/quality-sync.yml

@@ -0,0 +1,226 @@
+name: Quality sync
+
+on:
+  schedule:
+    # Every Monday at 07:00 UTC (offset from pricing-sync at 06:00)
+    - cron: "0 7 * * 1"
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  sync:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
+
+      - name: Install project
+        run: uv sync
+
+      - name: Capture old quality.json (pre-fetch baseline)
+        id: old_seed
+        run: |
+          # Export the committed version of the seed before the fetch overwrites it.
+          # If git show fails (seed never existed), OLD_QUALITY is left empty and
+          # classify_quality_update receives old=None (first-run path → MINOR).
+          if git show HEAD:src/frugon/data/quality.json > /tmp/quality_old.json 2>/dev/null; then
+            echo "has_old=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "has_old=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Fetch new quality tiers
+        id: fetch
+        run: |
+          uv run python - <<'EOF'
+          from frugon.quality import fetch_and_update_quality, _HF_BASE_URL
+          from datetime import date
+          from pathlib import Path
+          result = fetch_and_update_quality(
+              _HF_BASE_URL,
+              Path("src/frugon/data/quality.json"),
+              date.today().isoformat(),
+          )
+          print(f"Synced {result['models_synced']} models")
+          EOF
+
+      - name: Check for changes
+        id: diff
+        run: |
+          if git diff --quiet src/frugon/data/quality.json; then
+            echo "changed=false" >> "$GITHUB_OUTPUT"
+          else
+            echo "changed=true" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Validate and classify update
+        id: classify
+        if: steps.diff.outputs.changed == 'true'
+        run: |
+          uv run python - <<'EOF'
+          import json
+          import os
+          import sys
+          from pathlib import Path
+
+          from frugon.quality import (
+              VERDICT_INVALID,
+              VERDICT_MAJOR,
+              VERDICT_MINOR,
+              classify_quality_update,
+          )
+
+          # Load the newly fetched seed.
+          new_path = Path("src/frugon/data/quality.json")
+          with new_path.open(encoding="utf-8") as fh:
+              new_data: dict[str, object] = json.load(fh)
+
+          # Load the old seed (may be absent on first run).
+          old_data: dict[str, object] | None = None
+          old_path = Path("/tmp/quality_old.json")
+          if old_path.exists():
+              try:
+                  with old_path.open(encoding="utf-8") as fh:
+                      old_data = json.load(fh)
+              except json.JSONDecodeError:
+                  old_data = None  # treat corrupt baseline as missing
+
+          verdict, reason = classify_quality_update(new_data, old_data)
+
+          # Emit outputs for downstream steps.
+          # Use a randomized heredoc delimiter so that fetched content in
+          # *reason* cannot escape through the delimiter boundary.
+          import secrets
+          github_output = os.environ.get("GITHUB_OUTPUT", "")
+          if github_output:
+              with open(github_output, "a", encoding="utf-8") as fh:
+                  fh.write(f"verdict={verdict}\n")
+                  # Reason may contain newlines — use a multiline delimiter.
+                  # The delimiter is randomized so fetched model names in the
+                  # reason string cannot terminate it prematurely.
+                  # Invariant: classify_quality_update reasons must never embed
+                  # raw fetched model names (enforced in quality.py).
+                  delimiter = f"EOF_REASON_{secrets.token_hex(8)}"
+                  fh.write(f"reason<<{delimiter}\n{reason}\n{delimiter}\n")
+
+          # Emit to step summary for visibility in the Actions UI.
+          step_summary = os.environ.get("GITHUB_STEP_SUMMARY", "")
+          if step_summary:
+              with open(step_summary, "a", encoding="utf-8") as fh:
+                  emoji = {"INVALID": "🔴", "MAJOR": "🟡", "MINOR": "🟢"}.get(verdict, "⚪")
+                  fh.write(f"## Quality sync verdict: {emoji} {verdict}\n\n")
+                  fh.write(f"**Reason:** {reason}\n")
+
+          print(f"Verdict: {verdict}")
+          print(f"Reason:  {reason}")
+
+          # INVALID — discard the new seed and fail so CI turns red.
+          if verdict == VERDICT_INVALID:
+              print("ERROR: Fetch produced invalid data — discarding and failing job.", file=sys.stderr)
+              sys.exit(1)
+          EOF
+
+      - name: Discard seed on INVALID verdict
+        if: steps.diff.outputs.changed == 'true' && failure() && steps.classify.outcome == 'failure'
+        run: |
+          git checkout -- src/frugon/data/quality.json
+          echo "Seed discarded — invalid fetch data was NOT committed." >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Open PR — MINOR (CI-gated auto-merge enabled)
+        if: steps.diff.outputs.changed == 'true' && steps.classify.outputs.verdict == 'MINOR'
+        id: pr_minor
+        uses: peter-evans/create-pull-request@84ae59a2cdc2258d6fa0732dd66352dddae2a412 # v7.0.9
+        with:
+          commit-message: "chore(quality): sync LMArena quality tiers"
+          branch: chore/quality-sync
+          delete-branch: true
+          title: "chore(quality): weekly quality tier sync [auto-merge]"
+          body: |
+            Automated weekly sync of `src/frugon/data/quality.json` from the
+            [LMArena leaderboard dataset](https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset)
+            (CC-BY-4.0).
+
+            **Verdict: MINOR** — change classified as within expected weekly drift bounds.
+            Auto-merge is enabled; this PR will merge automatically after the CI matrix passes.
+
+            **Reason:** ${{ steps.classify.outputs.reason }}
+          labels: |
+            quality
+            automated
+
+      - name: In-job validation gate (MINOR path)
+        # Run the full quality-data safety gate in-process before enabling
+        # auto-merge.  This is belt-and-braces: gh pr merge --auto only waits
+        # for branch-protection-required checks, and PRs created by the
+        # default GITHUB_TOKEN do not trigger ci.yml on the same repo.
+        # Running linting + type-checking + the quality/classify tests here
+        # gives a synchronous correctness gate before the merge is enabled.
+        #
+        # Operational prerequisites: for auto-merge to be truly CI-gated,
+        # the repository's `main` branch protection must also require the
+        # `CI` matrix status checks.  That is server-side configuration, not
+        # in-repo config — see docs/quality-tiering.md §Operational prerequisites.
+        if: steps.diff.outputs.changed == 'true' && steps.classify.outputs.verdict == 'MINOR' && steps.pr_minor.outputs.pull-request-number != ''
+        run: |
+          uv run ruff check .
+          uv run mypy src
+          uv run pytest tests/test_quality.py tests/test_model_id_drift.py -q
+
+      - name: Enable auto-merge on MINOR PR
+        if: steps.diff.outputs.changed == 'true' && steps.classify.outputs.verdict == 'MINOR' && steps.pr_minor.outputs.pull-request-number != ''
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          gh pr merge "${{ steps.pr_minor.outputs.pull-request-number }}" \
+            --auto \
+            --squash \
+            --repo "${{ github.repository }}"
+
+      - name: Open PR — MAJOR (human review required, auto-merge NOT enabled)
+        if: steps.diff.outputs.changed == 'true' && steps.classify.outputs.verdict == 'MAJOR'
+        uses: peter-evans/create-pull-request@84ae59a2cdc2258d6fa0732dd66352dddae2a412 # v7.0.9
+        with:
+          commit-message: "chore(quality): sync LMArena quality tiers"
+          branch: chore/quality-sync
+          delete-branch: true
+          title: "chore(quality): weekly quality tier sync [needs-review]"
+          body: |
+            Automated weekly sync of `src/frugon/data/quality.json` from the
+            [LMArena leaderboard dataset](https://huggingface.co/datasets/lmarena-ai/leaderboard-dataset)
+            (CC-BY-4.0).
+
+            **Verdict: MAJOR** — the change exceeds expected weekly drift thresholds.
+            **Auto-merge is NOT enabled.** A human must review and merge this PR.
+
+            This may indicate:
+            - A leaderboard rebaseline (scoring mechanism changed)
+            - A significant model count shift (new cohort added / old cohort retired)
+            - A distribution-wide tier rescore
+
+            **Reason:** ${{ steps.classify.outputs.reason }}
+
+            Review the diff carefully before merging. If the change looks correct,
+            merge manually. If it looks wrong, close the PR and investigate.
+          labels: |
+            quality
+            automated
+            needs-review
+
+      - name: No changes — skip PR
+        if: steps.diff.outputs.changed == 'false'
+        run: |
+          echo "## Quality sync: no changes" >> "$GITHUB_STEP_SUMMARY"
+          echo "quality.json is already up to date — no PR opened." >> "$GITHUB_STEP_SUMMARY"
+          echo "quality.json unchanged — no PR opened."

frugon-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,112 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - "v*.*.*"
+
+# Workflow-level write access for the release job to create GitHub Releases.
+# The publish job overrides with id-token: write only (Trusted Publishing).
+permissions:
+  contents: write
+
+jobs:
+  test:
+    name: Test before release
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
+
+      - name: Install project with dev dependencies
+        run: uv sync --extra dev --extra measure --frozen
+
+      - name: Lint — ruff check
+        run: uv run ruff check .
+
+      - name: Type-check — mypy
+        run: uv run mypy src
+
+      - name: Test — pytest
+        run: uv run pytest
+
+      - name: Test — strict cost coverage
+        run: uv run pytest --cov-config=.coveragerc-strict --cov-fail-under=90
+
+  build:
+    name: Build sdist and wheel
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
+        with:
+          python-version: "3.12"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@d4b2f3b6ecc6e67c4457f6d3e41ec42d3d0fcb86 # v5.4.2
+
+      # Align the package version with the pushed git tag so the published
+      # wheel/sdist matches the release. Strips a leading "v" from the tag
+      # (v1.2.3 -> 1.2.3) and rewrites the version line in pyproject.toml.
+      - name: Set version from tag
+        run: |
+          VERSION="${GITHUB_REF_NAME#v}"
+          echo "Setting pyproject version to ${VERSION} (from tag ${GITHUB_REF_NAME})"
+          sed -i -E "s/^version = \".*\"/version = \"${VERSION}\"/" pyproject.toml
+          grep -E '^version = ' pyproject.toml
+
+      - name: Build
+        run: uv build
+
+      - name: Upload dist artifacts
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    name: Publish to PyPI
+    needs: [test, build]
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write  # required for Trusted Publishing — no API token used
+
+    steps:
+      - name: Download dist artifacts
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
+        with:
+          name: dist
+          path: dist/
+
+      - name: Publish
+        uses: pypa/gh-action-pypi-publish@76f52bc884231f62b9a034ebfe128415bbaabdfc # v1.12.4
+
+  release-notes:
+    name: Create GitHub Release
+    needs: [build, publish]
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Download dist artifacts
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0
+        with:
+          name: dist
+          path: dist/
+
+      - name: Create GitHub Release
+        uses: softprops/action-gh-release@3bb12739c298aeb8a4eeaf626c5b8d85266b0e65 # v2.6.2
+        with:
+          files: dist/*
+          generate_release_notes: true

frugon-0.1.0/.gitignore

@@ -0,0 +1,45 @@
+# Internal files that must never enter this public repo.
+/CLAUDE.md
+/docs/
+
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+*.egg
+
+# Environments
+.venv/
+venv/
+.env
+.python-version
+
+# Test / coverage
+.pytest_cache/
+.coverage
+coverage.xml
+htmlcov/
+.mypy_cache/
+.ruff_cache/
+.tox/
+
+# Captured logs / user data — never commit a user's traffic
+*.capture.jsonl
+captures/
+/report.html
+/report.md
+
+# Local agent config — never ship
+.claude/
+
+# OS / editor
+.DS_Store
+Thumbs.db
+.idea/
+.vscode/
+.demo.env
+DOGFOOD_FINDINGS.md
+DEMO_SCRIPT.md