llmsim 0.7.0__tar.gz

llmsim-0.7.0/.claude/commands/ship.md

@@ -0,0 +1,103 @@
+# Ship
+
+Run the full ship flow: verify quality, ensure test coverage, update artifacts, smoke test, then push, create PR, and merge when CI is green.
+
+This command implements the complete "Shipping" definition and Pre-PR Checklist from AGENTS.md. When the user says "ship" or "fix and ship", execute ALL phases below — not just the push/merge steps.
+
+## Arguments
+
+- `$ARGUMENTS` - Optional: description of what is being shipped (used for PR title/body context and to scope the quality checks)
+
+## Instructions
+
+### Phase 1: Pre-flight
+
+1. Confirm we're NOT on `main` or `master`
+2. Confirm there are no uncommitted changes (`git diff --quiet && git diff --cached --quiet`)
+3. If uncommitted changes exist, stop and tell the user
+
+### Phase 2: Test Coverage
+
+Review the changes on this branch (use `git diff origin/main...HEAD` and `git log origin/main..HEAD`) and ensure comprehensive test coverage:
+
+1. **Identify all changed code paths** — every new/modified function, endpoint, handler
+2. **Verify existing tests cover the changes** — run `cargo test` and check for failures
+3. **Write missing tests** for any uncovered code paths:
+   - **Positive tests**: happy path, valid inputs, expected state transitions
+   - **Negative tests**: invalid inputs, error conditions, boundary cases
+   - **Integration tests**: API endpoint tests in `tests/`
+4. **Run all tests** to confirm green: `cargo test`
+5. If any test fails, fix the code or test until green
+
+### Phase 3: Artifact Updates
+
+Review the changes and update project artifacts where applicable. Skip items that aren't affected.
+
+1. **Knowledge** (`knowledge/`): if the change adds/modifies behavior covered by a concept, update it to stay in sync, add a `knowledge/log.md` entry for notable decisions, and run `just check-okf`
+2. **AGENTS.md**: if the change adds new knowledge domains, skills, commands, or modifies development workflows — update the relevant section
+3. **Documentation** (`docs/`): if the change affects user-facing APIs, configuration, or features — update the relevant docs
+
+### Phase 4: Smoke Testing
+
+Smoke test impacted functionality to verify it works end-to-end:
+
+1. **Build**: `cargo build`
+2. **API/server changes**: run `./tests/smoke_test.sh` to verify all endpoints work
+3. If smoke testing reveals issues, fix them and loop back to Phase 2 (tests must still pass)
+
+### Phase 5: Quality Gates
+
+```bash
+git fetch origin main && git rebase origin/main
+```
+
+- If rebase fails with conflicts, abort and tell the user to resolve manually
+
+```bash
+just check
+```
+
+- If formatting fails, run `just fmt` to auto-fix, then retry once
+- If still failing, stop and report
+
+### Phase 6: Push and PR
+
+```bash
+git push -u origin <current-branch>
+```
+
+Check for existing PR:
+
+```bash
+gh pr view --json url 2>/dev/null
+```
+
+If no PR exists, create one using the PR template (`.github/pull_request_template.md`):
+
+- **Title**: conventional commit style from the branch commits
+- **Body**: fill in the PR template sections based on the actual changes; center it on functional change and impact, not a code-location walkthrough. Include what tests were added/verified.
+- **Evidence**: attach a Before / After with proof — `curl`/API output, server stats/metrics, or a TUI screenshot. Say so when there is no observable behavior change.
+- Use `gh pr create`
+
+If a PR already exists, update it if needed and report its URL.
+
+### Phase 7: Wait for CI and Merge
+
+- Check CI status with `gh pr checks` (poll every 30s, up to 15 minutes)
+- If CI is green, merge with `gh pr merge --squash --auto`
+- If CI fails, report the failing checks and stop
+- **NEVER** merge when CI is red
+
+### Phase 8: Post-merge
+
+After successful merge:
+
+- Report the merged PR URL
+- Done
+
+## Notes
+
+- This is the canonical shipping workflow. It implements the full "Shipping" definition and Pre-PR Checklist from AGENTS.md.
+- Phases 2-4 (tests, artifacts, smoke testing) are the quality core — do NOT skip them.
+- The `$ARGUMENTS` context helps scope which tests, specs, and smoke tests are relevant.
+- For "fix and ship" requests: implement the fix first, then run `/ship` to validate and merge.

llmsim-0.7.0/.claude/hooks/fix-git-identity.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Exit early if not in a git repo
+git rev-parse --git-dir >/dev/null 2>&1 || exit 0
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
+
+source "$REPO_ROOT/scripts/lib/common.sh"
+configure_commit_git_identity_if_needed 2>/dev/null || true

llmsim-0.7.0/.claude/settings.json

@@ -0,0 +1,20 @@
+{
+  "gitAttribution": false,
+  "attribution": {
+    "commit": "",
+    "pr": ""
+  },
+  "hooks": {
+    "SessionStart": [
+      {
+        "matcher": "",
+        "hooks": [
+          {
+            "type": "command",
+            "command": ".claude/hooks/fix-git-identity.sh"
+          }
+        ]
+      }
+    ]
+  }
+}

llmsim-0.7.0/.claude/skills/load-test/SKILL.md

@@ -0,0 +1,154 @@
+---
+name: load-test
+description: Run load and stress tests for llmsim using k6. Use this skill when you need to benchmark performance, validate high concurrency handling, or run smoke tests for CI validation.
+---
+
+# Load Testing Skill
+
+## Environment Setup
+
+### Prerequisites
+
+**k6** must be installed. Install options:
+
+```bash
+# Linux (binary)
+curl -sLO https://github.com/grafana/k6/releases/download/v0.54.0/k6-v0.54.0-linux-amd64.tar.gz
+tar -xzf k6-v0.54.0-linux-amd64.tar.gz
+sudo mv k6-v0.54.0-linux-amd64/k6 /usr/local/bin/
+
+# macOS
+brew install k6
+
+# Verify installation
+k6 version
+```
+
+**oha** is required only for the `throughput` profile (peak req/s ceiling +
+parallelisation sweep). It is a Rust load generator with near-zero per-request
+overhead, used instead of k6 when saturating the server:
+
+```bash
+cargo install oha   # or: brew install oha
+oha --version
+```
+
+### Build llmsim
+
+```bash
+cargo build --release
+```
+
+## Running Tests
+
+### Quick Smoke Test (Recommended for CI)
+
+Validates the load testing setup works correctly:
+
+```bash
+./benchmarks/smoke-test.sh
+```
+
+Ultra-fast variant (3 iterations):
+```bash
+./benchmarks/smoke-test.sh --quick
+```
+
+### Load Test Profiles
+
+| Profile | Command | Duration | VUs | Purpose |
+|---------|---------|----------|-----|---------|
+| Smoke | `./benchmarks/run-benchmark.sh smoke` | ~10s | 1 | Quick validation |
+| Load | `./benchmarks/run-benchmark.sh load` | ~4 min | 10-50 | Normal load |
+| Stress | `./benchmarks/run-benchmark.sh stress` | ~7 min | 50-500 | Breaking points |
+| Spike | `./benchmarks/run-benchmark.sh spike` | ~2 min | 10-500 | Burst traffic |
+| Soak | `./benchmarks/run-benchmark.sh soak` | ~32 min | 50 | Memory leaks |
+| High Concurrency | `./benchmarks/run-benchmark.sh high-concurrency` | ~4 min | 500 | Max throughput |
+| Throughput | `./benchmarks/run-benchmark.sh throughput` | varies | n/a | Peak req/s ceiling + parallelisation scaling (uses oha) |
+
+### Common Options
+
+```bash
+# Use existing server (don't start new one)
+./benchmarks/run-benchmark.sh load --no-server --port 8080
+
+# Custom port
+./benchmarks/run-benchmark.sh smoke --port 9000
+
+# Output results to JSON
+./benchmarks/run-benchmark.sh load --output results.json
+
+# Custom VU count for high-concurrency
+./benchmarks/run-benchmark.sh high-concurrency --vus 1000
+
+# Chaos mode - enable error injection (rate limits, server errors)
+./benchmarks/run-benchmark.sh load --chaos
+
+# Custom llmsim config file
+./benchmarks/run-benchmark.sh load --config path/to/config.toml
+```
+
+### Error Injection (Chaos Mode)
+
+By default, benchmarks run with error injection **disabled** for consistent results.
+
+Use `--chaos` to enable error injection:
+- 10% rate limit errors (429)
+- 5% server errors (500/503)
+- 2% timeout errors (504)
+
+This tests client error handling and retry logic under realistic conditions.
+
+### Remote Testing
+
+```bash
+export K6_TARGET_URL="https://llmsim.example.com"
+./benchmarks/run-benchmark.sh load --no-server
+```
+
+### Direct k6 Usage
+
+```bash
+# Run specific test script
+k6 run benchmarks/k6/chat-completions.js
+
+# With custom profile
+k6 run --env PROFILE=stress benchmarks/k6/chat-completions.js
+
+# With custom target
+k6 run --env K6_TARGET_URL=http://localhost:9000 benchmarks/k6/endpoints.js
+```
+
+## Test Scripts
+
+| Script | Description |
+|--------|-------------|
+| `chat-completions.js` | Main test - streaming/non-streaming chat completions |
+| `endpoints.js` | All endpoints with weighted distribution |
+| `high-concurrency.js` | Maximum concurrent connections |
+
+## Success Criteria
+
+Tests pass if thresholds are met:
+
+| Profile | Max Error Rate | P95 Latency |
+|---------|----------------|-------------|
+| smoke | 1% | 2s |
+| load | 5% | 3s |
+| stress | 15% | 10s |
+
+## Troubleshooting
+
+**Server not responding**: Check port availability and build status
+**High error rates in stress tests**: Expected behavior at capacity limits
+**k6 not found**: Install k6 following instructions above
+
+## Files
+
+- `benchmarks/run-benchmark.sh` - Main runner script
+- `benchmarks/smoke-test.sh` - Quick smoke test
+- `benchmarks/config/benchmark.toml` - Default config (no errors)
+- `benchmarks/config/chaos.toml` - Chaos mode config (error injection)
+- `benchmarks/k6/config.js` - k6 test configuration
+- `benchmarks/k6/*.js` - k6 test scripts
+- `knowledge/performance/load-testing.md` - Full specification

llmsim-0.7.0/.claude/skills/maintenance/SKILL.md

@@ -0,0 +1,115 @@
+---
+name: maintenance
+description: Run the routine maintenance workflow for llmsim — dependency updates, model profile refresh, code-quality + audit/deny checks, and spec alignment. Use when performing the periodic maintenance pass defined in knowledge/project/maintenance.md.
+---
+
+# Routine Maintenance Skill
+
+This skill walks the routine-maintenance workflow defined in
+[`knowledge/project/maintenance.md`](../../../knowledge/project/maintenance.md). Use it monthly,
+before a minor/major release, or whenever a major-provider releases new
+models.
+
+## Prerequisites
+
+The maintenance flow needs three Rust tools beyond the default toolchain.
+Install them once per machine; they are not cargo-installed automatically:
+
+```bash
+# Security advisory checks (RustSec)
+cargo install cargo-audit
+
+# License + duplicate-dep + ban policy checks
+cargo install cargo-deny
+```
+
+CI runs both of these on every PR (`Security Audit`, `License & Dependency
+Check` jobs), but running them locally surfaces issues *before* pushing.
+
+The Rust toolchain itself is pinned via `rust-toolchain.toml` at the repo
+root — rustup will pick the right channel automatically.
+
+## Workflow
+
+Follow the order below. Each step maps to a requirement in
+`knowledge/project/maintenance.md`.
+
+### 1. Dependency updates (R1)
+
+```bash
+# Bring the lockfile up to the latest semver-compatible versions.
+cargo update
+
+# Surface major-version updates blocked by version constraints.
+cargo update --dry-run --verbose
+
+# Security advisories. Fails the run if RustSec has reported anything.
+cargo audit
+
+# License + duplicate-dep + advisory policy. Reads ./deny.toml.
+cargo deny check
+```
+
+For each entry in the `cargo update --dry-run` output, decide:
+
+- **Apply the major bump** if the diff is small (a renamed trait import,
+  a tweaked signature) — see PR #34 for the pattern.
+- **Skip and note** if it requires a non-trivial refactor; record the
+  reason in the maintenance PR description so the next agent doesn't
+  re-evaluate from scratch.
+- **Watch for archived crates.** `serde_yaml` was at `0.9.34+deprecated`
+  before PR #37 migrated to TOML — `+deprecated` build metadata is a
+  red flag that no semver bump exists.
+
+### 2. Model profile refresh (R2)
+
+- Cross-check `src/openai/models.rs` against [models.dev](https://models.dev)
+  — context window, max output tokens, capabilities, knowledge cutoff,
+  release timestamp.
+- Add new models from OpenAI, Anthropic, and Google (and DeepSeek where
+  appropriate). Mirror the new IDs in `default_models()` in
+  `src/cli/config.rs` so they appear in `/openai/v1/models`.
+- Update the model table in `knowledge/foundations/architecture.md`.
+- Add focused unit tests for each new profile (one assertion per field
+  worth caring about; see existing tests as a template).
+
+### 3. Code-quality sweep (R3)
+
+```bash
+cargo fmt --check
+cargo clippy --all-targets -- -D warnings
+cargo test
+```
+
+- Grep for `TODO`, `FIXME`, `HACK`, `XXX` — resolve or document.
+- Review API handlers for input validation, panicking paths on user
+  input, and resource limits (body sizes, in-memory collection growth).
+- Verify CORS configuration is still appropriate for the deployment
+  posture.
+
+### 4. Spec + AGENTS.md alignment (R5 + R6)
+
+- Read each concept in `knowledge/` and confirm it still matches the code.
+  Run `just check-okf` to keep the bundle OKF v0.2 conformant.
+  Update tables, requirement lists, and example snippets where they've
+  drifted.
+- Verify `AGENTS.md` lists every knowledge domain, every skill, and every command.
+- Run a quick `grep -rn "PORT_NUMBER\|version-like-string"` for known
+  stale references (e.g., the AGENTS.md default-port fix in PR #36).
+
+## Shape of a maintenance pass
+
+A standard pass produces three or four small PRs rather than one omnibus
+diff:
+
+1. **`chore(deps)`** — `cargo update` + any major bumps you decided to
+   apply. Include the clippy fix if `dtolnay/rust-toolchain` rolled to
+   a new version and surfaced new lints.
+2. **`feat(models)`** — new model profiles + spec table updates.
+3. **`docs(knowledge)`** — any knowledge drift discovered while reviewing the
+   code.
+4. *(optional)* **`chore(...)` / `refactor(...)`** for anything bigger
+   that came out of the code-quality sweep.
+
+Open each PR independently against `main` so they can be reviewed and
+merged in isolation. See PRs #34 / #35 / #36 / #37 for a worked example.

llmsim-0.7.0/.dockerignore

@@ -0,0 +1,9 @@
+target/
+.git/
+.github/
+*.md
+!README.md
+Dockerfile
+docker-compose.yml
+.dockerignore
+.gitignore

llmsim-0.7.0/.github/FUNDING.yml

@@ -0,0 +1,3 @@
+# These are supported funding model platforms
+
+github: [chaliy]

llmsim-0.7.0/.github/pull_request_template.md

@@ -0,0 +1,20 @@
+## What changed
+Describe the change functionally — what behavior changes and its impact on users
+or callers. Lead with outcomes; don't walk through code locations, the diff shows
+where and how. Keep any code-level notes short and specific.
+
+## Why
+Problem or motivation.
+
+## Before / After
+Show the effect with evidence. Include before and after whenever behavior changes —
+CLI/API output, logs, metrics, or screenshots for UI (attach working screenshots
+when possible). For changes with no observable behavior (pure refactor, docs), say so.
+
+## Risk
+- Low / Medium / High
+- What can break
+
+## Checklist
+- [ ] Tests added or updated
+- [ ] Backward compatibility considered

llmsim-0.7.0/.github/workflows/bindings.yml

@@ -0,0 +1,145 @@
+# CI for the Python (PyO3/maturin) and Node (NAPI-RS) packages in crates/.
+# Release builds for every platform live in publish-python.yml / publish-js.yml.
+
+name: Bindings
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+env:
+  CARGO_TERM_COLOR: always
+
+jobs:
+  versions:
+    name: Package versions match Cargo
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - name: Check package.json version
+        run: |
+          CARGO_VERSION=$(grep '^version = ' Cargo.toml | head -1 | sed 's/version = "\(.*\)"/\1/')
+          PKG_VERSION=$(node -p "require('./crates/llmsim-js/package.json').version")
+          if [ "$CARGO_VERSION" != "$PKG_VERSION" ]; then
+            echo "::error::crates/llmsim-js/package.json version ($PKG_VERSION) != Cargo.toml ($CARGO_VERSION)"
+            exit 1
+          fi
+          echo "Versions match: $CARGO_VERSION"
+
+  python:
+    name: Python ${{ matrix.python-version }} (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - os: ubuntu-24.04
+            python-version: "3.9"
+          - os: ubuntu-24.04
+            python-version: "3.13"
+          - os: macos-latest
+            python-version: "3.13"
+          - os: windows-latest
+            python-version: "3.13"
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@1.99.0
+
+      - name: Cache cargo registry
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ runner.os }}-bindings-py-${{ hashFiles('**/Cargo.toml') }}
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Lint
+        if: matrix.python-version == '3.13' && runner.os == 'Linux'
+        working-directory: crates/llmsim-python
+        run: |
+          pip install ruff
+          ruff check .
+          ruff format --check .
+
+      - name: Build and install wheel
+        shell: bash
+        working-directory: crates/llmsim-python
+        run: |
+          python -m venv .venv
+          if [ "$RUNNER_OS" = "Windows" ]; then . .venv/Scripts/activate; else . .venv/bin/activate; fi
+          pip install maturin
+          maturin build --out dist
+          pip install --find-links dist llmsim
+          pip install pytest pytest-timeout openai anthropic
+
+      - name: Test
+        shell: bash
+        working-directory: crates/llmsim-python
+        run: |
+          if [ "$RUNNER_OS" = "Windows" ]; then . .venv/Scripts/activate; else . .venv/bin/activate; fi
+          # Run from tests/ so `import llmsim` resolves to the installed wheel,
+          # not the source directory (which has no native module).
+          cd tests && pytest -v .
+
+  node:
+    name: Node ${{ matrix.node }} (${{ matrix.os }})
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - os: ubuntu-latest
+            node: "20"
+          - os: ubuntu-latest
+            node: "22"
+          - os: ubuntu-latest
+            node: "24"
+          - os: macos-latest
+            node: "22"
+          - os: windows-latest
+            node: "22"
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@1.99.0
+
+      - name: Cache cargo registry
+        uses: actions/cache@v4
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            target
+          key: ${{ runner.os }}-bindings-js-${{ hashFiles('**/Cargo.toml') }}
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: ${{ matrix.node }}
+
+      - name: Install dependencies
+        working-directory: crates/llmsim-js
+        run: npm ci
+
+      - name: Build (debug)
+        working-directory: crates/llmsim-js
+        run: npm run build:debug
+
+      - name: Test
+        shell: bash
+        working-directory: crates/llmsim-js
+        run: node --test __test__/*.test.mjs
+
+      - name: Pack dry run
+        if: runner.os == 'Linux' && matrix.node == '22'
+        working-directory: crates/llmsim-js
+        run: npm pack --dry-run