rubric-load-tester 0.2.0__tar.gz

rubric_load_tester-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

rubric_load_tester-0.2.0/PKG-INFO

@@ -0,0 +1,292 @@
+Metadata-Version: 2.4
+Name: rubric-load-tester
+Version: 0.2.0
+Summary: Zero-config, LLM-powered API load tester.
+Author: Rubric Team
+License: MIT License
+        
+        Copyright (c) 2026
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
+Project-URL: Homepage, https://github.com/adetu-siyan/rubric_lt
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Testing :: Traffic Generation
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click
+Requires-Dist: rich
+Requires-Dist: aiohttp
+Requires-Dist: numpy
+Requires-Dist: groq
+Requires-Dist: python-dotenv
+Requires-Dist: pyyaml
+Requires-Dist: pydantic
+Requires-Dist: prompt_toolkit
+Dynamic: license-file
+
+# Rubric
+
+Rubric points at your API, reads its OpenAPI schema, and hits it with realistic load — no manual test scripts to write. It uses an LLM to generate the actual request payloads (valid data, edge cases, and garbage inputs), fires them concurrently, and gives you latency percentiles plus AI notes on what broke and why.
+
+---
+
+## How it works
+
+1. **Schema ingestion** — pulls `/openapi.json`, sorts routes into buckets (does it need a body? does it depend on an ID from an earlier request?)
+2. **Payload synthesis** — the LLM writes ~50 payloads per endpoint across three tiers, generated in parallel so this doesn't become the slow part
+3. **Load engine** — async workers fire the requests and stream live numbers to your terminal as it runs
+4. **Report** — latency percentiles, pass/fail per endpoint, and a short AI writeup of likely root causes for anything that failed
+
+### Route buckets
+
+| Bucket | Methods | What it means |
+|--------|---------|----------------|
+| A | GET, DELETE | No body — just path/query params |
+| B | POST, PUT, PATCH | Has a request body |
+| C | Any | Depends on a resource ID from an earlier POST in the run |
+
+### Payload tiers
+
+| Tier | Count | What it's testing |
+|------|-------|--------------------|
+| `happy_path` | 17 | Normal, valid data — should just return 2xx |
+| `edge_cases` | 17 | Boundaries: empty strings, huge ints, unicode, very long strings |
+| `malformed` | 16 | Wrong types, missing fields, broken structure |
+
+---
+
+## Getting set up
+
+```bash
+cd rubric
+pip install -r requirements.txt
+# or install as a package so you get the `rubric` command:
+pip install -e .
+```
+
+Then copy the env file and fill in your provider:
+
+```bash
+cp .env.example .env
+```
+
+```env
+LLM_PROVIDER=groq          # or openai
+
+# Groq — fast, has a free tier, this is what most people use
+GROQ_API_KEY=your_groq_key_here
+GROQ_MODEL=meta-llama/llama-4-scout-17b-16e-instruct
+
+# OpenAI, if you'd rather use that
+OPENAI_API_KEY=your_openai_key_here
+OPENAI_MODEL=gpt-4o
+OPENAI_BASE_URL=https://api.openai.com/v1
+```
+
+---
+
+## Using it
+
+The simplest case:
+
+```bash
+rubric run http://localhost:8000
+```
+
+Tuning the run:
+
+```bash
+rubric run http://localhost:8000 \
+  --requests 500 \
+  --concurrency 50 \
+  --tier happy_path \
+  --p95-threshold 200 \
+  --error-rate 0.01
+```
+
+Run every tier in one go:
+
+```bash
+rubric run http://localhost:8000 --tier all --requests 1000
+```
+
+Just want to see what Rubric found without actually load testing:
+
+```bash
+rubric inspect http://localhost:8000
+```
+
+Skip the AI diagnostics step if you just want raw numbers fast:
+
+```bash
+rubric run http://localhost:8000 --no-diagnostics
+```
+
+Switch providers per run instead of editing `.env`:
+
+```bash
+rubric run http://localhost:8000 --provider groq
+rubric run http://localhost:8000 --provider openai
+```
+
+### Testing against a self-signed cert (local/dev)
+
+Rubric checks the target's TLS certificate before sending login credentials, the same way a browser would — if it can't verify the cert, it won't send anything. That's fine for a real API, but it'll block you if you're testing against `localhost` or a dev server on a self-signed cert. In that case, opt in explicitly:
+
+```bash
+rubric run https://localhost:8443 --allow-self-signed
+```
+
+Only do this against something you control. It turns off the check that stops your credentials and tokens from being sent to an unverified server — don't use it against a real target over a network you don't fully trust.
+
+---
+
+## Flags
+
+| Flag | Default | What it does |
+|------|---------|---------------|
+| `--requests` | 100 | Total requests spread across endpoints |
+| `--concurrency` | 20 | Max concurrent connections |
+| `--tier` | `happy_path` | `happy_path`, `edge_cases`, `malformed`, or `all` |
+| `--p95-threshold` | 200 | P95 latency (ms) — above this, endpoint is marked DEGRADED |
+| `--error-rate` | 0.01 | 5xx rate above this marks the endpoint FAILED |
+| `--no-diagnostics` | False | Skip the AI failure writeup |
+| `--save` | True | Write the JSON report to disk |
+| `--provider` | from `.env` | Override the LLM provider for this run |
+| `--allow-self-signed` | False | Skip TLS verification for login/token requests — local/self-signed dev servers only |
+
+---
+
+## What you get back
+
+While it's running, you'll see something like this update live:
+
+```
+RUBRIC · 847 requests · 142.3 RPS · 5.9s
+┌────────────────────────┬──────┬──────┬──────┬──────┬──────┬───────┐
+│ Endpoint               │  RPS │  P95 │  2xx │  4xx │  5xx │ Total │
+├────────────────────────┼──────┼──────┼──────┼──────┼──────┼───────┤
+│ POST /users            │  142 │  84ms│  891 │   23 │    6 │   920 │
+│ GET  /users/{user_id}  │   89 │  31ms│  412 │    0 │    1 │   413 │
+│ POST /products         │  201 │  19ms│ 1204 │   11 │    0 │  1215 │
+└────────────────────────┴──────┴──────┴──────┴──────┴──────┴───────┘
+```
+
+And a summary once it's done:
+
+```
+┌────────────────────────┬────────────┬─────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┐
+│ Endpoint               │ Status     │ RPS │  P50 │  P95 │  P99 │  2xx │  4xx │  5xx │ Err% │
+├────────────────────────┼────────────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
+│ POST /users            │ PASSED     │ 142 │ 34ms │ 84ms │201ms │  891 │   23 │    6 │0.67% │
+│ GET  /users/{user_id}  │ DEGRADED   │  89 │ 12ms │312ms │ 67ms │  412 │    0 │    1 │0.24% │
+│ POST /products         │ FAILED     │ 201 │ 18ms │ 44ms │ 98ms │  980 │   11 │   24 │2.10% │
+└────────────────────────┴────────────┴─────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┘
+
+SUITE FAILED
+Passed: 1  Degraded: 1  Failed: 1
+Total: 2507 requests   Duration: 17.6s   RPS: 142.4
+Thresholds — P95 < 200ms, error rate < 1%
+```
+
+For anything that failed, you'll get a short explanation of the likely cause, not just a status code:
+
+```
+1. POST /users
+   Payload: {"name": "", "email": "x@y.com", "role": ""}
+   Cause: empty 'role' string hits a NOT NULL constraint at the DB layer.
+   Fix: add a validator to reject empty strings for that field.
+
+2. POST /products
+   Payload: {"price": -1, "name": "Widget", "category": "tools"}
+   Cause: negative price trips a CHECK constraint in Postgres.
+   Fix: add Field(gt=0) to the price field in the schema.
+```
+
+---
+
+## Trying it against the sample API
+
+There's a small FastAPI app included if you want to try Rubric without pointing it at a real project yet:
+
+```bash
+pip install fastapi uvicorn
+uvicorn sample_api:app --reload
+
+# then, in another terminal:
+rubric run http://localhost:8000
+rubric inspect http://localhost:8000
+```
+
+---
+
+## The JSON report
+
+Every run writes `rubric_report_<url>_<timestamp>.json` to disk:
+
+```json
+{
+  "suite_result": "SUITE FAILED",
+  "thresholds": { "p95_ms": 200, "error_rate_pct": 1.0 },
+  "summary": {
+    "total_requests": 2507,
+    "duration_seconds": 17.6,
+    "overall_rps": 142.4,
+    "connection_drops": 0,
+    "passed": 1,
+    "degraded": 1,
+    "failed": 1
+  },
+  "endpoints": [ ... ],
+  "failure_diagnostics": [ ... ]
+}
+```
+
+`suite_result` is meant to be read by CI — Rubric exits `1` on a failed suite.
+
+---
+
+## Layout
+
+```
+rubric/
+├── core/
+│   ├── schema.py         # OpenAPI ingestion & route categorization
+│   ├── synthesizer.py    # LLM payload synthesis
+│   └── llm.py            # Groq / OpenAI client wrapper
+├── engine/
+│   └── runner.py         # Async load engine & live streaming
+├── report/
+│   └── reporter.py       # Reports & AI diagnostics
+├── cli/
+│   └── main.py           # Click entry point
+├── sample_api.py         # Demo FastAPI server
+├── requirements.txt
+├── setup.py
+└── .env.example
+```
+
+---
+

rubric_load_tester-0.2.0/README.md

@@ -0,0 +1,244 @@
+# Rubric
+
+Rubric points at your API, reads its OpenAPI schema, and hits it with realistic load — no manual test scripts to write. It uses an LLM to generate the actual request payloads (valid data, edge cases, and garbage inputs), fires them concurrently, and gives you latency percentiles plus AI notes on what broke and why.
+
+---
+
+## How it works
+
+1. **Schema ingestion** — pulls `/openapi.json`, sorts routes into buckets (does it need a body? does it depend on an ID from an earlier request?)
+2. **Payload synthesis** — the LLM writes ~50 payloads per endpoint across three tiers, generated in parallel so this doesn't become the slow part
+3. **Load engine** — async workers fire the requests and stream live numbers to your terminal as it runs
+4. **Report** — latency percentiles, pass/fail per endpoint, and a short AI writeup of likely root causes for anything that failed
+
+### Route buckets
+
+| Bucket | Methods | What it means |
+|--------|---------|----------------|
+| A | GET, DELETE | No body — just path/query params |
+| B | POST, PUT, PATCH | Has a request body |
+| C | Any | Depends on a resource ID from an earlier POST in the run |
+
+### Payload tiers
+
+| Tier | Count | What it's testing |
+|------|-------|--------------------|
+| `happy_path` | 17 | Normal, valid data — should just return 2xx |
+| `edge_cases` | 17 | Boundaries: empty strings, huge ints, unicode, very long strings |
+| `malformed` | 16 | Wrong types, missing fields, broken structure |
+
+---
+
+## Getting set up
+
+```bash
+cd rubric
+pip install -r requirements.txt
+# or install as a package so you get the `rubric` command:
+pip install -e .
+```
+
+Then copy the env file and fill in your provider:
+
+```bash
+cp .env.example .env
+```
+
+```env
+LLM_PROVIDER=groq          # or openai
+
+# Groq — fast, has a free tier, this is what most people use
+GROQ_API_KEY=your_groq_key_here
+GROQ_MODEL=meta-llama/llama-4-scout-17b-16e-instruct
+
+# OpenAI, if you'd rather use that
+OPENAI_API_KEY=your_openai_key_here
+OPENAI_MODEL=gpt-4o
+OPENAI_BASE_URL=https://api.openai.com/v1
+```
+
+---
+
+## Using it
+
+The simplest case:
+
+```bash
+rubric run http://localhost:8000
+```
+
+Tuning the run:
+
+```bash
+rubric run http://localhost:8000 \
+  --requests 500 \
+  --concurrency 50 \
+  --tier happy_path \
+  --p95-threshold 200 \
+  --error-rate 0.01
+```
+
+Run every tier in one go:
+
+```bash
+rubric run http://localhost:8000 --tier all --requests 1000
+```
+
+Just want to see what Rubric found without actually load testing:
+
+```bash
+rubric inspect http://localhost:8000
+```
+
+Skip the AI diagnostics step if you just want raw numbers fast:
+
+```bash
+rubric run http://localhost:8000 --no-diagnostics
+```
+
+Switch providers per run instead of editing `.env`:
+
+```bash
+rubric run http://localhost:8000 --provider groq
+rubric run http://localhost:8000 --provider openai
+```
+
+### Testing against a self-signed cert (local/dev)
+
+Rubric checks the target's TLS certificate before sending login credentials, the same way a browser would — if it can't verify the cert, it won't send anything. That's fine for a real API, but it'll block you if you're testing against `localhost` or a dev server on a self-signed cert. In that case, opt in explicitly:
+
+```bash
+rubric run https://localhost:8443 --allow-self-signed
+```
+
+Only do this against something you control. It turns off the check that stops your credentials and tokens from being sent to an unverified server — don't use it against a real target over a network you don't fully trust.
+
+---
+
+## Flags
+
+| Flag | Default | What it does |
+|------|---------|---------------|
+| `--requests` | 100 | Total requests spread across endpoints |
+| `--concurrency` | 20 | Max concurrent connections |
+| `--tier` | `happy_path` | `happy_path`, `edge_cases`, `malformed`, or `all` |
+| `--p95-threshold` | 200 | P95 latency (ms) — above this, endpoint is marked DEGRADED |
+| `--error-rate` | 0.01 | 5xx rate above this marks the endpoint FAILED |
+| `--no-diagnostics` | False | Skip the AI failure writeup |
+| `--save` | True | Write the JSON report to disk |
+| `--provider` | from `.env` | Override the LLM provider for this run |
+| `--allow-self-signed` | False | Skip TLS verification for login/token requests — local/self-signed dev servers only |
+
+---
+
+## What you get back
+
+While it's running, you'll see something like this update live:
+
+```
+RUBRIC · 847 requests · 142.3 RPS · 5.9s
+┌────────────────────────┬──────┬──────┬──────┬──────┬──────┬───────┐
+│ Endpoint               │  RPS │  P95 │  2xx │  4xx │  5xx │ Total │
+├────────────────────────┼──────┼──────┼──────┼──────┼──────┼───────┤
+│ POST /users            │  142 │  84ms│  891 │   23 │    6 │   920 │
+│ GET  /users/{user_id}  │   89 │  31ms│  412 │    0 │    1 │   413 │
+│ POST /products         │  201 │  19ms│ 1204 │   11 │    0 │  1215 │
+└────────────────────────┴──────┴──────┴──────┴──────┴──────┴───────┘
+```
+
+And a summary once it's done:
+
+```
+┌────────────────────────┬────────────┬─────┬──────┬──────┬──────┬──────┬──────┬──────┬──────┐
+│ Endpoint               │ Status     │ RPS │  P50 │  P95 │  P99 │  2xx │  4xx │  5xx │ Err% │
+├────────────────────────┼────────────┼─────┼──────┼──────┼──────┼──────┼──────┼──────┼──────┤
+│ POST /users            │ PASSED     │ 142 │ 34ms │ 84ms │201ms │  891 │   23 │    6 │0.67% │
+│ GET  /users/{user_id}  │ DEGRADED   │  89 │ 12ms │312ms │ 67ms │  412 │    0 │    1 │0.24% │
+│ POST /products         │ FAILED     │ 201 │ 18ms │ 44ms │ 98ms │  980 │   11 │   24 │2.10% │
+└────────────────────────┴────────────┴─────┴──────┴──────┴──────┴──────┴──────┴──────┴──────┘
+
+SUITE FAILED
+Passed: 1  Degraded: 1  Failed: 1
+Total: 2507 requests   Duration: 17.6s   RPS: 142.4
+Thresholds — P95 < 200ms, error rate < 1%
+```
+
+For anything that failed, you'll get a short explanation of the likely cause, not just a status code:
+
+```
+1. POST /users
+   Payload: {"name": "", "email": "x@y.com", "role": ""}
+   Cause: empty 'role' string hits a NOT NULL constraint at the DB layer.
+   Fix: add a validator to reject empty strings for that field.
+
+2. POST /products
+   Payload: {"price": -1, "name": "Widget", "category": "tools"}
+   Cause: negative price trips a CHECK constraint in Postgres.
+   Fix: add Field(gt=0) to the price field in the schema.
+```
+
+---
+
+## Trying it against the sample API
+
+There's a small FastAPI app included if you want to try Rubric without pointing it at a real project yet:
+
+```bash
+pip install fastapi uvicorn
+uvicorn sample_api:app --reload
+
+# then, in another terminal:
+rubric run http://localhost:8000
+rubric inspect http://localhost:8000
+```
+
+---
+
+## The JSON report
+
+Every run writes `rubric_report_<url>_<timestamp>.json` to disk:
+
+```json
+{
+  "suite_result": "SUITE FAILED",
+  "thresholds": { "p95_ms": 200, "error_rate_pct": 1.0 },
+  "summary": {
+    "total_requests": 2507,
+    "duration_seconds": 17.6,
+    "overall_rps": 142.4,
+    "connection_drops": 0,
+    "passed": 1,
+    "degraded": 1,
+    "failed": 1
+  },
+  "endpoints": [ ... ],
+  "failure_diagnostics": [ ... ]
+}
+```
+
+`suite_result` is meant to be read by CI — Rubric exits `1` on a failed suite.
+
+---
+
+## Layout
+
+```
+rubric/
+├── core/
+│   ├── schema.py         # OpenAPI ingestion & route categorization
+│   ├── synthesizer.py    # LLM payload synthesis
+│   └── llm.py            # Groq / OpenAI client wrapper
+├── engine/
+│   └── runner.py         # Async load engine & live streaming
+├── report/
+│   └── reporter.py       # Reports & AI diagnostics
+├── cli/
+│   └── main.py           # Click entry point
+├── sample_api.py         # Demo FastAPI server
+├── requirements.txt
+├── setup.py
+└── .env.example
+```
+
+---
+

rubric_load_tester-0.2.0/pyproject.toml

@@ -0,0 +1,44 @@
+[build-system]
+requires = ["setuptools>=42"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "rubric-load-tester"
+version = "0.2.0"
+description = "Zero-config, LLM-powered API load tester."
+authors = [
+    {name = "Rubric Team"}
+]
+readme = "README.md"
+license = {file = "LICENSE"}
+requires-python = ">=3.11"
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3.11",
+    "Topic :: Software Development :: Testing",
+    "Topic :: Software Development :: Testing :: Traffic Generation",
+]
+dependencies = [
+    "click",
+    "rich",
+    "aiohttp",
+    "numpy",
+    "groq",
+    "python-dotenv",
+    "pyyaml",
+    "pydantic",
+    "prompt_toolkit",
+]
+
+[project.scripts]
+rubric = "rubric.cli.main:main"
+
+[project.urls]
+Homepage = "https://github.com/adetu-siyan/rubric_lt"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["rubric*"]
+exclude = ["rubric_traces*", "tests*"]

rubric_load_tester-0.2.0/rubric/auth/__init__.py

@@ -0,0 +1,8 @@
+from .store import load_auth, save_auth, delete_credentials, auth_file_exists
+from .interviewer import run_auth_interview
+from .resolver import resolve_headers
+
+__all__ = [
+    "load_auth", "save_auth", "delete_credentials", "auth_file_exists",
+    "run_auth_interview", "resolve_headers",
+]