hubric 0.5.0__tar.gz

hubric-0.5.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hubric-0.5.0/PKG-INFO

@@ -0,0 +1,395 @@
+Metadata-Version: 2.4
+Name: hubric
+Version: 0.5.0
+Summary: Zero-config, LLM-powered API load tester.
+Author: Hubric Team
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/adetu-siyan/rubric_lt
+Project-URL: Repository, https://github.com/adetu-siyan/rubric_lt
+Project-URL: Issues, https://github.com/adetu-siyan/rubric_lt/issues
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Testing :: Traffic Generation
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.1.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: aiohttp>=3.9.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: prompt_toolkit>=3.0
+Requires-Dist: keyring>=24.0.0
+Requires-Dist: certifi>=2024.0.0
+Provides-Extra: groq
+Requires-Dist: groq>=0.9.0; extra == "groq"
+Provides-Extra: bedrock
+Requires-Dist: boto3>=1.34.0; extra == "bedrock"
+Provides-Extra: pdf
+Requires-Dist: weasyprint>=60.0; extra == "pdf"
+Provides-Extra: all
+Requires-Dist: groq>=0.9.0; extra == "all"
+Requires-Dist: boto3>=1.34.0; extra == "all"
+Requires-Dist: weasyprint>=60.0; extra == "all"
+Dynamic: license-file
+
+# Hubric
+
+[![PyPI version](https://img.shields.io/pypi/v/hubric.svg)](https://pypi.org/project/hubric/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
+
+**Hubric** is a zero-config, LLM-powered API load tester and benchmark engine.
+
+It ingests your live OpenAPI/Swagger schema, leverages fast LLMs (Groq, OpenAI, Anthropic, Bedrock) to synthesize realistic multi-tier test payloads, and executes high-concurrency stateful user flows—benchmarking latency against the **Apdex (Application Performance Index)** standard.
+
+---
+
+## Key Features
+
+- ⚡ **Zero-Scripting Schema Ingestion**: Automatically reads `/openapi.json` or local spec files (OpenAPI, Swagger, Postman, Blueprint) to categorize endpoints, path parameters, and request bodies.
+- 🧠 **LLM-Powered Multi-Tier Payload Synthesis**:
+  - `happy_path`: Realistic, schema-valid data targeting production success.
+  - `edge_cases`: Boundary values, unicode, extreme numbers, and max-length strings.
+  - `malformed`: Invalid types and schema violations to test API error handling.
+- 🚀 **High-Throughput Asynchronous Engine**: Built on Python `asyncio` and `aiohttp` proactor runtimes, featuring an adaptive auto-concurrency heuristic and ramp scheduler for high-volume sustained traffic.
+- 🔄 **Stateful Multi-Step Journeys**: Automatically sequences dependent endpoints (`POST /orders` $\rightarrow$ `GET /orders/{id}`) by dynamically extracting identifiers, tokens, and cookies across requests.
+- 📊 **Apdex Standard Scoring**: Calculates objective 0–100 user satisfaction ratings with dynamic latency baseline scaling for AI, streaming, and media workloads.
+- 🛠️ **Dedicated CI/CD Pipeline Mode (`hubric ci`)**: Headless execution, configurable regression thresholds, and automated **JUnit XML** test artifact export.
+- 📦 **Payload Pre-Caching (`hubric payload`)**: Pre-generate and cache synthetic payloads to a local reusable pool (`.hubric_cache/`), eliminating redundant LLM calls during repeated benchmark runs.
+- 🔒 **Enterprise-Grade Security Architecture**: Multi-layer credential protection, atomic OS file permissions, header injection prevention, and linear-time PII/secret redaction.
+
+---
+
+## Installation
+
+Requires Python 3.11+.
+
+```bash
+pip install hubric
+```
+
+> **Windows Note:** To prevent Windows Defender or execution policy wrappers from interfering, you can always run Hubric directly via Python:
+> ```bash
+> python -m hubric run <url>
+> ```
+
+---
+
+## Quickstart
+
+Run a zero-config load test against any live API:
+
+```bash
+hubric run http://localhost:8000
+```
+
+### What Hubric Does:
+1. **Schema Ingestion**: Fetches `/openapi.json` and classifies endpoints into Route Buckets:
+   - **Bucket A**: Script-generated parameter routes (GET, HEAD, DELETE).
+   - **Bucket B**: Independent mutations (POST, PUT, PATCH) requiring synthesized bodies.
+   - **Bucket C**: Stateful endpoints that depend on upstream resource IDs.
+2. **Auth Discovery & Interview**: Automatically detects authentication schemes (Bearer JWT, API Key, Basic Auth, Cookies) and securely acquires/verifies a session token.
+3. **Payload Synthesis**: Pre-flight LLM pass synthesizes realistic payloads and caches them in `.hubric_cache/`.
+4. **4-Phase Traffic Dispatch**:
+   - *Phase 1a*: Independent writes (`POST`)
+   - *Phase 2*: Concurrent reads and updates (`GET`, `PUT`, `PATCH`)
+   - *Phase 3*: Destructive operations (`DELETE`)
+   - *Phase 4*: Session teardowns (`POST /logout`)
+5. **Report Generation**: Outputs rich terminal statistics, standalone interactive HTML reports, and JSON telemetry.
+
+---
+
+## Commands & Usage
+
+### 1. `hubric run` — Full Load Test
+
+```bash
+# Run with 50 concurrent users
+hubric run http://localhost:8000 --users 50
+
+# Target a specific request-per-second (RPS) rate
+hubric run http://localhost:8000 --rps 100
+
+# Choose traffic pattern (ramp, spike, constant, step)
+hubric run http://localhost:8000 --pattern ramp --users 100
+
+# Test a specific endpoint only
+hubric run http://localhost:8000 --endpoint "POST /api/items"
+
+# Run all payload tiers (happy_path, edge_cases, malformed)
+hubric run http://localhost:8000 --tier all
+
+# Pass a local OpenAPI spec or Postman collection
+hubric run http://localhost:8000 --spec ./openapi.yaml
+
+# Pass authentication token directly
+hubric run http://localhost:8000 --token "your-jwt-token"
+
+# Multi-account simulation using a Token Pool (distributes rate limits across users)
+hubric run https://api.example.com --tokens ./tokens.txt
+
+# Skip brute-force login limits or session-destroying routes (/logout)
+hubric run https://api.example.com --skip "POST /auth/login,POST /auth/logout"
+```
+
+### Handling Authentication & Rate Limits Under Load
+
+When load testing APIs with protected routes and concurrent virtual users:
+
+#### 1. Responsive Single-User Protection (Automatic)
+When Hubric detects you are testing with a single authenticated account (and no `--tokens` multi-account pool):
+- It **automatically shields** routes that would invalidate your session (e.g., `POST /logout`, `/signout`) or trigger 1-account brute-force rate limiters (e.g., `POST /auth/login`, `/forgot-password`, `/reset-password`).
+- This guarantees your valid token remains intact throughout all testing phases and concurrent stages without getting hit by `429 Too Many Requests` or cascade `401 Unauthorized` errors.
+- At the end of the test, Hubric displays an intelligent guidance nudge showing what was shielded and how to test those endpoints with `--tokens`.
+- *(Note: If you explicitly specify an endpoint like `--endpoint "POST /auth/login"`, Hubric will honor your direct command and test it).*
+
+#### 2. Multi-Account Virtual User Pools (`--tokens <file>`)
+Testing with a single account causes all concurrent virtual users to share the same identity. On production-like APIs, this triggers brute-force rate limiters (e.g., 10 login requests/minute) or invalidates sessions for all virtual users when `/logout` is called.
+
+Supply a `tokens.txt` file containing pre-generated tokens for multiple test users (one token per line):
+```text
+eyJhbGciOi... (user 1)
+eyJhbGciOi... (user 2)
+eyJhbGciOi... (user 3)
+```
+
+Run with `--tokens`:
+```bash
+hubric run https://api.example.com --tokens ./tokens.txt --users 25
+```
+Hubric round-robins requests across the pool, giving each virtual user their own rate-limit bucket and isolated session state.
+
+#### 3. Exclude Lifecycle Routes (`--skip <endpoints>`)
+If you want to manually exclude specific routes from testing:
+```bash
+hubric run https://api.example.com --skip "POST /auth/login,POST /auth/logout"
+```
+You can pass exact keys (`POST /auth/login`) or route substrings (`logout,login`).
+
+#### 4. Staging Environment Best Practices
+In staging environments, backend teams should either:
+- Configure relaxed rate limits (e.g., `1000/min` instead of `10/min`).
+- Add a rate-limit bypass header (e.g., `X-RateLimit-Bypass`) for automated load test suites.
+- Whitelist the test runner IP address from rate limit middleware.
+
+### 2. `hubric ci` — Non-Interactive CI/CD Automation
+
+Designed specifically for automated deployment pipelines (GitHub Actions, GitLab CI, Jenkins):
+
+```bash
+hubric ci http://localhost:8000 --users 20 --junit junit.xml --fail-on-degraded
+```
+
+- **Non-interactive & Headless**: Automatically disables browser popups (`--no-open`) and interactive credential interviews.
+- **JUnit XML Export**: Saves standard JUnit XML test results (`--junit <path>`) for native test reporting in CI/CD platforms.
+- **Fail Gates**: Exits with exit code `1` if any endpoint fails SLA or if `--fail-on-degraded` is tripped.
+
+#### Example: GitHub Actions Workflow
+```yaml
+name: API Load & Regression Test
+on: [push, pull_request]
+
+jobs:
+  load-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+      - name: Install Hubric
+        run: pip install hubric
+      - name: Run CI Load Test
+        env:
+          HUBRIC_AUTH_TOKEN: ${{ secrets.TEST_API_TOKEN }}
+          GROQ_API_KEY: ${{ secrets.GROQ_API_KEY }}
+        run: |
+          hubric ci http://localhost:8000 --users 25 --junit report.xml --fail-on-degraded
+      - name: Publish Test Results
+        uses: EnricoMi/publish-unit-test-result-action@v2
+        if: always()
+        with:
+          files: report.xml
+```
+
+### 3. `hubric payload` — Pre-Flight Payload Synthesis
+
+Generate and cache synthetic test payloads before running a test:
+
+```bash
+# Pre-synthesize 5 payloads per endpoint across all tiers
+hubric payload http://localhost:8000 --count 5 --tier all
+
+# Export generated payloads to a custom JSON file
+hubric payload http://localhost:8000 --out payloads.json
+```
+
+### 4. `hubric inspect` — API Schema Inspection
+
+Inspect discovered routes, parameters, and auth requirements without sending test traffic:
+
+```bash
+hubric inspect http://localhost:8000
+```
+
+### 5. `hubric settings` — LLM & Provider Configuration
+
+Configure LLM provider keys and default inference models:
+
+```bash
+# Interactive setup wizard
+hubric settings --configure-llm
+
+# View active provider and configuration
+hubric settings
+```
+
+---
+
+## LLM Provider Configuration
+
+Hubric supports **Groq** (ultra-fast LPU inference, default), **OpenAI**, **Anthropic**, **OpenRouter**, and **AWS Bedrock**.
+
+You can configure your provider via:
+
+### Option A: Interactive Wizard
+```bash
+hubric settings --configure-llm
+```
+
+### Option B: Environment Variables
+Create a `.env` file or export environment variables:
+
+```bash
+# Provider Selection (groq, openai, anthropic, openrouter, bedrock)
+HUBRIC_PROVIDER=groq
+
+# Groq (Default)
+GROQ_API_KEY=gsk_...
+GROQ_MODEL=qwen/qwen3.8-27b
+GROQ_FAST_MODEL=openai/gpt-oss-20b
+
+# OpenAI
+OPENAI_API_KEY=sk-...
+OPENAI_MODEL=gpt-4o-mini
+
+# Anthropic
+ANTHROPIC_API_KEY=sk-ant-...
+ANTHROPIC_MODEL=claude-haiku-4-5-20251001
+
+# AWS Bedrock
+AWS_REGION=us-east-1
+AWS_ACCESS_KEY_ID=AKIA...
+AWS_SECRET_ACCESS_KEY=...
+BEDROCK_MODEL=us.meta.llama3-1-70b-instruct-v1:0
+```
+
+---
+
+## Declarative Configuration (`hubric.json`)
+
+Standardize test parameters across your repository by committing a `hubric.json` or `hubric.yaml` to your project root:
+
+```json
+{
+  "target": "http://localhost:8000",
+  "users": 50,
+  "pattern": "ramp",
+  "tier": "happy_path",
+  "assertions": [
+    "status < 500",
+    "p95 < 500"
+  ],
+  "fail_on_degraded": true
+}
+```
+
+Whenever `hubric.json` is present in your working directory, simply run:
+```bash
+hubric run
+```
+
+---
+
+## Performance Benchmarks & Apdex Scoring
+
+Hubric measures user satisfaction using the **Apdex (Application Performance Index)** standard:
+
+$$\text{Apdex} = \frac{\text{Satisfied} + \frac{\text{Tolerating}}{2}}{\text{Total Requests}}$$
+
+Where requests are categorized against target latency threshold $T$:
+- **Satisfied**: Response time $\le T$, zero HTTP error.
+- **Tolerating**: $T < \text{Response time} \le 4T$, zero HTTP error.
+- **Frustrated**: Response time $> 4T$ or HTTP failure ($5xx$, connection drops, non-validation $4xx$).
+
+### Target Latencies ($T$)
+- **Read routes (GET/HEAD/DELETE)**: $T = 300\text{ms}$, degraded $\le 999\text{ms}$.
+- **Write mutations (POST/PUT/PATCH)**: $T = 1000\text{ms}$, degraded $\le 3999\text{ms}$.
+- **Media / File Uploads**: $T = 5000\text{ms}$, degraded $\le 15000\text{ms}$.
+- **Generative AI & Streaming**: Specialized thresholds ($1000\text{ms}$ for SSE streams, $5000\text{ms}$ for chat, $15000\text{ms}$ for image diffusion).
+- **Dynamic SLA Scaling**: If observed single-request baseline latency $L_{\text{base}} \ge 1500\text{ms}$, thresholds scale proportionally ($2.0 \times L_{\text{base}}$ pass, $4.5 \times L_{\text{base}}$ degraded).
+
+### Suite Score vs. Suite Verdict (Weighting)
+- **Suite Apdex Score**: Arithmetic average across all *tested* endpoints:
+  $$\text{Suite Score} = \text{round}\left(\frac{\sum_{i=1}^N \text{score}_i}{N}\right)$$
+  Endpoints skipped by auth or classified as untested report `score: None` and do not skew the average.
+- **Suite Verdict**: Evaluated via strictest-gate policy:
+  - If any tested endpoint receives `FAILED` (e.g. error rate $\ge 5\%$ or $p95 > 4T$), the overall suite verdict is **`SUITE FAILED`**, ensuring critical service failures are never masked by high scores on secondary endpoints.
+  - If any endpoint is `DEGRADED`, the suite verdict is **`PARTIAL PASS`**.
+  - Only when all tested endpoints achieve `PASSED` does the suite receive **`ALL PASSED`**.
+
+---
+
+## Security Architecture
+
+Hubric is engineered under strict Application Security (AppSec) guidelines to ensure that testing untrusted or external APIs never compromises the developer's machine, credentials, or privacy.
+
+### 1. Atomic Kernel-Level Private Writes (`write_private`)
+- All secret-bearing configuration files (`~/.hubric/config.json`) and session tokens are written with **POSIX `0o600` (owner-only)** permissions at creation time (`os.open`).
+- **Windows ACL Hardening**: On Windows, where POSIX permissions are ignored, Hubric invokes native `icacls` to strip inherited directory permissions and explicitly restrict access to the active user SID only, eliminating file-access race conditions.
+
+### 2. Multi-Pass PII & Credential Redaction Engine
+- Before request telemetry is rendered to HTML, saved to disk, or sent to an LLM for root-cause diagnosis, it passes through recursive redaction:
+  - **Header & Token Sanitization**: Masks JWTs, Bearer tokens, PEM private keys, and Basic Auth headers.
+  - **Free-Text JSON/KV Scanners**: Linear-time scanners identify `"key": value` and `key=value` patterns even in truncated or malformed HTTP response bodies.
+  - **Bounded Redaction**: Strict word-boundary checks prevent false-positive masking on normal metric keys like `passed` or `pass_threshold`.
+
+### 3. Header & Cookie Smuggling Protection (`_safe_token_text`)
+- Prevents malicious or hostile OpenAPI specs from injecting unauthorized headers or smuggling cookies:
+  - Header names and auth prefixes are validated against strict RFC 7230 token specifications.
+  - Dangerous hop-by-hop and routing headers (`Host`, `Transfer-Encoding`, etc.) are explicitly blocked from being hijacked by an API schema.
+
+### 4. Terminal Injection Defenses (`terminal_safe`)
+- When printing server responses or error diagnostics to stdout, all ANSI/VT C0/C1 control characters are stripped. This neutralizes terminal escape sequence injection attacks from untrusted remote APIs.
+
+### 5. Provider Isolation
+- LLM API keys and AWS credentials configured for one provider (e.g. Groq) are strictly isolated and never forwarded if you switch providers via environment variable or CLI flag.
+
+---
+
+## Project Architecture
+
+```text
+hubric/
+├── auth/           # Auth scheme detection, interview wizard, and token pools
+├── cli/            # Click CLI commands (run, ci, payload, inspect, settings, trace)
+├── config/         # Declarative hubric.json/yaml loader
+├── core/           # Schema parser, LLM client, Pydantic synthesizers, security layer
+├── engine/         # Async runner, stateful user journeys, reservoir aggregator, dispatcher
+├── forecast/       # Predictive performance analyzer
+└── report/         # Modular HTML report generator, Apdex metrics, JUnit XML exporter
+```
+
+---
+
+## License
+
+Distributed under the MIT License. See `LICENSE` for more information.