infergauge 0.7.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- infergauge-0.7.3/LICENSE +90 -0
- infergauge-0.7.3/PKG-INFO +518 -0
- infergauge-0.7.3/README.md +494 -0
- infergauge-0.7.3/infergauge/__init__.py +7 -0
- infergauge-0.7.3/infergauge/__main__.py +10 -0
- infergauge-0.7.3/infergauge/agent.py +237 -0
- infergauge-0.7.3/infergauge/ci.py +105 -0
- infergauge-0.7.3/infergauge/cli.py +1031 -0
- infergauge-0.7.3/infergauge/config.py +172 -0
- infergauge-0.7.3/infergauge/engine.py +194 -0
- infergauge-0.7.3/infergauge/insights.py +258 -0
- infergauge-0.7.3/infergauge/licensing.py +361 -0
- infergauge-0.7.3/infergauge/metrics.py +482 -0
- infergauge-0.7.3/infergauge/providers.py +322 -0
- infergauge-0.7.3/infergauge/quality.py +67 -0
- infergauge-0.7.3/infergauge/report.py +490 -0
- infergauge-0.7.3/infergauge/server.py +160 -0
- infergauge-0.7.3/infergauge/static/dashboard.html +517 -0
- infergauge-0.7.3/infergauge/static/uplot.css +1 -0
- infergauge-0.7.3/infergauge/static/uplot.js +2 -0
- infergauge-0.7.3/infergauge/updatecheck.py +96 -0
- infergauge-0.7.3/infergauge/webapp/__init__.py +7 -0
- infergauge-0.7.3/infergauge/webapp/app.py +971 -0
- infergauge-0.7.3/infergauge/webapp/mailer.py +107 -0
- infergauge-0.7.3/infergauge/webapp/mongo_store.py +523 -0
- infergauge-0.7.3/infergauge/webapp/quickstart-email.html +182 -0
- infergauge-0.7.3/infergauge/webapp/static/console.html +739 -0
- infergauge-0.7.3/infergauge/webapp/static/login.html +327 -0
- infergauge-0.7.3/infergauge/webapp/store.py +639 -0
- infergauge-0.7.3/infergauge.egg-info/PKG-INFO +518 -0
- infergauge-0.7.3/infergauge.egg-info/SOURCES.txt +38 -0
- infergauge-0.7.3/infergauge.egg-info/dependency_links.txt +1 -0
- infergauge-0.7.3/infergauge.egg-info/entry_points.txt +3 -0
- infergauge-0.7.3/infergauge.egg-info/requires.txt +8 -0
- infergauge-0.7.3/infergauge.egg-info/top_level.txt +1 -0
- infergauge-0.7.3/pyproject.toml +38 -0
- infergauge-0.7.3/setup.cfg +4 -0
- infergauge-0.7.3/tests/test_console.py +1097 -0
- infergauge-0.7.3/tests/test_mongo_e2e.py +221 -0
- infergauge-0.7.3/tests/test_mongo_store.py +318 -0
infergauge-0.7.3/LICENSE
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
Business Source License 1.1
|
|
2
|
+
|
|
3
|
+
License text copyright (c) 2017 MariaDB Corporation Ab, All Rights Reserved.
|
|
4
|
+
"Business Source License" is a trademark of MariaDB Corporation Ab.
|
|
5
|
+
|
|
6
|
+
-----------------------------------------------------------------------------
|
|
7
|
+
|
|
8
|
+
Parameters
|
|
9
|
+
|
|
10
|
+
Licensor: InferGauge
|
|
11
|
+
Licensed Work: InferGauge
|
|
12
|
+
The Licensed Work is (c) 2026 InferGauge
|
|
13
|
+
Additional Use Grant: You may make production use of the Licensed Work,
|
|
14
|
+
provided that you do not offer it to third parties as a
|
|
15
|
+
hosted, managed, or embedded service whose value derives
|
|
16
|
+
substantially from the functionality of the Licensed
|
|
17
|
+
Work, and provided that you do not remove, disable, or
|
|
18
|
+
circumvent its licensing, entitlement, or quota
|
|
19
|
+
enforcement.
|
|
20
|
+
|
|
21
|
+
Change Date: 2030-08-01
|
|
22
|
+
|
|
23
|
+
Change License: Apache License, Version 2.0
|
|
24
|
+
|
|
25
|
+
-----------------------------------------------------------------------------
|
|
26
|
+
|
|
27
|
+
Terms
|
|
28
|
+
|
|
29
|
+
The Licensor hereby grants you the right to copy, modify, create derivative
|
|
30
|
+
works, redistribute, and make non-production use of the Licensed Work. The
|
|
31
|
+
Licensor may make an Additional Use Grant, above, permitting limited
|
|
32
|
+
production use.
|
|
33
|
+
|
|
34
|
+
Effective on the Change Date, or the fourth anniversary of the first publicly
|
|
35
|
+
available distribution of a specific version of the Licensed Work under this
|
|
36
|
+
License, whichever comes first, the Licensor hereby grants you rights under
|
|
37
|
+
the terms of the Change License, and the rights granted in the paragraph
|
|
38
|
+
above terminate.
|
|
39
|
+
|
|
40
|
+
If your use of the Licensed Work does not comply with the requirements
|
|
41
|
+
currently in effect as described in this License, you must purchase a
|
|
42
|
+
commercial license from the Licensor, its affiliated entities, or authorized
|
|
43
|
+
resellers, or you must refrain from using the Licensed Work.
|
|
44
|
+
|
|
45
|
+
All copies of the original and modified Licensed Work, and derivative works
|
|
46
|
+
of the Licensed Work, are subject to this License. This License applies
|
|
47
|
+
separately for each version of the Licensed Work and the Change Date may vary
|
|
48
|
+
for each version of the Licensed Work released by Licensor.
|
|
49
|
+
|
|
50
|
+
You must conspicuously display this License on each original or modified copy
|
|
51
|
+
of the Licensed Work. If you receive the Licensed Work in original or
|
|
52
|
+
modified form from a third party, the terms and conditions set forth in this
|
|
53
|
+
License apply to your use of that work.
|
|
54
|
+
|
|
55
|
+
Any use of the Licensed Work in violation of this License will automatically
|
|
56
|
+
terminate your rights under this License for the current and all other
|
|
57
|
+
versions of the Licensed Work.
|
|
58
|
+
|
|
59
|
+
This License does not grant you any right in any trademark or logo of
|
|
60
|
+
Licensor or its affiliates (provided that you may use a trademark or logo of
|
|
61
|
+
Licensor as expressly required by this License).
|
|
62
|
+
|
|
63
|
+
TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON
|
|
64
|
+
AN "AS IS" BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS,
|
|
65
|
+
EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF
|
|
66
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND
|
|
67
|
+
TITLE.
|
|
68
|
+
|
|
69
|
+
-----------------------------------------------------------------------------
|
|
70
|
+
|
|
71
|
+
Covenants of Licensor
|
|
72
|
+
|
|
73
|
+
In consideration of the right to use this License's text and the "Business
|
|
74
|
+
Source License" name and trademark, Licensor covenants to MariaDB, and to all
|
|
75
|
+
other recipients of the licensed work to be provided by Licensor:
|
|
76
|
+
|
|
77
|
+
1. To specify as the Change License the GPL Version 2.0 or any later version,
|
|
78
|
+
or a license that is compatible with GPL Version 2.0 or a later version,
|
|
79
|
+
where "compatible" means that software provided under the Change License
|
|
80
|
+
can be included in a program with software provided under GPL Version 2.0
|
|
81
|
+
or a later version. Licensor may specify additional Change Licenses
|
|
82
|
+
without limitation.
|
|
83
|
+
|
|
84
|
+
2. To either: (a) specify an additional grant of rights to use that does not
|
|
85
|
+
impose any additional restriction on the right granted in this License, as
|
|
86
|
+
the Additional Use Grant; or (b) insert the text "None".
|
|
87
|
+
|
|
88
|
+
3. To specify a Change Date.
|
|
89
|
+
|
|
90
|
+
4. Not to modify this License in any other way.
|
|
@@ -0,0 +1,518 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: infergauge
|
|
3
|
+
Version: 0.7.3
|
|
4
|
+
Summary: InferGauge - AI performance testing with a live dashboard: load, stress, spike, and endurance tests for LLM endpoints
|
|
5
|
+
License-Expression: BUSL-1.1
|
|
6
|
+
Project-URL: Homepage, https://infergauge.dev
|
|
7
|
+
Project-URL: Documentation, https://infergauge.dev/docs
|
|
8
|
+
Keywords: llm,load-testing,performance,ai,benchmarking
|
|
9
|
+
Classifier: Development Status :: 4 - Beta
|
|
10
|
+
Classifier: Intended Audience :: Developers
|
|
11
|
+
Classifier: Programming Language :: Python :: 3
|
|
12
|
+
Classifier: Topic :: Software Development :: Testing :: Traffic Generation
|
|
13
|
+
Requires-Python: >=3.10
|
|
14
|
+
Description-Content-Type: text/markdown
|
|
15
|
+
License-File: LICENSE
|
|
16
|
+
Requires-Dist: fastapi>=0.110
|
|
17
|
+
Requires-Dist: uvicorn>=0.29
|
|
18
|
+
Requires-Dist: httpx>=0.27
|
|
19
|
+
Requires-Dist: pyyaml>=6.0
|
|
20
|
+
Requires-Dist: cryptography>=42.0
|
|
21
|
+
Provides-Extra: mongo
|
|
22
|
+
Requires-Dist: pymongo>=4.6; extra == "mongo"
|
|
23
|
+
Dynamic: license-file
|
|
24
|
+
|
|
25
|
+
# InferGauge
|
|
26
|
+
|
|
27
|
+
<p align="center">
|
|
28
|
+
<img src="demo/infergauge-dashboard.gif" width="760"
|
|
29
|
+
alt="InferGauge live dashboard during a stress test: users ramp to 900, the saturation knee appears at 380, SLAs fail, quality drops from 97% to 71%">
|
|
30
|
+
</p>
|
|
31
|
+
<p align="center"><em>A stress test running live — the saturation knee at 380 users, SLAs failing at 800, quality collapsing under load.</em></p>
|
|
32
|
+
|
|
33
|
+
<p align="center"><img src="demo/quickstart.svg" alt="InferGauge quickstart: pip install, init, run" width="720"></p>
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
pip install infergauge && infergauge init -y && infergauge run
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
> Source-available under [BSL 1.1](LICENSE) — free to read, modify and run
|
|
40
|
+
> (including in production and CI); not to resell as a hosted service.
|
|
41
|
+
> Converts to Apache 2.0 in 2030. See [COMMERCIAL.md](docs/business/COMMERCIAL.md).
|
|
42
|
+
|
|
43
|
+
**See it without installing:**
|
|
44
|
+
- [`demo/infergauge-dashboard.mp4`](demo/infergauge-dashboard.mp4) — 24-second video of the
|
|
45
|
+
dashboard during a live stress test (also as a GIF, above)
|
|
46
|
+
- [`demo/infergauge-demo.html`](demo/infergauge-demo.html) — self-playing 100-second product
|
|
47
|
+
tour with narration, runs in any browser
|
|
48
|
+
- [`docs/DEMO_SCRIPT.md`](docs/DEMO_SCRIPT.md) — shot list for recording your own
|
|
49
|
+
- [`demo/render_video.py`](demo/render_video.py) — regenerates the video/GIF
|
|
50
|
+
|
|
51
|
+
Three commands. No API key needed. Your browser opens to a live dashboard while
|
|
52
|
+
InferGauge finds your AI application's breaking point.
|
|
53
|
+
|
|
54
|
+
**AI performance testing with a live dashboard.** Load, stress, spike, and
|
|
55
|
+
endurance tests for AI applications — real-time visibility into latency
|
|
56
|
+
(E2E / TTFT / inter-token), throughput, token usage, cost, goodput, and SLA
|
|
57
|
+
validation. CLI-first for CI/CD; the dashboard is the lens.
|
|
58
|
+
|
|
59
|
+
## Install
|
|
60
|
+
|
|
61
|
+
**Download and run (recommended)** - a single native binary, no Python needed.
|
|
62
|
+
Grab the file for your OS from the [latest release](https://github.com/Nexus-InferGauge/infergauge-releases/releases/latest), then:
|
|
63
|
+
|
|
64
|
+
```bash
|
|
65
|
+
./infergauge init -y && ./infergauge run # dashboard opens at localhost:8710
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Binaries are compiled to native code (Nuitka) and code-signed for Windows and
|
|
69
|
+
macOS, so there is no interpreter to install and no OS trust warning.
|
|
70
|
+
|
|
71
|
+
**Homebrew (macOS / Linux)**
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
brew install Nexus-InferGauge/infergauge/infergauge
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
**pip - for CI/CD pipelines.** Load tests belong in your deploy gate; the
|
|
78
|
+
wheel is the right channel for that:
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
pip install infergauge
|
|
82
|
+
infergauge run ci.yaml --headless --report out/ # exit code 0/1 by SLA
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
See `ci/github-actions.yml` for a ready-made workflow. Installing from the
|
|
86
|
+
downloaded wheel file works the same way: `pip install infergauge-0.7.0-py3-none-any.whl`.
|
|
87
|
+
|
|
88
|
+
|
|
89
|
+
## Web console (accounts, teams, licensing)
|
|
90
|
+
|
|
91
|
+
```bash
|
|
92
|
+
infergauge serve # http://localhost:8720
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
Team members create an account, sign in, configure a test, and watch it run —
|
|
96
|
+
no CLI, no YAML. Every tier limit is enforced on the server, so the browser
|
|
97
|
+
cannot bypass it.
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
# operator: create the signing secret once, keep it in your secret manager
|
|
101
|
+
export INFERGAUGE_FERNET_KEY=$(infergauge license keygen)
|
|
102
|
+
infergauge serve # the console can now validate license tokens
|
|
103
|
+
|
|
104
|
+
# mint a customer/deployment token
|
|
105
|
+
infergauge license generate --tier team --key-id acme-prod --expires-at 2027-01-01T00:00:00Z
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
Users paste that token into **License** in the console, or set
|
|
109
|
+
`INFERGAUGE_API_KEY` (alias `INFERGAUGE_LICENSE_KEY`) plus `INFERGAUGE_FERNET_KEY`
|
|
110
|
+
for the CLI. Check the effective tier any time with `infergauge license status`.
|
|
111
|
+
|
|
112
|
+
See [`docs/LICENSING.md`](docs/LICENSING.md) for the tier table and the hosted
|
|
113
|
+
deployment note.
|
|
114
|
+
|
|
115
|
+
## macOS install
|
|
116
|
+
|
|
117
|
+
The wheel is platform-independent (`py3-none-any`), so install failures on macOS are
|
|
118
|
+
always environment issues. This one sequence avoids all of them:
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
python3 --version # must be 3.10+ (macOS ships 3.9 - see below)
|
|
122
|
+
python3 -m venv ~/infergauge-env
|
|
123
|
+
source ~/infergauge-env/bin/activate
|
|
124
|
+
pip install ~/Downloads/infergauge-0.7.0-py3-none-any.whl
|
|
125
|
+
infergauge init -y && infergauge run
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
Re-activate later with `source ~/infergauge-env/bin/activate`.
|
|
129
|
+
|
|
130
|
+
| Error you saw | Cause | Fix |
|
|
131
|
+
|---|---|---|
|
|
132
|
+
| `requires a different Python: 3.9.6 not in '>=3.10'` | macOS system Python is 3.9 | `brew install python@3.12`, then use `python3.12 -m venv ...` (or install from python.org) |
|
|
133
|
+
| `error: externally-managed-environment` | Homebrew Python blocks global installs (PEP 668) | Use the venv above — do **not** use `--break-system-packages` |
|
|
134
|
+
| `pip: command not found` | macOS has no `pip` alias | Use `python3 -m pip` |
|
|
135
|
+
| `is not a valid wheel filename` | Safari renamed or unzipped the file | Re-download with "Download Linked File As…", keep the `.whl` name |
|
|
136
|
+
| `no such file or directory` | wrong path | `cd ~/Downloads` first, or drag the file into Terminal to paste its path |
|
|
137
|
+
|
|
138
|
+
## Quick start
|
|
139
|
+
|
|
140
|
+
```bash
|
|
141
|
+
infergauge init # interactive setup -> infergauge.yaml
|
|
142
|
+
infergauge run # run it; live dashboard opens automatically
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
Pick the **Simulator** provider during init to try InferGauge instantly with no
|
|
146
|
+
API key — it behaves like a real endpoint, degrading past a saturation point
|
|
147
|
+
and rate-limiting under quota pressure, so every chart and insight lights up.
|
|
148
|
+
|
|
149
|
+
Override anything from the command line:
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
infergauge run --users 500 --type load
|
|
153
|
+
infergauge run --users 1000 --type stress --duration 300
|
|
154
|
+
infergauge run prod-test.yaml --model gpt-4o-mini
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
Revisit past results any time:
|
|
158
|
+
|
|
159
|
+
```bash
|
|
160
|
+
infergauge runs # list saved runs (score + SLA verdict)
|
|
161
|
+
infergauge dashboard # replay the latest run in the dashboard
|
|
162
|
+
infergauge dashboard --run 20260717-011813
|
|
163
|
+
infergauge compare # delta table: last two runs
|
|
164
|
+
infergauge run --baseline latest --max-regression-pct 15 # CI regression gate
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
## Quality under load
|
|
168
|
+
|
|
169
|
+
InferGauge can sample responses during a load test and run deterministic quality
|
|
170
|
+
checks (valid JSON, required fields, contains/regex, length bounds) - measuring
|
|
171
|
+
whether answer quality degrades as concurrency rises. Failed quality samples
|
|
172
|
+
count against goodput, the `min_quality_pct` SLA gates CI, and the report gets
|
|
173
|
+
a quality-vs-load chart. Configure it with:
|
|
174
|
+
|
|
175
|
+
```yaml
|
|
176
|
+
quality:
|
|
177
|
+
sample_rate: 0.25
|
|
178
|
+
checks:
|
|
179
|
+
- type: valid_json
|
|
180
|
+
- type: json_required_fields
|
|
181
|
+
value: answer, confidence
|
|
182
|
+
- type: contains
|
|
183
|
+
value: order
|
|
184
|
+
sla:
|
|
185
|
+
min_quality_pct: 92
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
## Providers
|
|
189
|
+
|
|
190
|
+
| Provider | `provider.kind` | Notes |
|
|
191
|
+
|---|---|---|
|
|
192
|
+
| Simulator | `simulator` | no key needed; tunable saturation & rate limits |
|
|
193
|
+
| OpenAI | `openai-compatible` | `base_url: https://api.openai.com/v1` |
|
|
194
|
+
| Anthropic Claude | `anthropic` | native Messages API with streaming TTFT/ITL |
|
|
195
|
+
| Azure OpenAI | `openai-compatible` | point `base_url` at your resource |
|
|
196
|
+
| Google Gemini | `openai-compatible` | Gemini's OpenAI-compatible endpoint |
|
|
197
|
+
| Local (Ollama / vLLM / LM Studio) | `openai-compatible` | e.g. `http://localhost:11434/v1` |
|
|
198
|
+
|
|
199
|
+
`infergauge init` scaffolds the right block for each of these.
|
|
200
|
+
|
|
201
|
+
## CI/CD — run on your org's infrastructure
|
|
202
|
+
|
|
203
|
+
Run InferGauge from your CI runners instead of laptops: runners sit close to the
|
|
204
|
+
app (stable network, no Wi-Fi noise), have consistent hardware, and turn every
|
|
205
|
+
merge into a performance checkpoint.
|
|
206
|
+
|
|
207
|
+
```bash
|
|
208
|
+
# PR pipeline: gate against the main-branch baseline
|
|
209
|
+
infergauge run ci-test.yaml --headless \
|
|
210
|
+
--baseline main --max-regression-pct 15 \
|
|
211
|
+
--junit results/junit.xml --report results/
|
|
212
|
+
echo $? # 0 = SLAs met & no regression · 1 = breach or regression
|
|
213
|
+
|
|
214
|
+
# main-branch pipeline: refresh the baseline after merge
|
|
215
|
+
infergauge run ci-test.yaml --headless --save-baseline main
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
**Ready-made templates** (baseline caching, native test reports, artifacts):
|
|
219
|
+
`.github/workflows/perf-gate.yml` (GitHub Actions), `ci/gitlab-ci.yml`,
|
|
220
|
+
`ci/Jenkinsfile`, `ci/azure-pipelines.yml`.
|
|
221
|
+
|
|
222
|
+
**CI features:**
|
|
223
|
+
- `--junit FILE` — SLA checks + regression checks as JUnit test cases, rendered
|
|
224
|
+
natively by Jenkins, GitLab MR widgets, and Azure DevOps Tests tab
|
|
225
|
+
- `--save-baseline NAME` / `--baseline NAME` — named baselines in
|
|
226
|
+
`.infergauge/baselines/`; persist them across ephemeral runners with your CI
|
|
227
|
+
cache (all four templates do this)
|
|
228
|
+
- GitHub step summary — when `$GITHUB_STEP_SUMMARY` is set, a markdown scorecard
|
|
229
|
+
is posted to the job summary automatically
|
|
230
|
+
- `INFERGAUGE_*` env overrides — one committed config, many environments:
|
|
231
|
+
`INFERGAUGE_BASE_URL`, `INFERGAUGE_MODEL`, `INFERGAUGE_USERS`, `INFERGAUGE_DURATION_S`,
|
|
232
|
+
`INFERGAUGE_TEST_TYPE`, `INFERGAUGE_PROVIDER`, `INFERGAUGE_API_KEY_ENV` — so staging
|
|
233
|
+
and prod pipelines share a config and differ only in variables
|
|
234
|
+
- API keys come from your CI secret store via `api_key_env` — never in the repo
|
|
235
|
+
- No implicit network calls: `--headless` and CI environments (`CI` set) skip
|
|
236
|
+
the update-nag check on their own; set `INFERGAUGE_NO_UPDATE_CHECK=1`
|
|
237
|
+
anywhere else you want it off too
|
|
238
|
+
|
|
239
|
+
## What the dashboard shows
|
|
240
|
+
|
|
241
|
+
- **Performance score** (0-100) with an explainable component breakdown
|
|
242
|
+
(latency 30 / errors 25 / goodput 20 / token perf 15 / cost 10). A component is
|
|
243
|
+
graded **only when the config defines the SLA it needs** — InferGauge will not
|
|
244
|
+
invent a threshold you didn't choose and grade you against it. Ungraded
|
|
245
|
+
components show `no SLA`, are excluded from the average, and the score is
|
|
246
|
+
labelled **provisional** with the exact settings to add:
|
|
247
|
+
|
|
248
|
+
```
|
|
249
|
+
score 97/100 (healthy) PROVISIONAL - only 45% of weight graded
|
|
250
|
+
ungraded: set sla.p95_latency_ms
|
|
251
|
+
ungraded: set sla.itl_p95_ms
|
|
252
|
+
ungraded: set sla.max_cost_usd
|
|
253
|
+
```
|
|
254
|
+
- **KPI cards**: avg/p95/p99 latency, TTFT, inter-token latency; active users,
|
|
255
|
+
in-flight, error rate, **goodput** (% of requests meeting *all* SLOs);
|
|
256
|
+
token in/out and tokens/sec; spend, cost/request, projected monthly
|
|
257
|
+
- **Latency & load timeline** with SLA threshold line and event markers for
|
|
258
|
+
every manual intervention (results stay honest)
|
|
259
|
+
- **Saturation curve** — live p95-vs-users scatter; the knee is your ceiling
|
|
260
|
+
- **SLA validation panel**, **error taxonomy** (429 / 5xx / timeout /
|
|
261
|
+
connection + retries), and **rule-based insights** (saturation knee,
|
|
262
|
+
rate limiting, TTFT-vs-ITL bottleneck diagnosis, drift at steady load,
|
|
263
|
+
quality degradation under load, unreachable-endpoint and bad-model
|
|
264
|
+
diagnostics, small-sample percentile warnings, cost pressure)
|
|
265
|
+
- **Statistical honesty**: p95/p99 are flagged with `*` until there are ~100
|
|
266
|
+
samples, because a percentile from 8 requests is just the slowest request
|
|
267
|
+
- **Live controls**: adjust users or stop the test mid-run
|
|
268
|
+
- **Exports**: HTML report + JSON
|
|
269
|
+
|
|
270
|
+
## Accounts and the web console
|
|
271
|
+
|
|
272
|
+
InferGauge can run entirely standalone. Teams that want shared history, plan
|
|
273
|
+
management and a usage dashboard can also run the console:
|
|
274
|
+
|
|
275
|
+
```bash
|
|
276
|
+
infergauge serve # http://localhost:8720
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
**The console never executes tests.** It authorises runs, meters your plan and
|
|
280
|
+
stores results. Every test executes on your own machine, with your own API
|
|
281
|
+
keys, through the CLI acting as a local agent:
|
|
282
|
+
|
|
283
|
+
```
|
|
284
|
+
browser (console) your machine (agent)
|
|
285
|
+
───────────────── ────────────────────
|
|
286
|
+
sign up / sign in infergauge login --server <console>
|
|
287
|
+
build a config ──── yaml ───▶ infergauge run infergauge.yaml --sync
|
|
288
|
+
plan + quota check ◀─── authorise ──
|
|
289
|
+
watch live progress ◀─── progress ─── (test runs here, against your endpoint)
|
|
290
|
+
history + reports ◀─── summary ────
|
|
291
|
+
```
|
|
292
|
+
|
|
293
|
+
### Building a test in the console
|
|
294
|
+
|
|
295
|
+
The **New test** tab is a form that produces a ready-to-run config. It checks
|
|
296
|
+
the test against your plan *before* generating anything, so you find out about
|
|
297
|
+
a limit here rather than halfway through a run.
|
|
298
|
+
|
|
299
|
+
- **Provider presets** for OpenAI, Anthropic, Groq, Together AI, Fireworks AI
|
|
300
|
+
and Ollama fill in the right base URL, a sensible model and the conventional
|
|
301
|
+
API-key variable name. Anything else that speaks the OpenAI API works via
|
|
302
|
+
*Other OpenAI-compatible*.
|
|
303
|
+
- **Prompt** is yours to set. Token counts, TTFT and cost all scale with the
|
|
304
|
+
prompt, so a generic one gives you generic numbers — paste the prompt your
|
|
305
|
+
application actually sends. Multi-line is fine. Left blank, a sample is used.
|
|
306
|
+
- **Download infergauge.yaml** writes the config straight to disk (or copy it
|
|
307
|
+
and save it yourself). The file is generated in your browser: the config names
|
|
308
|
+
an environment variable, never the key itself.
|
|
309
|
+
|
|
310
|
+
Two SLA thresholds are not on the form — `itl_p95_ms` and `max_cost_usd`. Add
|
|
311
|
+
them by hand if you want a fully graded score rather than a provisional one
|
|
312
|
+
(see [Config reference](#config-reference)).
|
|
313
|
+
|
|
314
|
+
Why this shape:
|
|
315
|
+
|
|
316
|
+
- **Honest measurements.** Results reflect your infrastructure and your network,
|
|
317
|
+
not ours. A shared server would inject its own latency and noise.
|
|
318
|
+
- **Your provider spend stays yours.** Inference is billed to your account, and
|
|
319
|
+
we never hold your API keys.
|
|
320
|
+
- **Privacy by construction.** Only metrics and summaries are uploaded. Prompts,
|
|
321
|
+
responses, keys and private base URLs are stripped on the agent *and* again on
|
|
322
|
+
the server, so a modified client cannot push them either.
|
|
323
|
+
- **Quota you cannot fake.** Plan limits are metered server-side, because the
|
|
324
|
+
agent runs on hardware we do not control. Deleting local state does not grant
|
|
325
|
+
extra runs.
|
|
326
|
+
|
|
327
|
+
```bash
|
|
328
|
+
infergauge login --server https://console.example.com # once per machine
|
|
329
|
+
infergauge run infergauge.yaml --sync # runs locally, records centrally
|
|
330
|
+
infergauge whoami # tier, quota, account
|
|
331
|
+
infergauge logout
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
For CI, create an agent token in the console (Agents tab) and set
|
|
335
|
+
`INFERGAUGE_AGENT_TOKEN` in your pipeline instead of logging in interactively.
|
|
336
|
+
|
|
337
|
+
Deploying the console for a team? See [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md)
|
|
338
|
+
for TLS, email verification, systemd, backups and the security controls in place.
|
|
339
|
+
|
|
340
|
+
## Tests
|
|
341
|
+
|
|
342
|
+
```bash
|
|
343
|
+
python tests/regression.py # 119 checks: config, metrics, quality, insights,
|
|
344
|
+
# engine, providers (vs a mock endpoint), CLI,
|
|
345
|
+
# server, reports, CI exports, legacy compatibility
|
|
346
|
+
npm install jsdom --no-save
|
|
347
|
+
node tests/dashboard_dom.js # 20 checks: dashboard charts in a real DOM
|
|
348
|
+
python tests/test_console.py # 66 checks: licensing, accounts, agent API, quota
|
|
349
|
+
node tests/console_ui.js # 22 checks: console pages against a live server
|
|
350
|
+
# (start the console first: infergauge serve --port 8822)
|
|
351
|
+
|
|
352
|
+
pip install pymongo mongomock --break-system-packages
|
|
353
|
+
python tests/test_mongo_store.py # 14 checks: MongoStore against mongomock, no Atlas
|
|
354
|
+
# account or network access needed - see
|
|
355
|
+
# docs/MONGODB_BACKEND.md
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
Both run automatically in every CI template.
|
|
359
|
+
|
|
360
|
+
## Architecture
|
|
361
|
+
|
|
362
|
+
```
|
|
363
|
+
InferGauge CLI
|
|
364
|
+
init | run | dashboard | runs | validate
|
|
365
|
+
|
|
|
366
|
+
+--------------+---------------+
|
|
367
|
+
| |
|
|
368
|
+
Test Engine Metrics Engine
|
|
369
|
+
asyncio virtual users 1s window aggregates,
|
|
370
|
+
load/stress/spike/endurance exact percentiles, goodput,
|
|
371
|
+
| cost, health score, insights
|
|
372
|
+
| |
|
|
373
|
+
AI Workload Providers |
|
|
374
|
+
+---------+----------+----+ |
|
|
375
|
+
| | | | |
|
|
376
|
+
OpenAI Anthropic Azure Local FastAPI + SSE
|
|
377
|
+
Gemini (native) OpenAI (Ollama, |
|
|
378
|
+
(compat) vLLM) v
|
|
379
|
+
InferGauge Dashboard
|
|
380
|
+
Performance | Tokens | Cost | SLA
|
|
381
|
+
(+ report.html / report.json / CI exit code)
|
|
382
|
+
```
|
|
383
|
+
|
|
384
|
+
Runs are persisted to `.infergauge/runs/` so `infergauge dashboard` and
|
|
385
|
+
`infergauge runs` work across sessions.
|
|
386
|
+
|
|
387
|
+
## Config reference
|
|
388
|
+
|
|
389
|
+
See `examples/` for complete files. All keys:
|
|
390
|
+
|
|
391
|
+
```yaml
|
|
392
|
+
name: My Test
|
|
393
|
+
application: My AI App
|
|
394
|
+
test_type: stress # load | stress | spike | endurance
|
|
395
|
+
duration_s: 120
|
|
396
|
+
users: 1000
|
|
397
|
+
ramp_up_s: 30
|
|
398
|
+
think_time_s: 0.6
|
|
399
|
+
stress_step_users: 200 # stress staircase (optional)
|
|
400
|
+
stress_step_duration_s: 24
|
|
401
|
+
spike_baseline_users: 100 # spike shape (optional)
|
|
402
|
+
spike_at_s: 30
|
|
403
|
+
spike_duration_s: 20
|
|
404
|
+
|
|
405
|
+
provider:
|
|
406
|
+
kind: simulator # simulator | openai-compatible | anthropic
|
|
407
|
+
base_url: https://api.openai.com/v1
|
|
408
|
+
api_key_env: OPENAI_API_KEY
|
|
409
|
+
model: gpt-4o-mini
|
|
410
|
+
stream: true # enables TTFT / inter-token measurement
|
|
411
|
+
max_tokens: 200
|
|
412
|
+
prompt: "..."
|
|
413
|
+
system_prompt: ""
|
|
414
|
+
timeout_s: 60
|
|
415
|
+
sim_ttft_ms: 320 # simulator knobs
|
|
416
|
+
sim_itl_ms: 16
|
|
417
|
+
sim_output_tokens: 180
|
|
418
|
+
sim_input_tokens: 950
|
|
419
|
+
sim_saturation_users: 700
|
|
420
|
+
sim_rate_limit_users: 880
|
|
421
|
+
|
|
422
|
+
sla: # omit any you don't need
|
|
423
|
+
p95_latency_ms: 6000
|
|
424
|
+
avg_latency_ms: null
|
|
425
|
+
ttft_p95_ms: 900
|
|
426
|
+
itl_p95_ms: 60
|
|
427
|
+
error_rate_pct: 1.0
|
|
428
|
+
min_throughput_rps: null
|
|
429
|
+
max_cost_usd: 5.0
|
|
430
|
+
min_goodput_pct: 90
|
|
431
|
+
|
|
432
|
+
pricing: # USD per 1M tokens
|
|
433
|
+
input_per_1m: 2.0
|
|
434
|
+
output_per_1m: 8.0
|
|
435
|
+
|
|
436
|
+
projection: # optional: realistic monthly cost estimate
|
|
437
|
+
requests_per_day: 50000 # expected production volume (best option)
|
|
438
|
+
# peak_hours_per_day: 8 # alternative: sustain test rate N hours/day
|
|
439
|
+
days_per_month: 30.4
|
|
440
|
+
```
|
|
441
|
+
|
|
442
|
+
### How cost is calculated
|
|
443
|
+
|
|
444
|
+
```
|
|
445
|
+
total = input_tokens/1M × pricing.input_per_1m
|
|
446
|
+
+ output_tokens/1M × pricing.output_per_1m
|
|
447
|
+
per_req = total ÷ successful requests ← a direct measurement, trust this
|
|
448
|
+
hourly = total ÷ test_duration × 3600
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
Monthly cost is reported two ways, never conflated:
|
|
452
|
+
|
|
453
|
+
- **At expected volume** (when `projection.requests_per_day` is set) —
|
|
454
|
+
`per_req × requests_per_day × days_per_month`. This is the number to budget with.
|
|
455
|
+
- **Peak ceiling** (the fallback) — `hourly × 730`, i.e. this test's spend rate
|
|
456
|
+
sustained *every hour of the month*. Shown in amber and labelled `PEAK CEILING`
|
|
457
|
+
because a stress test extrapolated 24/7 is a worst case, not a forecast.
|
|
458
|
+
|
|
459
|
+
Both figures state the **request volume they imply**, so a dollar number is always
|
|
460
|
+
sanity-checkable at a glance:
|
|
461
|
+
|
|
462
|
+
```
|
|
463
|
+
projected $211,529.82/mo PEAK CEILING - implies 63,597,600 requests/month
|
|
464
|
+
set projection.requests_per_day for a realistic estimate
|
|
465
|
+
```
|
|
466
|
+
|
|
467
|
+
"Do we serve 64 million requests a month?" is a far easier question than staring
|
|
468
|
+
at a six-figure total.
|
|
469
|
+
|
|
470
|
+
Prices come from your config — nothing is looked up. Prompt-caching and batch
|
|
471
|
+
discounts are not modelled, so real bills are often lower. Failed requests are
|
|
472
|
+
counted as $0 (their tokens are unknown), so heavily-erroring runs may understate
|
|
473
|
+
slightly. A month is 365/12 = 30.4167 days = 730 hours (configurable via
|
|
474
|
+
`projection.days_per_month`), used by both projections.
|
|
475
|
+
|
|
476
|
+
**If the endpoint doesn't report usage**, InferGauge estimates tokens from text
|
|
477
|
+
length (~4 chars/token, ±20%), marks the cost `ESTIMATED`, and raises an insight —
|
|
478
|
+
rather than silently reporting $0 input cost, which is what a missing usage block
|
|
479
|
+
would otherwise produce.
|
|
480
|
+
|
|
481
|
+
Currency is rounded to 4 significant digits, not a fixed number of decimals, so
|
|
482
|
+
cheap-model costs survive: a $0.00012864 test reports `$0.00012864`, not `$0.0001`.
|
|
483
|
+
When cost per request is below a tenth of a cent, a per-1,000-requests figure is
|
|
484
|
+
shown alongside it. Displayed values stay internally consistent — cost/request ×
|
|
485
|
+
implied requests reproduces the monthly total.
|
|
486
|
+
|
|
487
|
+
## Roadmap
|
|
488
|
+
|
|
489
|
+
- Prompt-mix workloads (weighted pools, multi-turn conversations)
|
|
490
|
+
- Multi-provider comparison runs
|
|
491
|
+
- Distributed load generation
|
|
492
|
+
- Prometheus / OpenTelemetry export
|
|
493
|
+
- **Admin dashboard for the console**: today, discovering who signed up
|
|
494
|
+
requesting a paid plan and minting/sending their license is an entirely
|
|
495
|
+
manual, inbound process (see `docs/ADMIN_GUIDE.md`) — there is no built-in
|
|
496
|
+
view listing accounts, requested plans, or license status across users.
|
|
497
|
+
Worth prioritizing before volume makes the manual loop unmanageable.
|
|
498
|
+
|
|
499
|
+
## License
|
|
500
|
+
|
|
501
|
+
InferGauge is **source-available** under the [Business Source License 1.1](LICENSE),
|
|
502
|
+
not open source.
|
|
503
|
+
|
|
504
|
+
- **Allowed:** reading and auditing the source, internal and production use,
|
|
505
|
+
CI/CD gating, testing your own or your employer's systems, and consulting
|
|
506
|
+
work delivered to your clients.
|
|
507
|
+
- **Not allowed:** offering InferGauge to third parties as a hosted or managed
|
|
508
|
+
service, embedding it in a product whose value comes mainly from InferGauge,
|
|
509
|
+
or removing/bypassing its tier and quota enforcement.
|
|
510
|
+
|
|
511
|
+
Each released version converts to the **Apache License 2.0 on 2030-08-01** (or
|
|
512
|
+
four years after that version was published, whichever comes first).
|
|
513
|
+
|
|
514
|
+
See [COMMERCIAL.md](docs/business/COMMERCIAL.md) for the full picture and commercial licensing.
|
|
515
|
+
Bundled uPlot is MIT; runtime dependencies keep their own licenses.
|
|
516
|
+
|
|
517
|
+
**"InferGauge" is a trademark of InferGauge.** The license grants no trademark rights, and pre-release material shared under NDA may not be redistributed. See [NOTICE](NOTICE) for the full copyright, trademark, and confidentiality notice.
|
|
518
|
+
|