infergauge 0.7.3__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. infergauge-0.7.3/LICENSE +90 -0
  2. infergauge-0.7.3/PKG-INFO +518 -0
  3. infergauge-0.7.3/README.md +494 -0
  4. infergauge-0.7.3/infergauge/__init__.py +7 -0
  5. infergauge-0.7.3/infergauge/__main__.py +10 -0
  6. infergauge-0.7.3/infergauge/agent.py +237 -0
  7. infergauge-0.7.3/infergauge/ci.py +105 -0
  8. infergauge-0.7.3/infergauge/cli.py +1031 -0
  9. infergauge-0.7.3/infergauge/config.py +172 -0
  10. infergauge-0.7.3/infergauge/engine.py +194 -0
  11. infergauge-0.7.3/infergauge/insights.py +258 -0
  12. infergauge-0.7.3/infergauge/licensing.py +361 -0
  13. infergauge-0.7.3/infergauge/metrics.py +482 -0
  14. infergauge-0.7.3/infergauge/providers.py +322 -0
  15. infergauge-0.7.3/infergauge/quality.py +67 -0
  16. infergauge-0.7.3/infergauge/report.py +490 -0
  17. infergauge-0.7.3/infergauge/server.py +160 -0
  18. infergauge-0.7.3/infergauge/static/dashboard.html +517 -0
  19. infergauge-0.7.3/infergauge/static/uplot.css +1 -0
  20. infergauge-0.7.3/infergauge/static/uplot.js +2 -0
  21. infergauge-0.7.3/infergauge/updatecheck.py +96 -0
  22. infergauge-0.7.3/infergauge/webapp/__init__.py +7 -0
  23. infergauge-0.7.3/infergauge/webapp/app.py +971 -0
  24. infergauge-0.7.3/infergauge/webapp/mailer.py +107 -0
  25. infergauge-0.7.3/infergauge/webapp/mongo_store.py +523 -0
  26. infergauge-0.7.3/infergauge/webapp/quickstart-email.html +182 -0
  27. infergauge-0.7.3/infergauge/webapp/static/console.html +739 -0
  28. infergauge-0.7.3/infergauge/webapp/static/login.html +327 -0
  29. infergauge-0.7.3/infergauge/webapp/store.py +639 -0
  30. infergauge-0.7.3/infergauge.egg-info/PKG-INFO +518 -0
  31. infergauge-0.7.3/infergauge.egg-info/SOURCES.txt +38 -0
  32. infergauge-0.7.3/infergauge.egg-info/dependency_links.txt +1 -0
  33. infergauge-0.7.3/infergauge.egg-info/entry_points.txt +3 -0
  34. infergauge-0.7.3/infergauge.egg-info/requires.txt +8 -0
  35. infergauge-0.7.3/infergauge.egg-info/top_level.txt +1 -0
  36. infergauge-0.7.3/pyproject.toml +38 -0
  37. infergauge-0.7.3/setup.cfg +4 -0
  38. infergauge-0.7.3/tests/test_console.py +1097 -0
  39. infergauge-0.7.3/tests/test_mongo_e2e.py +221 -0
  40. infergauge-0.7.3/tests/test_mongo_store.py +318 -0
@@ -0,0 +1,90 @@
1
+ Business Source License 1.1
2
+
3
+ License text copyright (c) 2017 MariaDB Corporation Ab, All Rights Reserved.
4
+ "Business Source License" is a trademark of MariaDB Corporation Ab.
5
+
6
+ -----------------------------------------------------------------------------
7
+
8
+ Parameters
9
+
10
+ Licensor: InferGauge
11
+ Licensed Work: InferGauge
12
+ The Licensed Work is (c) 2026 InferGauge
13
+ Additional Use Grant: You may make production use of the Licensed Work,
14
+ provided that you do not offer it to third parties as a
15
+ hosted, managed, or embedded service whose value derives
16
+ substantially from the functionality of the Licensed
17
+ Work, and provided that you do not remove, disable, or
18
+ circumvent its licensing, entitlement, or quota
19
+ enforcement.
20
+
21
+ Change Date: 2030-08-01
22
+
23
+ Change License: Apache License, Version 2.0
24
+
25
+ -----------------------------------------------------------------------------
26
+
27
+ Terms
28
+
29
+ The Licensor hereby grants you the right to copy, modify, create derivative
30
+ works, redistribute, and make non-production use of the Licensed Work. The
31
+ Licensor may make an Additional Use Grant, above, permitting limited
32
+ production use.
33
+
34
+ Effective on the Change Date, or the fourth anniversary of the first publicly
35
+ available distribution of a specific version of the Licensed Work under this
36
+ License, whichever comes first, the Licensor hereby grants you rights under
37
+ the terms of the Change License, and the rights granted in the paragraph
38
+ above terminate.
39
+
40
+ If your use of the Licensed Work does not comply with the requirements
41
+ currently in effect as described in this License, you must purchase a
42
+ commercial license from the Licensor, its affiliated entities, or authorized
43
+ resellers, or you must refrain from using the Licensed Work.
44
+
45
+ All copies of the original and modified Licensed Work, and derivative works
46
+ of the Licensed Work, are subject to this License. This License applies
47
+ separately for each version of the Licensed Work and the Change Date may vary
48
+ for each version of the Licensed Work released by Licensor.
49
+
50
+ You must conspicuously display this License on each original or modified copy
51
+ of the Licensed Work. If you receive the Licensed Work in original or
52
+ modified form from a third party, the terms and conditions set forth in this
53
+ License apply to your use of that work.
54
+
55
+ Any use of the Licensed Work in violation of this License will automatically
56
+ terminate your rights under this License for the current and all other
57
+ versions of the Licensed Work.
58
+
59
+ This License does not grant you any right in any trademark or logo of
60
+ Licensor or its affiliates (provided that you may use a trademark or logo of
61
+ Licensor as expressly required by this License).
62
+
63
+ TO THE EXTENT PERMITTED BY APPLICABLE LAW, THE LICENSED WORK IS PROVIDED ON
64
+ AN "AS IS" BASIS. LICENSOR HEREBY DISCLAIMS ALL WARRANTIES AND CONDITIONS,
65
+ EXPRESS OR IMPLIED, INCLUDING (WITHOUT LIMITATION) WARRANTIES OF
66
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT, AND
67
+ TITLE.
68
+
69
+ -----------------------------------------------------------------------------
70
+
71
+ Covenants of Licensor
72
+
73
+ In consideration of the right to use this License's text and the "Business
74
+ Source License" name and trademark, Licensor covenants to MariaDB, and to all
75
+ other recipients of the licensed work to be provided by Licensor:
76
+
77
+ 1. To specify as the Change License the GPL Version 2.0 or any later version,
78
+ or a license that is compatible with GPL Version 2.0 or a later version,
79
+ where "compatible" means that software provided under the Change License
80
+ can be included in a program with software provided under GPL Version 2.0
81
+ or a later version. Licensor may specify additional Change Licenses
82
+ without limitation.
83
+
84
+ 2. To either: (a) specify an additional grant of rights to use that does not
85
+ impose any additional restriction on the right granted in this License, as
86
+ the Additional Use Grant; or (b) insert the text "None".
87
+
88
+ 3. To specify a Change Date.
89
+
90
+ 4. Not to modify this License in any other way.
@@ -0,0 +1,518 @@
1
+ Metadata-Version: 2.4
2
+ Name: infergauge
3
+ Version: 0.7.3
4
+ Summary: InferGauge - AI performance testing with a live dashboard: load, stress, spike, and endurance tests for LLM endpoints
5
+ License-Expression: BUSL-1.1
6
+ Project-URL: Homepage, https://infergauge.dev
7
+ Project-URL: Documentation, https://infergauge.dev/docs
8
+ Keywords: llm,load-testing,performance,ai,benchmarking
9
+ Classifier: Development Status :: 4 - Beta
10
+ Classifier: Intended Audience :: Developers
11
+ Classifier: Programming Language :: Python :: 3
12
+ Classifier: Topic :: Software Development :: Testing :: Traffic Generation
13
+ Requires-Python: >=3.10
14
+ Description-Content-Type: text/markdown
15
+ License-File: LICENSE
16
+ Requires-Dist: fastapi>=0.110
17
+ Requires-Dist: uvicorn>=0.29
18
+ Requires-Dist: httpx>=0.27
19
+ Requires-Dist: pyyaml>=6.0
20
+ Requires-Dist: cryptography>=42.0
21
+ Provides-Extra: mongo
22
+ Requires-Dist: pymongo>=4.6; extra == "mongo"
23
+ Dynamic: license-file
24
+
25
+ # InferGauge
26
+
27
+ <p align="center">
28
+ <img src="demo/infergauge-dashboard.gif" width="760"
29
+ alt="InferGauge live dashboard during a stress test: users ramp to 900, the saturation knee appears at 380, SLAs fail, quality drops from 97% to 71%">
30
+ </p>
31
+ <p align="center"><em>A stress test running live — the saturation knee at 380 users, SLAs failing at 800, quality collapsing under load.</em></p>
32
+
33
+ <p align="center"><img src="demo/quickstart.svg" alt="InferGauge quickstart: pip install, init, run" width="720"></p>
34
+
35
+ ```bash
36
+ pip install infergauge && infergauge init -y && infergauge run
37
+ ```
38
+
39
+ > Source-available under [BSL 1.1](LICENSE) — free to read, modify and run
40
+ > (including in production and CI); not to resell as a hosted service.
41
+ > Converts to Apache 2.0 in 2030. See [COMMERCIAL.md](docs/business/COMMERCIAL.md).
42
+
43
+ **See it without installing:**
44
+ - [`demo/infergauge-dashboard.mp4`](demo/infergauge-dashboard.mp4) — 24-second video of the
45
+ dashboard during a live stress test (also as a GIF, above)
46
+ - [`demo/infergauge-demo.html`](demo/infergauge-demo.html) — self-playing 100-second product
47
+ tour with narration, runs in any browser
48
+ - [`docs/DEMO_SCRIPT.md`](docs/DEMO_SCRIPT.md) — shot list for recording your own
49
+ - [`demo/render_video.py`](demo/render_video.py) — regenerates the video/GIF
50
+
51
+ Three commands. No API key needed. Your browser opens to a live dashboard while
52
+ InferGauge finds your AI application's breaking point.
53
+
54
+ **AI performance testing with a live dashboard.** Load, stress, spike, and
55
+ endurance tests for AI applications — real-time visibility into latency
56
+ (E2E / TTFT / inter-token), throughput, token usage, cost, goodput, and SLA
57
+ validation. CLI-first for CI/CD; the dashboard is the lens.
58
+
59
+ ## Install
60
+
61
+ **Download and run (recommended)** - a single native binary, no Python needed.
62
+ Grab the file for your OS from the [latest release](https://github.com/Nexus-InferGauge/infergauge-releases/releases/latest), then:
63
+
64
+ ```bash
65
+ ./infergauge init -y && ./infergauge run # dashboard opens at localhost:8710
66
+ ```
67
+
68
+ Binaries are compiled to native code (Nuitka) and code-signed for Windows and
69
+ macOS, so there is no interpreter to install and no OS trust warning.
70
+
71
+ **Homebrew (macOS / Linux)**
72
+
73
+ ```bash
74
+ brew install Nexus-InferGauge/infergauge/infergauge
75
+ ```
76
+
77
+ **pip - for CI/CD pipelines.** Load tests belong in your deploy gate; the
78
+ wheel is the right channel for that:
79
+
80
+ ```bash
81
+ pip install infergauge
82
+ infergauge run ci.yaml --headless --report out/ # exit code 0/1 by SLA
83
+ ```
84
+
85
+ See `ci/github-actions.yml` for a ready-made workflow. Installing from the
86
+ downloaded wheel file works the same way: `pip install infergauge-0.7.0-py3-none-any.whl`.
87
+
88
+
89
+ ## Web console (accounts, teams, licensing)
90
+
91
+ ```bash
92
+ infergauge serve # http://localhost:8720
93
+ ```
94
+
95
+ Team members create an account, sign in, configure a test, and watch it run —
96
+ no CLI, no YAML. Every tier limit is enforced on the server, so the browser
97
+ cannot bypass it.
98
+
99
+ ```bash
100
+ # operator: create the signing secret once, keep it in your secret manager
101
+ export INFERGAUGE_FERNET_KEY=$(infergauge license keygen)
102
+ infergauge serve # the console can now validate license tokens
103
+
104
+ # mint a customer/deployment token
105
+ infergauge license generate --tier team --key-id acme-prod --expires-at 2027-01-01T00:00:00Z
106
+ ```
107
+
108
+ Users paste that token into **License** in the console, or set
109
+ `INFERGAUGE_API_KEY` (alias `INFERGAUGE_LICENSE_KEY`) plus `INFERGAUGE_FERNET_KEY`
110
+ for the CLI. Check the effective tier any time with `infergauge license status`.
111
+
112
+ See [`docs/LICENSING.md`](docs/LICENSING.md) for the tier table and the hosted
113
+ deployment note.
114
+
115
+ ## macOS install
116
+
117
+ The wheel is platform-independent (`py3-none-any`), so install failures on macOS are
118
+ always environment issues. This one sequence avoids all of them:
119
+
120
+ ```bash
121
+ python3 --version # must be 3.10+ (macOS ships 3.9 - see below)
122
+ python3 -m venv ~/infergauge-env
123
+ source ~/infergauge-env/bin/activate
124
+ pip install ~/Downloads/infergauge-0.7.0-py3-none-any.whl
125
+ infergauge init -y && infergauge run
126
+ ```
127
+
128
+ Re-activate later with `source ~/infergauge-env/bin/activate`.
129
+
130
+ | Error you saw | Cause | Fix |
131
+ |---|---|---|
132
+ | `requires a different Python: 3.9.6 not in '>=3.10'` | macOS system Python is 3.9 | `brew install python@3.12`, then use `python3.12 -m venv ...` (or install from python.org) |
133
+ | `error: externally-managed-environment` | Homebrew Python blocks global installs (PEP 668) | Use the venv above — do **not** use `--break-system-packages` |
134
+ | `pip: command not found` | macOS has no `pip` alias | Use `python3 -m pip` |
135
+ | `is not a valid wheel filename` | Safari renamed or unzipped the file | Re-download with "Download Linked File As…", keep the `.whl` name |
136
+ | `no such file or directory` | wrong path | `cd ~/Downloads` first, or drag the file into Terminal to paste its path |
137
+
138
+ ## Quick start
139
+
140
+ ```bash
141
+ infergauge init # interactive setup -> infergauge.yaml
142
+ infergauge run # run it; live dashboard opens automatically
143
+ ```
144
+
145
+ Pick the **Simulator** provider during init to try InferGauge instantly with no
146
+ API key — it behaves like a real endpoint, degrading past a saturation point
147
+ and rate-limiting under quota pressure, so every chart and insight lights up.
148
+
149
+ Override anything from the command line:
150
+
151
+ ```bash
152
+ infergauge run --users 500 --type load
153
+ infergauge run --users 1000 --type stress --duration 300
154
+ infergauge run prod-test.yaml --model gpt-4o-mini
155
+ ```
156
+
157
+ Revisit past results any time:
158
+
159
+ ```bash
160
+ infergauge runs # list saved runs (score + SLA verdict)
161
+ infergauge dashboard # replay the latest run in the dashboard
162
+ infergauge dashboard --run 20260717-011813
163
+ infergauge compare # delta table: last two runs
164
+ infergauge run --baseline latest --max-regression-pct 15 # CI regression gate
165
+ ```
166
+
167
+ ## Quality under load
168
+
169
+ InferGauge can sample responses during a load test and run deterministic quality
170
+ checks (valid JSON, required fields, contains/regex, length bounds) - measuring
171
+ whether answer quality degrades as concurrency rises. Failed quality samples
172
+ count against goodput, the `min_quality_pct` SLA gates CI, and the report gets
173
+ a quality-vs-load chart. Configure it with:
174
+
175
+ ```yaml
176
+ quality:
177
+ sample_rate: 0.25
178
+ checks:
179
+ - type: valid_json
180
+ - type: json_required_fields
181
+ value: answer, confidence
182
+ - type: contains
183
+ value: order
184
+ sla:
185
+ min_quality_pct: 92
186
+ ```
187
+
188
+ ## Providers
189
+
190
+ | Provider | `provider.kind` | Notes |
191
+ |---|---|---|
192
+ | Simulator | `simulator` | no key needed; tunable saturation & rate limits |
193
+ | OpenAI | `openai-compatible` | `base_url: https://api.openai.com/v1` |
194
+ | Anthropic Claude | `anthropic` | native Messages API with streaming TTFT/ITL |
195
+ | Azure OpenAI | `openai-compatible` | point `base_url` at your resource |
196
+ | Google Gemini | `openai-compatible` | Gemini's OpenAI-compatible endpoint |
197
+ | Local (Ollama / vLLM / LM Studio) | `openai-compatible` | e.g. `http://localhost:11434/v1` |
198
+
199
+ `infergauge init` scaffolds the right block for each of these.
200
+
201
+ ## CI/CD — run on your org's infrastructure
202
+
203
+ Run InferGauge from your CI runners instead of laptops: runners sit close to the
204
+ app (stable network, no Wi-Fi noise), have consistent hardware, and turn every
205
+ merge into a performance checkpoint.
206
+
207
+ ```bash
208
+ # PR pipeline: gate against the main-branch baseline
209
+ infergauge run ci-test.yaml --headless \
210
+ --baseline main --max-regression-pct 15 \
211
+ --junit results/junit.xml --report results/
212
+ echo $? # 0 = SLAs met & no regression · 1 = breach or regression
213
+
214
+ # main-branch pipeline: refresh the baseline after merge
215
+ infergauge run ci-test.yaml --headless --save-baseline main
216
+ ```
217
+
218
+ **Ready-made templates** (baseline caching, native test reports, artifacts):
219
+ `.github/workflows/perf-gate.yml` (GitHub Actions), `ci/gitlab-ci.yml`,
220
+ `ci/Jenkinsfile`, `ci/azure-pipelines.yml`.
221
+
222
+ **CI features:**
223
+ - `--junit FILE` — SLA checks + regression checks as JUnit test cases, rendered
224
+ natively by Jenkins, GitLab MR widgets, and Azure DevOps Tests tab
225
+ - `--save-baseline NAME` / `--baseline NAME` — named baselines in
226
+ `.infergauge/baselines/`; persist them across ephemeral runners with your CI
227
+ cache (all four templates do this)
228
+ - GitHub step summary — when `$GITHUB_STEP_SUMMARY` is set, a markdown scorecard
229
+ is posted to the job summary automatically
230
+ - `INFERGAUGE_*` env overrides — one committed config, many environments:
231
+ `INFERGAUGE_BASE_URL`, `INFERGAUGE_MODEL`, `INFERGAUGE_USERS`, `INFERGAUGE_DURATION_S`,
232
+ `INFERGAUGE_TEST_TYPE`, `INFERGAUGE_PROVIDER`, `INFERGAUGE_API_KEY_ENV` — so staging
233
+ and prod pipelines share a config and differ only in variables
234
+ - API keys come from your CI secret store via `api_key_env` — never in the repo
235
+ - No implicit network calls: `--headless` and CI environments (`CI` set) skip
236
+ the update-nag check on their own; set `INFERGAUGE_NO_UPDATE_CHECK=1`
237
+ anywhere else you want it off too
238
+
239
+ ## What the dashboard shows
240
+
241
+ - **Performance score** (0-100) with an explainable component breakdown
242
+ (latency 30 / errors 25 / goodput 20 / token perf 15 / cost 10). A component is
243
+ graded **only when the config defines the SLA it needs** — InferGauge will not
244
+ invent a threshold you didn't choose and grade you against it. Ungraded
245
+ components show `no SLA`, are excluded from the average, and the score is
246
+ labelled **provisional** with the exact settings to add:
247
+
248
+ ```
249
+ score 97/100 (healthy) PROVISIONAL - only 45% of weight graded
250
+ ungraded: set sla.p95_latency_ms
251
+ ungraded: set sla.itl_p95_ms
252
+ ungraded: set sla.max_cost_usd
253
+ ```
254
+ - **KPI cards**: avg/p95/p99 latency, TTFT, inter-token latency; active users,
255
+ in-flight, error rate, **goodput** (% of requests meeting *all* SLOs);
256
+ token in/out and tokens/sec; spend, cost/request, projected monthly
257
+ - **Latency & load timeline** with SLA threshold line and event markers for
258
+ every manual intervention (results stay honest)
259
+ - **Saturation curve** — live p95-vs-users scatter; the knee is your ceiling
260
+ - **SLA validation panel**, **error taxonomy** (429 / 5xx / timeout /
261
+ connection + retries), and **rule-based insights** (saturation knee,
262
+ rate limiting, TTFT-vs-ITL bottleneck diagnosis, drift at steady load,
263
+ quality degradation under load, unreachable-endpoint and bad-model
264
+ diagnostics, small-sample percentile warnings, cost pressure)
265
+ - **Statistical honesty**: p95/p99 are flagged with `*` until there are ~100
266
+ samples, because a percentile from 8 requests is just the slowest request
267
+ - **Live controls**: adjust users or stop the test mid-run
268
+ - **Exports**: HTML report + JSON
269
+
270
+ ## Accounts and the web console
271
+
272
+ InferGauge can run entirely standalone. Teams that want shared history, plan
273
+ management and a usage dashboard can also run the console:
274
+
275
+ ```bash
276
+ infergauge serve # http://localhost:8720
277
+ ```
278
+
279
+ **The console never executes tests.** It authorises runs, meters your plan and
280
+ stores results. Every test executes on your own machine, with your own API
281
+ keys, through the CLI acting as a local agent:
282
+
283
+ ```
284
+ browser (console) your machine (agent)
285
+ ───────────────── ────────────────────
286
+ sign up / sign in infergauge login --server <console>
287
+ build a config ──── yaml ───▶ infergauge run infergauge.yaml --sync
288
+ plan + quota check ◀─── authorise ──
289
+ watch live progress ◀─── progress ─── (test runs here, against your endpoint)
290
+ history + reports ◀─── summary ────
291
+ ```
292
+
293
+ ### Building a test in the console
294
+
295
+ The **New test** tab is a form that produces a ready-to-run config. It checks
296
+ the test against your plan *before* generating anything, so you find out about
297
+ a limit here rather than halfway through a run.
298
+
299
+ - **Provider presets** for OpenAI, Anthropic, Groq, Together AI, Fireworks AI
300
+ and Ollama fill in the right base URL, a sensible model and the conventional
301
+ API-key variable name. Anything else that speaks the OpenAI API works via
302
+ *Other OpenAI-compatible*.
303
+ - **Prompt** is yours to set. Token counts, TTFT and cost all scale with the
304
+ prompt, so a generic one gives you generic numbers — paste the prompt your
305
+ application actually sends. Multi-line is fine. Left blank, a sample is used.
306
+ - **Download infergauge.yaml** writes the config straight to disk (or copy it
307
+ and save it yourself). The file is generated in your browser: the config names
308
+ an environment variable, never the key itself.
309
+
310
+ Two SLA thresholds are not on the form — `itl_p95_ms` and `max_cost_usd`. Add
311
+ them by hand if you want a fully graded score rather than a provisional one
312
+ (see [Config reference](#config-reference)).
313
+
314
+ Why this shape:
315
+
316
+ - **Honest measurements.** Results reflect your infrastructure and your network,
317
+ not ours. A shared server would inject its own latency and noise.
318
+ - **Your provider spend stays yours.** Inference is billed to your account, and
319
+ we never hold your API keys.
320
+ - **Privacy by construction.** Only metrics and summaries are uploaded. Prompts,
321
+ responses, keys and private base URLs are stripped on the agent *and* again on
322
+ the server, so a modified client cannot push them either.
323
+ - **Quota you cannot fake.** Plan limits are metered server-side, because the
324
+ agent runs on hardware we do not control. Deleting local state does not grant
325
+ extra runs.
326
+
327
+ ```bash
328
+ infergauge login --server https://console.example.com # once per machine
329
+ infergauge run infergauge.yaml --sync # runs locally, records centrally
330
+ infergauge whoami # tier, quota, account
331
+ infergauge logout
332
+ ```
333
+
334
+ For CI, create an agent token in the console (Agents tab) and set
335
+ `INFERGAUGE_AGENT_TOKEN` in your pipeline instead of logging in interactively.
336
+
337
+ Deploying the console for a team? See [`docs/DEPLOYMENT.md`](docs/DEPLOYMENT.md)
338
+ for TLS, email verification, systemd, backups and the security controls in place.
339
+
340
+ ## Tests
341
+
342
+ ```bash
343
+ python tests/regression.py # 119 checks: config, metrics, quality, insights,
344
+ # engine, providers (vs a mock endpoint), CLI,
345
+ # server, reports, CI exports, legacy compatibility
346
+ npm install jsdom --no-save
347
+ node tests/dashboard_dom.js # 20 checks: dashboard charts in a real DOM
348
+ python tests/test_console.py # 66 checks: licensing, accounts, agent API, quota
349
+ node tests/console_ui.js # 22 checks: console pages against a live server
350
+ # (start the console first: infergauge serve --port 8822)
351
+
352
+ pip install pymongo mongomock --break-system-packages
353
+ python tests/test_mongo_store.py # 14 checks: MongoStore against mongomock, no Atlas
354
+ # account or network access needed - see
355
+ # docs/MONGODB_BACKEND.md
356
+ ```
357
+
358
+ Both run automatically in every CI template.
359
+
360
+ ## Architecture
361
+
362
+ ```
363
+ InferGauge CLI
364
+ init | run | dashboard | runs | validate
365
+ |
366
+ +--------------+---------------+
367
+ | |
368
+ Test Engine Metrics Engine
369
+ asyncio virtual users 1s window aggregates,
370
+ load/stress/spike/endurance exact percentiles, goodput,
371
+ | cost, health score, insights
372
+ | |
373
+ AI Workload Providers |
374
+ +---------+----------+----+ |
375
+ | | | | |
376
+ OpenAI Anthropic Azure Local FastAPI + SSE
377
+ Gemini (native) OpenAI (Ollama, |
378
+ (compat) vLLM) v
379
+ InferGauge Dashboard
380
+ Performance | Tokens | Cost | SLA
381
+ (+ report.html / report.json / CI exit code)
382
+ ```
383
+
384
+ Runs are persisted to `.infergauge/runs/` so `infergauge dashboard` and
385
+ `infergauge runs` work across sessions.
386
+
387
+ ## Config reference
388
+
389
+ See `examples/` for complete files. All keys:
390
+
391
+ ```yaml
392
+ name: My Test
393
+ application: My AI App
394
+ test_type: stress # load | stress | spike | endurance
395
+ duration_s: 120
396
+ users: 1000
397
+ ramp_up_s: 30
398
+ think_time_s: 0.6
399
+ stress_step_users: 200 # stress staircase (optional)
400
+ stress_step_duration_s: 24
401
+ spike_baseline_users: 100 # spike shape (optional)
402
+ spike_at_s: 30
403
+ spike_duration_s: 20
404
+
405
+ provider:
406
+ kind: simulator # simulator | openai-compatible | anthropic
407
+ base_url: https://api.openai.com/v1
408
+ api_key_env: OPENAI_API_KEY
409
+ model: gpt-4o-mini
410
+ stream: true # enables TTFT / inter-token measurement
411
+ max_tokens: 200
412
+ prompt: "..."
413
+ system_prompt: ""
414
+ timeout_s: 60
415
+ sim_ttft_ms: 320 # simulator knobs
416
+ sim_itl_ms: 16
417
+ sim_output_tokens: 180
418
+ sim_input_tokens: 950
419
+ sim_saturation_users: 700
420
+ sim_rate_limit_users: 880
421
+
422
+ sla: # omit any you don't need
423
+ p95_latency_ms: 6000
424
+ avg_latency_ms: null
425
+ ttft_p95_ms: 900
426
+ itl_p95_ms: 60
427
+ error_rate_pct: 1.0
428
+ min_throughput_rps: null
429
+ max_cost_usd: 5.0
430
+ min_goodput_pct: 90
431
+
432
+ pricing: # USD per 1M tokens
433
+ input_per_1m: 2.0
434
+ output_per_1m: 8.0
435
+
436
+ projection: # optional: realistic monthly cost estimate
437
+ requests_per_day: 50000 # expected production volume (best option)
438
+ # peak_hours_per_day: 8 # alternative: sustain test rate N hours/day
439
+ days_per_month: 30.4
440
+ ```
441
+
442
+ ### How cost is calculated
443
+
444
+ ```
445
+ total = input_tokens/1M × pricing.input_per_1m
446
+ + output_tokens/1M × pricing.output_per_1m
447
+ per_req = total ÷ successful requests ← a direct measurement, trust this
448
+ hourly = total ÷ test_duration × 3600
449
+ ```
450
+
451
+ Monthly cost is reported two ways, never conflated:
452
+
453
+ - **At expected volume** (when `projection.requests_per_day` is set) —
454
+ `per_req × requests_per_day × days_per_month`. This is the number to budget with.
455
+ - **Peak ceiling** (the fallback) — `hourly × 730`, i.e. this test's spend rate
456
+ sustained *every hour of the month*. Shown in amber and labelled `PEAK CEILING`
457
+ because a stress test extrapolated 24/7 is a worst case, not a forecast.
458
+
459
+ Both figures state the **request volume they imply**, so a dollar number is always
460
+ sanity-checkable at a glance:
461
+
462
+ ```
463
+ projected $211,529.82/mo PEAK CEILING - implies 63,597,600 requests/month
464
+ set projection.requests_per_day for a realistic estimate
465
+ ```
466
+
467
+ "Do we serve 64 million requests a month?" is a far easier question than staring
468
+ at a six-figure total.
469
+
470
+ Prices come from your config — nothing is looked up. Prompt-caching and batch
471
+ discounts are not modelled, so real bills are often lower. Failed requests are
472
+ counted as $0 (their tokens are unknown), so heavily-erroring runs may understate
473
+ slightly. A month is 365/12 = 30.4167 days = 730 hours (configurable via
474
+ `projection.days_per_month`), used by both projections.
475
+
476
+ **If the endpoint doesn't report usage**, InferGauge estimates tokens from text
477
+ length (~4 chars/token, ±20%), marks the cost `ESTIMATED`, and raises an insight —
478
+ rather than silently reporting $0 input cost, which is what a missing usage block
479
+ would otherwise produce.
480
+
481
+ Currency is rounded to 4 significant digits, not a fixed number of decimals, so
482
+ cheap-model costs survive: a $0.00012864 test reports `$0.00012864`, not `$0.0001`.
483
+ When cost per request is below a tenth of a cent, a per-1,000-requests figure is
484
+ shown alongside it. Displayed values stay internally consistent — cost/request ×
485
+ implied requests reproduces the monthly total.
486
+
487
+ ## Roadmap
488
+
489
+ - Prompt-mix workloads (weighted pools, multi-turn conversations)
490
+ - Multi-provider comparison runs
491
+ - Distributed load generation
492
+ - Prometheus / OpenTelemetry export
493
+ - **Admin dashboard for the console**: today, discovering who signed up
494
+ requesting a paid plan and minting/sending their license is an entirely
495
+ manual, inbound process (see `docs/ADMIN_GUIDE.md`) — there is no built-in
496
+ view listing accounts, requested plans, or license status across users.
497
+ Worth prioritizing before volume makes the manual loop unmanageable.
498
+
499
+ ## License
500
+
501
+ InferGauge is **source-available** under the [Business Source License 1.1](LICENSE),
502
+ not open source.
503
+
504
+ - **Allowed:** reading and auditing the source, internal and production use,
505
+ CI/CD gating, testing your own or your employer's systems, and consulting
506
+ work delivered to your clients.
507
+ - **Not allowed:** offering InferGauge to third parties as a hosted or managed
508
+ service, embedding it in a product whose value comes mainly from InferGauge,
509
+ or removing/bypassing its tier and quota enforcement.
510
+
511
+ Each released version converts to the **Apache License 2.0 on 2030-08-01** (or
512
+ four years after that version was published, whichever comes first).
513
+
514
+ See [COMMERCIAL.md](docs/business/COMMERCIAL.md) for the full picture and commercial licensing.
515
+ Bundled uPlot is MIT; runtime dependencies keep their own licenses.
516
+
517
+ **"InferGauge" is a trademark of InferGauge.** The license grants no trademark rights, and pre-release material shared under NDA may not be redistributed. See [NOTICE](NOTICE) for the full copyright, trademark, and confidentiality notice.
518
+