testmcpy 0.7.2__tar.gz → 0.7.4__tar.gz

{testmcpy-0.7.2/testmcpy.egg-info → testmcpy-0.7.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: testmcpy
-Version: 0.7.2
+Version: 0.7.4
 Summary: A comprehensive testing framework for validating LLM tool calling capabilities with MCP services
 Author: Amin Ghadersohi
 License-Expression: Apache-2.0
@@ -86,7 +86,7 @@ Dynamic: license-file
   <a href="https://pypi.org/project/testmcpy/"><img src="https://img.shields.io/badge/pypi-testmcpy-blue" alt="PyPI"></a>
 </p>
 
-![MCP Explorer](https://raw.githubusercontent.com/preset-io/testmcpy/main/context/images/web-ui-explorer.png)
+![MCP Explorer — tools, resources, and prompts from a connected MCP service](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/mcp-explorer.png)
 
 ---
 
@@ -131,7 +131,7 @@ Test with **Claude**, **GPT-4**, **Llama**, and other models. Works with both pa
 | Ollama | Llama, Mistral, etc. (local) | Free, local execution, no API costs |
 | Claude SDK | claude-cli, claude-code | Subprocess-based, full MCP support |
 
-![LLM Profiles](https://raw.githubusercontent.com/preset-io/testmcpy/main/context/images/model-selector.png)
+![LLM Profiles — manage Anthropic, OpenAI, Ollama and Claude SDK provider configurations](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/llm-profiles.png)
 
 ### Built-in Evaluators
 
@@ -154,7 +154,7 @@ Comprehensive validation out of the box. Each evaluator returns a score from 0.0
 
 **Extensible:** Extend `BaseEvaluator` and implement `evaluate(context) -> EvalResult` to create custom evaluators for your domain.
 
-![Test Results](https://raw.githubusercontent.com/preset-io/testmcpy/main/context/images/test-results.png)
+![Reports — combined view of every test run, evaluator scores, and cost analysis](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/reports.png)
 
 ### YAML Test Definitions
 
@@ -200,32 +200,13 @@ tests:
       duration: 60
 ```
 
-### Interactive TUI Dashboard
-
-Beautiful terminal interface for MCP testing — no browser required:
-
-```bash
-testmcpy dash                    # Launch interactive dashboard
-testmcpy dash --auto-refresh     # Live connection monitoring
-testmcpy dash --profile prod     # Use specific MCP profile
-```
-
-**TUI Features:**
-- Real-time MCP connection status
-- Interactive tool exploration
-- Live test execution with progress
-- Configuration editor
-- Global search across tools, tests, and settings
-- Help system with keyboard shortcuts (press `?`)
-- Multiple themes (default, light, high contrast)
-
 ### CLI & Web UI
 
 - **Rich terminal UI**: Progress bars, colored output, formatted tables
 - **Optional web interface**: Visual tool explorer, interactive chat, analytics dashboards
 - **Real-time feedback**: Watch tests execute with live updates via WebSocket
 
-![Chat Interface](https://raw.githubusercontent.com/preset-io/testmcpy/main/context/images/cli-interface.png)
+![Chat Interface — interactive chat against your MCP service from the browser](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/chat.png)
 
 ## Architecture
 
@@ -394,13 +375,20 @@ testmcpy run tests/ --model claude-haiku-4-5
 | `testmcpy chat` | Interactive chat with MCP tools |
 | `testmcpy compare` | Multi-model comparison |
 | **Advanced** | |
-| `testmcpy baseline` | Save and compare against baselines |
+| `testmcpy baseline-save` | Save current test results as a named baseline |
+| `testmcpy baseline-compare` | Compare a run against a saved baseline |
+| `testmcpy baseline-list` | List saved baselines |
 | `testmcpy mutate` | Prompt mutation testing |
 | `testmcpy metamorphic` | Metamorphic testing |
+| `testmcpy generate` | AI-assisted test generation |
+| `testmcpy smoke-test` | Quick smoke test against an MCP service |
+| `testmcpy coverage` | Tool coverage report for a test suite |
+| `testmcpy multi-env` | Run the same suite against multiple MCP profiles |
+| `testmcpy export-db` | Export the SQLite results database |
 | **UI** | |
-| `testmcpy serve` | Start web UI server (port 8000) |
-| `testmcpy dash` | Launch terminal UI dashboard |
+| `testmcpy serve` | Start web UI server (default port 8000) |
 | `testmcpy config-cmd` | View current configuration |
+| `testmcpy config-mcp` | Print MCP client snippets for Claude Desktop / Code |
 
 **Common options:** `--profile`, `--llm-profile`, `--model`, `--provider`, `--timeout`, `--verbose`, `--output`
 
@@ -433,9 +421,9 @@ Environment variables are also supported: `MCP_AUTH_TOKEN`, `MCP_JWT_URL`, `MCP_
 
 ## Web Interface
 
-Optional React-based UI with 15+ pages for visual testing and analytics:
+Optional React-based UI with 14 pages for visual testing and analytics:
 
-![Test Manager](https://raw.githubusercontent.com/preset-io/testmcpy/main/context/images/web-ui-dashboard.png)
+![Test Manager — browse YAML suites, kick off runs, watch results stream in](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/test-manager.png)
 
 ```bash
 # Install with UI support
@@ -462,7 +450,20 @@ testmcpy serve
 | `/mcp-profiles` | MCP Profiles | MCP server configuration |
 | `/llm-profiles` | LLM Profiles | LLM provider configuration |
 
-Access at `http://localhost:8000`
+Access at `http://localhost:8000`.
+
+#### More screenshots
+
+<table>
+  <tr>
+    <td align="center"><img src="https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/generation-history.png" alt="Generation History page"><br><sub>Generation History — AI-assisted test generation runs</sub></td>
+    <td align="center"><img src="https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/auth-debugger.png" alt="Auth Debugger page"><br><sub>Auth Debugger — step through OAuth / JWT / Bearer flows</sub></td>
+  </tr>
+  <tr>
+    <td align="center"><img src="https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/mcp-profiles.png" alt="MCP Profiles page"><br><sub>MCP Profiles — manage MCP service connections</sub></td>
+    <td align="center"><img src="https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/config.png" alt="Configuration page"><br><sub>Configuration — current settings and client snippets</sub></td>
+  </tr>
+</table>
 
 ## LLM Providers
 

{testmcpy-0.7.2 → testmcpy-0.7.4}/README.md

@@ -16,7 +16,7 @@
   <a href="https://pypi.org/project/testmcpy/"><img src="https://img.shields.io/badge/pypi-testmcpy-blue" alt="PyPI"></a>
 </p>
 
-![MCP Explorer](https://raw.githubusercontent.com/preset-io/testmcpy/main/context/images/web-ui-explorer.png)
+![MCP Explorer — tools, resources, and prompts from a connected MCP service](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/mcp-explorer.png)
 
 ---
 
@@ -61,7 +61,7 @@ Test with **Claude**, **GPT-4**, **Llama**, and other models. Works with both pa
 | Ollama | Llama, Mistral, etc. (local) | Free, local execution, no API costs |
 | Claude SDK | claude-cli, claude-code | Subprocess-based, full MCP support |
 
-![LLM Profiles](https://raw.githubusercontent.com/preset-io/testmcpy/main/context/images/model-selector.png)
+![LLM Profiles — manage Anthropic, OpenAI, Ollama and Claude SDK provider configurations](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/llm-profiles.png)
 
 ### Built-in Evaluators
 
@@ -84,7 +84,7 @@ Comprehensive validation out of the box. Each evaluator returns a score from 0.0
 
 **Extensible:** Extend `BaseEvaluator` and implement `evaluate(context) -> EvalResult` to create custom evaluators for your domain.
 
-![Test Results](https://raw.githubusercontent.com/preset-io/testmcpy/main/context/images/test-results.png)
+![Reports — combined view of every test run, evaluator scores, and cost analysis](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/reports.png)
 
 ### YAML Test Definitions
 
@@ -130,32 +130,13 @@ tests:
       duration: 60
 ```
 
-### Interactive TUI Dashboard
-
-Beautiful terminal interface for MCP testing — no browser required:
-
-```bash
-testmcpy dash                    # Launch interactive dashboard
-testmcpy dash --auto-refresh     # Live connection monitoring
-testmcpy dash --profile prod     # Use specific MCP profile
-```
-
-**TUI Features:**
-- Real-time MCP connection status
-- Interactive tool exploration
-- Live test execution with progress
-- Configuration editor
-- Global search across tools, tests, and settings
-- Help system with keyboard shortcuts (press `?`)
-- Multiple themes (default, light, high contrast)
-
 ### CLI & Web UI
 
 - **Rich terminal UI**: Progress bars, colored output, formatted tables
 - **Optional web interface**: Visual tool explorer, interactive chat, analytics dashboards
 - **Real-time feedback**: Watch tests execute with live updates via WebSocket
 
-![Chat Interface](https://raw.githubusercontent.com/preset-io/testmcpy/main/context/images/cli-interface.png)
+![Chat Interface — interactive chat against your MCP service from the browser](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/chat.png)
 
 ## Architecture
 
@@ -324,13 +305,20 @@ testmcpy run tests/ --model claude-haiku-4-5
 | `testmcpy chat` | Interactive chat with MCP tools |
 | `testmcpy compare` | Multi-model comparison |
 | **Advanced** | |
-| `testmcpy baseline` | Save and compare against baselines |
+| `testmcpy baseline-save` | Save current test results as a named baseline |
+| `testmcpy baseline-compare` | Compare a run against a saved baseline |
+| `testmcpy baseline-list` | List saved baselines |
 | `testmcpy mutate` | Prompt mutation testing |
 | `testmcpy metamorphic` | Metamorphic testing |
+| `testmcpy generate` | AI-assisted test generation |
+| `testmcpy smoke-test` | Quick smoke test against an MCP service |
+| `testmcpy coverage` | Tool coverage report for a test suite |
+| `testmcpy multi-env` | Run the same suite against multiple MCP profiles |
+| `testmcpy export-db` | Export the SQLite results database |
 | **UI** | |
-| `testmcpy serve` | Start web UI server (port 8000) |
-| `testmcpy dash` | Launch terminal UI dashboard |
+| `testmcpy serve` | Start web UI server (default port 8000) |
 | `testmcpy config-cmd` | View current configuration |
+| `testmcpy config-mcp` | Print MCP client snippets for Claude Desktop / Code |
 
 **Common options:** `--profile`, `--llm-profile`, `--model`, `--provider`, `--timeout`, `--verbose`, `--output`
 
@@ -363,9 +351,9 @@ Environment variables are also supported: `MCP_AUTH_TOKEN`, `MCP_JWT_URL`, `MCP_
 
 ## Web Interface
 
-Optional React-based UI with 15+ pages for visual testing and analytics:
+Optional React-based UI with 14 pages for visual testing and analytics:
 
-![Test Manager](https://raw.githubusercontent.com/preset-io/testmcpy/main/context/images/web-ui-dashboard.png)
+![Test Manager — browse YAML suites, kick off runs, watch results stream in](https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/test-manager.png)
 
 ```bash
 # Install with UI support
@@ -392,7 +380,20 @@ testmcpy serve
 | `/mcp-profiles` | MCP Profiles | MCP server configuration |
 | `/llm-profiles` | LLM Profiles | LLM provider configuration |
 
-Access at `http://localhost:8000`
+Access at `http://localhost:8000`.
+
+#### More screenshots
+
+<table>
+  <tr>
+    <td align="center"><img src="https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/generation-history.png" alt="Generation History page"><br><sub>Generation History — AI-assisted test generation runs</sub></td>
+    <td align="center"><img src="https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/auth-debugger.png" alt="Auth Debugger page"><br><sub>Auth Debugger — step through OAuth / JWT / Bearer flows</sub></td>
+  </tr>
+  <tr>
+    <td align="center"><img src="https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/mcp-profiles.png" alt="MCP Profiles page"><br><sub>MCP Profiles — manage MCP service connections</sub></td>
+    <td align="center"><img src="https://raw.githubusercontent.com/preset-io/testmcpy/main/docs/screenshots/config.png" alt="Configuration page"><br><sub>Configuration — current settings and client snippets</sub></td>
+  </tr>
+</table>
 
 ## LLM Providers
 

{testmcpy-0.7.2 → testmcpy-0.7.4}/pyproject.toml

@@ -93,7 +93,7 @@ testmcpy = [
 
 [project]
 name = "testmcpy"
-version = "0.7.2"
+version = "0.7.4"
 description = "A comprehensive testing framework for validating LLM tool calling capabilities with MCP services"
 authors = [{name = "Amin Ghadersohi"}]
 license = "Apache-2.0"

{testmcpy-0.7.2 → testmcpy-0.7.4}/testmcpy/__init__.py

@@ -11,6 +11,6 @@ try:
     __version__ = version("testmcpy")
 except Exception:
     # Fallback for development or when package not installed
-    __version__ = "0.7.2"
+    __version__ = "0.7.4"
 
 __author__ = "testmcpy Contributors"

{testmcpy-0.7.2 → testmcpy-0.7.4}/testmcpy/cli/commands/run.py

@@ -296,6 +296,18 @@ def run(
             "provider (default: provider class's _DEFAULT_COMPLETIONS_PATH)"
         ),
     ),
+    max_concurrent_streams: Optional[int] = typer.Option(
+        None,
+        "--max-concurrent-streams",
+        help=(
+            "Process-wide cap on concurrent SSE streams for the "
+            "assistant/chatbot provider. Useful when a parent harness "
+            "spawns many testmcpy children at once and the chatbot "
+            "endpoint stalls under load. None / 0 = unbounded (default). "
+            "Limit applies across all AssistantProvider instances in the "
+            "process. (SC-106138)"
+        ),
+    ),
 ):
     """
     Run test cases against MCP service.
@@ -470,6 +482,27 @@ def run(
                 if value is not None:
                     effective_provider_config[key] = value
 
+            # Apply the concurrency limit at the class level so every
+            # AssistantProvider instance in this process shares the cap.
+            # SC-106138: agor harness fan-out can stall the chatbot when
+            # too many SSE streams open at once.
+            #
+            # Always call configure_concurrency_limit() (even with None)
+            # so the CLI flag is a true override/reset — otherwise a
+            # prior in-process configuration could leak when run() is
+            # invoked multiple times within the same Python process.
+            from testmcpy.src.llm_integration import AssistantProvider
+
+            AssistantProvider.configure_concurrency_limit(max_concurrent_streams)
+            if verbose:
+                if max_concurrent_streams:
+                    console.print(
+                        f"[cyan]Concurrency cap:[/cyan] "
+                        f"max {max_concurrent_streams} concurrent SSE streams"
+                    )
+                else:
+                    console.print("[cyan]Concurrency cap:[/cyan] unbounded")
+
         if suite_provider and verbose:
             console.print(f"[yellow]Suite-level provider override:[/yellow] {suite_provider}")
             if suite_provider_config:
@@ -507,6 +540,47 @@ def run(
 
         # Run tests with progress output
         results = []
+
+        # Progressive checkpoint: dump partial results to a JSON file under
+        # tests/.results/.checkpoints/ after every test completes, so an
+        # outer harness (or this process after a crash) can recover what
+        # finished even if the run is killed mid-stream. The matching
+        # <session_id>.done sentinel is written immediately after the run
+        # summary prints, before optional post-processing (DB save, report
+        # generation) — so harnesses see "done" as soon as the test loop
+        # itself finishes, even if a later non-test step hangs or fails.
+        # (SC-107284 / c33 issues 3 & 6)
+        checkpoint_dir = Path("tests/.results/.checkpoints")
+        checkpoint_path: Optional[Path] = None
+        done_path: Optional[Path] = None
+        try:
+            checkpoint_dir.mkdir(parents=True, exist_ok=True)
+            checkpoint_path = checkpoint_dir / f"{session_id}.json"
+            done_path = checkpoint_dir / f"{session_id}.done"
+        except OSError as e:
+            console.print(f"[dim]Note: Could not create checkpoint dir: {e}[/dim]")
+
+        def _write_checkpoint(completed: list, total: int) -> None:
+            """Atomically write current results to the checkpoint file."""
+            if checkpoint_path is None:
+                return
+            payload = {
+                "session_id": session_id,
+                "test_file": str(test_path),
+                "provider": effective_provider,
+                "model": effective_model,
+                "mcp_profile": profile or "default",
+                "completed": len(completed),
+                "total": total,
+                "results": [r.to_dict() for r in completed],
+            }
+            try:
+                tmp = checkpoint_path.with_suffix(".json.tmp")
+                tmp.write_text(json.dumps(payload, indent=2, default=str))
+                tmp.replace(checkpoint_path)
+            except OSError as e:
+                console.print(f"[dim]Note: Could not write checkpoint: {e}[/dim]")
+
         # Only initialize LLM provider if there are non-auth-only tests
         has_non_auth_only = any(not tc.is_auth_only for tc in test_cases)
         if has_non_auth_only:
@@ -556,6 +630,7 @@ def run(
                     _status.stop()
 
             results.append(result)
+            _write_checkpoint(results, len(test_cases))
 
             # Show immediate result
             if result.passed:
@@ -631,6 +706,24 @@ def run(
 
         console.print(f"\n[bold]Summary:[/bold] {' | '.join(summary_parts)}")
 
+        # Drop a .done sentinel immediately after the summary, before any
+        # optional post-processing (DB save, report gen) so outer harnesses
+        # can treat the run as finished even if a later step fails or hangs.
+        if done_path is not None:
+            try:
+                done_path.write_text(
+                    json.dumps(
+                        {
+                            "session_id": session_id,
+                            "total": len(results),
+                            "passed": total_passed,
+                            "failed": len(results) - total_passed,
+                        }
+                    )
+                )
+            except OSError as e:
+                console.print(f"[dim]Note: Could not write .done marker: {e}[/dim]")
+
         # Always auto-save results to tests/.results/ so the UI can see them
         try:
             from testmcpy.server.routers.results import save_test_run_to_file