fusiontest 0.2.7__tar.gz → 0.2.8__tar.gz

fusiontest-0.2.8/PKG-INFO

@@ -0,0 +1,201 @@
+Metadata-Version: 2.4
+Name: fusiontest
+Version: 0.2.8
+Summary: AI-powered UI testing for mobile and desktop — by FusionLeap.io
+Author-email: FusionLeap <hello@fusionleap.io>
+License: MIT
+Project-URL: Homepage, https://fusionleap.io
+Project-URL: Repository, https://github.com/mohigo/fusion-test
+Project-URL: Issues, https://github.com/mohigo/fusion-test/issues
+Keywords: testing,AI,mobile,desktop,LLM,automation,QA
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Testing
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: click>=8.1
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: httpx>=0.27
+Requires-Dist: python-dotenv>=1.0
+Requires-Dist: rich>=13.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: openai>=1.58
+Provides-Extra: mobile
+Requires-Dist: Appium-Python-Client>=3.1; extra == "mobile"
+Requires-Dist: selenium>=4.18; extra == "mobile"
+Provides-Extra: desktop
+Requires-Dist: playwright>=1.43; extra == "desktop"
+Provides-Extra: windows
+Requires-Dist: Appium-Python-Client>=3.1; extra == "windows"
+Requires-Dist: selenium>=4.18; extra == "windows"
+Provides-Extra: mpnet
+Requires-Dist: sentence-transformers>=2.7; extra == "mpnet"
+Requires-Dist: torch>=2.2; extra == "mpnet"
+Provides-Extra: api
+Requires-Dist: fastapi>=0.111; extra == "api"
+Requires-Dist: uvicorn[standard]>=0.29; extra == "api"
+Requires-Dist: pydantic[email]>=2.0; extra == "api"
+Requires-Dist: python-jose[cryptography]>=3.3; extra == "api"
+Requires-Dist: cryptography>=42; extra == "api"
+Requires-Dist: passlib[bcrypt]>=1.7; extra == "api"
+Requires-Dist: psycopg2-binary>=2.9; extra == "api"
+Provides-Extra: training
+Requires-Dist: sentence-transformers>=2.7; extra == "training"
+Requires-Dist: torch>=2.2; extra == "training"
+Requires-Dist: datasets>=2.19; extra == "training"
+Requires-Dist: accelerate>=0.30; extra == "training"
+Requires-Dist: scikit-learn>=1.4; extra == "training"
+Requires-Dist: pandas>=2.2; extra == "training"
+Requires-Dist: numpy>=1.26; extra == "training"
+Requires-Dist: tqdm>=4.66; extra == "training"
+Requires-Dist: wandb>=0.17; extra == "training"
+Provides-Extra: openai
+Requires-Dist: openai>=1.58; extra == "openai"
+Provides-Extra: groq
+Requires-Dist: openai>=1.58; extra == "groq"
+Provides-Extra: claude
+Requires-Dist: anthropic>=0.25; extra == "claude"
+Provides-Extra: gemini
+Requires-Dist: google-genai>=1.0; extra == "gemini"
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
+Requires-Dist: pytest-cov>=5.0; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: mypy>=1.10; extra == "dev"
+Requires-Dist: pre-commit>=3.7; extra == "dev"
+Requires-Dist: fusiontest[desktop,mobile,openai]; extra == "dev"
+Provides-Extra: all
+Requires-Dist: fusiontest[api,claude,desktop,gemini,groq,mobile,mpnet,openai,training,windows]; extra == "all"
+
+# 🧪 FusionTest
+
+> AI-powered UI testing for web, desktop and mobile — built by [FusionLeap.io](https://fusionleap.io)
+
+FusionTest runs UI tests written as plain-English goals. An LLM reads the
+screen and decides what to do, a verifier confirms each goal against what's
+actually on screen, and goals that pass are recorded so later runs replay them
+without calling the AI at all.
+
+---
+
+## ✨ Features
+
+| Feature | Notes |
+|---|---|
+| Natural-language goals | `"Open the Pricing page and verify all three plans are shown"` |
+| Goal discovery | Crawls a site and proposes goals grounded in its real content |
+| Record & replay | Passing goals replay with no LLM calls; the AI repairs them when the page changes |
+| Honest results | Capacity or infrastructure problems are reported as **errors**, not test failures |
+| Multi-provider LLMs | Groq, Claude, Gemini, OpenAI or local Ollama, with automatic fallback |
+| Safe secrets | `{{PASSWORD}}` placeholders — the AI never sees values, which are only typed on allowed sites |
+| Guardrails | Invalid-action checks, loop detection, backtracking, no-progress detection |
+| Reporting | JSON reports, dashboard uploads, Slack, GitHub Actions annotations, webhooks |
+| Platforms | Web (Playwright), Electron, Android & iOS (Appium), Windows, macOS |
+
+---
+
+## 🏗 Architecture
+
+```
+fusiontest/
+├── core/         # Runner, LLM action model + fallback chain, goal verifier,
+│                 # screen parser, secrets, replay, SSRF guard (netguard)
+├── desktop/      # Playwright (web/Electron), Windows, macOS adapters
+├── mobile/       # Android (UIAutomator2) and iOS (XCUITest) adapters via Appium
+├── discovery/    # Goal discovery from a site's pages
+├── guardrails/   # Action validation, loop detection, backtracking
+├── reporting/    # JSON reports, Slack, GitHub annotations
+└── training/     # Experimental MPNet fine-tuning (not wired into runs)
+api/              # SaaS backend (FastAPI + PostgreSQL) — see docs/api.md
+dashboard/        # React dashboard
+website/          # Marketing site (Next.js)
+```
+
+Design decisions are recorded in [docs/adr](docs/adr) — start with
+[ADR-007](docs/adr/ADR-007-llm-capacity-and-execution.md).
+
+---
+
+## 🚀 Quickstart
+
+```bash
+pip install "fusiontest[desktop]"
+playwright install chromium
+
+export GROQ_API_KEY=gsk_...          # or ANTHROPIC_API_KEY / GEMINI_API_KEY / OPENAI_API_KEY
+
+fusiontest run --url https://example.com \
+  --goal "Verify the page heading says Example Domain" --headless
+```
+
+Set `FUSIONTEST_API_KEY` (from the dashboard's API Keys page) to upload each
+run to your FusionTest dashboard.
+
+Exit codes: `0` all goals passed · `1` a goal failed · `2` some goals couldn't
+be evaluated (LLM capacity or infrastructure) — not a test failure.
+
+> **Full guide:** [docs/getting-started.md](docs/getting-started.md) — platforms,
+> goal files, configuration, secrets, replay, dashboard and CI.
+
+---
+
+## 📝 Writing goals
+
+```yaml
+# tests/goals/smoke.yaml
+name: Marketing site smoke test
+platform: playwright
+url: https://example.com
+goals:
+  - "Open the Pricing page and verify three plan tiers are shown"
+  - "Open the Contact page, submit the form empty, and verify required-field errors appear"
+  - name: Log in
+    goal: "Log in with {{EMAIL}} and {{PASSWORD}} and verify the dashboard loads"
+```
+
+```bash
+fusiontest run --file tests/goals/smoke.yaml --headless
+```
+
+Tips: say what to verify, not just where to go; one outcome per goal; don't
+submit real forms on production sites.
+
+---
+
+## 🛠 Configuration
+
+`fusiontest init` creates `fusiontest.config.yaml`:
+
+```yaml
+backend: groq               # unset = auto-detect from API keys
+guardrails:
+  loop_detection_window: 5
+  max_retries: 3
+replay:
+  enabled: true
+  trace_dir: .fusiontest/traces
+  reverify_every: 10
+```
+
+---
+
+## 🔌 CI/CD
+
+Run `fusiontest run --headless` in any CI system and cache
+`.fusiontest/traces` so runs can replay. Guides for GitHub Actions, Bitrise,
+Fastlane and Jenkins: [docs/ci-integrations.md](docs/ci-integrations.md).
+
+---
+
+## 🤝 Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) and [ROADMAP.md](ROADMAP.md).
+
+## 📄 License
+
+MIT © [FusionLeap.io](https://fusionleap.io)

fusiontest-0.2.8/README.md

@@ -0,0 +1,127 @@
+# 🧪 FusionTest
+
+> AI-powered UI testing for web, desktop and mobile — built by [FusionLeap.io](https://fusionleap.io)
+
+FusionTest runs UI tests written as plain-English goals. An LLM reads the
+screen and decides what to do, a verifier confirms each goal against what's
+actually on screen, and goals that pass are recorded so later runs replay them
+without calling the AI at all.
+
+---
+
+## ✨ Features
+
+| Feature | Notes |
+|---|---|
+| Natural-language goals | `"Open the Pricing page and verify all three plans are shown"` |
+| Goal discovery | Crawls a site and proposes goals grounded in its real content |
+| Record & replay | Passing goals replay with no LLM calls; the AI repairs them when the page changes |
+| Honest results | Capacity or infrastructure problems are reported as **errors**, not test failures |
+| Multi-provider LLMs | Groq, Claude, Gemini, OpenAI or local Ollama, with automatic fallback |
+| Safe secrets | `{{PASSWORD}}` placeholders — the AI never sees values, which are only typed on allowed sites |
+| Guardrails | Invalid-action checks, loop detection, backtracking, no-progress detection |
+| Reporting | JSON reports, dashboard uploads, Slack, GitHub Actions annotations, webhooks |
+| Platforms | Web (Playwright), Electron, Android & iOS (Appium), Windows, macOS |
+
+---
+
+## 🏗 Architecture
+
+```
+fusiontest/
+├── core/         # Runner, LLM action model + fallback chain, goal verifier,
+│                 # screen parser, secrets, replay, SSRF guard (netguard)
+├── desktop/      # Playwright (web/Electron), Windows, macOS adapters
+├── mobile/       # Android (UIAutomator2) and iOS (XCUITest) adapters via Appium
+├── discovery/    # Goal discovery from a site's pages
+├── guardrails/   # Action validation, loop detection, backtracking
+├── reporting/    # JSON reports, Slack, GitHub annotations
+└── training/     # Experimental MPNet fine-tuning (not wired into runs)
+api/              # SaaS backend (FastAPI + PostgreSQL) — see docs/api.md
+dashboard/        # React dashboard
+website/          # Marketing site (Next.js)
+```
+
+Design decisions are recorded in [docs/adr](docs/adr) — start with
+[ADR-007](docs/adr/ADR-007-llm-capacity-and-execution.md).
+
+---
+
+## 🚀 Quickstart
+
+```bash
+pip install "fusiontest[desktop]"
+playwright install chromium
+
+export GROQ_API_KEY=gsk_...          # or ANTHROPIC_API_KEY / GEMINI_API_KEY / OPENAI_API_KEY
+
+fusiontest run --url https://example.com \
+  --goal "Verify the page heading says Example Domain" --headless
+```
+
+Set `FUSIONTEST_API_KEY` (from the dashboard's API Keys page) to upload each
+run to your FusionTest dashboard.
+
+Exit codes: `0` all goals passed · `1` a goal failed · `2` some goals couldn't
+be evaluated (LLM capacity or infrastructure) — not a test failure.
+
+> **Full guide:** [docs/getting-started.md](docs/getting-started.md) — platforms,
+> goal files, configuration, secrets, replay, dashboard and CI.
+
+---
+
+## 📝 Writing goals
+
+```yaml
+# tests/goals/smoke.yaml
+name: Marketing site smoke test
+platform: playwright
+url: https://example.com
+goals:
+  - "Open the Pricing page and verify three plan tiers are shown"
+  - "Open the Contact page, submit the form empty, and verify required-field errors appear"
+  - name: Log in
+    goal: "Log in with {{EMAIL}} and {{PASSWORD}} and verify the dashboard loads"
+```
+
+```bash
+fusiontest run --file tests/goals/smoke.yaml --headless
+```
+
+Tips: say what to verify, not just where to go; one outcome per goal; don't
+submit real forms on production sites.
+
+---
+
+## 🛠 Configuration
+
+`fusiontest init` creates `fusiontest.config.yaml`:
+
+```yaml
+backend: groq               # unset = auto-detect from API keys
+guardrails:
+  loop_detection_window: 5
+  max_retries: 3
+replay:
+  enabled: true
+  trace_dir: .fusiontest/traces
+  reverify_every: 10
+```
+
+---
+
+## 🔌 CI/CD
+
+Run `fusiontest run --headless` in any CI system and cache
+`.fusiontest/traces` so runs can replay. Guides for GitHub Actions, Bitrise,
+Fastlane and Jenkins: [docs/ci-integrations.md](docs/ci-integrations.md).
+
+---
+
+## 🤝 Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) and [ROADMAP.md](ROADMAP.md).
+
+## 📄 License
+
+MIT © [FusionLeap.io](https://fusionleap.io)

{fusiontest-0.2.7 → fusiontest-0.2.8}/fusiontest/cli.py

@@ -263,8 +263,9 @@ def run(goal, goal_file, platform, app, url, locale, backend, headless, output,
         finally:
             progress.stop()
 
-    # Print results table
+    # Print results table, then Slack / GitHub annotations if configured
     _print_results(result)
+    _notify(result)
 
     # Save JSON report locally
     if output:
@@ -473,22 +474,19 @@ def init():
         detected_backend = "groq"   # recommended default (free)
 
     config = {
-        # backend: which LLM to use — groq | claude | gemini | gpt4o | ollama | mpnet
+        # backend: which LLM to use — groq | claude | gemini | gpt4o | ollama
         # Free options: groq (recommended), gemini, ollama
         # Paid options: claude, gpt4o
         # Leave unset to auto-detect from whichever API key is present.
         "backend": detected_backend,
-        "runner": {
-            "max_steps_per_goal": 25,
-        },
         "guardrails": {
-            "loop_detection": True,
             "loop_detection_window": 5,
             "max_retries": 3,
-            "backtrack_on_invalid": True,
         },
-        "reporting": {
-            "save_recordings": True,
+        "replay": {
+            "enabled": True,
+            "trace_dir": DEFAULT_TRACE_DIR,
+            "reverify_every": 10,
         },
     }
 
@@ -594,6 +592,14 @@ def _exit_code(result) -> int:
 _MODE_LABELS = {"llm": "AI", "replayed": "[cyan]replay[/cyan]", "repaired": "[blue]repaired[/blue]"}
 
 
+def _notify(result) -> None:
+    """Slack (FUSIONTEST_SLACK_WEBHOOK) and GitHub Actions annotations, when configured."""
+    import os
+
+    from fusiontest.reporting.reporter import Reporter
+    Reporter(github_annotations=os.getenv("GITHUB_ACTIONS") == "true").notify(result)
+
+
 def _print_results(result) -> None:
     if result.success:
         status = "[green]PASS ✓[/green]"

{fusiontest-0.2.7 → fusiontest-0.2.8}/fusiontest/core/runner.py

@@ -99,7 +99,6 @@ class RunResult:
     steps: list[StepResult] = field(default_factory=list)
     total_duration_seconds: float = 0.0
     bugs_detected: list[str] = field(default_factory=list)
-    video_path: str = ""
     cancelled: bool = False
     # Set when the run stopped early because it could not be evaluated
     # (e.g. no LLM backend available). Remaining goals were not run.
@@ -160,9 +159,6 @@ class RunnerConfig:
     max_goal_seconds: int = 180
     # Path to write step-level JSONL logs (used by the training pipeline)
     step_log_path: str | None = None
-    # Video recording
-    record_video: bool = False
-    video_output_dir: str = "fusiontest-reports/videos"
     # Named secrets for {{VAR}} template resolution in goal strings.
     # Keys are variable names, values are secret strings.
     # Env vars are also searched when a key is missing here, unless
@@ -221,14 +217,6 @@ class FusionTestRunner:
             pathlib.Path(self.config.step_log_path).parent.mkdir(parents=True, exist_ok=True)
             self._step_log = open(self.config.step_log_path, "a")  # noqa: SIM115
 
-        self._recorder = None
-        if self.config.record_video:
-            from fusiontest.recording.recorder import Recorder, RecordingConfig
-            self._recorder = Recorder(
-                platform=getattr(adapter, "_platform", "unknown"),
-                config=RecordingConfig(output_dir=self.config.video_output_dir),
-            )
-
     def run(
         self,
         goals: list[str],
@@ -255,12 +243,6 @@ class FusionTestRunner:
         run_start = time.time()
         result = RunResult(name=name, platform=platform, locale=locale, success=True, url=url)
 
-        if self._recorder:
-            try:
-                self._recorder.start(getattr(self.adapter, "_driver", self.adapter))
-            except Exception as e:
-                logger.warning(f"Could not start recorder: {e}")
-
         logger.info(f"Starting run: {name} ({len(goals)} goals)")
 
         # Goals keep their {{VAR}} placeholders everywhere — model prompts, logs,
@@ -390,14 +372,6 @@ class FusionTestRunner:
                 )
                 break
 
-        if self._recorder:
-            try:
-                video_path = self._recorder.stop()
-                if video_path:
-                    result.video_path = str(video_path)
-            except Exception as e:
-                logger.warning(f"Could not stop recorder: {e}")
-
         result.total_duration_seconds = time.time() - run_start
         logger.info(
             f"Run complete: {'PASS' if result.success else 'FAIL'} "

{fusiontest-0.2.7 → fusiontest-0.2.8}/fusiontest/desktop/playwright_adapter.py

@@ -14,7 +14,6 @@ Supports:
 from __future__ import annotations
 
 import asyncio
-import base64
 import logging
 import os
 import re
@@ -636,47 +635,3 @@ class PlaywrightAdapter:
             ).first
 
         raise ValueError(f"Cannot resolve locator for action: {action}")
-
-    # ── Vision fallback ────────────────────────────────────────────────────────
-
-    async def get_vision_description(self) -> str:
-        """
-        Fallback for pages with poor accessibility trees.
-        Takes a screenshot and returns a GPT-4V/Claude description of the UI.
-        """
-        import os
-
-        screenshot = await self.async_get_screenshot()
-        b64 = base64.b64encode(screenshot).decode()
-
-        # Use whichever vision LLM is configured
-        api_key = os.getenv("OPENAI_API_KEY")
-        if not api_key:
-            return "=== SCREEN STATE ===\n-- Vision fallback not configured --"
-
-        try:
-            from openai import OpenAI
-            client = OpenAI(api_key=api_key)
-            resp = client.chat.completions.create(
-                model="gpt-4o",
-                messages=[{
-                    "role": "user",
-                    "content": [
-                        {
-                            "type": "text",
-                            "text": (
-                                "Describe all interactive UI elements visible on this screen. "
-                                "For each element, note: type (button, input, link, etc.), "
-                                "visible label or text, and approximate position (top/center/bottom, left/center/right). "
-                                "Format as a list."
-                            ),
-                        },
-                        {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{b64}"}},
-                    ],
-                }],
-                max_tokens=512,
-            )
-            return f"=== SCREEN STATE (vision) ===\n{resp.choices[0].message.content}"
-        except Exception as e:
-            logger.warning(f"Vision fallback failed: {e}")
-            return "=== SCREEN STATE ===\n-- Vision fallback error --"

{fusiontest-0.2.7 → fusiontest-0.2.8}/fusiontest/reporting/reporter.py

@@ -64,6 +64,13 @@ class Reporter:
 
         return report_path
 
+    def notify(self, result: RunResult) -> None:
+        """Send notifications only (Slack, GitHub annotations) — no files written."""
+        if self.slack_webhook:
+            self._notify_slack(result)
+        if self.github_annotations:
+            self._emit_github_annotations(result)
+
     # ── JSON ───────────────────────────────────────────────────────────────────
 
     def _save_json(self, result: RunResult) -> Path:

{fusiontest-0.2.7 → fusiontest-0.2.8}/fusiontest/training/__init__.py

@@ -1,6 +1,11 @@
 """
 fusiontest.training — MPNet fine-tuning pipeline for FusionTest action ranking.
 
+EXPERIMENTAL — not wired into runs. The action strings it learns from step
+logs don't match the candidates ActionModel(backend="mpnet") ranks at run
+time, so a model trained here won't help yet. Kept for the distillation plan
+in ADR-001 (amended by ADR-007, which makes replay the main cost lever).
+
 Workflow:
   1. Collect  — gather (screen_text, goal, action, outcome) tuples from live runs
   2. Build    — convert raw logs into contrastive training pairs