screenforge 0.4.0__tar.gz

screenforge-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 jhinzzz
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

screenforge-0.4.0/PKG-INFO

@@ -0,0 +1,43 @@
+Metadata-Version: 2.4
+Name: screenforge
+Version: 0.4.0
+Summary: AI-driven cross-platform UI automation engine with test script generation
+License: MIT
+Project-URL: Homepage, https://github.com/jhinzzz/ScreenForge
+Project-URL: Repository, https://github.com/jhinzzz/ScreenForge
+Project-URL: Issues, https://github.com/jhinzzz/ScreenForge/issues
+Requires-Python: >=3.11
+License-File: LICENSE
+Requires-Dist: playwright>=1.50
+Requires-Dist: openai>=2.0
+Requires-Dist: allure-pytest>=2.15
+Requires-Dist: loguru>=0.7
+Requires-Dist: pydantic>=2.0
+Requires-Dist: python-dotenv>=1.0
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: pillow>=12.0
+Requires-Dist: lxml>=5.0
+Requires-Dist: requests>=2.30
+Requires-Dist: rich>=14.0
+Requires-Dist: typer>=0.24
+Requires-Dist: retry2>=0.9
+Requires-Dist: opencv-python>=4.10
+Requires-Dist: numpy>=1.24
+Requires-Dist: filelock>=3.12
+Provides-Extra: android
+Requires-Dist: uiautomator2>=3.5; extra == "android"
+Requires-Dist: adbutils>=2.12; extra == "android"
+Provides-Extra: ios
+Requires-Dist: facebook-wda>=1.0; extra == "ios"
+Provides-Extra: ml
+Requires-Dist: torch>=2.0; extra == "ml"
+Requires-Dist: transformers>=5.0; extra == "ml"
+Requires-Dist: sentence-transformers>=5.0; extra == "ml"
+Requires-Dist: scikit-learn>=1.8; extra == "ml"
+Provides-Extra: playground
+Requires-Dist: fastapi>=0.115; extra == "playground"
+Requires-Dist: uvicorn>=0.34; extra == "playground"
+Requires-Dist: websockets>=14.0; extra == "playground"
+Provides-Extra: all
+Requires-Dist: screenforge[android,ios,ml,playground]; extra == "all"
+Dynamic: license-file

screenforge-0.4.0/README.md

@@ -0,0 +1,172 @@
+# ScreenForge
+
+[![CI](https://github.com/jhinzzz/ScreenForge/actions/workflows/ci.yml/badge.svg)](https://github.com/jhinzzz/ScreenForge/actions/workflows/ci.yml)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://www.python.org/downloads/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
+
+**[中文](./README_CN.md)** | English
+
+> Describe what to test. Watch it happen. Get a pytest script.
+
+ScreenForge is an AI-driven UI automation engine that turns natural language into executable test scripts. Unlike record-and-replay tools, you don't perform the actions yourself — the AI does it for you.
+
+![ScreenForge Demo](docs/assets/demo.gif)
+
+## Why ScreenForge?
+
+| | Playwright Codegen | Browser Use | Midscene.js | **ScreenForge** |
+|---|---|---|---|---|
+| Need to perform actions yourself? | Yes | No | No | **No** |
+| Generates replayable test scripts? | Yes | No | No | **Yes (pytest)** |
+| Self-healing when UI changes? | No | No | No | **Yes** |
+| Works as AI Agent tool (MCP)? | No | Yes | No | **Yes** |
+
+**Core architecture**: Your AI Agent is the brain (understands requirements, makes decisions). ScreenForge is the hands (executes UI actions, generates code).
+
+## Quick Start
+
+```bash
+pip install screenforge
+
+# See the magic without any API key:
+screenforge --demo
+
+# For real usage, set your LLM key:
+export OPENAI_API_KEY=sk-...
+
+# Inspect the current page (returns DOM tree for your Agent to analyze):
+echo '{"operation":"inspect_ui","platform":"web"}' | screenforge --tool-stdin
+
+# Execute a single action:
+screenforge --action click --platform web --locator-type text --locator-value "Login"
+```
+
+## How It Works
+
+```
+You (or your AI Agent)          ScreenForge
+        │                            │
+        ├──── "Test the login" ─────►│
+        │                            ├── inspect_ui (get DOM tree)
+        │◄── DOM tree ──────────────┤
+        │                            │
+        ├──── click #email ─────────►│
+        ├──── input "user@..." ─────►│
+        ├──── click "Sign In" ──────►│
+        │                            │
+        │◄── pytest script ─────────┤
+        │◄── Allure report ─────────┤
+```
+
+Each step: **inspect → decide → act → verify**. The AI decides, ScreenForge executes.
+
+## Features
+
+- **Cross-platform**: Android (uiautomator2), iOS (wda), Web (Playwright)
+- **Self-healing engine**: When tests break due to UI changes, the engine auto-repairs locators with confidence scoring and AST validation
+- **L1/L2 semantic cache**: Same page + same instruction = instant response, no LLM call needed
+- **Visual fallback**: When DOM can't locate elements (Canvas, games), VLM parses screenshots
+- **MCP server**: Any MCP-compatible Agent can drive ScreenForge natively
+- **Structured output**: JSON Lines events + `report/runs/<id>/` artifacts for CI integration
+
+## Agent Integration (Claude Code / Cursor / Codex)
+
+ScreenForge exposes itself as a tool for AI Agents. The standard loop:
+
+```bash
+# 1. Get page structure (your Agent analyzes it)
+echo '{"operation":"inspect_ui","platform":"web"}' | screenforge --tool-stdin
+
+# 2. Your Agent decides what to do, sends precise actions
+screenforge --action click --platform web --locator-type text --locator-value "Login"
+
+# 3. Verify the result, repeat
+echo '{"operation":"inspect_ui","platform":"web"}' | screenforge --tool-stdin
+```
+
+For batch operations, use workflows:
+
+```bash
+screenforge --workflow ./workflows/login.yaml --platform web --json
+```
+
+Or start the MCP server for native Agent integration:
+
+```bash
+screenforge --mcp-server
+```
+
+## GitHub Actions
+
+Add ScreenForge to your CI pipeline:
+
+```yaml
+- uses: jhinzzz/ScreenForge@v1
+  with:
+    platform: web
+    workflow: ./workflows/login.yaml
+    openai-api-key: ${{ secrets.OPENAI_API_KEY }}
+```
+
+Results are auto-uploaded as Allure artifacts. See [action.yml](action.yml) for all inputs.
+
+See [Agent Integration Guide](docs/agent_guide.md) for the complete protocol.
+
+## Installation (from source)
+
+```bash
+git clone https://github.com/jhinzzz/ScreenForge.git
+cd ScreenForge
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[all]"
+```
+
+Platform-specific extras:
+- Web only: `pip install -e .` (default, includes Playwright)
+- Android: `pip install -e ".[android]"`
+- iOS: `pip install -e ".[ios]"`
+- ML/cache: `pip install -e ".[ml]"` (sentence-transformers for semantic cache)
+
+## Configuration
+
+```bash
+# Required: LLM API key (OpenAI-compatible endpoint)
+export OPENAI_API_KEY=sk-...
+
+# Optional: custom endpoint (defaults to api.openai.com)
+export OPENAI_BASE_URL=https://api.openai.com/v1
+
+# Optional: model (defaults to gpt-4o)
+export MODEL_NAME=gpt-4o
+```
+
+Or create a `.env` file (copy from `.env_template`).
+
+## Badge
+
+If ScreenForge generates tests for your project, add this badge to your README:
+
+```markdown
+[![Tests by ScreenForge](https://img.shields.io/badge/tests%20by-ScreenForge-blue?logo=pytest)](https://github.com/jhinzzz/ScreenForge)
+```
+
+[![Tests by ScreenForge](https://img.shields.io/badge/tests%20by-ScreenForge-blue?logo=pytest)](https://github.com/jhinzzz/ScreenForge)
+
+## Learn More
+
+| Resource | Description |
+|----------|-------------|
+| [Mobile Setup](docs/mobile-setup.md) | Android & iOS device connection guide |
+| [MCP Setup (3 min)](docs/mcp-setup.md) | Connect to Claude Desktop / Cursor / Cline / Claude Code |
+| [Agent Guide](docs/agent_guide.md) | Integration protocol for AI Agents |
+| [Capability Matrix](docs/capability-matrix.md) | Supported platforms, actions, and locators |
+| [Workflow Examples](docs/workflows/) | YAML workflow templates |
+| [CHANGELOG](CHANGELOG.md) | Version history |
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines. Issues and PRs welcome!
+
+## License
+
+[MIT](LICENSE)

screenforge-0.4.0/cli/_version.py

@@ -0,0 +1 @@
+__version__ = "0.4.0"

screenforge-0.4.0/cli/dispatch.py

@@ -0,0 +1,266 @@
+"""Execution dispatcher and CLI entry point."""
+
+import sys
+
+from cli.doctor import run_capabilities_mode, run_doctor_mode
+from cli.modes.action import run_action_default_mode
+from cli.modes.default import run_default_mode
+from cli.modes.dry_run import run_action_dry_run_mode, run_dry_run_mode
+from cli.modes.plan import run_action_plan_only_mode, run_plan_only_mode
+from cli.modes.workflow import (
+    run_workflow_default_mode,
+    run_workflow_dry_run_mode,
+    run_workflow_plan_only_mode,
+)
+from cli.parser import build_parser, validate_cli_args
+from cli.reporter import _load_context_content, _resolve_output_script_path
+from cli.shared import _SharedAdapterManager, config, log
+from cli.tool_protocol_handlers import (
+    _requires_model_runtime,
+    run_mcp_server_mode,
+    run_tool_request_mode,
+    run_tool_stdin_mode,
+)
+from common.run_resume import RunContextLoadError
+from common.runtime_modes import (
+    MODE_DOCTOR,
+    MODE_DRY_RUN,
+    MODE_PLAN_ONLY,
+    resolve_execution_mode,
+)
+
+
+def _dispatch_execution(
+    args,
+    execution_mode: str,
+    output_script_path: str,
+    context_content: str,
+    resume_context: dict,
+    shared_adapter_manager: _SharedAdapterManager | None = None,
+) -> int:
+    if execution_mode == MODE_DOCTOR:
+        return run_doctor_mode(args, output_script_path)
+    if args.workflow and execution_mode == MODE_PLAN_ONLY:
+        return run_workflow_plan_only_mode(
+            args,
+            output_script_path,
+            resume_context,
+        )
+    if args.workflow and execution_mode == MODE_DRY_RUN:
+        return run_workflow_dry_run_mode(
+            args,
+            output_script_path,
+            resume_context,
+        )
+    if args.workflow:
+        return run_workflow_default_mode(
+            args,
+            output_script_path,
+            resume_context,
+        )
+    if args.action and execution_mode == MODE_PLAN_ONLY:
+        return run_action_plan_only_mode(
+            args,
+            output_script_path,
+            resume_context,
+        )
+    if args.action and execution_mode == MODE_DRY_RUN:
+        return run_action_dry_run_mode(
+            args,
+            output_script_path,
+            resume_context,
+        )
+    if args.action:
+        return run_action_default_mode(
+            args,
+            output_script_path,
+            resume_context,
+            shared_adapter_manager=shared_adapter_manager,
+        )
+    if execution_mode == MODE_PLAN_ONLY:
+        return run_plan_only_mode(
+            args,
+            output_script_path,
+            context_content,
+            resume_context,
+        )
+    if execution_mode == MODE_DRY_RUN:
+        return run_dry_run_mode(
+            args,
+            output_script_path,
+            context_content,
+            resume_context,
+        )
+    return run_default_mode(
+        args,
+        output_script_path,
+        context_content,
+        resume_context,
+    )
+
+
+def _run_session_end(args) -> int:
+    from cli.session import delete_session, load_session, stop_session_recording
+
+    session_id = args.session_end
+    session = load_session(session_id)
+    if not session:
+        log.error(f"❌ Session not found: {session_id}")
+        return 1
+
+    output_path = session["output_path"]
+
+    video_path = stop_session_recording(session_id)
+    if video_path:
+        log.info(f"🎬 [Session] Recording saved: {video_path}")
+
+    delete_session(session_id)
+    log.info(f"✅ [Session] Ended session '{session_id}'")
+    log.info(f"📄 [Session] Test script: {output_path}")
+    log.info(f"📊 [Session] Total steps: {session.get('steps', 0)}")
+    return 0
+
+
+def main():
+    from cli.shorthand import preprocess_argv
+
+    processed_argv = preprocess_argv(sys.argv)
+
+    parser = build_parser()
+    args = parser.parse_args(processed_argv[1:])
+
+    if getattr(args, "session_end", ""):
+        sys.exit(_run_session_end(args))
+
+    if getattr(args, "web_stop", False):
+        from common.adapters.web_adapter import stop_persistent_browser
+
+        # Idempotent: stopping a browser, or finding none to stop, both succeed.
+        stop_persistent_browser()
+        sys.exit(0)
+
+    try:
+        validate_cli_args(args)
+    except ValueError as e:
+        log.error(f"[E010] Invalid CLI arguments: {e}. Fix: run 'screenforge --help' to see valid options")
+        sys.exit(2)
+
+    if getattr(args, "init", False):
+        from cli.modes.init import run_init_mode
+
+        sys.exit(run_init_mode())
+
+    if getattr(args, "demo", False):
+        from cli.modes.demo import run_demo_mode
+
+        sys.exit(run_demo_mode())
+
+    if getattr(args, "playground", False):
+        try:
+            from playground.app import run_server
+        except ImportError:
+            log.error("[E013] Playground requires extra dependencies. Fix: pip install screenforge[playground]")
+            sys.exit(1)
+        cdp_url = "http://127.0.0.1:9333"
+        log.info(f"Starting Playground on http://127.0.0.1:{args.playground_port}")
+        log.info(f"CDP screencast target: {cdp_url}")
+        run_server(port=args.playground_port, cdp_url=cdp_url)
+        sys.exit(0)
+
+    if args.tool_stdin:
+        from common.progress import set_tool_mode
+        set_tool_mode(True)
+        if sys.stdin.isatty():
+            import io
+            import json as _json
+            sys.stdin = io.StringIO(_json.dumps({"operation": "inspect_ui", "platform": args.platform}))
+        sys.exit(run_tool_stdin_mode(args))
+
+    if args.mcp_server:
+        from common.progress import set_tool_mode
+        set_tool_mode(True)
+        sys.exit(run_mcp_server_mode(args))
+
+    if args.tool_request:
+        from common.progress import set_tool_mode
+        set_tool_mode(True)
+        sys.exit(run_tool_request_mode(args))
+
+    if args.capabilities:
+        sys.exit(run_capabilities_mode(args))
+
+    execution_mode = resolve_execution_mode(
+        doctor=args.doctor,
+        plan_only=args.plan_only,
+        dry_run=args.dry_run,
+    )
+
+    session_id = getattr(args, "session_id", "")
+    shared_adapter_mgr = None
+    if session_id and args.action:
+        from cli.session import (
+            create_session,
+            load_session,
+            resolve_session_output_path,
+            update_session,
+        )
+
+        session = load_session(session_id)
+        if session:
+            output_script_path = session["output_path"]
+            shared_adapter_mgr = _SharedAdapterManager()
+        else:
+            from cli.session import start_session_recording
+
+            output_script_path = resolve_session_output_path(session_id, args.platform)
+            session = create_session(session_id, args.platform, output_script_path)
+            shared_adapter_mgr = _SharedAdapterManager()
+            start_session_recording(session_id, args.platform)
+    else:
+        output_script_path = _resolve_output_script_path(args)
+
+    log.info("=" * 60)
+    log.info("Starting ScreenForge UI automation engine")
+    target_label = (
+        args.goal
+        or getattr(args, "action_name", "")
+        or getattr(args, "action", "")
+        or args.workflow
+        or "doctor / no-goal mode"
+    )
+    log.info(f"Target: {target_label}")
+    log.info(f"Circuit breaker: max {args.max_retries} retries per step")
+    log.info(
+        f"Platform: {args.platform} | Vision: {'on' if args.vision else 'off'}"
+    )
+    log.info(f"Mode: {execution_mode}")
+    log.info(f"Output: {output_script_path}")
+    log.info("=" * 60)
+
+    try:
+        context_content, resume_context = _load_context_content(args)
+    except RunContextLoadError as e:
+        log.error(f"[E011] Failed to restore run context: {e}. Fix: check that report/runs/<run_id>/ exists and contains summary.json")
+        sys.exit(2)
+
+    if _requires_model_runtime(args, execution_mode) and not config.validate_config():
+        log.error("[E012] Configuration validation failed. See errors above for details and fix instructions")
+        sys.exit(1)
+
+    exit_code = _dispatch_execution(
+        args,
+        execution_mode,
+        output_script_path,
+        context_content,
+        resume_context,
+        shared_adapter_manager=shared_adapter_mgr,
+    )
+
+    if session_id and args.action:
+        from cli.session import load_session, update_session
+
+        session = load_session(session_id)
+        if session and exit_code == 0:
+            update_session(session_id, steps=session.get("steps", 0) + 1)
+
+    sys.exit(exit_code)