haliosai-cli 2.0.0__tar.gz → 2.0.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (32) hide show
  1. haliosai_cli-2.0.2/PKG-INFO +138 -0
  2. haliosai_cli-2.0.2/README.md +101 -0
  3. haliosai_cli-2.0.2/halios_cli/_version.py +1 -0
  4. haliosai_cli-2.0.2/halios_cli/cli_auth.py +171 -0
  5. haliosai_cli-2.0.2/haliosai_cli.egg-info/PKG-INFO +138 -0
  6. haliosai_cli-2.0.0/PKG-INFO +0 -101
  7. haliosai_cli-2.0.0/README.md +0 -64
  8. haliosai_cli-2.0.0/halios_cli/_version.py +0 -1
  9. haliosai_cli-2.0.0/halios_cli/cli_auth.py +0 -162
  10. haliosai_cli-2.0.0/haliosai_cli.egg-info/PKG-INFO +0 -101
  11. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/LICENSE +0 -0
  12. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/MANIFEST.in +0 -0
  13. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/__init__.py +0 -0
  14. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli.py +0 -0
  15. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_eval.py +0 -0
  16. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_optimize.py +0 -0
  17. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_project.py +0 -0
  18. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_scenario.py +0 -0
  19. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_support.py +0 -0
  20. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_trace.py +0 -0
  21. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/py.typed +0 -0
  22. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/schemas/__init__.py +0 -0
  23. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/schemas/eval.schema.json +0 -0
  24. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/schemas/scenarios.schema.json +0 -0
  25. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/haliosai_cli.egg-info/SOURCES.txt +0 -0
  26. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/haliosai_cli.egg-info/dependency_links.txt +0 -0
  27. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/haliosai_cli.egg-info/entry_points.txt +0 -0
  28. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/haliosai_cli.egg-info/requires.txt +0 -0
  29. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/haliosai_cli.egg-info/top_level.txt +0 -0
  30. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/pyproject.toml +0 -0
  31. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/setup.cfg +0 -0
  32. {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/tests/test_cli_surface.py +0 -0
@@ -0,0 +1,138 @@
1
+ Metadata-Version: 2.4
2
+ Name: haliosai-cli
3
+ Version: 2.0.2
4
+ Summary: Halios CLI for coding-agent evaluations, guardrails, and OpenTelemetry evidence
5
+ Author-email: HaliosLabs <support@halios.ai>
6
+ License-Expression: Apache-2.0
7
+ Project-URL: Homepage, https://halios.ai
8
+ Project-URL: Documentation, https://docs.halios.ai
9
+ Project-URL: Repository, https://github.com/HaliosAI/halios
10
+ Project-URL: Changelog, https://github.com/HaliosAI/halios/releases
11
+ Keywords: ai,agents,evaluation,guardrails,opentelemetry,cli
12
+ Classifier: Development Status :: 4 - Beta
13
+ Classifier: Environment :: Console
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: Programming Language :: Python :: 3 :: Only
16
+ Classifier: Programming Language :: Python :: 3.10
17
+ Classifier: Programming Language :: Python :: 3.11
18
+ Classifier: Programming Language :: Python :: 3.12
19
+ Classifier: Programming Language :: Python :: 3.13
20
+ Classifier: Programming Language :: Python :: 3.14
21
+ Classifier: Topic :: Software Development :: Testing
22
+ Requires-Python: >=3.10
23
+ Description-Content-Type: text/markdown
24
+ License-File: LICENSE
25
+ Requires-Dist: httpx<1,>=0.27
26
+ Requires-Dist: jsonschema<5,>=4.18
27
+ Requires-Dist: PyYAML<7,>=6.0
28
+ Requires-Dist: rich<15,>=13
29
+ Requires-Dist: typer<1,>=0.12
30
+ Requires-Dist: tomli>=2.0; python_version < "3.11"
31
+ Provides-Extra: dev
32
+ Requires-Dist: build>=1.2; extra == "dev"
33
+ Requires-Dist: pytest>=8; extra == "dev"
34
+ Requires-Dist: ruff>=0.9; extra == "dev"
35
+ Requires-Dist: twine>=6; extra == "dev"
36
+ Dynamic: license-file
37
+
38
+ # Halios
39
+
40
+ [![PyPI version](https://img.shields.io/pypi/v/haliosai-cli.svg)](https://pypi.org/project/haliosai-cli/)
41
+ [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
42
+
43
+ **Your pair programmer for AI agent evaluations.**
44
+
45
+ Halios helps you build reliable AI agents in minutes. Add the Halios skill to **Codex**, **Claude Code**, **Cursor**, or your favorite coding agent, and create eval suites, run multi-turn simulations, investigate failures, and gate pull requests directly from your repository.
46
+
47
+ Halios provides the evaluation framework, developer tools, and hosted evaluation runtime underneath.
48
+
49
+ ---
50
+
51
+ ## Quickstart
52
+
53
+ ### 1. Install the Agent Skill
54
+
55
+ Add the Halios skill to your coding agent environment:
56
+
57
+ ```bash
58
+ npx skills add HaliosAI/halios --skill halios
59
+ ```
60
+
61
+ *(Works with Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and any harness supporting the open Agent Skills format).*
62
+
63
+ ### 2. Ask Your Coding Agent
64
+
65
+ Prompt your agent to set up evaluations for your project:
66
+
67
+ ```text
68
+ "Set up evals for this agent: inspect the repository, create realistic test scenarios and checks, and run a baseline."
69
+ ```
70
+
71
+ Your agent will inspect your application entrypoint, configure standard OpenTelemetry, draft scenarios in `.halios/`, and run baseline evaluations.
72
+
73
+ ---
74
+
75
+ ### Standalone CLI Installation
76
+
77
+ If you prefer to drive evaluations directly from the command line or CI:
78
+
79
+ ```bash
80
+ # Recommended: Install with uv tool
81
+ uv tool install 'haliosai-cli>=2.0.0'
82
+
83
+ # Or install with pipx
84
+ pipx install 'haliosai-cli>=2.0.0'
85
+
86
+ # See available commands and usage
87
+ halios --help
88
+ ```
89
+
90
+ ---
91
+
92
+ ## How It Works
93
+
94
+ 1. **Inspect & Connect**: Your coding agent inspects your agent's tools, policies, and runtime, then configures standard OpenTelemetry export.
95
+ 2. **Author Scenarios & Checks**: Test cases, rubrics, and failure criteria are stored directly in your repository (`.halios/scenarios.yml` and `.halios/eval.yml`). You own the evaluation suite.
96
+ 3. **Simulate Multi-Turn Trajectories**: Halios runs fresh simulation passes against your agent across edge cases, tool dependencies, and user personas.
97
+ 4. **Investigate & Fix Failures**: Pinpoint hallucinated parameters, broken tool handoffs, or policy violations from complete trace evidence.
98
+ 5. **Gate Pull Requests**: Run protected checks in CI to block regressions before merging to production.
99
+ 6. **Monitor Production**: Evaluate production traces using the same checks, and turn real-world failures into new regression test scenarios.
100
+
101
+ ---
102
+
103
+ ## Key Principles
104
+
105
+ - **You own the evaluation suite**: Scenarios and checks are clean YAML files stored in your Git repository.
106
+ - **Fresh simulations, not static replays**: Halios tests real agent execution across multiple turns, rather than replaying outdated completions.
107
+ - **Framework agnostic**: Works with any agent architecture—OpenAI Agents SDK, LangChain, LlamaIndex, PydanticAI, or custom workflows.
108
+ - **Stock OpenTelemetry**: Applications emit standard OpenTelemetry GenAI spans; no proprietary vendor lock-in in your production runtime.
109
+ - **Unified local, CI, and production loop**: The same checks run during local development, gate merge requests in CI, and monitor live production traces.
110
+
111
+ ---
112
+
113
+ ## Resources
114
+
115
+ - **Website**: [halios.ai](https://halios.ai)
116
+ - **Documentation**: [docs.halios.ai](https://docs.halios.ai)
117
+ - **Python SDK**: [github.com/HaliosAI/haliosai-python-sdk](https://github.com/HaliosAI/haliosai-python-sdk)
118
+ - **Public Skill Source**: [`skills/halios/`](skills/halios/)
119
+
120
+ ---
121
+
122
+ ## Development
123
+
124
+ ```bash
125
+ # Clone and install locally in editable mode
126
+ git clone https://github.com/HaliosAI/halios.git
127
+ cd halios
128
+ python -m pip install -e '.[dev]'
129
+
130
+ # Run the test suite
131
+ python -m pytest -q
132
+ ```
133
+
134
+ ---
135
+
136
+ ## License
137
+
138
+ [Apache 2.0](LICENSE) © Anomalytica Inc. 2026
@@ -0,0 +1,101 @@
1
+ # Halios
2
+
3
+ [![PyPI version](https://img.shields.io/pypi/v/haliosai-cli.svg)](https://pypi.org/project/haliosai-cli/)
4
+ [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
5
+
6
+ **Your pair programmer for AI agent evaluations.**
7
+
8
+ Halios helps you build reliable AI agents in minutes. Add the Halios skill to **Codex**, **Claude Code**, **Cursor**, or your favorite coding agent, and create eval suites, run multi-turn simulations, investigate failures, and gate pull requests directly from your repository.
9
+
10
+ Halios provides the evaluation framework, developer tools, and hosted evaluation runtime underneath.
11
+
12
+ ---
13
+
14
+ ## Quickstart
15
+
16
+ ### 1. Install the Agent Skill
17
+
18
+ Add the Halios skill to your coding agent environment:
19
+
20
+ ```bash
21
+ npx skills add HaliosAI/halios --skill halios
22
+ ```
23
+
24
+ *(Works with Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and any harness supporting the open Agent Skills format).*
25
+
26
+ ### 2. Ask Your Coding Agent
27
+
28
+ Prompt your agent to set up evaluations for your project:
29
+
30
+ ```text
31
+ "Set up evals for this agent: inspect the repository, create realistic test scenarios and checks, and run a baseline."
32
+ ```
33
+
34
+ Your agent will inspect your application entrypoint, configure standard OpenTelemetry, draft scenarios in `.halios/`, and run baseline evaluations.
35
+
36
+ ---
37
+
38
+ ### Standalone CLI Installation
39
+
40
+ If you prefer to drive evaluations directly from the command line or CI:
41
+
42
+ ```bash
43
+ # Recommended: Install with uv tool
44
+ uv tool install 'haliosai-cli>=2.0.0'
45
+
46
+ # Or install with pipx
47
+ pipx install 'haliosai-cli>=2.0.0'
48
+
49
+ # See available commands and usage
50
+ halios --help
51
+ ```
52
+
53
+ ---
54
+
55
+ ## How It Works
56
+
57
+ 1. **Inspect & Connect**: Your coding agent inspects your agent's tools, policies, and runtime, then configures standard OpenTelemetry export.
58
+ 2. **Author Scenarios & Checks**: Test cases, rubrics, and failure criteria are stored directly in your repository (`.halios/scenarios.yml` and `.halios/eval.yml`). You own the evaluation suite.
59
+ 3. **Simulate Multi-Turn Trajectories**: Halios runs fresh simulation passes against your agent across edge cases, tool dependencies, and user personas.
60
+ 4. **Investigate & Fix Failures**: Pinpoint hallucinated parameters, broken tool handoffs, or policy violations from complete trace evidence.
61
+ 5. **Gate Pull Requests**: Run protected checks in CI to block regressions before merging to production.
62
+ 6. **Monitor Production**: Evaluate production traces using the same checks, and turn real-world failures into new regression test scenarios.
63
+
64
+ ---
65
+
66
+ ## Key Principles
67
+
68
+ - **You own the evaluation suite**: Scenarios and checks are clean YAML files stored in your Git repository.
69
+ - **Fresh simulations, not static replays**: Halios tests real agent execution across multiple turns, rather than replaying outdated completions.
70
+ - **Framework agnostic**: Works with any agent architecture—OpenAI Agents SDK, LangChain, LlamaIndex, PydanticAI, or custom workflows.
71
+ - **Stock OpenTelemetry**: Applications emit standard OpenTelemetry GenAI spans; no proprietary vendor lock-in in your production runtime.
72
+ - **Unified local, CI, and production loop**: The same checks run during local development, gate merge requests in CI, and monitor live production traces.
73
+
74
+ ---
75
+
76
+ ## Resources
77
+
78
+ - **Website**: [halios.ai](https://halios.ai)
79
+ - **Documentation**: [docs.halios.ai](https://docs.halios.ai)
80
+ - **Python SDK**: [github.com/HaliosAI/haliosai-python-sdk](https://github.com/HaliosAI/haliosai-python-sdk)
81
+ - **Public Skill Source**: [`skills/halios/`](skills/halios/)
82
+
83
+ ---
84
+
85
+ ## Development
86
+
87
+ ```bash
88
+ # Clone and install locally in editable mode
89
+ git clone https://github.com/HaliosAI/halios.git
90
+ cd halios
91
+ python -m pip install -e '.[dev]'
92
+
93
+ # Run the test suite
94
+ python -m pytest -q
95
+ ```
96
+
97
+ ---
98
+
99
+ ## License
100
+
101
+ [Apache 2.0](LICENSE) © Anomalytica Inc. 2026
@@ -0,0 +1 @@
1
+ __version__ = "2.0.2"
@@ -0,0 +1,171 @@
1
+ """User-level CLI authentication commands."""
2
+
3
+ from __future__ import annotations
4
+
5
+ import time
6
+
7
+ import httpx
8
+ import typer
9
+
10
+ from .cli_support import (
11
+ DEFAULT_BASE_URL,
12
+ ApiClient,
13
+ ApiError,
14
+ delete_profile,
15
+ normalize_url,
16
+ resolve_credentials,
17
+ save_profile,
18
+ stored_profile_credentials,
19
+ )
20
+
21
+ app = typer.Typer(help="Authenticate the Halios CLI.", no_args_is_help=True)
22
+
23
+ # INVARIANT: Match the server-side poll interval from CLIAuthRequestResponse.interval.
24
+ _POLL_INTERVAL_S = 2
25
+
26
+
27
+ @app.command("login")
28
+ def login(
29
+ profile: str = typer.Option("default", "--profile"),
30
+ base_url: str = typer.Option(DEFAULT_BASE_URL, "--base-url"),
31
+ api_key: str | None = typer.Option(None, "--api-key", envvar="HALIOS_API_KEY", hidden=True),
32
+ ) -> None:
33
+ """Authorize this machine and store credentials outside the repository.
34
+
35
+ Opens your browser to complete authorization. No local HTTP server is
36
+ started; the CLI polls the Halios API until you approve or the session expires.
37
+ """
38
+ normalized_url = normalize_url(base_url)
39
+
40
+ # Fast path: explicit API key provided (e.g. CI environments or manual --api-key flag).
41
+ if api_key:
42
+ save_profile(profile, base_url=normalized_url, api_key=api_key)
43
+ typer.echo(f"Logged in as profile '{profile}'.")
44
+ return
45
+
46
+ # --- Step 1: Request a device-code session from the backend ---
47
+ try:
48
+ with httpx.Client(timeout=10) as http:
49
+ resp = http.post(
50
+ f"{normalized_url}/api/v1/cli-auth/request",
51
+ params={"base_url": normalized_url},
52
+ )
53
+ resp.raise_for_status()
54
+ session_data = resp.json()
55
+ except httpx.HTTPStatusError as exc:
56
+ if exc.response.content:
57
+ detail = exc.response.json().get("detail", exc.response.text)
58
+ else:
59
+ detail = str(exc)
60
+ raise typer.BadParameter(f"Could not start login session: {detail}") from exc
61
+ except httpx.HTTPError as exc:
62
+ raise typer.BadParameter(
63
+ f"Could not reach {normalized_url}. Check your network or --base-url."
64
+ ) from exc
65
+
66
+ device_code: str = session_data["device_code"]
67
+ # INTENT: CLI builds verification_url itself so it uses its own configured base_url,
68
+ # not whatever the server returned (which may differ in local dev setups).
69
+ verification_url = f"{normalized_url}/cli-auth?code={device_code}&profile={profile}"
70
+ expires_in: int = session_data.get("expires_in", 300)
71
+
72
+ # --- Step 2: Open the browser and tell the user what's happening ---
73
+ typer.echo(f"\nOpening your browser to authorize the CLI:\n {verification_url}\n")
74
+ typer.echo("Waiting for authorization in browser… (Ctrl+C to cancel)\n")
75
+
76
+ import webbrowser
77
+ webbrowser.open(verification_url)
78
+
79
+ # --- Step 3: Poll until authorized, denied, or expired ---
80
+ poll_url = f"{normalized_url}/api/v1/cli-auth/poll"
81
+ deadline = time.monotonic() + expires_in
82
+
83
+ try:
84
+ with httpx.Client(timeout=10) as http:
85
+ while time.monotonic() < deadline:
86
+ time.sleep(_POLL_INTERVAL_S)
87
+ try:
88
+ poll_resp = http.get(poll_url, params={"code": device_code})
89
+ poll_resp.raise_for_status()
90
+ result = poll_resp.json()
91
+ except httpx.HTTPError:
92
+ # WHY: Transient network errors should not abort the wait.
93
+ continue
94
+
95
+ poll_status: str = result.get("status", "expired")
96
+
97
+ if poll_status == "pending":
98
+ continue
99
+
100
+ if poll_status == "authorized":
101
+ save_profile(
102
+ profile,
103
+ base_url=normalized_url,
104
+ api_key=result["api_key"],
105
+ organization_id=result.get("organization_id"),
106
+ api_key_id=int(result["api_key_id"]) if result.get("api_key_id") else None,
107
+ expires_at=result.get("expires_at"),
108
+ )
109
+ typer.echo(f"Logged in as profile '{profile}'.")
110
+ return
111
+
112
+ if poll_status == "denied":
113
+ raise typer.BadParameter("Authorization was cancelled in the browser.")
114
+
115
+ # status == "expired" or unknown
116
+ raise typer.BadParameter(
117
+ "Login session expired. Please run 'halios auth login' again."
118
+ )
119
+
120
+ except KeyboardInterrupt:
121
+ typer.echo("\nLogin cancelled.")
122
+ raise typer.Exit(code=1)
123
+
124
+ raise typer.BadParameter("Login timed out. Please run 'halios auth login' again.")
125
+
126
+
127
+ @app.command("status")
128
+ def status(profile: str = typer.Option("default", "--profile")) -> None:
129
+ """Verify the selected profile without exposing its credential."""
130
+ credentials = resolve_credentials(profile)
131
+ with ApiClient(credentials) as api:
132
+ api.request("GET", "/api/v1/agents", params={"limit": 1})
133
+ expiry = f" Credential expires {credentials.expires_at}." if credentials.expires_at else ""
134
+ typer.echo(f"Authenticated profile '{profile}' at {credentials.base_url}.{expiry}")
135
+
136
+
137
+ @app.command("logout")
138
+ def logout(
139
+ profile: str = typer.Option("default", "--profile"),
140
+ local_only: bool = typer.Option(
141
+ False,
142
+ "--local-only",
143
+ help="Remove local credentials without revoking the remote key.",
144
+ ),
145
+ ) -> None:
146
+ """Revoke one CLI credential, then remove its local profile."""
147
+ credentials = stored_profile_credentials(profile)
148
+ if credentials is None:
149
+ typer.echo(f"Profile '{profile}' was not stored.")
150
+ return
151
+
152
+ if not local_only:
153
+ try:
154
+ with ApiClient(credentials) as api:
155
+ api.request("DELETE", "/api/v1/api-keys/current")
156
+ except ApiError as exc:
157
+ if exc.status_code != 401:
158
+ raise typer.BadParameter(
159
+ "Remote revocation failed; local credentials were kept. "
160
+ "Retry, or use --local-only if the server is permanently unavailable."
161
+ ) from exc
162
+ typer.echo("The remote credential was already invalid or expired.")
163
+ except httpx.HTTPError as exc:
164
+ raise typer.BadParameter(
165
+ "Could not reach Halios to revoke the credential; local credentials were kept. "
166
+ "Retry, or use --local-only if the server is permanently unavailable."
167
+ ) from exc
168
+
169
+ delete_profile(profile)
170
+ suffix = " locally (remote key unchanged)" if local_only else " and revoked its API key"
171
+ typer.echo(f"Removed profile '{profile}'{suffix}.")
@@ -0,0 +1,138 @@
1
+ Metadata-Version: 2.4
2
+ Name: haliosai-cli
3
+ Version: 2.0.2
4
+ Summary: Halios CLI for coding-agent evaluations, guardrails, and OpenTelemetry evidence
5
+ Author-email: HaliosLabs <support@halios.ai>
6
+ License-Expression: Apache-2.0
7
+ Project-URL: Homepage, https://halios.ai
8
+ Project-URL: Documentation, https://docs.halios.ai
9
+ Project-URL: Repository, https://github.com/HaliosAI/halios
10
+ Project-URL: Changelog, https://github.com/HaliosAI/halios/releases
11
+ Keywords: ai,agents,evaluation,guardrails,opentelemetry,cli
12
+ Classifier: Development Status :: 4 - Beta
13
+ Classifier: Environment :: Console
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: Programming Language :: Python :: 3 :: Only
16
+ Classifier: Programming Language :: Python :: 3.10
17
+ Classifier: Programming Language :: Python :: 3.11
18
+ Classifier: Programming Language :: Python :: 3.12
19
+ Classifier: Programming Language :: Python :: 3.13
20
+ Classifier: Programming Language :: Python :: 3.14
21
+ Classifier: Topic :: Software Development :: Testing
22
+ Requires-Python: >=3.10
23
+ Description-Content-Type: text/markdown
24
+ License-File: LICENSE
25
+ Requires-Dist: httpx<1,>=0.27
26
+ Requires-Dist: jsonschema<5,>=4.18
27
+ Requires-Dist: PyYAML<7,>=6.0
28
+ Requires-Dist: rich<15,>=13
29
+ Requires-Dist: typer<1,>=0.12
30
+ Requires-Dist: tomli>=2.0; python_version < "3.11"
31
+ Provides-Extra: dev
32
+ Requires-Dist: build>=1.2; extra == "dev"
33
+ Requires-Dist: pytest>=8; extra == "dev"
34
+ Requires-Dist: ruff>=0.9; extra == "dev"
35
+ Requires-Dist: twine>=6; extra == "dev"
36
+ Dynamic: license-file
37
+
38
+ # Halios
39
+
40
+ [![PyPI version](https://img.shields.io/pypi/v/haliosai-cli.svg)](https://pypi.org/project/haliosai-cli/)
41
+ [![License](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](LICENSE)
42
+
43
+ **Your pair programmer for AI agent evaluations.**
44
+
45
+ Halios helps you build reliable AI agents in minutes. Add the Halios skill to **Codex**, **Claude Code**, **Cursor**, or your favorite coding agent, and create eval suites, run multi-turn simulations, investigate failures, and gate pull requests directly from your repository.
46
+
47
+ Halios provides the evaluation framework, developer tools, and hosted evaluation runtime underneath.
48
+
49
+ ---
50
+
51
+ ## Quickstart
52
+
53
+ ### 1. Install the Agent Skill
54
+
55
+ Add the Halios skill to your coding agent environment:
56
+
57
+ ```bash
58
+ npx skills add HaliosAI/halios --skill halios
59
+ ```
60
+
61
+ *(Works with Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and any harness supporting the open Agent Skills format).*
62
+
63
+ ### 2. Ask Your Coding Agent
64
+
65
+ Prompt your agent to set up evaluations for your project:
66
+
67
+ ```text
68
+ "Set up evals for this agent: inspect the repository, create realistic test scenarios and checks, and run a baseline."
69
+ ```
70
+
71
+ Your agent will inspect your application entrypoint, configure standard OpenTelemetry, draft scenarios in `.halios/`, and run baseline evaluations.
72
+
73
+ ---
74
+
75
+ ### Standalone CLI Installation
76
+
77
+ If you prefer to drive evaluations directly from the command line or CI:
78
+
79
+ ```bash
80
+ # Recommended: Install with uv tool
81
+ uv tool install 'haliosai-cli>=2.0.0'
82
+
83
+ # Or install with pipx
84
+ pipx install 'haliosai-cli>=2.0.0'
85
+
86
+ # See available commands and usage
87
+ halios --help
88
+ ```
89
+
90
+ ---
91
+
92
+ ## How It Works
93
+
94
+ 1. **Inspect & Connect**: Your coding agent inspects your agent's tools, policies, and runtime, then configures standard OpenTelemetry export.
95
+ 2. **Author Scenarios & Checks**: Test cases, rubrics, and failure criteria are stored directly in your repository (`.halios/scenarios.yml` and `.halios/eval.yml`). You own the evaluation suite.
96
+ 3. **Simulate Multi-Turn Trajectories**: Halios runs fresh simulation passes against your agent across edge cases, tool dependencies, and user personas.
97
+ 4. **Investigate & Fix Failures**: Pinpoint hallucinated parameters, broken tool handoffs, or policy violations from complete trace evidence.
98
+ 5. **Gate Pull Requests**: Run protected checks in CI to block regressions before merging to production.
99
+ 6. **Monitor Production**: Evaluate production traces using the same checks, and turn real-world failures into new regression test scenarios.
100
+
101
+ ---
102
+
103
+ ## Key Principles
104
+
105
+ - **You own the evaluation suite**: Scenarios and checks are clean YAML files stored in your Git repository.
106
+ - **Fresh simulations, not static replays**: Halios tests real agent execution across multiple turns, rather than replaying outdated completions.
107
+ - **Framework agnostic**: Works with any agent architecture—OpenAI Agents SDK, LangChain, LlamaIndex, PydanticAI, or custom workflows.
108
+ - **Stock OpenTelemetry**: Applications emit standard OpenTelemetry GenAI spans; no proprietary vendor lock-in in your production runtime.
109
+ - **Unified local, CI, and production loop**: The same checks run during local development, gate merge requests in CI, and monitor live production traces.
110
+
111
+ ---
112
+
113
+ ## Resources
114
+
115
+ - **Website**: [halios.ai](https://halios.ai)
116
+ - **Documentation**: [docs.halios.ai](https://docs.halios.ai)
117
+ - **Python SDK**: [github.com/HaliosAI/haliosai-python-sdk](https://github.com/HaliosAI/haliosai-python-sdk)
118
+ - **Public Skill Source**: [`skills/halios/`](skills/halios/)
119
+
120
+ ---
121
+
122
+ ## Development
123
+
124
+ ```bash
125
+ # Clone and install locally in editable mode
126
+ git clone https://github.com/HaliosAI/halios.git
127
+ cd halios
128
+ python -m pip install -e '.[dev]'
129
+
130
+ # Run the test suite
131
+ python -m pytest -q
132
+ ```
133
+
134
+ ---
135
+
136
+ ## License
137
+
138
+ [Apache 2.0](LICENSE) © Anomalytica Inc. 2026
@@ -1,101 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: haliosai-cli
3
- Version: 2.0.0
4
- Summary: Halios CLI for coding-agent evaluations, guardrails, and OpenTelemetry evidence
5
- Author-email: HaliosLabs <support@halios.ai>
6
- License-Expression: Apache-2.0
7
- Project-URL: Homepage, https://halios.ai
8
- Project-URL: Documentation, https://docs.halios.ai
9
- Project-URL: Repository, https://github.com/HaliosAI/halios
10
- Project-URL: Changelog, https://github.com/HaliosAI/halios/releases
11
- Keywords: ai,agents,evaluation,guardrails,opentelemetry,cli
12
- Classifier: Development Status :: 4 - Beta
13
- Classifier: Environment :: Console
14
- Classifier: Intended Audience :: Developers
15
- Classifier: Programming Language :: Python :: 3 :: Only
16
- Classifier: Programming Language :: Python :: 3.10
17
- Classifier: Programming Language :: Python :: 3.11
18
- Classifier: Programming Language :: Python :: 3.12
19
- Classifier: Programming Language :: Python :: 3.13
20
- Classifier: Programming Language :: Python :: 3.14
21
- Classifier: Topic :: Software Development :: Testing
22
- Requires-Python: >=3.10
23
- Description-Content-Type: text/markdown
24
- License-File: LICENSE
25
- Requires-Dist: httpx<1,>=0.27
26
- Requires-Dist: jsonschema<5,>=4.18
27
- Requires-Dist: PyYAML<7,>=6.0
28
- Requires-Dist: rich<15,>=13
29
- Requires-Dist: typer<1,>=0.12
30
- Requires-Dist: tomli>=2.0; python_version < "3.11"
31
- Provides-Extra: dev
32
- Requires-Dist: build>=1.2; extra == "dev"
33
- Requires-Dist: pytest>=8; extra == "dev"
34
- Requires-Dist: ruff>=0.9; extra == "dev"
35
- Requires-Dist: twine>=6; extra == "dev"
36
- Dynamic: license-file
37
-
38
- # Halios CLI and Agent Skill
39
-
40
- The `halios` command is the machine-facing client used by the Halios Agent Skill and CI. It owns
41
- repository setup, evaluation-suite authoring, fresh scenario execution, evidence inspection, and
42
- prompt-optimization handoffs. Applications export their runtime telemetry with stock
43
- OpenTelemetry; the CLI is not an application tracing SDK.
44
-
45
- ```bash
46
- uv tool install 'haliosai-cli>=2.0.0'
47
- halios --version
48
- ```
49
-
50
- Use `pipx install 'haliosai-cli>=2.0.0'` when `uv` is unavailable. Install the cross-harness skill
51
- separately:
52
-
53
- ```bash
54
- npx skills add HaliosAI/halios --skill halios
55
- ```
56
-
57
- The stable project workflow is:
58
-
59
- ```bash
60
- halios auth login
61
- halios project init --agent support-assistant --command "python tests/halios_adapter.py"
62
- halios eval review --json
63
- halios project configure --json
64
- halios project check --json
65
- halios eval run -k 3 --json
66
- ```
67
-
68
- Prompt optimization is driven by the coding agent and the normal immutable eval workflow:
69
-
70
- ```bash
71
- halios optimize start --baseline-run <run-id> --prompt-file path/to/system-prompt.txt --json
72
- # make one bounded prompt edit from the returned guidance, then run the unchanged suite
73
- halios eval run -k 3 --json
74
- halios optimize record <optimization-run-id> \
75
- --evaluation-run <candidate-eval-run-id> \
76
- --prompt-file path/to/system-prompt.txt \
77
- --json
78
- ```
79
-
80
- Credentials are stored outside the repository. The repository contains only `.halios/config.toml`,
81
- `.halios/eval.yml`, and `.halios/scenarios.yml`.
82
-
83
- In non-interactive CI, use `HALIOS_API_KEY` for control-plane commands and the separate
84
- agent-scoped `HALIOS_OTLP_TOKEN` for adapter trace export. Only trusted default-branch jobs should
85
- receive `HALIOS_CI_PUBLISH_TOKEN`. The jsonl adapter is used by local/CI simulation only; deployed
86
- staging and production applications export their own stock OpenTelemetry directly to Halios.
87
-
88
- See the repository's
89
- [`skills/halios`](https://github.com/HaliosAI/halios/tree/main/skills/halios) workflow for
90
- coding-agent setup and
91
- [`OTEL_SCHEMA.md`](https://github.com/HaliosAI/haliosai-python-sdk/blob/main/OTEL_SCHEMA.md)
92
- for the telemetry accepted by Halios.
93
-
94
- ## Development
95
-
96
- ```bash
97
- python -m pip install -e '.[dev]'
98
- python -m pytest -q
99
- ```
100
-
101
- See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution and release process.
@@ -1,64 +0,0 @@
1
- # Halios CLI and Agent Skill
2
-
3
- The `halios` command is the machine-facing client used by the Halios Agent Skill and CI. It owns
4
- repository setup, evaluation-suite authoring, fresh scenario execution, evidence inspection, and
5
- prompt-optimization handoffs. Applications export their runtime telemetry with stock
6
- OpenTelemetry; the CLI is not an application tracing SDK.
7
-
8
- ```bash
9
- uv tool install 'haliosai-cli>=2.0.0'
10
- halios --version
11
- ```
12
-
13
- Use `pipx install 'haliosai-cli>=2.0.0'` when `uv` is unavailable. Install the cross-harness skill
14
- separately:
15
-
16
- ```bash
17
- npx skills add HaliosAI/halios --skill halios
18
- ```
19
-
20
- The stable project workflow is:
21
-
22
- ```bash
23
- halios auth login
24
- halios project init --agent support-assistant --command "python tests/halios_adapter.py"
25
- halios eval review --json
26
- halios project configure --json
27
- halios project check --json
28
- halios eval run -k 3 --json
29
- ```
30
-
31
- Prompt optimization is driven by the coding agent and the normal immutable eval workflow:
32
-
33
- ```bash
34
- halios optimize start --baseline-run <run-id> --prompt-file path/to/system-prompt.txt --json
35
- # make one bounded prompt edit from the returned guidance, then run the unchanged suite
36
- halios eval run -k 3 --json
37
- halios optimize record <optimization-run-id> \
38
- --evaluation-run <candidate-eval-run-id> \
39
- --prompt-file path/to/system-prompt.txt \
40
- --json
41
- ```
42
-
43
- Credentials are stored outside the repository. The repository contains only `.halios/config.toml`,
44
- `.halios/eval.yml`, and `.halios/scenarios.yml`.
45
-
46
- In non-interactive CI, use `HALIOS_API_KEY` for control-plane commands and the separate
47
- agent-scoped `HALIOS_OTLP_TOKEN` for adapter trace export. Only trusted default-branch jobs should
48
- receive `HALIOS_CI_PUBLISH_TOKEN`. The jsonl adapter is used by local/CI simulation only; deployed
49
- staging and production applications export their own stock OpenTelemetry directly to Halios.
50
-
51
- See the repository's
52
- [`skills/halios`](https://github.com/HaliosAI/halios/tree/main/skills/halios) workflow for
53
- coding-agent setup and
54
- [`OTEL_SCHEMA.md`](https://github.com/HaliosAI/haliosai-python-sdk/blob/main/OTEL_SCHEMA.md)
55
- for the telemetry accepted by Halios.
56
-
57
- ## Development
58
-
59
- ```bash
60
- python -m pip install -e '.[dev]'
61
- python -m pytest -q
62
- ```
63
-
64
- See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution and release process.
@@ -1 +0,0 @@
1
- __version__ = "2.0.0"
@@ -1,162 +0,0 @@
1
- """User-level CLI authentication commands."""
2
-
3
- from __future__ import annotations
4
-
5
- import json
6
- import secrets
7
- import threading
8
- import urllib.parse
9
- import webbrowser
10
- from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
11
-
12
- import httpx
13
- import typer
14
-
15
- from .cli_support import (
16
- DEFAULT_BASE_URL,
17
- ApiClient,
18
- ApiError,
19
- delete_profile,
20
- normalize_url,
21
- resolve_credentials,
22
- save_profile,
23
- stored_profile_credentials,
24
- )
25
-
26
- app = typer.Typer(help="Authenticate the Halios CLI.", no_args_is_help=True)
27
-
28
-
29
- @app.command("login")
30
- def login(
31
- profile: str = typer.Option("default", "--profile"),
32
- base_url: str = typer.Option(DEFAULT_BASE_URL, "--base-url"),
33
- api_key: str | None = typer.Option(None, "--api-key", envvar="HALIOS_API_KEY", hidden=True),
34
- ) -> None:
35
- """Authorize this machine and store credentials outside the repository."""
36
- normalized_url = normalize_url(base_url)
37
- if api_key:
38
- save_profile(profile, base_url=normalized_url, api_key=api_key)
39
- typer.echo(f"Logged in as profile '{profile}'.")
40
- return
41
-
42
- result: dict[str, str] = {}
43
- completed = threading.Event()
44
- state = secrets.token_urlsafe(24)
45
-
46
- class CallbackHandler(BaseHTTPRequestHandler):
47
- def _cors(self) -> None:
48
- self.send_header("Access-Control-Allow-Origin", normalized_url)
49
- self.send_header("Access-Control-Allow-Methods", "POST, OPTIONS")
50
- self.send_header("Access-Control-Allow-Headers", "Content-Type")
51
-
52
- def do_OPTIONS(self) -> None: # noqa: N802
53
- self.send_response(204)
54
- self._cors()
55
- self.end_headers()
56
-
57
- def do_POST(self) -> None: # noqa: N802
58
- if self.path != "/callback":
59
- self.send_error(404)
60
- return
61
- length = int(self.headers.get("content-length") or 0)
62
- try:
63
- body = json.loads(self.rfile.read(length))
64
- except (ValueError, UnicodeDecodeError):
65
- self.send_error(400)
66
- return
67
- if not isinstance(body, dict) or not secrets.compare_digest(
68
- str(body.get("state") or ""), state
69
- ):
70
- self.send_error(422)
71
- return
72
- if body.get("error"):
73
- result["error"] = str(body["error"])
74
- elif body.get("api_key"):
75
- result.update({key: str(value) for key, value in body.items() if value is not None})
76
- else:
77
- self.send_error(422)
78
- return
79
- self.send_response(204)
80
- self._cors()
81
- self.end_headers()
82
- completed.set()
83
-
84
- def log_message(self, _format: str, *_args: object) -> None:
85
- return
86
-
87
- server = ThreadingHTTPServer(("127.0.0.1", 0), CallbackHandler)
88
- thread = threading.Thread(target=server.serve_forever, daemon=True)
89
- thread.start()
90
- login_url = f"{normalized_url}/cli-login?" + urllib.parse.urlencode(
91
- {"port": server.server_port, "profile": profile, "state": state}
92
- )
93
- typer.echo(f"Open this URL to authorize the CLI:\n{login_url}")
94
- webbrowser.open(login_url)
95
- try:
96
- if not completed.wait(timeout=300):
97
- raise typer.BadParameter("Login timed out after 5 minutes")
98
- finally:
99
- server.shutdown()
100
- server.server_close()
101
-
102
- if result.get("error") == "access_denied":
103
- raise typer.BadParameter("Authorization was cancelled in the browser")
104
- if result.get("error"):
105
- raise typer.BadParameter(f"Authorization failed: {result['error']}")
106
-
107
- save_profile(
108
- profile,
109
- base_url=normalized_url,
110
- api_key=result["api_key"],
111
- organization_id=result.get("organization_id"),
112
- api_key_id=int(result["api_key_id"]) if result.get("api_key_id") else None,
113
- expires_at=result.get("expires_at"),
114
- )
115
- typer.echo(f"Logged in as profile '{profile}'.")
116
-
117
-
118
- @app.command("status")
119
- def status(profile: str = typer.Option("default", "--profile")) -> None:
120
- """Verify the selected profile without exposing its credential."""
121
- credentials = resolve_credentials(profile)
122
- with ApiClient(credentials) as api:
123
- api.request("GET", "/api/v1/agents", params={"limit": 1})
124
- expiry = f" Credential expires {credentials.expires_at}." if credentials.expires_at else ""
125
- typer.echo(f"Authenticated profile '{profile}' at {credentials.base_url}.{expiry}")
126
-
127
-
128
- @app.command("logout")
129
- def logout(
130
- profile: str = typer.Option("default", "--profile"),
131
- local_only: bool = typer.Option(
132
- False,
133
- "--local-only",
134
- help="Remove local credentials without revoking the remote key.",
135
- ),
136
- ) -> None:
137
- """Revoke one CLI credential, then remove its local profile."""
138
- credentials = stored_profile_credentials(profile)
139
- if credentials is None:
140
- typer.echo(f"Profile '{profile}' was not stored.")
141
- return
142
-
143
- if not local_only:
144
- try:
145
- with ApiClient(credentials) as api:
146
- api.request("DELETE", "/api/v1/api-keys/current")
147
- except ApiError as exc:
148
- if exc.status_code != 401:
149
- raise typer.BadParameter(
150
- "Remote revocation failed; local credentials were kept. "
151
- "Retry, or use --local-only if the server is permanently unavailable."
152
- ) from exc
153
- typer.echo("The remote credential was already invalid or expired.")
154
- except httpx.HTTPError as exc:
155
- raise typer.BadParameter(
156
- "Could not reach Halios to revoke the credential; local credentials were kept. "
157
- "Retry, or use --local-only if the server is permanently unavailable."
158
- ) from exc
159
-
160
- delete_profile(profile)
161
- suffix = " locally (remote key unchanged)" if local_only else " and revoked its API key"
162
- typer.echo(f"Removed profile '{profile}'{suffix}.")
@@ -1,101 +0,0 @@
1
- Metadata-Version: 2.4
2
- Name: haliosai-cli
3
- Version: 2.0.0
4
- Summary: Halios CLI for coding-agent evaluations, guardrails, and OpenTelemetry evidence
5
- Author-email: HaliosLabs <support@halios.ai>
6
- License-Expression: Apache-2.0
7
- Project-URL: Homepage, https://halios.ai
8
- Project-URL: Documentation, https://docs.halios.ai
9
- Project-URL: Repository, https://github.com/HaliosAI/halios
10
- Project-URL: Changelog, https://github.com/HaliosAI/halios/releases
11
- Keywords: ai,agents,evaluation,guardrails,opentelemetry,cli
12
- Classifier: Development Status :: 4 - Beta
13
- Classifier: Environment :: Console
14
- Classifier: Intended Audience :: Developers
15
- Classifier: Programming Language :: Python :: 3 :: Only
16
- Classifier: Programming Language :: Python :: 3.10
17
- Classifier: Programming Language :: Python :: 3.11
18
- Classifier: Programming Language :: Python :: 3.12
19
- Classifier: Programming Language :: Python :: 3.13
20
- Classifier: Programming Language :: Python :: 3.14
21
- Classifier: Topic :: Software Development :: Testing
22
- Requires-Python: >=3.10
23
- Description-Content-Type: text/markdown
24
- License-File: LICENSE
25
- Requires-Dist: httpx<1,>=0.27
26
- Requires-Dist: jsonschema<5,>=4.18
27
- Requires-Dist: PyYAML<7,>=6.0
28
- Requires-Dist: rich<15,>=13
29
- Requires-Dist: typer<1,>=0.12
30
- Requires-Dist: tomli>=2.0; python_version < "3.11"
31
- Provides-Extra: dev
32
- Requires-Dist: build>=1.2; extra == "dev"
33
- Requires-Dist: pytest>=8; extra == "dev"
34
- Requires-Dist: ruff>=0.9; extra == "dev"
35
- Requires-Dist: twine>=6; extra == "dev"
36
- Dynamic: license-file
37
-
38
- # Halios CLI and Agent Skill
39
-
40
- The `halios` command is the machine-facing client used by the Halios Agent Skill and CI. It owns
41
- repository setup, evaluation-suite authoring, fresh scenario execution, evidence inspection, and
42
- prompt-optimization handoffs. Applications export their runtime telemetry with stock
43
- OpenTelemetry; the CLI is not an application tracing SDK.
44
-
45
- ```bash
46
- uv tool install 'haliosai-cli>=2.0.0'
47
- halios --version
48
- ```
49
-
50
- Use `pipx install 'haliosai-cli>=2.0.0'` when `uv` is unavailable. Install the cross-harness skill
51
- separately:
52
-
53
- ```bash
54
- npx skills add HaliosAI/halios --skill halios
55
- ```
56
-
57
- The stable project workflow is:
58
-
59
- ```bash
60
- halios auth login
61
- halios project init --agent support-assistant --command "python tests/halios_adapter.py"
62
- halios eval review --json
63
- halios project configure --json
64
- halios project check --json
65
- halios eval run -k 3 --json
66
- ```
67
-
68
- Prompt optimization is driven by the coding agent and the normal immutable eval workflow:
69
-
70
- ```bash
71
- halios optimize start --baseline-run <run-id> --prompt-file path/to/system-prompt.txt --json
72
- # make one bounded prompt edit from the returned guidance, then run the unchanged suite
73
- halios eval run -k 3 --json
74
- halios optimize record <optimization-run-id> \
75
- --evaluation-run <candidate-eval-run-id> \
76
- --prompt-file path/to/system-prompt.txt \
77
- --json
78
- ```
79
-
80
- Credentials are stored outside the repository. The repository contains only `.halios/config.toml`,
81
- `.halios/eval.yml`, and `.halios/scenarios.yml`.
82
-
83
- In non-interactive CI, use `HALIOS_API_KEY` for control-plane commands and the separate
84
- agent-scoped `HALIOS_OTLP_TOKEN` for adapter trace export. Only trusted default-branch jobs should
85
- receive `HALIOS_CI_PUBLISH_TOKEN`. The jsonl adapter is used by local/CI simulation only; deployed
86
- staging and production applications export their own stock OpenTelemetry directly to Halios.
87
-
88
- See the repository's
89
- [`skills/halios`](https://github.com/HaliosAI/halios/tree/main/skills/halios) workflow for
90
- coding-agent setup and
91
- [`OTEL_SCHEMA.md`](https://github.com/HaliosAI/haliosai-python-sdk/blob/main/OTEL_SCHEMA.md)
92
- for the telemetry accepted by Halios.
93
-
94
- ## Development
95
-
96
- ```bash
97
- python -m pip install -e '.[dev]'
98
- python -m pytest -q
99
- ```
100
-
101
- See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution and release process.
File without changes
File without changes
File without changes