haliosai-cli 2.0.0__tar.gz → 2.0.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- haliosai_cli-2.0.2/PKG-INFO +138 -0
- haliosai_cli-2.0.2/README.md +101 -0
- haliosai_cli-2.0.2/halios_cli/_version.py +1 -0
- haliosai_cli-2.0.2/halios_cli/cli_auth.py +171 -0
- haliosai_cli-2.0.2/haliosai_cli.egg-info/PKG-INFO +138 -0
- haliosai_cli-2.0.0/PKG-INFO +0 -101
- haliosai_cli-2.0.0/README.md +0 -64
- haliosai_cli-2.0.0/halios_cli/_version.py +0 -1
- haliosai_cli-2.0.0/halios_cli/cli_auth.py +0 -162
- haliosai_cli-2.0.0/haliosai_cli.egg-info/PKG-INFO +0 -101
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/LICENSE +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/MANIFEST.in +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/__init__.py +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli.py +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_eval.py +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_optimize.py +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_project.py +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_scenario.py +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_support.py +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/cli_trace.py +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/py.typed +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/schemas/__init__.py +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/schemas/eval.schema.json +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/halios_cli/schemas/scenarios.schema.json +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/haliosai_cli.egg-info/SOURCES.txt +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/haliosai_cli.egg-info/dependency_links.txt +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/haliosai_cli.egg-info/entry_points.txt +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/haliosai_cli.egg-info/requires.txt +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/haliosai_cli.egg-info/top_level.txt +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/pyproject.toml +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/setup.cfg +0 -0
- {haliosai_cli-2.0.0 → haliosai_cli-2.0.2}/tests/test_cli_surface.py +0 -0
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: haliosai-cli
|
|
3
|
+
Version: 2.0.2
|
|
4
|
+
Summary: Halios CLI for coding-agent evaluations, guardrails, and OpenTelemetry evidence
|
|
5
|
+
Author-email: HaliosLabs <support@halios.ai>
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Project-URL: Homepage, https://halios.ai
|
|
8
|
+
Project-URL: Documentation, https://docs.halios.ai
|
|
9
|
+
Project-URL: Repository, https://github.com/HaliosAI/halios
|
|
10
|
+
Project-URL: Changelog, https://github.com/HaliosAI/halios/releases
|
|
11
|
+
Keywords: ai,agents,evaluation,guardrails,opentelemetry,cli
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Environment :: Console
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
21
|
+
Classifier: Topic :: Software Development :: Testing
|
|
22
|
+
Requires-Python: >=3.10
|
|
23
|
+
Description-Content-Type: text/markdown
|
|
24
|
+
License-File: LICENSE
|
|
25
|
+
Requires-Dist: httpx<1,>=0.27
|
|
26
|
+
Requires-Dist: jsonschema<5,>=4.18
|
|
27
|
+
Requires-Dist: PyYAML<7,>=6.0
|
|
28
|
+
Requires-Dist: rich<15,>=13
|
|
29
|
+
Requires-Dist: typer<1,>=0.12
|
|
30
|
+
Requires-Dist: tomli>=2.0; python_version < "3.11"
|
|
31
|
+
Provides-Extra: dev
|
|
32
|
+
Requires-Dist: build>=1.2; extra == "dev"
|
|
33
|
+
Requires-Dist: pytest>=8; extra == "dev"
|
|
34
|
+
Requires-Dist: ruff>=0.9; extra == "dev"
|
|
35
|
+
Requires-Dist: twine>=6; extra == "dev"
|
|
36
|
+
Dynamic: license-file
|
|
37
|
+
|
|
38
|
+
# Halios
|
|
39
|
+
|
|
40
|
+
[](https://pypi.org/project/haliosai-cli/)
|
|
41
|
+
[](LICENSE)
|
|
42
|
+
|
|
43
|
+
**Your pair programmer for AI agent evaluations.**
|
|
44
|
+
|
|
45
|
+
Halios helps you build reliable AI agents in minutes. Add the Halios skill to **Codex**, **Claude Code**, **Cursor**, or your favorite coding agent, and create eval suites, run multi-turn simulations, investigate failures, and gate pull requests directly from your repository.
|
|
46
|
+
|
|
47
|
+
Halios provides the evaluation framework, developer tools, and hosted evaluation runtime underneath.
|
|
48
|
+
|
|
49
|
+
---
|
|
50
|
+
|
|
51
|
+
## Quickstart
|
|
52
|
+
|
|
53
|
+
### 1. Install the Agent Skill
|
|
54
|
+
|
|
55
|
+
Add the Halios skill to your coding agent environment:
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
npx skills add HaliosAI/halios --skill halios
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
*(Works with Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and any harness supporting the open Agent Skills format).*
|
|
62
|
+
|
|
63
|
+
### 2. Ask Your Coding Agent
|
|
64
|
+
|
|
65
|
+
Prompt your agent to set up evaluations for your project:
|
|
66
|
+
|
|
67
|
+
```text
|
|
68
|
+
"Set up evals for this agent: inspect the repository, create realistic test scenarios and checks, and run a baseline."
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Your agent will inspect your application entrypoint, configure standard OpenTelemetry, draft scenarios in `.halios/`, and run baseline evaluations.
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
### Standalone CLI Installation
|
|
76
|
+
|
|
77
|
+
If you prefer to drive evaluations directly from the command line or CI:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
# Recommended: Install with uv tool
|
|
81
|
+
uv tool install 'haliosai-cli>=2.0.0'
|
|
82
|
+
|
|
83
|
+
# Or install with pipx
|
|
84
|
+
pipx install 'haliosai-cli>=2.0.0'
|
|
85
|
+
|
|
86
|
+
# See available commands and usage
|
|
87
|
+
halios --help
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## How It Works
|
|
93
|
+
|
|
94
|
+
1. **Inspect & Connect**: Your coding agent inspects your agent's tools, policies, and runtime, then configures standard OpenTelemetry export.
|
|
95
|
+
2. **Author Scenarios & Checks**: Test cases, rubrics, and failure criteria are stored directly in your repository (`.halios/scenarios.yml` and `.halios/eval.yml`). You own the evaluation suite.
|
|
96
|
+
3. **Simulate Multi-Turn Trajectories**: Halios runs fresh simulation passes against your agent across edge cases, tool dependencies, and user personas.
|
|
97
|
+
4. **Investigate & Fix Failures**: Pinpoint hallucinated parameters, broken tool handoffs, or policy violations from complete trace evidence.
|
|
98
|
+
5. **Gate Pull Requests**: Run protected checks in CI to block regressions before merging to production.
|
|
99
|
+
6. **Monitor Production**: Evaluate production traces using the same checks, and turn real-world failures into new regression test scenarios.
|
|
100
|
+
|
|
101
|
+
---
|
|
102
|
+
|
|
103
|
+
## Key Principles
|
|
104
|
+
|
|
105
|
+
- **You own the evaluation suite**: Scenarios and checks are clean YAML files stored in your Git repository.
|
|
106
|
+
- **Fresh simulations, not static replays**: Halios tests real agent execution across multiple turns, rather than replaying outdated completions.
|
|
107
|
+
- **Framework agnostic**: Works with any agent architecture—OpenAI Agents SDK, LangChain, LlamaIndex, PydanticAI, or custom workflows.
|
|
108
|
+
- **Stock OpenTelemetry**: Applications emit standard OpenTelemetry GenAI spans; no proprietary vendor lock-in in your production runtime.
|
|
109
|
+
- **Unified local, CI, and production loop**: The same checks run during local development, gate merge requests in CI, and monitor live production traces.
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
|
113
|
+
## Resources
|
|
114
|
+
|
|
115
|
+
- **Website**: [halios.ai](https://halios.ai)
|
|
116
|
+
- **Documentation**: [docs.halios.ai](https://docs.halios.ai)
|
|
117
|
+
- **Python SDK**: [github.com/HaliosAI/haliosai-python-sdk](https://github.com/HaliosAI/haliosai-python-sdk)
|
|
118
|
+
- **Public Skill Source**: [`skills/halios/`](skills/halios/)
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
## Development
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
# Clone and install locally in editable mode
|
|
126
|
+
git clone https://github.com/HaliosAI/halios.git
|
|
127
|
+
cd halios
|
|
128
|
+
python -m pip install -e '.[dev]'
|
|
129
|
+
|
|
130
|
+
# Run the test suite
|
|
131
|
+
python -m pytest -q
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
|
|
136
|
+
## License
|
|
137
|
+
|
|
138
|
+
[Apache 2.0](LICENSE) © Anomalytica Inc. 2026
|
|
@@ -0,0 +1,101 @@
|
|
|
1
|
+
# Halios
|
|
2
|
+
|
|
3
|
+
[](https://pypi.org/project/haliosai-cli/)
|
|
4
|
+
[](LICENSE)
|
|
5
|
+
|
|
6
|
+
**Your pair programmer for AI agent evaluations.**
|
|
7
|
+
|
|
8
|
+
Halios helps you build reliable AI agents in minutes. Add the Halios skill to **Codex**, **Claude Code**, **Cursor**, or your favorite coding agent, and create eval suites, run multi-turn simulations, investigate failures, and gate pull requests directly from your repository.
|
|
9
|
+
|
|
10
|
+
Halios provides the evaluation framework, developer tools, and hosted evaluation runtime underneath.
|
|
11
|
+
|
|
12
|
+
---
|
|
13
|
+
|
|
14
|
+
## Quickstart
|
|
15
|
+
|
|
16
|
+
### 1. Install the Agent Skill
|
|
17
|
+
|
|
18
|
+
Add the Halios skill to your coding agent environment:
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
npx skills add HaliosAI/halios --skill halios
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
*(Works with Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and any harness supporting the open Agent Skills format).*
|
|
25
|
+
|
|
26
|
+
### 2. Ask Your Coding Agent
|
|
27
|
+
|
|
28
|
+
Prompt your agent to set up evaluations for your project:
|
|
29
|
+
|
|
30
|
+
```text
|
|
31
|
+
"Set up evals for this agent: inspect the repository, create realistic test scenarios and checks, and run a baseline."
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Your agent will inspect your application entrypoint, configure standard OpenTelemetry, draft scenarios in `.halios/`, and run baseline evaluations.
|
|
35
|
+
|
|
36
|
+
---
|
|
37
|
+
|
|
38
|
+
### Standalone CLI Installation
|
|
39
|
+
|
|
40
|
+
If you prefer to drive evaluations directly from the command line or CI:
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
# Recommended: Install with uv tool
|
|
44
|
+
uv tool install 'haliosai-cli>=2.0.0'
|
|
45
|
+
|
|
46
|
+
# Or install with pipx
|
|
47
|
+
pipx install 'haliosai-cli>=2.0.0'
|
|
48
|
+
|
|
49
|
+
# See available commands and usage
|
|
50
|
+
halios --help
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
## How It Works
|
|
56
|
+
|
|
57
|
+
1. **Inspect & Connect**: Your coding agent inspects your agent's tools, policies, and runtime, then configures standard OpenTelemetry export.
|
|
58
|
+
2. **Author Scenarios & Checks**: Test cases, rubrics, and failure criteria are stored directly in your repository (`.halios/scenarios.yml` and `.halios/eval.yml`). You own the evaluation suite.
|
|
59
|
+
3. **Simulate Multi-Turn Trajectories**: Halios runs fresh simulation passes against your agent across edge cases, tool dependencies, and user personas.
|
|
60
|
+
4. **Investigate & Fix Failures**: Pinpoint hallucinated parameters, broken tool handoffs, or policy violations from complete trace evidence.
|
|
61
|
+
5. **Gate Pull Requests**: Run protected checks in CI to block regressions before merging to production.
|
|
62
|
+
6. **Monitor Production**: Evaluate production traces using the same checks, and turn real-world failures into new regression test scenarios.
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
66
|
+
## Key Principles
|
|
67
|
+
|
|
68
|
+
- **You own the evaluation suite**: Scenarios and checks are clean YAML files stored in your Git repository.
|
|
69
|
+
- **Fresh simulations, not static replays**: Halios tests real agent execution across multiple turns, rather than replaying outdated completions.
|
|
70
|
+
- **Framework agnostic**: Works with any agent architecture—OpenAI Agents SDK, LangChain, LlamaIndex, PydanticAI, or custom workflows.
|
|
71
|
+
- **Stock OpenTelemetry**: Applications emit standard OpenTelemetry GenAI spans; no proprietary vendor lock-in in your production runtime.
|
|
72
|
+
- **Unified local, CI, and production loop**: The same checks run during local development, gate merge requests in CI, and monitor live production traces.
|
|
73
|
+
|
|
74
|
+
---
|
|
75
|
+
|
|
76
|
+
## Resources
|
|
77
|
+
|
|
78
|
+
- **Website**: [halios.ai](https://halios.ai)
|
|
79
|
+
- **Documentation**: [docs.halios.ai](https://docs.halios.ai)
|
|
80
|
+
- **Python SDK**: [github.com/HaliosAI/haliosai-python-sdk](https://github.com/HaliosAI/haliosai-python-sdk)
|
|
81
|
+
- **Public Skill Source**: [`skills/halios/`](skills/halios/)
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Development
|
|
86
|
+
|
|
87
|
+
```bash
|
|
88
|
+
# Clone and install locally in editable mode
|
|
89
|
+
git clone https://github.com/HaliosAI/halios.git
|
|
90
|
+
cd halios
|
|
91
|
+
python -m pip install -e '.[dev]'
|
|
92
|
+
|
|
93
|
+
# Run the test suite
|
|
94
|
+
python -m pytest -q
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
---
|
|
98
|
+
|
|
99
|
+
## License
|
|
100
|
+
|
|
101
|
+
[Apache 2.0](LICENSE) © Anomalytica Inc. 2026
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
__version__ = "2.0.2"
|
|
@@ -0,0 +1,171 @@
|
|
|
1
|
+
"""User-level CLI authentication commands."""
|
|
2
|
+
|
|
3
|
+
from __future__ import annotations
|
|
4
|
+
|
|
5
|
+
import time
|
|
6
|
+
|
|
7
|
+
import httpx
|
|
8
|
+
import typer
|
|
9
|
+
|
|
10
|
+
from .cli_support import (
|
|
11
|
+
DEFAULT_BASE_URL,
|
|
12
|
+
ApiClient,
|
|
13
|
+
ApiError,
|
|
14
|
+
delete_profile,
|
|
15
|
+
normalize_url,
|
|
16
|
+
resolve_credentials,
|
|
17
|
+
save_profile,
|
|
18
|
+
stored_profile_credentials,
|
|
19
|
+
)
|
|
20
|
+
|
|
21
|
+
app = typer.Typer(help="Authenticate the Halios CLI.", no_args_is_help=True)
|
|
22
|
+
|
|
23
|
+
# INVARIANT: Match the server-side poll interval from CLIAuthRequestResponse.interval.
|
|
24
|
+
_POLL_INTERVAL_S = 2
|
|
25
|
+
|
|
26
|
+
|
|
27
|
+
@app.command("login")
|
|
28
|
+
def login(
|
|
29
|
+
profile: str = typer.Option("default", "--profile"),
|
|
30
|
+
base_url: str = typer.Option(DEFAULT_BASE_URL, "--base-url"),
|
|
31
|
+
api_key: str | None = typer.Option(None, "--api-key", envvar="HALIOS_API_KEY", hidden=True),
|
|
32
|
+
) -> None:
|
|
33
|
+
"""Authorize this machine and store credentials outside the repository.
|
|
34
|
+
|
|
35
|
+
Opens your browser to complete authorization. No local HTTP server is
|
|
36
|
+
started; the CLI polls the Halios API until you approve or the session expires.
|
|
37
|
+
"""
|
|
38
|
+
normalized_url = normalize_url(base_url)
|
|
39
|
+
|
|
40
|
+
# Fast path: explicit API key provided (e.g. CI environments or manual --api-key flag).
|
|
41
|
+
if api_key:
|
|
42
|
+
save_profile(profile, base_url=normalized_url, api_key=api_key)
|
|
43
|
+
typer.echo(f"Logged in as profile '{profile}'.")
|
|
44
|
+
return
|
|
45
|
+
|
|
46
|
+
# --- Step 1: Request a device-code session from the backend ---
|
|
47
|
+
try:
|
|
48
|
+
with httpx.Client(timeout=10) as http:
|
|
49
|
+
resp = http.post(
|
|
50
|
+
f"{normalized_url}/api/v1/cli-auth/request",
|
|
51
|
+
params={"base_url": normalized_url},
|
|
52
|
+
)
|
|
53
|
+
resp.raise_for_status()
|
|
54
|
+
session_data = resp.json()
|
|
55
|
+
except httpx.HTTPStatusError as exc:
|
|
56
|
+
if exc.response.content:
|
|
57
|
+
detail = exc.response.json().get("detail", exc.response.text)
|
|
58
|
+
else:
|
|
59
|
+
detail = str(exc)
|
|
60
|
+
raise typer.BadParameter(f"Could not start login session: {detail}") from exc
|
|
61
|
+
except httpx.HTTPError as exc:
|
|
62
|
+
raise typer.BadParameter(
|
|
63
|
+
f"Could not reach {normalized_url}. Check your network or --base-url."
|
|
64
|
+
) from exc
|
|
65
|
+
|
|
66
|
+
device_code: str = session_data["device_code"]
|
|
67
|
+
# INTENT: CLI builds verification_url itself so it uses its own configured base_url,
|
|
68
|
+
# not whatever the server returned (which may differ in local dev setups).
|
|
69
|
+
verification_url = f"{normalized_url}/cli-auth?code={device_code}&profile={profile}"
|
|
70
|
+
expires_in: int = session_data.get("expires_in", 300)
|
|
71
|
+
|
|
72
|
+
# --- Step 2: Open the browser and tell the user what's happening ---
|
|
73
|
+
typer.echo(f"\nOpening your browser to authorize the CLI:\n {verification_url}\n")
|
|
74
|
+
typer.echo("Waiting for authorization in browser… (Ctrl+C to cancel)\n")
|
|
75
|
+
|
|
76
|
+
import webbrowser
|
|
77
|
+
webbrowser.open(verification_url)
|
|
78
|
+
|
|
79
|
+
# --- Step 3: Poll until authorized, denied, or expired ---
|
|
80
|
+
poll_url = f"{normalized_url}/api/v1/cli-auth/poll"
|
|
81
|
+
deadline = time.monotonic() + expires_in
|
|
82
|
+
|
|
83
|
+
try:
|
|
84
|
+
with httpx.Client(timeout=10) as http:
|
|
85
|
+
while time.monotonic() < deadline:
|
|
86
|
+
time.sleep(_POLL_INTERVAL_S)
|
|
87
|
+
try:
|
|
88
|
+
poll_resp = http.get(poll_url, params={"code": device_code})
|
|
89
|
+
poll_resp.raise_for_status()
|
|
90
|
+
result = poll_resp.json()
|
|
91
|
+
except httpx.HTTPError:
|
|
92
|
+
# WHY: Transient network errors should not abort the wait.
|
|
93
|
+
continue
|
|
94
|
+
|
|
95
|
+
poll_status: str = result.get("status", "expired")
|
|
96
|
+
|
|
97
|
+
if poll_status == "pending":
|
|
98
|
+
continue
|
|
99
|
+
|
|
100
|
+
if poll_status == "authorized":
|
|
101
|
+
save_profile(
|
|
102
|
+
profile,
|
|
103
|
+
base_url=normalized_url,
|
|
104
|
+
api_key=result["api_key"],
|
|
105
|
+
organization_id=result.get("organization_id"),
|
|
106
|
+
api_key_id=int(result["api_key_id"]) if result.get("api_key_id") else None,
|
|
107
|
+
expires_at=result.get("expires_at"),
|
|
108
|
+
)
|
|
109
|
+
typer.echo(f"Logged in as profile '{profile}'.")
|
|
110
|
+
return
|
|
111
|
+
|
|
112
|
+
if poll_status == "denied":
|
|
113
|
+
raise typer.BadParameter("Authorization was cancelled in the browser.")
|
|
114
|
+
|
|
115
|
+
# status == "expired" or unknown
|
|
116
|
+
raise typer.BadParameter(
|
|
117
|
+
"Login session expired. Please run 'halios auth login' again."
|
|
118
|
+
)
|
|
119
|
+
|
|
120
|
+
except KeyboardInterrupt:
|
|
121
|
+
typer.echo("\nLogin cancelled.")
|
|
122
|
+
raise typer.Exit(code=1)
|
|
123
|
+
|
|
124
|
+
raise typer.BadParameter("Login timed out. Please run 'halios auth login' again.")
|
|
125
|
+
|
|
126
|
+
|
|
127
|
+
@app.command("status")
|
|
128
|
+
def status(profile: str = typer.Option("default", "--profile")) -> None:
|
|
129
|
+
"""Verify the selected profile without exposing its credential."""
|
|
130
|
+
credentials = resolve_credentials(profile)
|
|
131
|
+
with ApiClient(credentials) as api:
|
|
132
|
+
api.request("GET", "/api/v1/agents", params={"limit": 1})
|
|
133
|
+
expiry = f" Credential expires {credentials.expires_at}." if credentials.expires_at else ""
|
|
134
|
+
typer.echo(f"Authenticated profile '{profile}' at {credentials.base_url}.{expiry}")
|
|
135
|
+
|
|
136
|
+
|
|
137
|
+
@app.command("logout")
|
|
138
|
+
def logout(
|
|
139
|
+
profile: str = typer.Option("default", "--profile"),
|
|
140
|
+
local_only: bool = typer.Option(
|
|
141
|
+
False,
|
|
142
|
+
"--local-only",
|
|
143
|
+
help="Remove local credentials without revoking the remote key.",
|
|
144
|
+
),
|
|
145
|
+
) -> None:
|
|
146
|
+
"""Revoke one CLI credential, then remove its local profile."""
|
|
147
|
+
credentials = stored_profile_credentials(profile)
|
|
148
|
+
if credentials is None:
|
|
149
|
+
typer.echo(f"Profile '{profile}' was not stored.")
|
|
150
|
+
return
|
|
151
|
+
|
|
152
|
+
if not local_only:
|
|
153
|
+
try:
|
|
154
|
+
with ApiClient(credentials) as api:
|
|
155
|
+
api.request("DELETE", "/api/v1/api-keys/current")
|
|
156
|
+
except ApiError as exc:
|
|
157
|
+
if exc.status_code != 401:
|
|
158
|
+
raise typer.BadParameter(
|
|
159
|
+
"Remote revocation failed; local credentials were kept. "
|
|
160
|
+
"Retry, or use --local-only if the server is permanently unavailable."
|
|
161
|
+
) from exc
|
|
162
|
+
typer.echo("The remote credential was already invalid or expired.")
|
|
163
|
+
except httpx.HTTPError as exc:
|
|
164
|
+
raise typer.BadParameter(
|
|
165
|
+
"Could not reach Halios to revoke the credential; local credentials were kept. "
|
|
166
|
+
"Retry, or use --local-only if the server is permanently unavailable."
|
|
167
|
+
) from exc
|
|
168
|
+
|
|
169
|
+
delete_profile(profile)
|
|
170
|
+
suffix = " locally (remote key unchanged)" if local_only else " and revoked its API key"
|
|
171
|
+
typer.echo(f"Removed profile '{profile}'{suffix}.")
|
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: haliosai-cli
|
|
3
|
+
Version: 2.0.2
|
|
4
|
+
Summary: Halios CLI for coding-agent evaluations, guardrails, and OpenTelemetry evidence
|
|
5
|
+
Author-email: HaliosLabs <support@halios.ai>
|
|
6
|
+
License-Expression: Apache-2.0
|
|
7
|
+
Project-URL: Homepage, https://halios.ai
|
|
8
|
+
Project-URL: Documentation, https://docs.halios.ai
|
|
9
|
+
Project-URL: Repository, https://github.com/HaliosAI/halios
|
|
10
|
+
Project-URL: Changelog, https://github.com/HaliosAI/halios/releases
|
|
11
|
+
Keywords: ai,agents,evaluation,guardrails,opentelemetry,cli
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Environment :: Console
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
21
|
+
Classifier: Topic :: Software Development :: Testing
|
|
22
|
+
Requires-Python: >=3.10
|
|
23
|
+
Description-Content-Type: text/markdown
|
|
24
|
+
License-File: LICENSE
|
|
25
|
+
Requires-Dist: httpx<1,>=0.27
|
|
26
|
+
Requires-Dist: jsonschema<5,>=4.18
|
|
27
|
+
Requires-Dist: PyYAML<7,>=6.0
|
|
28
|
+
Requires-Dist: rich<15,>=13
|
|
29
|
+
Requires-Dist: typer<1,>=0.12
|
|
30
|
+
Requires-Dist: tomli>=2.0; python_version < "3.11"
|
|
31
|
+
Provides-Extra: dev
|
|
32
|
+
Requires-Dist: build>=1.2; extra == "dev"
|
|
33
|
+
Requires-Dist: pytest>=8; extra == "dev"
|
|
34
|
+
Requires-Dist: ruff>=0.9; extra == "dev"
|
|
35
|
+
Requires-Dist: twine>=6; extra == "dev"
|
|
36
|
+
Dynamic: license-file
|
|
37
|
+
|
|
38
|
+
# Halios
|
|
39
|
+
|
|
40
|
+
[](https://pypi.org/project/haliosai-cli/)
|
|
41
|
+
[](LICENSE)
|
|
42
|
+
|
|
43
|
+
**Your pair programmer for AI agent evaluations.**
|
|
44
|
+
|
|
45
|
+
Halios helps you build reliable AI agents in minutes. Add the Halios skill to **Codex**, **Claude Code**, **Cursor**, or your favorite coding agent, and create eval suites, run multi-turn simulations, investigate failures, and gate pull requests directly from your repository.
|
|
46
|
+
|
|
47
|
+
Halios provides the evaluation framework, developer tools, and hosted evaluation runtime underneath.
|
|
48
|
+
|
|
49
|
+
---
|
|
50
|
+
|
|
51
|
+
## Quickstart
|
|
52
|
+
|
|
53
|
+
### 1. Install the Agent Skill
|
|
54
|
+
|
|
55
|
+
Add the Halios skill to your coding agent environment:
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
npx skills add HaliosAI/halios --skill halios
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
*(Works with Codex, Claude Code, Cursor, GitHub Copilot, Gemini CLI, OpenCode, and any harness supporting the open Agent Skills format).*
|
|
62
|
+
|
|
63
|
+
### 2. Ask Your Coding Agent
|
|
64
|
+
|
|
65
|
+
Prompt your agent to set up evaluations for your project:
|
|
66
|
+
|
|
67
|
+
```text
|
|
68
|
+
"Set up evals for this agent: inspect the repository, create realistic test scenarios and checks, and run a baseline."
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
Your agent will inspect your application entrypoint, configure standard OpenTelemetry, draft scenarios in `.halios/`, and run baseline evaluations.
|
|
72
|
+
|
|
73
|
+
---
|
|
74
|
+
|
|
75
|
+
### Standalone CLI Installation
|
|
76
|
+
|
|
77
|
+
If you prefer to drive evaluations directly from the command line or CI:
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
# Recommended: Install with uv tool
|
|
81
|
+
uv tool install 'haliosai-cli>=2.0.0'
|
|
82
|
+
|
|
83
|
+
# Or install with pipx
|
|
84
|
+
pipx install 'haliosai-cli>=2.0.0'
|
|
85
|
+
|
|
86
|
+
# See available commands and usage
|
|
87
|
+
halios --help
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
92
|
+
## How It Works
|
|
93
|
+
|
|
94
|
+
1. **Inspect & Connect**: Your coding agent inspects your agent's tools, policies, and runtime, then configures standard OpenTelemetry export.
|
|
95
|
+
2. **Author Scenarios & Checks**: Test cases, rubrics, and failure criteria are stored directly in your repository (`.halios/scenarios.yml` and `.halios/eval.yml`). You own the evaluation suite.
|
|
96
|
+
3. **Simulate Multi-Turn Trajectories**: Halios runs fresh simulation passes against your agent across edge cases, tool dependencies, and user personas.
|
|
97
|
+
4. **Investigate & Fix Failures**: Pinpoint hallucinated parameters, broken tool handoffs, or policy violations from complete trace evidence.
|
|
98
|
+
5. **Gate Pull Requests**: Run protected checks in CI to block regressions before merging to production.
|
|
99
|
+
6. **Monitor Production**: Evaluate production traces using the same checks, and turn real-world failures into new regression test scenarios.
|
|
100
|
+
|
|
101
|
+
---
|
|
102
|
+
|
|
103
|
+
## Key Principles
|
|
104
|
+
|
|
105
|
+
- **You own the evaluation suite**: Scenarios and checks are clean YAML files stored in your Git repository.
|
|
106
|
+
- **Fresh simulations, not static replays**: Halios tests real agent execution across multiple turns, rather than replaying outdated completions.
|
|
107
|
+
- **Framework agnostic**: Works with any agent architecture—OpenAI Agents SDK, LangChain, LlamaIndex, PydanticAI, or custom workflows.
|
|
108
|
+
- **Stock OpenTelemetry**: Applications emit standard OpenTelemetry GenAI spans; no proprietary vendor lock-in in your production runtime.
|
|
109
|
+
- **Unified local, CI, and production loop**: The same checks run during local development, gate merge requests in CI, and monitor live production traces.
|
|
110
|
+
|
|
111
|
+
---
|
|
112
|
+
|
|
113
|
+
## Resources
|
|
114
|
+
|
|
115
|
+
- **Website**: [halios.ai](https://halios.ai)
|
|
116
|
+
- **Documentation**: [docs.halios.ai](https://docs.halios.ai)
|
|
117
|
+
- **Python SDK**: [github.com/HaliosAI/haliosai-python-sdk](https://github.com/HaliosAI/haliosai-python-sdk)
|
|
118
|
+
- **Public Skill Source**: [`skills/halios/`](skills/halios/)
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
## Development
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
# Clone and install locally in editable mode
|
|
126
|
+
git clone https://github.com/HaliosAI/halios.git
|
|
127
|
+
cd halios
|
|
128
|
+
python -m pip install -e '.[dev]'
|
|
129
|
+
|
|
130
|
+
# Run the test suite
|
|
131
|
+
python -m pytest -q
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
|
|
136
|
+
## License
|
|
137
|
+
|
|
138
|
+
[Apache 2.0](LICENSE) © Anomalytica Inc. 2026
|
haliosai_cli-2.0.0/PKG-INFO
DELETED
|
@@ -1,101 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: haliosai-cli
|
|
3
|
-
Version: 2.0.0
|
|
4
|
-
Summary: Halios CLI for coding-agent evaluations, guardrails, and OpenTelemetry evidence
|
|
5
|
-
Author-email: HaliosLabs <support@halios.ai>
|
|
6
|
-
License-Expression: Apache-2.0
|
|
7
|
-
Project-URL: Homepage, https://halios.ai
|
|
8
|
-
Project-URL: Documentation, https://docs.halios.ai
|
|
9
|
-
Project-URL: Repository, https://github.com/HaliosAI/halios
|
|
10
|
-
Project-URL: Changelog, https://github.com/HaliosAI/halios/releases
|
|
11
|
-
Keywords: ai,agents,evaluation,guardrails,opentelemetry,cli
|
|
12
|
-
Classifier: Development Status :: 4 - Beta
|
|
13
|
-
Classifier: Environment :: Console
|
|
14
|
-
Classifier: Intended Audience :: Developers
|
|
15
|
-
Classifier: Programming Language :: Python :: 3 :: Only
|
|
16
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
-
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
-
Classifier: Programming Language :: Python :: 3.14
|
|
21
|
-
Classifier: Topic :: Software Development :: Testing
|
|
22
|
-
Requires-Python: >=3.10
|
|
23
|
-
Description-Content-Type: text/markdown
|
|
24
|
-
License-File: LICENSE
|
|
25
|
-
Requires-Dist: httpx<1,>=0.27
|
|
26
|
-
Requires-Dist: jsonschema<5,>=4.18
|
|
27
|
-
Requires-Dist: PyYAML<7,>=6.0
|
|
28
|
-
Requires-Dist: rich<15,>=13
|
|
29
|
-
Requires-Dist: typer<1,>=0.12
|
|
30
|
-
Requires-Dist: tomli>=2.0; python_version < "3.11"
|
|
31
|
-
Provides-Extra: dev
|
|
32
|
-
Requires-Dist: build>=1.2; extra == "dev"
|
|
33
|
-
Requires-Dist: pytest>=8; extra == "dev"
|
|
34
|
-
Requires-Dist: ruff>=0.9; extra == "dev"
|
|
35
|
-
Requires-Dist: twine>=6; extra == "dev"
|
|
36
|
-
Dynamic: license-file
|
|
37
|
-
|
|
38
|
-
# Halios CLI and Agent Skill
|
|
39
|
-
|
|
40
|
-
The `halios` command is the machine-facing client used by the Halios Agent Skill and CI. It owns
|
|
41
|
-
repository setup, evaluation-suite authoring, fresh scenario execution, evidence inspection, and
|
|
42
|
-
prompt-optimization handoffs. Applications export their runtime telemetry with stock
|
|
43
|
-
OpenTelemetry; the CLI is not an application tracing SDK.
|
|
44
|
-
|
|
45
|
-
```bash
|
|
46
|
-
uv tool install 'haliosai-cli>=2.0.0'
|
|
47
|
-
halios --version
|
|
48
|
-
```
|
|
49
|
-
|
|
50
|
-
Use `pipx install 'haliosai-cli>=2.0.0'` when `uv` is unavailable. Install the cross-harness skill
|
|
51
|
-
separately:
|
|
52
|
-
|
|
53
|
-
```bash
|
|
54
|
-
npx skills add HaliosAI/halios --skill halios
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
The stable project workflow is:
|
|
58
|
-
|
|
59
|
-
```bash
|
|
60
|
-
halios auth login
|
|
61
|
-
halios project init --agent support-assistant --command "python tests/halios_adapter.py"
|
|
62
|
-
halios eval review --json
|
|
63
|
-
halios project configure --json
|
|
64
|
-
halios project check --json
|
|
65
|
-
halios eval run -k 3 --json
|
|
66
|
-
```
|
|
67
|
-
|
|
68
|
-
Prompt optimization is driven by the coding agent and the normal immutable eval workflow:
|
|
69
|
-
|
|
70
|
-
```bash
|
|
71
|
-
halios optimize start --baseline-run <run-id> --prompt-file path/to/system-prompt.txt --json
|
|
72
|
-
# make one bounded prompt edit from the returned guidance, then run the unchanged suite
|
|
73
|
-
halios eval run -k 3 --json
|
|
74
|
-
halios optimize record <optimization-run-id> \
|
|
75
|
-
--evaluation-run <candidate-eval-run-id> \
|
|
76
|
-
--prompt-file path/to/system-prompt.txt \
|
|
77
|
-
--json
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
Credentials are stored outside the repository. The repository contains only `.halios/config.toml`,
|
|
81
|
-
`.halios/eval.yml`, and `.halios/scenarios.yml`.
|
|
82
|
-
|
|
83
|
-
In non-interactive CI, use `HALIOS_API_KEY` for control-plane commands and the separate
|
|
84
|
-
agent-scoped `HALIOS_OTLP_TOKEN` for adapter trace export. Only trusted default-branch jobs should
|
|
85
|
-
receive `HALIOS_CI_PUBLISH_TOKEN`. The jsonl adapter is used by local/CI simulation only; deployed
|
|
86
|
-
staging and production applications export their own stock OpenTelemetry directly to Halios.
|
|
87
|
-
|
|
88
|
-
See the repository's
|
|
89
|
-
[`skills/halios`](https://github.com/HaliosAI/halios/tree/main/skills/halios) workflow for
|
|
90
|
-
coding-agent setup and
|
|
91
|
-
[`OTEL_SCHEMA.md`](https://github.com/HaliosAI/haliosai-python-sdk/blob/main/OTEL_SCHEMA.md)
|
|
92
|
-
for the telemetry accepted by Halios.
|
|
93
|
-
|
|
94
|
-
## Development
|
|
95
|
-
|
|
96
|
-
```bash
|
|
97
|
-
python -m pip install -e '.[dev]'
|
|
98
|
-
python -m pytest -q
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution and release process.
|
haliosai_cli-2.0.0/README.md
DELETED
|
@@ -1,64 +0,0 @@
|
|
|
1
|
-
# Halios CLI and Agent Skill
|
|
2
|
-
|
|
3
|
-
The `halios` command is the machine-facing client used by the Halios Agent Skill and CI. It owns
|
|
4
|
-
repository setup, evaluation-suite authoring, fresh scenario execution, evidence inspection, and
|
|
5
|
-
prompt-optimization handoffs. Applications export their runtime telemetry with stock
|
|
6
|
-
OpenTelemetry; the CLI is not an application tracing SDK.
|
|
7
|
-
|
|
8
|
-
```bash
|
|
9
|
-
uv tool install 'haliosai-cli>=2.0.0'
|
|
10
|
-
halios --version
|
|
11
|
-
```
|
|
12
|
-
|
|
13
|
-
Use `pipx install 'haliosai-cli>=2.0.0'` when `uv` is unavailable. Install the cross-harness skill
|
|
14
|
-
separately:
|
|
15
|
-
|
|
16
|
-
```bash
|
|
17
|
-
npx skills add HaliosAI/halios --skill halios
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
The stable project workflow is:
|
|
21
|
-
|
|
22
|
-
```bash
|
|
23
|
-
halios auth login
|
|
24
|
-
halios project init --agent support-assistant --command "python tests/halios_adapter.py"
|
|
25
|
-
halios eval review --json
|
|
26
|
-
halios project configure --json
|
|
27
|
-
halios project check --json
|
|
28
|
-
halios eval run -k 3 --json
|
|
29
|
-
```
|
|
30
|
-
|
|
31
|
-
Prompt optimization is driven by the coding agent and the normal immutable eval workflow:
|
|
32
|
-
|
|
33
|
-
```bash
|
|
34
|
-
halios optimize start --baseline-run <run-id> --prompt-file path/to/system-prompt.txt --json
|
|
35
|
-
# make one bounded prompt edit from the returned guidance, then run the unchanged suite
|
|
36
|
-
halios eval run -k 3 --json
|
|
37
|
-
halios optimize record <optimization-run-id> \
|
|
38
|
-
--evaluation-run <candidate-eval-run-id> \
|
|
39
|
-
--prompt-file path/to/system-prompt.txt \
|
|
40
|
-
--json
|
|
41
|
-
```
|
|
42
|
-
|
|
43
|
-
Credentials are stored outside the repository. The repository contains only `.halios/config.toml`,
|
|
44
|
-
`.halios/eval.yml`, and `.halios/scenarios.yml`.
|
|
45
|
-
|
|
46
|
-
In non-interactive CI, use `HALIOS_API_KEY` for control-plane commands and the separate
|
|
47
|
-
agent-scoped `HALIOS_OTLP_TOKEN` for adapter trace export. Only trusted default-branch jobs should
|
|
48
|
-
receive `HALIOS_CI_PUBLISH_TOKEN`. The jsonl adapter is used by local/CI simulation only; deployed
|
|
49
|
-
staging and production applications export their own stock OpenTelemetry directly to Halios.
|
|
50
|
-
|
|
51
|
-
See the repository's
|
|
52
|
-
[`skills/halios`](https://github.com/HaliosAI/halios/tree/main/skills/halios) workflow for
|
|
53
|
-
coding-agent setup and
|
|
54
|
-
[`OTEL_SCHEMA.md`](https://github.com/HaliosAI/haliosai-python-sdk/blob/main/OTEL_SCHEMA.md)
|
|
55
|
-
for the telemetry accepted by Halios.
|
|
56
|
-
|
|
57
|
-
## Development
|
|
58
|
-
|
|
59
|
-
```bash
|
|
60
|
-
python -m pip install -e '.[dev]'
|
|
61
|
-
python -m pytest -q
|
|
62
|
-
```
|
|
63
|
-
|
|
64
|
-
See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution and release process.
|
|
@@ -1 +0,0 @@
|
|
|
1
|
-
__version__ = "2.0.0"
|
|
@@ -1,162 +0,0 @@
|
|
|
1
|
-
"""User-level CLI authentication commands."""
|
|
2
|
-
|
|
3
|
-
from __future__ import annotations
|
|
4
|
-
|
|
5
|
-
import json
|
|
6
|
-
import secrets
|
|
7
|
-
import threading
|
|
8
|
-
import urllib.parse
|
|
9
|
-
import webbrowser
|
|
10
|
-
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
|
11
|
-
|
|
12
|
-
import httpx
|
|
13
|
-
import typer
|
|
14
|
-
|
|
15
|
-
from .cli_support import (
|
|
16
|
-
DEFAULT_BASE_URL,
|
|
17
|
-
ApiClient,
|
|
18
|
-
ApiError,
|
|
19
|
-
delete_profile,
|
|
20
|
-
normalize_url,
|
|
21
|
-
resolve_credentials,
|
|
22
|
-
save_profile,
|
|
23
|
-
stored_profile_credentials,
|
|
24
|
-
)
|
|
25
|
-
|
|
26
|
-
app = typer.Typer(help="Authenticate the Halios CLI.", no_args_is_help=True)
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
@app.command("login")
|
|
30
|
-
def login(
|
|
31
|
-
profile: str = typer.Option("default", "--profile"),
|
|
32
|
-
base_url: str = typer.Option(DEFAULT_BASE_URL, "--base-url"),
|
|
33
|
-
api_key: str | None = typer.Option(None, "--api-key", envvar="HALIOS_API_KEY", hidden=True),
|
|
34
|
-
) -> None:
|
|
35
|
-
"""Authorize this machine and store credentials outside the repository."""
|
|
36
|
-
normalized_url = normalize_url(base_url)
|
|
37
|
-
if api_key:
|
|
38
|
-
save_profile(profile, base_url=normalized_url, api_key=api_key)
|
|
39
|
-
typer.echo(f"Logged in as profile '{profile}'.")
|
|
40
|
-
return
|
|
41
|
-
|
|
42
|
-
result: dict[str, str] = {}
|
|
43
|
-
completed = threading.Event()
|
|
44
|
-
state = secrets.token_urlsafe(24)
|
|
45
|
-
|
|
46
|
-
class CallbackHandler(BaseHTTPRequestHandler):
|
|
47
|
-
def _cors(self) -> None:
|
|
48
|
-
self.send_header("Access-Control-Allow-Origin", normalized_url)
|
|
49
|
-
self.send_header("Access-Control-Allow-Methods", "POST, OPTIONS")
|
|
50
|
-
self.send_header("Access-Control-Allow-Headers", "Content-Type")
|
|
51
|
-
|
|
52
|
-
def do_OPTIONS(self) -> None: # noqa: N802
|
|
53
|
-
self.send_response(204)
|
|
54
|
-
self._cors()
|
|
55
|
-
self.end_headers()
|
|
56
|
-
|
|
57
|
-
def do_POST(self) -> None: # noqa: N802
|
|
58
|
-
if self.path != "/callback":
|
|
59
|
-
self.send_error(404)
|
|
60
|
-
return
|
|
61
|
-
length = int(self.headers.get("content-length") or 0)
|
|
62
|
-
try:
|
|
63
|
-
body = json.loads(self.rfile.read(length))
|
|
64
|
-
except (ValueError, UnicodeDecodeError):
|
|
65
|
-
self.send_error(400)
|
|
66
|
-
return
|
|
67
|
-
if not isinstance(body, dict) or not secrets.compare_digest(
|
|
68
|
-
str(body.get("state") or ""), state
|
|
69
|
-
):
|
|
70
|
-
self.send_error(422)
|
|
71
|
-
return
|
|
72
|
-
if body.get("error"):
|
|
73
|
-
result["error"] = str(body["error"])
|
|
74
|
-
elif body.get("api_key"):
|
|
75
|
-
result.update({key: str(value) for key, value in body.items() if value is not None})
|
|
76
|
-
else:
|
|
77
|
-
self.send_error(422)
|
|
78
|
-
return
|
|
79
|
-
self.send_response(204)
|
|
80
|
-
self._cors()
|
|
81
|
-
self.end_headers()
|
|
82
|
-
completed.set()
|
|
83
|
-
|
|
84
|
-
def log_message(self, _format: str, *_args: object) -> None:
|
|
85
|
-
return
|
|
86
|
-
|
|
87
|
-
server = ThreadingHTTPServer(("127.0.0.1", 0), CallbackHandler)
|
|
88
|
-
thread = threading.Thread(target=server.serve_forever, daemon=True)
|
|
89
|
-
thread.start()
|
|
90
|
-
login_url = f"{normalized_url}/cli-login?" + urllib.parse.urlencode(
|
|
91
|
-
{"port": server.server_port, "profile": profile, "state": state}
|
|
92
|
-
)
|
|
93
|
-
typer.echo(f"Open this URL to authorize the CLI:\n{login_url}")
|
|
94
|
-
webbrowser.open(login_url)
|
|
95
|
-
try:
|
|
96
|
-
if not completed.wait(timeout=300):
|
|
97
|
-
raise typer.BadParameter("Login timed out after 5 minutes")
|
|
98
|
-
finally:
|
|
99
|
-
server.shutdown()
|
|
100
|
-
server.server_close()
|
|
101
|
-
|
|
102
|
-
if result.get("error") == "access_denied":
|
|
103
|
-
raise typer.BadParameter("Authorization was cancelled in the browser")
|
|
104
|
-
if result.get("error"):
|
|
105
|
-
raise typer.BadParameter(f"Authorization failed: {result['error']}")
|
|
106
|
-
|
|
107
|
-
save_profile(
|
|
108
|
-
profile,
|
|
109
|
-
base_url=normalized_url,
|
|
110
|
-
api_key=result["api_key"],
|
|
111
|
-
organization_id=result.get("organization_id"),
|
|
112
|
-
api_key_id=int(result["api_key_id"]) if result.get("api_key_id") else None,
|
|
113
|
-
expires_at=result.get("expires_at"),
|
|
114
|
-
)
|
|
115
|
-
typer.echo(f"Logged in as profile '{profile}'.")
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
@app.command("status")
|
|
119
|
-
def status(profile: str = typer.Option("default", "--profile")) -> None:
|
|
120
|
-
"""Verify the selected profile without exposing its credential."""
|
|
121
|
-
credentials = resolve_credentials(profile)
|
|
122
|
-
with ApiClient(credentials) as api:
|
|
123
|
-
api.request("GET", "/api/v1/agents", params={"limit": 1})
|
|
124
|
-
expiry = f" Credential expires {credentials.expires_at}." if credentials.expires_at else ""
|
|
125
|
-
typer.echo(f"Authenticated profile '{profile}' at {credentials.base_url}.{expiry}")
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
@app.command("logout")
|
|
129
|
-
def logout(
|
|
130
|
-
profile: str = typer.Option("default", "--profile"),
|
|
131
|
-
local_only: bool = typer.Option(
|
|
132
|
-
False,
|
|
133
|
-
"--local-only",
|
|
134
|
-
help="Remove local credentials without revoking the remote key.",
|
|
135
|
-
),
|
|
136
|
-
) -> None:
|
|
137
|
-
"""Revoke one CLI credential, then remove its local profile."""
|
|
138
|
-
credentials = stored_profile_credentials(profile)
|
|
139
|
-
if credentials is None:
|
|
140
|
-
typer.echo(f"Profile '{profile}' was not stored.")
|
|
141
|
-
return
|
|
142
|
-
|
|
143
|
-
if not local_only:
|
|
144
|
-
try:
|
|
145
|
-
with ApiClient(credentials) as api:
|
|
146
|
-
api.request("DELETE", "/api/v1/api-keys/current")
|
|
147
|
-
except ApiError as exc:
|
|
148
|
-
if exc.status_code != 401:
|
|
149
|
-
raise typer.BadParameter(
|
|
150
|
-
"Remote revocation failed; local credentials were kept. "
|
|
151
|
-
"Retry, or use --local-only if the server is permanently unavailable."
|
|
152
|
-
) from exc
|
|
153
|
-
typer.echo("The remote credential was already invalid or expired.")
|
|
154
|
-
except httpx.HTTPError as exc:
|
|
155
|
-
raise typer.BadParameter(
|
|
156
|
-
"Could not reach Halios to revoke the credential; local credentials were kept. "
|
|
157
|
-
"Retry, or use --local-only if the server is permanently unavailable."
|
|
158
|
-
) from exc
|
|
159
|
-
|
|
160
|
-
delete_profile(profile)
|
|
161
|
-
suffix = " locally (remote key unchanged)" if local_only else " and revoked its API key"
|
|
162
|
-
typer.echo(f"Removed profile '{profile}'{suffix}.")
|
|
@@ -1,101 +0,0 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: haliosai-cli
|
|
3
|
-
Version: 2.0.0
|
|
4
|
-
Summary: Halios CLI for coding-agent evaluations, guardrails, and OpenTelemetry evidence
|
|
5
|
-
Author-email: HaliosLabs <support@halios.ai>
|
|
6
|
-
License-Expression: Apache-2.0
|
|
7
|
-
Project-URL: Homepage, https://halios.ai
|
|
8
|
-
Project-URL: Documentation, https://docs.halios.ai
|
|
9
|
-
Project-URL: Repository, https://github.com/HaliosAI/halios
|
|
10
|
-
Project-URL: Changelog, https://github.com/HaliosAI/halios/releases
|
|
11
|
-
Keywords: ai,agents,evaluation,guardrails,opentelemetry,cli
|
|
12
|
-
Classifier: Development Status :: 4 - Beta
|
|
13
|
-
Classifier: Environment :: Console
|
|
14
|
-
Classifier: Intended Audience :: Developers
|
|
15
|
-
Classifier: Programming Language :: Python :: 3 :: Only
|
|
16
|
-
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
-
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
-
Classifier: Programming Language :: Python :: 3.14
|
|
21
|
-
Classifier: Topic :: Software Development :: Testing
|
|
22
|
-
Requires-Python: >=3.10
|
|
23
|
-
Description-Content-Type: text/markdown
|
|
24
|
-
License-File: LICENSE
|
|
25
|
-
Requires-Dist: httpx<1,>=0.27
|
|
26
|
-
Requires-Dist: jsonschema<5,>=4.18
|
|
27
|
-
Requires-Dist: PyYAML<7,>=6.0
|
|
28
|
-
Requires-Dist: rich<15,>=13
|
|
29
|
-
Requires-Dist: typer<1,>=0.12
|
|
30
|
-
Requires-Dist: tomli>=2.0; python_version < "3.11"
|
|
31
|
-
Provides-Extra: dev
|
|
32
|
-
Requires-Dist: build>=1.2; extra == "dev"
|
|
33
|
-
Requires-Dist: pytest>=8; extra == "dev"
|
|
34
|
-
Requires-Dist: ruff>=0.9; extra == "dev"
|
|
35
|
-
Requires-Dist: twine>=6; extra == "dev"
|
|
36
|
-
Dynamic: license-file
|
|
37
|
-
|
|
38
|
-
# Halios CLI and Agent Skill
|
|
39
|
-
|
|
40
|
-
The `halios` command is the machine-facing client used by the Halios Agent Skill and CI. It owns
|
|
41
|
-
repository setup, evaluation-suite authoring, fresh scenario execution, evidence inspection, and
|
|
42
|
-
prompt-optimization handoffs. Applications export their runtime telemetry with stock
|
|
43
|
-
OpenTelemetry; the CLI is not an application tracing SDK.
|
|
44
|
-
|
|
45
|
-
```bash
|
|
46
|
-
uv tool install 'haliosai-cli>=2.0.0'
|
|
47
|
-
halios --version
|
|
48
|
-
```
|
|
49
|
-
|
|
50
|
-
Use `pipx install 'haliosai-cli>=2.0.0'` when `uv` is unavailable. Install the cross-harness skill
|
|
51
|
-
separately:
|
|
52
|
-
|
|
53
|
-
```bash
|
|
54
|
-
npx skills add HaliosAI/halios --skill halios
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
The stable project workflow is:
|
|
58
|
-
|
|
59
|
-
```bash
|
|
60
|
-
halios auth login
|
|
61
|
-
halios project init --agent support-assistant --command "python tests/halios_adapter.py"
|
|
62
|
-
halios eval review --json
|
|
63
|
-
halios project configure --json
|
|
64
|
-
halios project check --json
|
|
65
|
-
halios eval run -k 3 --json
|
|
66
|
-
```
|
|
67
|
-
|
|
68
|
-
Prompt optimization is driven by the coding agent and the normal immutable eval workflow:
|
|
69
|
-
|
|
70
|
-
```bash
|
|
71
|
-
halios optimize start --baseline-run <run-id> --prompt-file path/to/system-prompt.txt --json
|
|
72
|
-
# make one bounded prompt edit from the returned guidance, then run the unchanged suite
|
|
73
|
-
halios eval run -k 3 --json
|
|
74
|
-
halios optimize record <optimization-run-id> \
|
|
75
|
-
--evaluation-run <candidate-eval-run-id> \
|
|
76
|
-
--prompt-file path/to/system-prompt.txt \
|
|
77
|
-
--json
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
Credentials are stored outside the repository. The repository contains only `.halios/config.toml`,
|
|
81
|
-
`.halios/eval.yml`, and `.halios/scenarios.yml`.
|
|
82
|
-
|
|
83
|
-
In non-interactive CI, use `HALIOS_API_KEY` for control-plane commands and the separate
|
|
84
|
-
agent-scoped `HALIOS_OTLP_TOKEN` for adapter trace export. Only trusted default-branch jobs should
|
|
85
|
-
receive `HALIOS_CI_PUBLISH_TOKEN`. The jsonl adapter is used by local/CI simulation only; deployed
|
|
86
|
-
staging and production applications export their own stock OpenTelemetry directly to Halios.
|
|
87
|
-
|
|
88
|
-
See the repository's
|
|
89
|
-
[`skills/halios`](https://github.com/HaliosAI/halios/tree/main/skills/halios) workflow for
|
|
90
|
-
coding-agent setup and
|
|
91
|
-
[`OTEL_SCHEMA.md`](https://github.com/HaliosAI/haliosai-python-sdk/blob/main/OTEL_SCHEMA.md)
|
|
92
|
-
for the telemetry accepted by Halios.
|
|
93
|
-
|
|
94
|
-
## Development
|
|
95
|
-
|
|
96
|
-
```bash
|
|
97
|
-
python -m pip install -e '.[dev]'
|
|
98
|
-
python -m pytest -q
|
|
99
|
-
```
|
|
100
|
-
|
|
101
|
-
See [CONTRIBUTING.md](CONTRIBUTING.md) for the contribution and release process.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|