agento11y-harbor 0.20.0__tar.gz

agento11y_harbor-0.20.0/LICENSE

@@ -0,0 +1,3 @@
+SPDX-License-Identifier: Apache-2.0
+
+See /LICENSE at repository root for full license text.

agento11y_harbor-0.20.0/PKG-INFO

@@ -0,0 +1,213 @@
+Metadata-Version: 2.4
+Name: agento11y-harbor
+Version: 0.20.0
+Summary: Harbor live evaluation and job import for the Grafana Agent Observability Python SDK
+License: SPDX-License-Identifier: Apache-2.0
+        
+        See /LICENSE at repository root for full license text.
+        
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: agento11y>=0.21.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.2.0; extra == "dev"
+Dynamic: license-file
+
+# Harbor integration
+
+`agento11y-harbor` publishes [Harbor](https://github.com/harbor-framework/harbor)
+evaluations to Agent Observability in Grafana Cloud. You can publish a job
+while it runs, import a job that already finished, or run explicit Harbor
+trials from Python.
+
+## Install
+
+Install the package in the same Python environment as Harbor, so Harbor can
+discover the `agento11y` job plugin. With a `uv` tool install:
+
+```bash
+uv tool install harbor --with-executables-from agento11y-harbor
+```
+
+Or with `pip`, in Harbor's environment:
+
+```bash
+pip install agento11y-harbor
+```
+
+The package needs `agento11y` 0.20.0 or later and a Harbor release that
+supports job plugins (`harbor run --plugin`).
+
+## Configure Grafana Cloud
+
+Set these environment variables before you run or import a job:
+
+- `AGENTO11Y_ENDPOINT`: your Grafana Cloud Agent Observability endpoint.
+- `AGENTO11Y_AUTH_TOKEN` and, when your stack requires it,
+  `AGENTO11Y_AUTH_TENANT_ID`: your Grafana Cloud ingestion credentials.
+- `AGENTO11Y_GRAFANA_URL`: your Grafana Cloud stack URL, for experiment links.
+- `AGENTO11Y_SERVICE_ACCOUNT_TOKEN`: a Grafana service account token. You need
+  it only to publish a suite checkpoint with `--publish-suite`.
+
+Refer to the [Grafana Cloud setup guide](https://grafana.com/docs/grafana-cloud/machine-learning/agent-observability/get-started/grafana-cloud/)
+for the credentials. `agento11y harbor` also loads the credentials that
+`agento11y login` saved.
+
+## Publish a job while it runs
+
+Run Harbor through the `agento11y` CLI. Every argument after `--` goes to
+`harbor run` unchanged:
+
+```bash
+agento11y harbor -- \
+  --dataset terminal-bench@2.0 \
+  --agent codex \
+  --model openai/gpt-5
+```
+
+`agento11y harbor run -- ...` is the same command. The launcher adds
+`--plugin agento11y`, so `harbor run --plugin agento11y ...` works too. Pass
+plugin options with `--plugin-kwarg`, for example
+`--plugin-kwarg pass_threshold=0.5`. The options are `name`, `experiment_id`,
+`main_reward` (default `reward`), `pass_threshold` (default `1.0`), and
+`upload_artifacts` (default `true`).
+
+The plugin creates one experiment per agent, model, and dataset. Each trial
+appears when Harbor starts its agent, after the environment and agent setup,
+so the experiment records the agent version Harbor detects. A trial whose
+setup fails appears when it ends. When the trial ends, the plugin attaches its
+rewards, duration, tokens, cost, errors, and verifier artifacts. The main
+reward is the pass/fail verdict, and other rewards are diagnostics.
+
+If your agent publishes its own conversation, put its ID in
+`AgentContext.metadata["agento11y.conversation_id"]`, and the trial links to
+it. Otherwise, the plugin imports the trial's ATIF trajectory as a
+conversation. ATIF import creates generations, not traces. Each tool call's
+arguments keep their first 4 KiB, and each step's prompt and response keep
+their first 1 MiB, so one large tool result can't fail the import.
+
+## Import a completed job
+
+Publish a Harbor job directory that already finished. Harbor writes jobs to
+`jobs/` in the directory you ran it from, unless you pass `--jobs-dir`:
+
+```bash
+agento11y harbor export jobs/<job-name>
+```
+
+The importer reads Harbor's `result.json`, ATIF, reward, and verifier files.
+It doesn't need Harbor installed. Run `agento11y harbor export --help` for the
+flags:
+
+- `--experiment-id`: the base experiment ID. The default is the Harbor job ID.
+- `--suite-id`: the suite ID for the imported cases. The default is derived
+  from the dataset. On its own, it doesn't publish anything.
+- `--publish-suite subset|replace`: store the job's tasks as a published suite
+  checkpoint before the import. `subset` adds and updates cases. `replace` also
+  removes stored cases that aren't in the job.
+- `--main-reward`, `--pass-threshold`, and `--no-artifacts`: the same options
+  as the plugin.
+
+From Python, call `publish_harbor_job`:
+
+```python
+from agento11y.experiments import TestSuitesClient
+from agento11y_harbor import publish_harbor_job
+
+published = publish_harbor_job(
+    "jobs/my-job",
+    test_suites_client=TestSuitesClient(),
+    suite_publication_policy="subset",
+)
+for run in published.runs:
+    print(run.url)
+```
+
+## Experiment IDs
+
+The plugin and the importer use the same experiment ID:
+`<experiment_id or Harbor job ID>-<candidate hash>`. The hash comes from the
+agent name, model, and dataset, so it's the same for every run of the same
+candidate.
+
+A finalized experiment can't change. When you import a job, a candidate whose
+experiment is already finalized, either live or by an earlier import, is
+skipped with a warning, and the import fails if every candidate is. Pass a new
+`--experiment-id` to publish the job as separate experiments. Resuming a
+Harbor job also reuses the job ID, so pass a new `experiment_id` plugin option
+when the job's experiments already finalized.
+
+## Errors and retries
+
+Publishing problems never stop the Harbor job. When a trial can't be
+published, or one of its files can't be read, that trial is marked errored and
+the rest of the job continues. The plugin logs a warning. The errored trial's
+rewards stay diagnostic.
+
+An agent timeout or non-zero exit doesn't error a trial when Harbor's verifier
+still produced the main reward: like Harbor's own metrics, the reward decides
+pass or fail, and the agent error is kept in the score metadata. Any other
+Harbor exception, or a missing main reward, errors the trial.
+
+Any errored trial finalizes its experiment as failed, and the experiment error
+lists the errored trials. An importer skips a trial whose `result.json` it
+can't read and logs a warning. Each candidate is imported on its own, so one
+that fails to publish doesn't stop the others. The export command lists it and
+exits with status 1, and rerunning the command publishes only what's missing.
+
+If the plugin can't create an experiment, the candidate's next trial tries
+again. Once it exists, the experiment is finalized as failed, because it's
+missing the trials that started before then. If it never exists, the plugin logs
+how many trials it left out when the job ends. Either way, export the job
+directory to publish every trial, with a new `--experiment-id` if the
+experiment exists.
+
+If the Harbor process is interrupted, for example with Ctrl-C, the plugin marks
+trials without a result errored and finalizes the experiments as failed. To
+publish the finished trials, export the job directory with a new
+`--experiment-id`.
+
+Harbor retries a failed trial under the same trial name. The plugin keeps one
+trial per name and publishes only the final attempt, so a failure that passes
+on retry counts once, as a pass. The importer sees the same result, because
+Harbor deletes the files of superseded attempts. Separate attempts from
+`--n-attempts` are separate trials. Their attempt numbers follow the order
+their agents start live and trial-name order on import, so compare them by experiment, not by
+attempt number.
+
+## Run trials from Python
+
+`run_harbor` runs explicit Harbor `TrialConfig` objects for one agent and
+model, one case at a time. It needs Harbor installed:
+
+```python
+from agento11y_harbor import run_harbor
+
+run = await run_harbor(suite, configs, name="Harbor regression", agent_env=otel_env)
+print(run.url)
+```
+
+`configs` maps each suite case ID to a `harbor.models.trial.config.TrialConfig`.
+The runner adds `agent_env`, `TRACEPARENT`, `TRACESTATE`, and the
+`AGENTO11Y_*` trial identity to each agent's environment. A Python agent can
+continue the trace and read the trial with `TrialRef.from_env()`:
+
+```python
+import os
+from opentelemetry import context, propagate
+
+carrier = {key.lower(): os.environ[key] for key in ("TRACEPARENT", "TRACESTATE") if key in os.environ}
+token = context.attach(propagate.extract(carrier))
+try:
+    run_instrumented_agent()
+finally:
+    context.detach(token)
+```
+
+The live plugin can't add environment variables to agents, because Harbor
+creates each agent before the plugin's hooks run. With the plugin, link
+conversations through `AgentContext.metadata` instead.
+
+`run_harbor` doesn't retry trials or run them concurrently. Use
+`agento11y harbor` for Harbor's scheduler, retries, and datasets.

agento11y_harbor-0.20.0/README.md

@@ -0,0 +1,197 @@
+# Harbor integration
+
+`agento11y-harbor` publishes [Harbor](https://github.com/harbor-framework/harbor)
+evaluations to Agent Observability in Grafana Cloud. You can publish a job
+while it runs, import a job that already finished, or run explicit Harbor
+trials from Python.
+
+## Install
+
+Install the package in the same Python environment as Harbor, so Harbor can
+discover the `agento11y` job plugin. With a `uv` tool install:
+
+```bash
+uv tool install harbor --with-executables-from agento11y-harbor
+```
+
+Or with `pip`, in Harbor's environment:
+
+```bash
+pip install agento11y-harbor
+```
+
+The package needs `agento11y` 0.20.0 or later and a Harbor release that
+supports job plugins (`harbor run --plugin`).
+
+## Configure Grafana Cloud
+
+Set these environment variables before you run or import a job:
+
+- `AGENTO11Y_ENDPOINT`: your Grafana Cloud Agent Observability endpoint.
+- `AGENTO11Y_AUTH_TOKEN` and, when your stack requires it,
+  `AGENTO11Y_AUTH_TENANT_ID`: your Grafana Cloud ingestion credentials.
+- `AGENTO11Y_GRAFANA_URL`: your Grafana Cloud stack URL, for experiment links.
+- `AGENTO11Y_SERVICE_ACCOUNT_TOKEN`: a Grafana service account token. You need
+  it only to publish a suite checkpoint with `--publish-suite`.
+
+Refer to the [Grafana Cloud setup guide](https://grafana.com/docs/grafana-cloud/machine-learning/agent-observability/get-started/grafana-cloud/)
+for the credentials. `agento11y harbor` also loads the credentials that
+`agento11y login` saved.
+
+## Publish a job while it runs
+
+Run Harbor through the `agento11y` CLI. Every argument after `--` goes to
+`harbor run` unchanged:
+
+```bash
+agento11y harbor -- \
+  --dataset terminal-bench@2.0 \
+  --agent codex \
+  --model openai/gpt-5
+```
+
+`agento11y harbor run -- ...` is the same command. The launcher adds
+`--plugin agento11y`, so `harbor run --plugin agento11y ...` works too. Pass
+plugin options with `--plugin-kwarg`, for example
+`--plugin-kwarg pass_threshold=0.5`. The options are `name`, `experiment_id`,
+`main_reward` (default `reward`), `pass_threshold` (default `1.0`), and
+`upload_artifacts` (default `true`).
+
+The plugin creates one experiment per agent, model, and dataset. Each trial
+appears when Harbor starts its agent, after the environment and agent setup,
+so the experiment records the agent version Harbor detects. A trial whose
+setup fails appears when it ends. When the trial ends, the plugin attaches its
+rewards, duration, tokens, cost, errors, and verifier artifacts. The main
+reward is the pass/fail verdict, and other rewards are diagnostics.
+
+If your agent publishes its own conversation, put its ID in
+`AgentContext.metadata["agento11y.conversation_id"]`, and the trial links to
+it. Otherwise, the plugin imports the trial's ATIF trajectory as a
+conversation. ATIF import creates generations, not traces. Each tool call's
+arguments keep their first 4 KiB, and each step's prompt and response keep
+their first 1 MiB, so one large tool result can't fail the import.
+
+## Import a completed job
+
+Publish a Harbor job directory that already finished. Harbor writes jobs to
+`jobs/` in the directory you ran it from, unless you pass `--jobs-dir`:
+
+```bash
+agento11y harbor export jobs/<job-name>
+```
+
+The importer reads Harbor's `result.json`, ATIF, reward, and verifier files.
+It doesn't need Harbor installed. Run `agento11y harbor export --help` for the
+flags:
+
+- `--experiment-id`: the base experiment ID. The default is the Harbor job ID.
+- `--suite-id`: the suite ID for the imported cases. The default is derived
+  from the dataset. On its own, it doesn't publish anything.
+- `--publish-suite subset|replace`: store the job's tasks as a published suite
+  checkpoint before the import. `subset` adds and updates cases. `replace` also
+  removes stored cases that aren't in the job.
+- `--main-reward`, `--pass-threshold`, and `--no-artifacts`: the same options
+  as the plugin.
+
+From Python, call `publish_harbor_job`:
+
+```python
+from agento11y.experiments import TestSuitesClient
+from agento11y_harbor import publish_harbor_job
+
+published = publish_harbor_job(
+    "jobs/my-job",
+    test_suites_client=TestSuitesClient(),
+    suite_publication_policy="subset",
+)
+for run in published.runs:
+    print(run.url)
+```
+
+## Experiment IDs
+
+The plugin and the importer use the same experiment ID:
+`<experiment_id or Harbor job ID>-<candidate hash>`. The hash comes from the
+agent name, model, and dataset, so it's the same for every run of the same
+candidate.
+
+A finalized experiment can't change. When you import a job, a candidate whose
+experiment is already finalized, either live or by an earlier import, is
+skipped with a warning, and the import fails if every candidate is. Pass a new
+`--experiment-id` to publish the job as separate experiments. Resuming a
+Harbor job also reuses the job ID, so pass a new `experiment_id` plugin option
+when the job's experiments already finalized.
+
+## Errors and retries
+
+Publishing problems never stop the Harbor job. When a trial can't be
+published, or one of its files can't be read, that trial is marked errored and
+the rest of the job continues. The plugin logs a warning. The errored trial's
+rewards stay diagnostic.
+
+An agent timeout or non-zero exit doesn't error a trial when Harbor's verifier
+still produced the main reward: like Harbor's own metrics, the reward decides
+pass or fail, and the agent error is kept in the score metadata. Any other
+Harbor exception, or a missing main reward, errors the trial.
+
+Any errored trial finalizes its experiment as failed, and the experiment error
+lists the errored trials. An importer skips a trial whose `result.json` it
+can't read and logs a warning. Each candidate is imported on its own, so one
+that fails to publish doesn't stop the others. The export command lists it and
+exits with status 1, and rerunning the command publishes only what's missing.
+
+If the plugin can't create an experiment, the candidate's next trial tries
+again. Once it exists, the experiment is finalized as failed, because it's
+missing the trials that started before then. If it never exists, the plugin logs
+how many trials it left out when the job ends. Either way, export the job
+directory to publish every trial, with a new `--experiment-id` if the
+experiment exists.
+
+If the Harbor process is interrupted, for example with Ctrl-C, the plugin marks
+trials without a result errored and finalizes the experiments as failed. To
+publish the finished trials, export the job directory with a new
+`--experiment-id`.
+
+Harbor retries a failed trial under the same trial name. The plugin keeps one
+trial per name and publishes only the final attempt, so a failure that passes
+on retry counts once, as a pass. The importer sees the same result, because
+Harbor deletes the files of superseded attempts. Separate attempts from
+`--n-attempts` are separate trials. Their attempt numbers follow the order
+their agents start live and trial-name order on import, so compare them by experiment, not by
+attempt number.
+
+## Run trials from Python
+
+`run_harbor` runs explicit Harbor `TrialConfig` objects for one agent and
+model, one case at a time. It needs Harbor installed:
+
+```python
+from agento11y_harbor import run_harbor
+
+run = await run_harbor(suite, configs, name="Harbor regression", agent_env=otel_env)
+print(run.url)
+```
+
+`configs` maps each suite case ID to a `harbor.models.trial.config.TrialConfig`.
+The runner adds `agent_env`, `TRACEPARENT`, `TRACESTATE`, and the
+`AGENTO11Y_*` trial identity to each agent's environment. A Python agent can
+continue the trace and read the trial with `TrialRef.from_env()`:
+
+```python
+import os
+from opentelemetry import context, propagate
+
+carrier = {key.lower(): os.environ[key] for key in ("TRACEPARENT", "TRACESTATE") if key in os.environ}
+token = context.attach(propagate.extract(carrier))
+try:
+    run_instrumented_agent()
+finally:
+    context.detach(token)
+```
+
+The live plugin can't add environment variables to agents, because Harbor
+creates each agent before the plugin's hooks run. With the plugin, link
+conversations through `AgentContext.metadata` instead.
+
+`run_harbor` doesn't retry trials or run them concurrently. Use
+`agento11y harbor` for Harbor's scheduler, retries, and datasets.

agento11y_harbor-0.20.0/agento11y_harbor/__init__.py

@@ -0,0 +1,6 @@
+"""Harbor job import for Grafana Agent Observability."""
+
+from .integration import PublishedHarborJob, PublishedHarborRun, publish_harbor_job
+from .runner import run_harbor
+
+__all__ = ["PublishedHarborJob", "PublishedHarborRun", "publish_harbor_job", "run_harbor"]

agento11y_harbor-0.20.0/agento11y_harbor/cli.py

@@ -0,0 +1,114 @@
+"""Command-line bridge used by ``agento11y harbor``."""
+
+from __future__ import annotations
+
+import argparse
+import os
+import shutil
+import sys
+from collections.abc import Sequence
+
+from agento11y.experiments import TestSuitesClient
+
+from .integration import publish_harbor_job
+
+
+def _export(argv: Sequence[str]) -> int:
+    parser = argparse.ArgumentParser(
+        prog="agento11y harbor export",
+        description="Publish a completed Harbor job directory as one experiment per agent/model candidate.",
+    )
+    parser.add_argument("job_dir", help="Harbor job directory that contains the job result.json")
+    parser.add_argument(
+        "--experiment-id",
+        default="",
+        help="Base experiment ID (default: the Harbor job ID). A candidate hash is appended.",
+    )
+    parser.add_argument("--name", default="", help="Base experiment name (default: Harbor <job directory name>)")
+    parser.add_argument(
+        "--suite-id",
+        default="",
+        help="Test suite ID for the imported cases (default: derived from the Harbor dataset). "
+        "Combine with --publish-suite to store the cases in that suite.",
+    )
+    parser.add_argument(
+        "--publish-suite",
+        choices=("subset", "replace"),
+        default=None,
+        help="Publish the job's tasks as a stored suite checkpoint before importing. "
+        "subset adds or updates cases; replace also removes cases missing from the job. "
+        "Needs AGENTO11Y_GRAFANA_URL and AGENTO11Y_SERVICE_ACCOUNT_TOKEN.",
+    )
+    parser.add_argument("--main-reward", default="reward", help="Reward that decides pass or fail (default: reward)")
+    parser.add_argument(
+        "--pass-threshold",
+        type=float,
+        default=1.0,
+        help="Minimum main reward that counts as a pass (default: 1.0)",
+    )
+    parser.add_argument("--no-artifacts", action="store_true", help="Skip uploading trajectories and verifier logs")
+    args = parser.parse_args(list(argv))
+    try:
+        published = publish_harbor_job(
+            args.job_dir,
+            experiment_id=args.experiment_id,
+            experiment_name=args.name,
+            suite_id=args.suite_id,
+            test_suites_client=TestSuitesClient() if args.publish_suite else None,
+            suite_publication_policy=args.publish_suite,
+            main_reward=args.main_reward,
+            pass_threshold=args.pass_threshold,
+            upload_artifacts=not args.no_artifacts,
+        )
+    except ValueError as error:
+        print(f"agento11y: {error}", file=sys.stderr)
+        return 1
+    for run in published.runs:
+        print(run.url or run.experiment_id)
+    for label in published.skipped:
+        print(f"agento11y: skipped {label}: its experiment is already finalized", file=sys.stderr)
+    for failure in published.failed:
+        print(f"agento11y: could not publish {failure}", file=sys.stderr)
+    if published.failed:
+        print("agento11y: rerun the same command to publish only the missing experiments", file=sys.stderr)
+        return 1
+    return 0
+
+
+def _run(argv: Sequence[str]) -> int:
+    harbor = shutil.which("harbor")
+    if harbor is None:
+        print("agento11y: Harbor is not installed or is not on PATH", file=sys.stderr)
+        print("Install Harbor, then retry this command.", file=sys.stderr)
+        return 1
+    args = list(argv)
+    if args[:1] == ["--"]:
+        args = args[1:]
+    plugins = [args[index + 1] for index, arg in enumerate(args[:-1]) if arg == "--plugin"]
+    plugins.extend(arg.removeprefix("--plugin=") for arg in args if arg.startswith("--plugin="))
+    if "agento11y" not in plugins:
+        args = ["--plugin", "agento11y", *args]
+    os.execvpe(harbor, [harbor, "run", *args], os.environ)
+    return 0
+
+
+def main(argv: Sequence[str] | None = None) -> int:
+    args = list(sys.argv[1:] if argv is None else argv)
+    if not args or args[0] in {"-h", "--help", "help"}:
+        print(
+            "Usage:\n"
+            "  agento11y harbor [run] -- <harbor run arguments>\n"
+            "  agento11y harbor export <job-directory> [flags]\n"
+            "\n"
+            "Run 'agento11y harbor export --help' for export flags."
+        )
+        return 0
+    if args[0] == "export":
+        return _export(args[1:])
+    if args[0] == "run":
+        args = args[1:]
+    return _run(args)
+
+
+def console_main() -> None:
+    raise SystemExit(main())