agent-operations 0.1.0__py3-none-any.whl

agent_operations-0.1.0.dist-info/METADATA

@@ -0,0 +1,264 @@
+Metadata-Version: 2.4
+Name: agent-operations
+Version: 0.1.0
+Summary: Lightweight self-hosted Agent Operations (AgentOps / LLMOps) toolkit
+Author: WenSongWang
+License: MIT
+Project-URL: Homepage, https://github.com/WenSongWang/agent-operations
+Project-URL: Repository, https://github.com/WenSongWang/agent-operations
+Project-URL: Issues, https://github.com/WenSongWang/agent-operations/issues
+Keywords: agentops,llmops,observability,tracing,sqlite
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Dynamic: license-file
+
+# agent-operations
+
+Open-source **Agent Operations** (AgentOps / LLMOps) toolkit — trace runs, track latency / tokens / cost, and debug tool failures.
+
+Lightweight and self-hostable. Default storage is one SQLite file. No ClickHouse required.
+
+| 项 | 值 |
+| :--- | :--- |
+| Python | ≥ 3.11 |
+| Install | `pip install -e .`（当前建议从源码安装） |
+| Default UI | http://127.0.0.1:8787 |
+| Default DB | `./data/agentops.db` |
+
+## 5 分钟上手
+
+### 1. 安装
+
+```powershell
+git clone https://github.com/WenSongWang/agent-operations.git
+cd agent-operations
+python -m pip install -e .
+```
+
+可选：跑测试
+
+```powershell
+python -m unittest discover -s tests -v
+```
+
+### 2. 用 Python 写入一条 run
+
+```python
+from agentops import connect, ingest_run
+
+conn = connect("./data/agentops.db")
+ingest_run(conn, {
+    "id": "run-001",
+    "project": "demo",
+    "agent": "assistant",
+    "environment": "dev",          # dev | test | uat | prd
+    "name": "answer-question",
+    "status": "ok",                # running | ok | error | timeout | cancelled
+    "started_at": "2026-10-24T10:00:00+00:00",
+    "ended_at": "2026-10-24T10:00:02+00:00",
+    "provider": "openai",
+    "model": "gpt-4o-mini",
+    "input_tokens": 100,
+    "output_tokens": 50,
+    # cost_usd 可省略：会按内置单价估算
+    "steps": [
+        {
+            "seq": 1,
+            "kind": "llm",         # llm | tool | agent | human
+            "name": "gpt-4o-mini",
+            "status": "ok",
+            "started_at": "2026-10-24T10:00:00+00:00",
+            "ended_at": "2026-10-24T10:00:02+00:00",
+            "input_tokens": 100,
+            "output_tokens": 50,
+        }
+    ],
+})
+conn.close()
+```
+
+或直接跑示例：
+
+```powershell
+python examples/quickstart.py
+python examples/openai_wrap.py
+```
+
+### 2b. 更省事：用 `TracedRun`（推荐）
+
+不用手写 `running` / 终态时间戳；OpenAI 响应可直接抽 token：
+
+```python
+from agentops import connect, TracedRun
+
+conn = connect("./data/agentops.db")
+with TracedRun(
+    conn,
+    project="demo",
+    agent="assistant",
+    environment="dev",
+    name="chat",
+    task_input="Say hi",
+) as tr:
+    # resp = client.chat.completions.create(...)
+    # tr.record_openai(resp, output_text=resp.choices[0].message.content)
+    tr.finish(task_output="hi")
+conn.close()
+```
+
+异常退出时会自动记成 `status=error`。完整示例见 `examples/openai_wrap.py`。
+
+### 3. 启动本机服务与页面
+
+```powershell
+agentops-http
+# 等价：python -m agentops.http
+```
+
+浏览器打开 http://127.0.0.1:8787  
+
+### 4. 用 HTTP / curl 上报
+
+```powershell
+curl -s http://127.0.0.1:8787/api/runs -Method POST -ContentType "application/json" -Body @'
+{
+  "id": "run-002",
+  "project": "demo",
+  "agent": "assistant",
+  "environment": "dev",
+  "name": "via-http",
+  "status": "ok",
+  "started_at": "2026-10-24T11:00:00+00:00",
+  "ended_at": "2026-10-24T11:00:01+00:00",
+  "latency_ms": 1000,
+  "input_tokens": 10,
+  "output_tokens": 5
+}
+'@
+```
+
+查询：
+
+```powershell
+curl -s "http://127.0.0.1:8787/api/runs?project=demo"
+curl -s "http://127.0.0.1:8787/api/runs/summary?project=demo"
+curl -s "http://127.0.0.1:8787/api/failures?project=demo"
+curl -s "http://127.0.0.1:8787/api/runs/run-002"
+```
+
+## API 一览
+
+| 方法 | 路径 | 说明 |
+| :--- | :--- | :--- |
+| `GET` | `/` | 只读调试页 |
+| `POST` | `/api/runs` | 写入 / 终态更新一条 run |
+| `GET` | `/api/runs` | 列表（可按 project/agent/environment/status/时间筛选） |
+| `GET` | `/api/runs/summary` | 次数、成功率、token/费用、p50/p95/p99 |
+| `GET` | `/api/runs/{id}` | 单条详情（含 steps） |
+| `GET` | `/api/failures` | 工具失败归并 |
+
+### 必填字段（run）
+
+`id`, `project`, `agent`, `environment`, `name`, `status`, `started_at`  
+
+终态（`ok` / `error` / `timeout` / `cancelled`）还要 `ended_at`。  
+`status=running` 时可省略 `ended_at`；同一 `id` 再报终态会覆盖更新。
+
+### 常用可选字段
+
+| 字段 | 说明 |
+| :--- | :--- |
+| `latency_ms` / `ttft_ms` | 耗时；不传则尽量从时间戳 / steps 推算 |
+| `input_tokens` / `output_tokens` / `cached_tokens` / `reasoning_tokens` | token；未上报保持空，不是 0 |
+| `cost_usd` | 费用；不传则按 `provider`+`model` 估算 |
+| `provider` / `model` | 厂商与模型 |
+| `steps` | 步骤列表（排错用） |
+| `quality_score` | 0–1 |
+| `thumbs` | `up` / `down` |
+| `prompt_version` / `template_version` | 提示词 / 模板版本 |
+| `tags` | JSON，如 `{"end_user_id":"..."}` |
+| `extra` | 备用 JSON |
+
+步骤必填：`seq`, `kind`, `name`, `status`。
+
+## 可选能力
+
+### 脱敏
+
+```powershell
+$env:AGENTOPS_REDACT = "hash"   # off | hash | drop
+```
+
+作用于 `task_input` / `task_output`、步骤 `input` / `output` / `traceback`、`tags.end_user_id`。
+
+### 价目表
+
+未传 `cost_usd` 时用内置单价估算。自定义 JSON（键为 `provider/model`）：
+
+```powershell
+$env:AGENTOPS_PRICING_PATH = ".\pricing.json"
+```
+
+### 告警（适合 cron）
+
+```powershell
+$env:AGENTOPS_ALERT_WEBHOOK = "https://example.com/hook"
+$env:AGENTOPS_ALERT_FAILURE_RATE = "0.2"
+$env:AGENTOPS_ALERT_P95_MS = "3000"
+$env:AGENTOPS_ALERT_COST_USD = "10"
+agentops-alerts --db .\data\agentops.db
+```
+
+有告警时退出码 `2`。`--dry-run` 只打印不 POST。
+
+### 事件 outbox（Kafka 预留）
+
+```powershell
+$env:AGENTOPS_EVENT_OUTBOX = ".\data\events.jsonl"
+```
+
+每次成功写入追加一行 `agentops.run` v1 事件（`create` / `finalize`）。
+
+### API Key（外网绑定前必配）
+
+```powershell
+$env:AGENTOPS_API_KEYS = '{"dev-key":{"projects":["demo"]},"admin":{"projects":["*"]}}'
+agentops-http --host 0.0.0.0 --port 8787
+```
+
+请求头：`Authorization: Bearer <key>` 或 `X-API-Key: <key>`。  
+未配置密钥时：仅建议本机 `127.0.0.1`，接口开放。非 loopback 绑定必须配置密钥。
+
+## 限制（请先读）
+
+| 限制 | 说明 |
+| :--- | :--- |
+| Alpha | 接口可能继续演进；生产使用请自行评估 |
+| SQLite | 默认单机文件库；多机共享库尚未提供（Postgres 在路线图） |
+| 自报数据 | 延迟 / token / 费用依赖调用方或估算，不是云厂商账单 |
+| 非全链路 APM | 聚焦 agent run + steps，不是通用分布式 tracing |
+| Kafka | 目前只有 JSONL outbox，没有内置 broker |
+| 页面 | 只读调试页；配置了 API Key 后，浏览器调 API 需自行带密钥 |
+| 安装渠道 | 当前以源码 `pip install -e .` 为主；版本见 tag `v0.1.0` / [CHANGELOG](./CHANGELOG.md)，暂未上传 PyPI |
+
+更细的后续项见 [ROADMAP.md](./ROADMAP.md)。
+
+## 开发
+
+```powershell
+python -m pip install -e .
+python -m unittest discover -s tests -v
+```
+
+库入口常用符号：`connect`, `ingest_run`, `list_runs`, `summarize_runs`, `get_run`, `list_tool_failures`。

agent_operations-0.1.0.dist-info/RECORD

@@ -0,0 +1,18 @@
+agent_operations-0.1.0.dist-info/licenses/LICENSE,sha256=pltZSnCe9YoWzZVT7n0XL7fly4FDUgat8puYMKhwjLU,1089
+agentops/__init__.py,sha256=Qwf-7qj4YCELTBcwX3zsBYB_TFEjk-5futtTDt_dMkY,565
+agentops/alerts.py,sha256=FNzwr3EVoGYrpfoseQxDiIfD0EhbD9FnL_76SB6eVqs,5765
+agentops/auth.py,sha256=8_SzAQgA7oSC55LE0HLCJCfqjhsbPwaAXegNDU7fBdM,3739
+agentops/db.py,sha256=GMaSwfQCUp8hCeLTHXeFWdDL0T9hNocxBafiDiqtLrM,4354
+agentops/events.py,sha256=YDRzLCNrCHyRNwCYUvZQtLyXZmQVizMKzgcXE-OFdWM,1844
+agentops/http.py,sha256=oAmBQbIpeG27Nd8ZLbERTQDYTJrDfLWKW5esi2U6sq8,13680
+agentops/ingest.py,sha256=Oa_NoeD75VxNVyNS1f_ySdPltpaOpeZ-6pX-DEB23EM,10326
+agentops/metrics.py,sha256=3xxIXZNpqsqpciIY_AJak6pEkUQTp4_Jw9HqDhJbJGc,3729
+agentops/pricing.py,sha256=RVtEioOeCdfrwM1WfXx0XI30Wl99jmJBGqtn-mxi4Bc,3913
+agentops/query.py,sha256=_AI36H--imEwQQHfLL8-QRVDsum5K27L0CNXSM33zIw,7350
+agentops/redact.py,sha256=OwLPGIv5TM51JP1lLIekiJzPLSqRy--MFnomiI-V-7k,2852
+agentops/trace.py,sha256=TEsux9vTQW6vI2TpZJwTmcOhmVPXx-bwVlSeiIlzNQY,6499
+agent_operations-0.1.0.dist-info/METADATA,sha256=UiuV2XTe90fVAptHZedWNT0qwB_r_BuMj8PArVJdRUM,8116
+agent_operations-0.1.0.dist-info/WHEEL,sha256=mOdXR9C3wMVov7cSY8PY-XCzo3SmmWCHvGQr5csJoao,91
+agent_operations-0.1.0.dist-info/entry_points.txt,sha256=Q31dc6NYv5jw9XA_68trmIIO4Q5OJMoiwgYgs25Vb_s,92
+agent_operations-0.1.0.dist-info/top_level.txt,sha256=iveqXCIg3p0dqEDFVBzPclQhECEFXeEFbALdlVupVVk,9
+agent_operations-0.1.0.dist-info/RECORD,,

agent_operations-0.1.0.dist-info/WHEEL

@@ -0,0 +1,5 @@
+Wheel-Version: 1.0
+Generator: setuptools (84.2.1)
+Root-Is-Purelib: true
+Tag: py3-none-any
+

agent_operations-0.1.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+agentops-alerts = agentops.alerts:main
+agentops-http = agentops.http:main

agent_operations-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 WenSongWang
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

agent_operations-0.1.0.dist-info/top_level.txt

@@ -0,0 +1 @@
+agentops

agentops/__init__.py

@@ -0,0 +1,26 @@
+"""Lightweight Agent Operations (AgentOps / LLMOps) toolkit."""
+
+from agentops.db import connect
+from agentops.ingest import IngestError, ingest_run
+from agentops.query import (
+    get_run,
+    list_runs,
+    list_tool_failures,
+    summarize_runs,
+)
+from agentops.trace import TracedRun, usage_from_openai
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "IngestError",
+    "TracedRun",
+    "connect",
+    "get_run",
+    "ingest_run",
+    "list_runs",
+    "list_tool_failures",
+    "summarize_runs",
+    "usage_from_openai",
+    "__version__",
+]

agentops/alerts.py

@@ -0,0 +1,194 @@
+"""Threshold checks and webhook notifications for run health."""
+
+from __future__ import annotations
+
+import json
+import os
+import sqlite3
+import urllib.error
+import urllib.request
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any
+
+from agentops.db import connect
+from agentops.query import summarize_runs
+
+DEFAULT_DB = Path("data") / "agentops.db"
+
+
+@dataclass(frozen=True)
+class Alert:
+    kind: str
+    message: str
+    value: float
+    threshold: float
+
+
+@dataclass(frozen=True)
+class Thresholds:
+    failure_rate: float | None = 0.2
+    p95_ms: float | None = None
+    cost_usd: float | None = None
+    min_runs: int = 1
+
+
+def thresholds_from_env() -> Thresholds:
+    def _float(name: str, default: float | None) -> float | None:
+        raw = os.environ.get(name)
+        if raw is None or raw.strip() == "":
+            return default
+        return float(raw)
+
+    return Thresholds(
+        failure_rate=_float("AGENTOPS_ALERT_FAILURE_RATE", 0.2),
+        p95_ms=_float("AGENTOPS_ALERT_P95_MS", None),
+        cost_usd=_float("AGENTOPS_ALERT_COST_USD", None),
+        min_runs=int(os.environ.get("AGENTOPS_ALERT_MIN_RUNS", "1")),
+    )
+
+
+def evaluate_alerts(
+    summary: dict[str, Any], thresholds: Thresholds | None = None
+) -> list[Alert]:
+    """Return fired alerts for a summarize_runs() result."""
+    th = thresholds or thresholds_from_env()
+    count = int(summary.get("count") or 0)
+    if count < th.min_runs:
+        return []
+
+    alerts: list[Alert] = []
+    by_status = summary.get("by_status") or {}
+    failed = int(by_status.get("error", 0)) + int(by_status.get("timeout", 0))
+    failure_rate = failed / count
+
+    if th.failure_rate is not None and failure_rate >= th.failure_rate:
+        alerts.append(
+            Alert(
+                kind="failure_rate",
+                message=(
+                    f"failure rate {failure_rate:.2%} "
+                    f">= {th.failure_rate:.2%}"
+                ),
+                value=failure_rate,
+                threshold=th.failure_rate,
+            )
+        )
+
+    p95 = (summary.get("latency_ms") or {}).get("p95")
+    if th.p95_ms is not None and p95 is not None and float(p95) >= th.p95_ms:
+        alerts.append(
+            Alert(
+                kind="p95_ms",
+                message=f"latency p95 {p95}ms >= {th.p95_ms}ms",
+                value=float(p95),
+                threshold=th.p95_ms,
+            )
+        )
+
+    cost = (summary.get("cost_usd") or {}).get("total")
+    if (
+        th.cost_usd is not None
+        and cost is not None
+        and float(cost) >= th.cost_usd
+    ):
+        alerts.append(
+            Alert(
+                kind="cost_usd",
+                message=f"cost ${cost} >= ${th.cost_usd}",
+                value=float(cost),
+                threshold=th.cost_usd,
+            )
+        )
+    return alerts
+
+
+def send_webhook(url: str, payload: dict[str, Any], timeout: float = 5.0) -> None:
+    data = json.dumps(payload, ensure_ascii=False, default=str).encode("utf-8")
+    req = urllib.request.Request(
+        url,
+        data=data,
+        headers={"Content-Type": "application/json"},
+        method="POST",
+    )
+    with urllib.request.urlopen(req, timeout=timeout) as resp:
+        resp.read()
+
+
+def check_alerts(
+    conn: sqlite3.Connection,
+    *,
+    thresholds: Thresholds | None = None,
+    webhook: str | None = None,
+    notify: bool = True,
+    project: str | None = None,
+    agent: str | None = None,
+    environment: str | None = None,
+    started_after: str | None = None,
+    started_before: str | None = None,
+) -> list[Alert]:
+    """Evaluate summary filters; optionally POST alerts to webhook."""
+    summary = summarize_runs(
+        conn,
+        project=project,
+        agent=agent,
+        environment=environment,
+        started_after=started_after,
+        started_before=started_before,
+    )
+    alerts = evaluate_alerts(summary, thresholds)
+    url = webhook if webhook is not None else os.environ.get("AGENTOPS_ALERT_WEBHOOK")
+    if notify and alerts and url:
+        send_webhook(
+            url,
+            {
+                "source": "agent-operations",
+                "alerts": [asdict(a) for a in alerts],
+                "summary": summary,
+                "filters": {
+                    "project": project,
+                    "agent": agent,
+                    "environment": environment,
+                    "started_after": started_after,
+                    "started_before": started_before,
+                },
+            },
+        )
+    return alerts
+
+
+def main() -> None:
+    import argparse
+
+    parser = argparse.ArgumentParser(description="Check agentops alert thresholds")
+    parser.add_argument("--db", default=str(DEFAULT_DB))
+    parser.add_argument("--webhook", default=None)
+    parser.add_argument("--project", default=None)
+    parser.add_argument("--agent", default=None)
+    parser.add_argument("--environment", default=None)
+    parser.add_argument("--dry-run", action="store_true")
+    args = parser.parse_args()
+
+    conn = connect(args.db)
+    try:
+        alerts = check_alerts(
+            conn,
+            webhook=args.webhook,
+            notify=not args.dry_run,
+            project=args.project,
+            agent=args.agent,
+            environment=args.environment,
+        )
+    finally:
+        conn.close()
+
+    if not alerts:
+        print("ok: no alerts")
+        return
+    for alert in alerts:
+        print(f"ALERT {alert.kind}: {alert.message}")
+    raise SystemExit(2)
+
+
+if __name__ == "__main__":
+    main()

agentops/auth.py

@@ -0,0 +1,113 @@
+"""API key auth and per-key project ACL."""
+
+from __future__ import annotations
+
+import json
+import os
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+
+class AuthError(PermissionError):
+    """Missing/invalid credentials or project not allowed."""
+
+
+@dataclass(frozen=True)
+class Principal:
+    name: str
+    projects: frozenset[str]
+
+    def allows_project(self, project: str | None) -> bool:
+        if "*" in self.projects:
+            return True
+        if project is None:
+            return False
+        return project in self.projects
+
+    def require_project(self, project: str | None) -> None:
+        if not self.allows_project(project):
+            raise AuthError(f"project not allowed: {project!r}")
+
+
+def load_api_keys(
+    raw: str | None = None, path: str | Path | None = None
+) -> dict[str, Principal]:
+    """Load API keys from JSON object: {\"key\": {\"projects\": [\"demo\"]}}."""
+    text = raw
+    if text is None:
+        text = os.environ.get("AGENTOPS_API_KEYS")
+    file_path = path
+    if file_path is None:
+        env_path = os.environ.get("AGENTOPS_API_KEYS_PATH")
+        file_path = env_path
+    if (not text) and file_path:
+        text = Path(file_path).read_text(encoding="utf-8")
+    if not text:
+        return {}
+
+    data = json.loads(text)
+    if not isinstance(data, dict):
+        raise ValueError("API keys JSON must be an object")
+
+    keys: dict[str, Principal] = {}
+    for key, meta in data.items():
+        if not isinstance(key, str) or not key:
+            raise ValueError("API key names must be non-empty strings")
+        if isinstance(meta, list):
+            projects = meta
+            name = key[:8]
+        elif isinstance(meta, dict):
+            projects = meta.get("projects", ["*"])
+            name = str(meta.get("name") or key[:8])
+        else:
+            raise ValueError(f"invalid API key entry for {key!r}")
+        if not isinstance(projects, list) or not projects:
+            raise ValueError(f"projects required for API key {key!r}")
+        keys[key] = Principal(name=name, projects=frozenset(str(p) for p in projects))
+    return keys
+
+
+def extract_api_key(headers: Any) -> str | None:
+    """Read Bearer token or X-API-Key from headers mapping."""
+    auth = headers.get("Authorization") or headers.get("authorization")
+    if auth:
+        parts = str(auth).split(None, 1)
+        if len(parts) == 2 and parts[0].lower() == "bearer":
+            return parts[1].strip() or None
+    key = headers.get("X-API-Key") or headers.get("x-api-key")
+    if key:
+        return str(key).strip() or None
+    return None
+
+
+def authenticate(
+    headers: Any, keys: dict[str, Principal] | None = None
+) -> Principal | None:
+    """Return Principal when keys are configured; None when auth is disabled."""
+    table = keys if keys is not None else load_api_keys()
+    if not table:
+        return None
+    token = extract_api_key(headers)
+    if not token:
+        raise AuthError("missing API key")
+    principal = table.get(token)
+    if principal is None:
+        raise AuthError("invalid API key")
+    return principal
+
+
+def scope_project_filter(
+    principal: Principal | None, project: str | None
+) -> str | None:
+    """Resolve project filter under ACL. May raise AuthError."""
+    if principal is None:
+        return project
+    if "*" in principal.projects:
+        return project
+    if project is not None:
+        principal.require_project(project)
+        return project
+    if len(principal.projects) == 1:
+        return next(iter(principal.projects))
+    raise AuthError("project query parameter required")