devopsiq 0.1.0__py3-none-any.whl

devopsiq-0.1.0.dist-info/METADATA

@@ -0,0 +1,662 @@
+Metadata-Version: 2.4
+Name: devopsiq
+Version: 0.1.0
+Summary: A from-scratch, read-only AI agent that investigates real DevOps problems — Kubernetes, Docker, Helm, Argo CD, Istio, Terraform, GitHub Actions, cloud, New Relic and more.
+Author: DevOpsAbhii
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/DevOpsAbhii/devops-ai-agent
+Project-URL: Repository, https://github.com/DevOpsAbhii/devops-ai-agent
+Project-URL: Issues, https://github.com/DevOpsAbhii/devops-ai-agent/issues
+Keywords: devops,kubernetes,docker,helm,argocd,istio,terraform,github-actions,observability,newrelic,incident-response,root-cause-analysis,ai-agent,cli
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: System :: Systems Administration
+Classifier: Topic :: Software Development :: Debuggers
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: openai>=1.40.0
+Requires-Dist: python-dotenv>=1.0.1
+Dynamic: license-file
+
+# DevOps AI Agent
+
+An AI agent that investigates real DevOps problems. The end goal: ask it
+something like *"Why is my Kubernetes pod in CrashLoopBackOff?"* and have it
+gather evidence, reason about the evidence, identify the likely root cause,
+recommend remediation, and give verification steps.
+
+**Phase 5** broadens the agent from
+Kubernetes-only to the wider DevOps surface: Linux systemd services and
+journals, Docker containers, Terraform state/plan, and git/GitHub pull
+requests — every tool still read-only, still a fixed allowlisted argv
+template. **Phase 6 adds the automation surface:** a one-shot CLI mode
+(`python main.py "problem"` — exit-code driven, cron/CI-friendly) and a
+structured JSON export of the investigation report, so pipelines can act on
+the verdict instead of parsing markdown. **Phase 7 adds persistence:** the
+investigation record is auto-saved on every change to
+`~/.devops-ai-agent/investigations/` (overridable with `--store-dir` or
+`AGENT_STORE_DIR`), survives CLI exits, and the REPL resumes the newest
+in-progress record at startup. **Phase 8 broadens the tool surface to 47
+read-only tools:** Kubernetes depth (pod listing, `kubectl top` resource
+usage, HPAs, PVCs, contexts), Docker depth (networks, volumes, disk usage),
+GitHub Actions via `gh` (runs, run jobs, workflows), cloud identity
+(AWS/GCP/Azure), monitoring/logging (Prometheus, Loki, Grafana — endpoints
+from environment config), and Ansible listing (inventory, playbook tasks).
+**Phase 9 adds 11 trending-market tools (58 total):** New Relic (NRQL +
+open alerts over NerdGraph — credentials from env), Trivy image
+vulnerability scanning, Helm releases (list/status/history), Argo CD
+(GitOps app sync/health), Istio mesh proxy status, and Docker Compose
+(project and service listing). The repository is git-tracked.
+Every phase still built from scratch — no LangChain, LangGraph,
+AutoGen, CrewAI, or MCP.
+
+> **Want to use this in your own project?** See [INTEGRATION.md](INTEGRATION.md)
+> for the three integration levels: drive it as a CLI from cron/CI (exit
+> codes + `--json`), embed it as a Python library, or extend it with your
+> own read-only tools.
+
+---
+
+## 1. What this project is
+
+The agent is built intentionally: we own the core architecture — how the
+model is called, how conversation history flows, how tool selection +
+execution + evidence feedback work — instead of depending on a framework
+for it.
+
+In Phase 5 the agent:
+- holds a conversation with **GLM 5.3** through **OpenRouter**;
+- has **58 real, read-only tools** across fifteen domains: host facts,
+  Kubernetes (12 tools), Linux system (4), Docker + Compose (10),
+  Terraform (3), Helm (3), Argo CD (2), Istio (1), security/Trivy (1),
+  git/GitHub + Actions (7), cloud identity (4), monitoring/logging (3),
+  New Relic (2), Ansible (2), plus the 3 investigation tools;
+- runs a **tool-use loop**: when the model decides a question needs evidence,
+  it requests the tool, the application executes it locally, and the real
+  output is fed back to the model, which then answers from it;
+- runs an **investigation loop**: for a reported problem it plans a multi-step
+  investigation — open a record with initial hypotheses, gather evidence with
+  the right domain tools, update each hypothesis's verdict as evidence
+  builds, conclude with root cause + remediation + verification, and end with
+  a structured report. The tracked state lives in the application, so
+  `/investigation` and `/report` show it directly.
+
+It is **read-only by design**: every tool is a static, allowlisted command
+template — the model can never pass arbitrary command text, can never select
+a mutating verb (no delete/restart/edit/apply/scale/exec/run/rm/reset/
+destroy anywhere), and no mutating capability exists. Object and unit names
+are validated before reaching any CLI; flags are injected only by fixed
+templates, never by model text. The investigation tools mutate only the
+agent's in-memory record. Any future mutating capability will only ever
+arrive behind an explicit human-approval gate.
+
+## 2. Current architecture
+
+```
+You (terminal)
+   │  plain text
+   ▼
+main.py ......................... CLI REPL, .env loading, error handling
+   │
+   ▼
+agent/agent.py (DevOpsAgent) .... system prompt + conversation history
+   │
+   ▼  the tool-use loop, inside DevOpsAgent._complete():
+   │
+   │   attach tool schemas -> call model
+   │        │
+   │        ├─ model requests tool(s)? ── yes ─▶ tools/registry.execute_tool()
+   │        │                                 │   per call, in order
+   │        │                                 │      └─ allowlisted read-only
+   │        │                                 │         command runs for real
+   │        │                                 │      └─ result echoed back as
+   │        │                                 │         a "tool" message
+   │        │                                 ◀── loop calls the model again
+   │        └─ model answers in plain text? ── yes ─▶ done
+   │              (safety cap: max 10 tool-use turns, then abort)
+   ▼
+OpenRouter (https://openrouter.ai/api/v1)
+   ▼
+GLM 5.3 (z-ai/glm-5.3)
+```
+
+Module map:
+
+| Path                        | Responsibility                                              |
+| --------------------------- | ------------------------------------------------------------ |
+| `main.py`                   | REPL loop, one-shot CLI, environment loading, `/investigate` commands, error messages |
+| `agent/agent.py`            | `DevOpsAgent` — client, history, `ask()`, `_complete()` loop |
+| `agent/prompts.py`          | The system prompt (versioned/tested separately)              |
+| `agent/investigation.py`    | First-class investigation record: hypotheses, verdicts, evidence, report renderer (pure data) |
+| `agent/store.py`            | `InvestigationStore` — one JSON file per record, atomic writes, resume/list (Phase 7) |
+| `tools/base.py`             | Tool contract: `Tool`, `ToolError`, `read_command_output()`  |
+| `tools/registry.py`         | `register/get_tools/execute_tool` — tools declared & executed here |
+| `tools/preflight.py`        | `system_info` — host facts (allowlisted read-only commands)  |
+| `tools/kubernetes.py`       | 12 kubectl tools: pods, pod status/logs, deployments, events, nodes, top (cpu/mem), hpa, pvc, services, contexts |
+| `tools/system.py`           | systemd/journal/ss/ps tools — services, journals, ports, top processes |
+| `tools/docker.py`           | read-only docker tools — ps, inspect, logs, stats, images, networks, volumes, disk usage, compose ls/ps |
+| `tools/terraform.py`        | `tf_show` / `tf_state_list` / `tf_plan` (working directory)  |
+| `tools/helm.py`             | `helm_list` / `helm_status` / `helm_history` — release reads only (Phase 9) |
+| `tools/argocd.py`           | `argocd_apps` / `argocd_app_status` — GitOps app reads (Phase 9) |
+| `tools/istio.py`            | `istioctl_proxy_status` — mesh sync view, no arguments (Phase 9) |
+| `tools/trivy.py`            | `trivy_image_scan` — vulnerability report for one image (Phase 9) |
+| `tools/newrelic.py`         | `newrelic_nrql` / `newrelic_alerts` — NerdGraph over curl, credentials from env only (Phase 9) |
+| `tools/git_ci.py`           | git status/log/diff + `gh_prs`, `gh_runs`, `gh_run_view`, `gh_workflows` (working directory) |
+| `tools/cloud.py`            | read-only cloud identity/listing: `aws_identity`, `gcloud_identity`, `az_account`, `az_groups` |
+| `tools/monitoring.py`       | `prom_query`, `loki_query`, `grafana_health` — endpoints from env config only |
+| `tools/ansible.py`          | `ansible_inventory`, `ansible_playbook_tasks` — listing modes only |
+| `tools/investigation.py`    | `investigation_begin` / `investigation_record` / `investigation_conclude` — meta-tools for the investigation record |
+| `tests/test_phase2.py`      | Offline suite: contract, safety, loop (stdlib `unittest`)     |
+| `tests/test_phase3.py`      | Offline suite: k8s command lines + verb-allowlist (fake kubectl) |
+| `tests/test_phase4.py`      | Offline suite: investigation state, meta-tools, loop-driven investigation |
+| `tests/test_phase5.py`      | Offline suite: argv templates + name validation for all new domains (fake CLIs) |
+| `tests/test_automation.py`  | Offline suite: JSON reports + one-shot CLI (Phase 6)                        |
+| `tests/test_phase7.py`      | Offline suite: round-trip serialization, store files, auto-save, resume, CLI flags (Phase 7) |
+| `tests/test_phase8.py`      | Offline suite: argv templates + validation for the 21 Phase 8 tools (fake CLIs, env-based monitoring) |
+| `tests/test_phase9.py`      | Offline suite: New Relic env-credential + payload tests, trivy/helm/argocd/istio/compose argv templates (Phase 9) |
+
+### The tool-use loop
+
+All model calls funnel through `DevOpsAgent._complete()`. Each turn:
+
+1. The full history is sent with every tool's schema attached (`tools=`).
+2. If the reply contains tool calls, the assistant tool-request turn is
+   echoed into history verbatim, and each call is executed locally via
+   `tools/registry.execute_tool(name, arguments)`.
+3. Every real result is appended as a sibling `{"role": "tool"}` message
+   pinned to its `tool_call_id`.
+4. The loop calls the model again — giving it the evidence to reflect on —
+   and repeats until it answers in plain text.
+5. `MAX_TOOL_ITERATIONS = 10` aborts runaway loops.
+
+### The investigation loop (Phase 4)
+
+For a reported problem (a pod in CrashLoopBackOff, a broken rollout) the
+same loop carries a *plan* as well as evidence:
+
+1. The model opens a record with `investigation_begin` (one-line problem +
+   2–4 initial hypotheses, which become H1, H2, …).
+2. It plans what it needs, then calls the read-only tools one deliberate
+   step at a time.
+3. Every finding is recorded with `investigation_record`: evidence notes
+   (linked to a hypothesis when they bear on one) and verdicts
+   (supported / refuted / confirmed).
+4. Each record call returns the **live tracker** (`Problem`, status,
+   hypotheses with verdicts, evidence), so the model always knows where it
+   stands without inspecting all of history.
+5. When evidence is sufficient the model calls `investigation_conclude`
+   with root cause, remediation *recommendations* (nothing is ever
+   executed), verification steps, and confidence — then ends its answer
+   with the structured report.
+
+The record lives in the **application**, not just the conversation: the
+report is rendered deterministically from it (agent/investigation.py), and
+the CLI exposes it directly:
+
+| Command                  | What it does                                        |
+| ------------------------ | --------------------------------------------------- |
+| `/investigate <problem>` | Open a formal investigation (same path the model uses) |
+| `/investigation`         | Show the live tracked state (hypotheses/evidence)   |
+| `/investigations`        | List saved records on disk (newest first; `← active` marks the live one) |
+| `/report`                | Show the canonical report (once concluded)          |
+| `/endinvestigation`      | Clear the record (memory only — the saved copy stays as history) |
+
+The report the agent ends with and `/report` render are kept consistent by
+construction: `tool` results confirm each record call and the tracker, and
+`render_report()` is the single report format.
+
+Example (live, against a real cluster):
+
+```
+You: /investigate Pod crashloop-6f8c… is in CrashLoopBackOff in default ns.
+     (…) investigate.
+Agent: (investigation_begin → k8s_pod_status → k8s_pod_logs →
+       k8s_deployment_status → verdicts on H1/H2 → investigation_conclude)
+       ends with: Facts → Hypotheses → Root cause → Remediation (recommended)
+       → Verification steps.
+
+You: /report
+Agent: # Investigation report  … (canonical, rendered from the record)
+```
+
+### The tools (Phases 5, 8 and 9)
+
+Every tool runs a fixed, allowlisted command — the model only picks
+arguments (names, counts, namespaces), never command text.
+
+| Domain           | Tool                  | Backing command (read-only)                          |
+| ---------------- | --------------------- | ---------------------------------------------------- |
+| Host             | `system_info`         | `date`/`uname`/`uptime`/`df`/`free`                  |
+| Kubernetes       | `k8s_pod_status`      | `kubectl get pod <pod> -n <ns> -o json`              |
+|                  | `k8s_pod_logs`        | `kubectl logs <pod> -n <ns> --tail=<n>`              |
+|                  | `k8s_deployment_status` | `kubectl get deployment <dep> -n <ns> -o json`     |
+|                  | `k8s_events`          | `kubectl get events -n <ns> --sort-by=.lastTimestamp -o wide [--field-selector involvedObject.name=<obj>]` |
+|                  | `k8s_nodes`           | `kubectl get nodes -o json`                          |
+|                  | `k8s_services`        | `kubectl get services -n <ns> -o json`               |
+|                  | `k8s_pods`            | `kubectl get pods -n <ns> -o wide`                   |
+|                  | `k8s_top_pods`        | `kubectl top pods -n <ns> [--sort-by=cpu\|memory]`   |
+|                  | `k8s_top_nodes`       | `kubectl top nodes`                                  |
+|                  | `k8s_hpa`             | `kubectl get hpa [<name>] -n <ns> -o json`           |
+|                  | `k8s_pvc`             | `kubectl get pvc [<name>] -n <ns> -o json`           |
+|                  | `k8s_contexts`        | `kubectl config get-contexts` (listing only — never `use-context`) |
+| Linux system     | `sys_service_status`  | `systemctl status <unit> --no-pager`                 |
+|                  | `sys_service_logs`    | `journalctl -u <unit> --no-pager -n <n>`             |
+|                  | `sys_open_ports`      | `ss -tlnp`                                           |
+|                  | `sys_top_processes`   | `ps aux --sort=-%cpu --no-headers`                   |
+| Docker           | `docker_ps`           | `docker ps -a`                                       |
+|                  | `docker_inspect`      | `docker inspect <name>`                              |
+|                  | `docker_logs`         | `docker logs --tail <n> <name>`                      |
+|                  | `docker_stats`        | `docker stats --no-stream` (the flag is pinned — see below) |
+|                  | `docker_images`       | `docker images`                                      |
+|                  | `docker_networks`     | `docker network ls`                                  |
+|                  | `docker_volumes`      | `docker volume ls`                                   |
+|                  | `docker_disk_usage`   | `docker system df`                                   |
+|                  | `docker_compose_ls`   | `docker compose ls`                                  |
+|                  | `docker_compose_ps`   | `docker compose [-p <project>] ps -a`                |
+| Helm            | `helm_list`           | `helm list -n <ns>` / `helm list --all-namespaces`   |
+|                  | `helm_status`         | `helm status <release> -n <ns>`                      |
+|                  | `helm_history`        | `helm history <release> -n <ns> --max <n>`           |
+| Argo CD         | `argocd_apps`         | `argocd app list --output json`                      |
+|                  | `argocd_app_status`   | `argocd app get <app>`                               |
+| Istio           | `istioctl_proxy_status` | `istioctl proxy-status` (no arguments)             |
+| Security        | `trivy_image_scan`    | `trivy image --scanners vuln --format table <image>` |
+| New Relic       | `newrelic_nrql`       | `curl -H "API-Key: …" -d <json payload> https://api.newrelic.com/graphql` |
+|                  | `newrelic_alerts`     | same pinned curl + payload (no model input)          |
+| Terraform        | `tf_show`             | `terraform show -no-color`                           |
+|                  | `tf_state_list`       | `terraform state list`                               |
+|                  | `tf_plan`             | `terraform plan -no-color -input=false`              |
+| git / GitHub     | `git_repo_status`     | `git status --short --branch`                        |
+|                  | `git_log`             | `git log --oneline -n <n>`                           |
+|                  | `git_diff`            | `git diff --stat HEAD`                               |
+|                  | `gh_prs`              | `gh pr list --limit <n> --json number,title,state,...` |
+|                  | `gh_runs`             | `gh run list --limit <n> --json databaseId,displayTitle,status,...` |
+|                  | `gh_run_view`         | `gh run view <id> --json status,conclusion,jobs` (id digits-only) |
+|                  | `gh_workflows`        | `gh workflow list --limit <n> --json id,name,state`  |
+| Cloud identity   | `aws_identity`        | `aws sts get-caller-identity --output json`          |
+|                  | `gcloud_identity`     | `gcloud config list --format=json`                   |
+|                  | `az_account`          | `az account show`                                    |
+|                  | `az_groups`           | `az group list`                                      |
+| Monitoring       | `prom_query`          | `curl <PROMETHEUS_URL>/api/v1/query?query=<urlencoded PromQL>` |
+|                  | `loki_query`          | `curl <LOKI_URL>/loki/api/v1/query?query=<urlencoded LogQL>&limit=<n>` |
+|                  | `grafana_health`      | `curl <GRAFANA_URL>/api/health`                      |
+| Ansible          | `ansible_inventory`   | `ansible-inventory [--inventory <source>] --list`    |
+|                  | `ansible_playbook_tasks` | `ansible-playbook --list-tasks --list-hosts <playbook>` |
+| Investigation    | `investigation_begin` / `investigation_record` / `investigation_conclude` | record only — memory, no external command |
+
+Notes:
+
+- Output is the **real** CLI output — Python does not re-parse it; the model
+  reads exactly what an engineer would see. Verbatim quoting is a hard
+  prompt rule.
+- `docker stats` is **always** `--no-stream` (the pinned flag is what keeps
+  it from following forever and hanging the turn).
+- `terraform plan` is a **dry run** — it computes the diff, mutates nothing;
+  `-input=false` keeps it from ever prompting. Requires an initialized
+  directory (`terraform init`); on a fresh dir the model reports terraform's
+  own error, honestly.
+- Terraform and git/gh tools operate on the **current working directory** the
+  agent was launched from (no path parameters — that keeps traversal out).
+- `gh_prs` needs the `gh` CLI authenticated and a GitHub remote; otherwise
+  the exact CLI error is returned.
+- `kubectl top` needs **metrics-server** in the cluster; where it is absent,
+  kubectl's exact error is returned (that is honest behavior, not a bug).
+- `k8s_contexts` is a listing of kubeconfig contexts only — the agent can
+  never run `config use-context` and change which cluster is in use.
+- **Monitoring endpoints come from the environment, never the model:**
+  `prom_query`/`loki_query`/`grafana_health` read `PROMETHEUS_URL` /
+  `LOKI_URL` / `GRAFANA_URL`; unset or non-http(s) values are honest
+  ToolErrors naming the variable. The model supplies only the PromQL/LogQL
+  text, which is percent-encoded into the query string; the pinned curl argv
+  (`--proto =https,http`, GET only) blocks `file://` and everything but the
+  configured host.
+- Cloud tools are identity/account level only and take no arguments at all;
+  Ansible tools are listing modes only (`--list`, `--list-tasks`) with
+  relative-path validation (no `/`, no leading `-`, no `..`).
+- **Helm tools only read** (list/status/history — never install/upgrade/
+  rollback/uninstall); **Argo CD tools only list/get** (never sync/rollback/
+  delete); `istioctl_proxy_status` takes no arguments at all;
+  `trivy_image_scan` validates the image reference (no leading `-`, no
+  spaces) and warns its first run may take minutes (CVE DB download).
+- **New Relic credentials are environment-configured, never model-chosen:**
+  `NEW_RELIC_API_KEY` (passed only as a curl header value) and
+  `NEW_RELIC_ACCOUNT_ID` (digits only). The NRQL text travels inside a
+  `json.dumps`-built payload as a GraphQL *variable*, so it can never
+  escape its string slot; only NerdGraph queries are ever sent, never
+  mutations.
+- Log tails (`--tail`, `-n`, `--limit`) are bounded integers, validated
+  1–500 (git log 1–100, gh 1–50, Loki limit 1–1000, helm history 1–50).
+- Any missing CLI / unreachable target returns the exact error — nothing is
+  invented (verified live for kubectl, systemctl, journalctl, docker,
+  terraform, git).
+
+### How read-only is enforced (defense in depth)
+
+- The tool schemas only allow picking names/counts/namespaces from validated
+  arguments — there is no way to pass command text to any CLI.
+- Every tool is a fixed argv template. No `sh -c` anywhere, so nothing is
+  ever parsed by a shell; flags the model might abuse (e.g. `--no-stream`
+  for docker stats, `-input=false` for terraform plan) are hard-coded into
+  the template and cannot be removed or added.
+- Only read-only verbs exist per domain — `get`/`logs`/`top` and
+  `config get-contexts` (kubectl), `status`/`-u`/`ss`/`ps` (system),
+  `ps`/`inspect`/`logs`/`stats`/`images`/`network ls`/`volume ls`/
+  `system df`/`compose ls`/`compose ps` (docker), `show`/`state list`/
+  `plan` (terraform), `list`/`status`/`history` (helm), `app list`/
+  `app get` (argocd), `proxy-status` (istioctl), `image --scanners vuln`
+  (trivy), `status`/`log`/`diff`/`pr list`/`run list`/`run view`/
+  `workflow list` (git/gh), `--list`/`--list-tasks` (ansible), GET-only
+  curl with a pinned argv (monitoring), query-only NerdGraph with a pinned
+  curl (New Relic). No delete, restart, edit, apply, scale, exec, run, rm,
+  pull, push, commit, reset, merge, use-context, playbook-run, install,
+  upgrade, rollback, uninstall, sync, destroy — by construction.
+- Names are validated per domain before reaching any CLI: Kubernetes object
+  names (DNS style), systemd unit names (no `/`, no leading `-`), Docker
+  names (no `/`), GitHub run ids (digits only), Ansible sources (relative
+  paths, no `..`), and no path parameters at all for terraform/git/gh tools.
+  This blocks flag and path injection.
+- The executor re-validates every argument. Never trust the model.
+- `--request-timeout`/timeouts bound slow or hanging commands, and logs are
+  tail-bounded.
+- Unrecognized tools/arguments return `Tool error: ...` to the model instead
+  of executing.
+- Tool output is truncated at 8,000 characters per result.
+
+## 3. Why GLM 5.3
+
+- GLM 5.3 (`z-ai/glm-5.3` on OpenRouter) is a capable, cost-effective
+  general model — with strong instruction-following and **function-calling**
+  support, which the tool loop has demonstrated live in every phase.
+- Served by Zhipu AI through a single OpenRouter endpoint, so there is no
+  separate vendor API to manage.
+- An investigation loop sends many tokens (system prompt, tool schemas, tool
+  output, history); GLM 5.3 keeps that cost sustainable.
+
+## 4. How OpenRouter fits
+
+[OpenRouter](https://openrouter.ai) is a model gateway: one API key, one
+OpenAI-compatible endpoint, access to many models. We call
+
+```
+POST https://openrouter.ai/api/v1/chat/completions
+```
+
+with model `z-ai/glm-5.3` and standard `tools` / `tool` messages. The
+official `openai` Python SDK works as our client with two config lines:
+
+```python
+OpenAI(api_key=..., base_url="https://openrouter.ai/api/v1")
+```
+
+Swapping to another OpenRouter model later is a one-line change (an env var
+today); moving to any other OpenAI-compatible provider changes only
+`agent/agent.py` configuration.
+
+## 5. Installation
+
+Three ways — pick one. Whatever you choose, the agent also needs the CLIs of
+the domains you use (kubectl, docker, gh, … — each tool reports its exact
+error if its CLI is missing, so a partial install is fine).
+
+**From PyPI (no clone):**
+
+```bash
+pipx install devopsiq        # or: pip install devopsiq
+export OPENROUTER_API_KEY=sk-or-...
+devopsiq --json "why is api-5d6f crash-looping?"
+```
+
+The console command is `devopsiq`; the importable package is
+`agent` / `tools` / `main` (see INTEGRATION.md for library use).
+
+**Prebuilt Docker image (bundles kubectl, helm, gh, trivy, git, curl,
+docker CLI):**
+
+```bash
+docker run --rm \
+  -e OPENROUTER_API_KEY=sk-or-... \
+  -v "$HOME/.kube:/home/agent/.kube:ro" \
+  -v agent-records:/data \
+  ghcr.io/devopsabhii/devops-ai-agent --json "why is api-5d6f crash-looping?"
+```
+
+Images are multi-arch (amd64 + arm64), published on every `v*` tag
+(`:latest` tracks the newest release). Add
+`-v /var/run/docker.sock:/var/run/docker.sock` for the Docker/Compose tools.
+
+**From source (development):**
+
+Requires **Python 3.10+** (built-in `venv`). The tool layer itself uses only
+the standard library; the OpenAI SDK and python-dotenv are the only Python
+dependencies.
+
+```bash
+cd ~/devops-ai-agent
+python3 -m venv .venv
+.venv/bin/pip install --upgrade pip
+.venv/bin/pip install -r requirements.txt
+```
+
+Recommended CLIs, by domain:
+
+| Domain       | CLIs needed                                      |
+| ------------ | ------------------------------------------------ |
+| Kubernetes   | `kubectl` (configured cluster context; metrics-server for `top`) |
+| Linux system | `systemctl`, `journalctl`, `ss`, `ps` (systemd host) |
+| Docker       | `docker` (daemon running); `docker compose` for the compose tools |
+| Terraform    | `terraform` (initialized working directory)      |
+| Helm         | `helm` (kubeconfig)                              |
+| Argo CD      | `argocd` (installed + `argocd login`)            |
+| Istio        | `istioctl` (reachable mesh)                      |
+| Security     | `trivy` (first scan downloads the CVE DB)        |
+| git / GitHub | `git`; `gh` (authenticated, for PRs and Actions) |
+| Cloud        | `aws` / `gcloud` / `az` (only the ones you use)  |
+| Monitoring   | `curl` + endpoint env vars (see below)           |
+| New Relic    | `curl` + `NEW_RELIC_API_KEY` / `NEW_RELIC_ACCOUNT_ID` env vars |
+| Ansible      | `ansible-inventory`, `ansible-playbook`          |
+
+## 6. Environment setup
+
+```bash
+cp .env.example .env     # then edit it
+```
+
+`.env` must contain your real key — get one at <https://openrouter.ai/keys>:
+
+```
+OPENROUTER_API_KEY=sk-or-...
+```
+
+Optional overrides (defaults shown):
+
+```
+OPENROUTER_BASE_URL=https://openrouter.ai/api/v1
+OPENROUTER_MODEL=z-ai/glm-5.3
+AGENT_STORE_DIR=~/.devops-ai-agent/investigations   # (Phase 7) where records are saved
+PROMETHEUS_URL=http://prometheus:9090               # (Phase 8) enables prom_query
+LOKI_URL=http://loki:3100                           # (Phase 8) enables loki_query
+GRAFANA_URL=http://grafana:3000                     # (Phase 8) enables grafana_health
+NEW_RELIC_API_KEY=NRAK-...                          # (Phase 9) NerdGraph user key
+NEW_RELIC_ACCOUNT_ID=1234567                        # (Phase 9) numeric account id
+```
+
+`.env` is gitignored; the API key is never hard-coded in Python, printed, or
+logged. If `OPENROUTER_API_KEY` is already set in your shell, the shell value
+wins and `.env` is not consulted. Kubernetes tools use kubectl's own config
+(`~/.kube/config` or `KUBECONFIG`); no agent-side config is needed.
+
+## 7. How to run the agent
+
+**REPL (interactive):**
+
+```bash
+cd ~/devops-ai-agent
+.venv/bin/python main.py
+```
+
+**One-shot (cron / CI / scripts):** pass the problem as the first argument.
+The agent runs once, prints its answer, and exits 0 on success, 1 on
+setup/API errors — so it drops straight into a pipeline:
+
+```bash
+.venv/bin/python main.py "why is api-5d6f crash-looping?"
+.venv/bin/python main.py --json "why is api-5d6f crash-looping?"   # structured report
+.venv/bin/python main.py --resume --json "any update?"             # continue a prior run
+.venv/bin/python main.py --store-dir /tmp/runs --out report.json "..."  # pipeline paths
+```
+
+With `--json` the stdout is one JSON document (see `render_report_json` in
+`agent/investigation.py`): `problem`, `status`, `hypotheses`, `evidence`,
+and — once concluded — `conclusion` with `root_cause`, `remediation`,
+`verification`, `confidence`. If the model never opened an investigation,
+`--json`/`--out` fail with exit 1 rather than printing a malformed report.
+`--out PATH` additionally writes that JSON to an exact path for pipelines
+that want a known location; `--store-dir DIR` points persistence somewhere
+else for the run; `--resume` continues the newest in-progress record from
+the store before asking. Slash commands also work one-shot:
+`python main.py /report`.
+
+**Persistence (Phase 7):** every record mutation is auto-saved to
+`~/.devops-ai-agent/investigations/` — one JSON file per investigation,
+written atomically on every begin/record/conclude, so a crash mid-run
+loses nothing. The REPL resumes the newest *in-progress* record at startup
+(concluded records stay as history); `/investigations` lists everything on
+disk; `/endinvestigation` clears memory but keeps the saved file. A broken
+store (permissions, full disk) degrades to a warning in the tool result —
+it never interrupts an investigation.
+
+Example session:
+
+```
+DevOps AI Agent (Phase 9 — 58 read-only tools, persistent investigations)
+Model:   z-ai/glm-5.3
+Backend: https://openrouter.ai/api/v1
+Store:   /home/you/.devops-ai-agent/investigations
+Tools:   ansible_inventory, ansible_playbook_tasks, aws_identity,
+         az_account, az_groups, docker_disk_usage, docker_images,
+         docker_inspect, docker_logs, docker_networks, docker_ps,
+         docker_stats, docker_volumes, gcloud_identity, gh_prs,
+         gh_run_view, gh_runs, gh_workflows, git_diff, git_log,
+         git_repo_status, grafana_health, investigation_begin,
+         investigation_conclude, investigation_record, k8s_contexts,
+         k8s_deployment_status, k8s_events, k8s_hpa, k8s_nodes,
+         k8s_pod_logs, k8s_pod_status, k8s_pods, k8s_pvc, k8s_services,
+         k8s_top_nodes, k8s_top_pods, loki_query, prom_query,
+         sys_open_ports, sys_service_logs, sys_service_status,
+         sys_top_processes, system_info, tf_plan, tf_show, tf_state_list
+Commands: /investigate <problem>, /investigation, /investigations, /report, /endinvestigation
+One-shot: python main.py [--json] [--resume] [--out report.json] [--store-dir DIR] "<problem>"
+Type 'exit' to quit.
+
+You: The checkout service container keeps exiting in Docker. Investigate.
+Agent: (docker_ps → docker_inspect → docker_logs, tracks hypotheses, and
+       ends with the structured report: the crash command, exit code, and
+       the remediation recommendation)
+
+You: /report
+Agent: # Investigation report … (the canonical record, rendered from state)
+
+You: exit
+```
+
+Type `exit` / `quit`, or press Ctrl-D / Ctrl-C to leave. Slash commands are
+handled locally and never reach the model.
+
+## 8. Current limitations (Phase 9)
+
+- **Each domain needs its CLI installed and reachable.** Missing CLIs,
+  unauthenticated `gh`, a dead docker daemon, an uninitialized terraform
+  directory, or an unreachable cluster all return the exact error honestly —
+  nothing is invented, but a tool can't produce data without its backend.
+- **terraform/git/gh/ansible tools are working-directory scoped.** They read
+  the directory the agent was launched from — no path arguments by design
+  (keeps traversal out). To investigate another repo/module, launch the
+  agent there.
+- **Monitoring endpoints are environment-configured by design.** Unless the
+  operator sets `PROMETHEUS_URL` / `LOKI_URL` / `GRAFANA_URL`, those tools
+  fail with an honest message naming the variable; the agent never invents
+  a URL.
+- **`kubectl top` needs metrics-server.** Clusters without it return
+  kubectl's exact error — honest, but no usage data.
+- **Cloud tools are identity-level only.** `aws_identity`,
+  `gcloud_identity`, `az_account`, `az_groups` answer "which account am I
+  looking at" but do not sweep region-scoped resources (ec2 describe-*,
+  compute instances list, ...) yet.
+- **Phase 9 tools need their CLIs.** `helm`, `argocd` (plus `argocd
+  login`), `istioctl` and `trivy` are not bundled; without them those
+  tools return the exact "not found" error. New Relic tools need
+  `NEW_RELIC_API_KEY` + `NEW_RELIC_ACCOUNT_ID` in the environment; without
+  them they name the missing variable — nothing is invented.
+- **Trivy's first scan downloads the CVE database** and can take a couple
+  of minutes; subsequent scans are fast.
+- **The model can only select from the registered tools.** It can never run
+  an arbitrary verb of any CLI, or a mutating one — by construction.
+- **Raw CLI output to the model.** Python does not re-parse pod/docs/state;
+  the model interprets real output, truncated at 8,000 characters per result.
+- **Conversation history is short-term only.** The investigation record
+  persists across CLI exits (Phase 7), but chat history still lives in the
+  process and is lost when the CLI exits.
+- **Hypothesis tracking is model-driven.** The record is what the model
+  chose to record through the investigation tools; the live tracker returned
+  on every record call is designed to keep that complete, but it is still
+  the model's discipline.
+- **No streaming, no retries/backoff** yet.
+- **Read-only is enforced by construction today.** Mutating capabilities
+  will only be added behind an explicit human-approval gate, much later.
+
+## 9. Planned future phases
+
+- **Phase 2 — tool architecture. ✅ Done.** Tool contract, registry,
+  tool-use loop, first read-only tool, offline + live verification.
+- **Phase 3 — Kubernetes tooling. ✅ Done.** Read-only `kubectl`-backed
+  tools (pod status/logs, deployment status), name validation, verb
+  allowlist, offline + live verification.
+- **Phase 4 — investigation loop. ✅ Done.** The agent plans a multi-step
+  investigation, tracks hypotheses and evidence in a first-class in-memory
+  record, and produces a structured report (facts → observations →
+  hypotheses → conclusion → remediation → verification).
+- **Phase 5 — multi-domain tooling. ✅ Done.** Linux system (systemd/
+  journal/ports/processes), Docker (ps/inspect/logs/stats/images),
+  Kubernetes depth (events/nodes/services), Terraform (show/state/plan),
+  git/GitHub (status/log/diff/PRs) — all read-only, offline + live verified.
+- **Phase 6 — automation surface. ✅ Done.** One-shot CLI mode
+  (`python main.py [--json] "<problem>"`) with exit codes for cron/CI, and
+  a structured JSON export of the investigation report (`render_report_json`)
+  so pipelines can act on the verdict. Repository is git-tracked with a
+  clean `.gitignore` (secrets and the venv never ride along).
+- **Phase 7 — persistence. ✅ Done.** The investigation record survives CLI
+  exits: auto-saved on every mutation to
+  `~/.devops-ai-agent/investigations/` (`--store-dir`/`AGENT_STORE_DIR` to
+  override), atomic in-place file updates, REPL auto-resume of the newest
+  in-progress record, `/investigations` listing, one-shot `--resume` /
+  `--out`, and lossless `to_dict`/`from_dict` round-tripping — all offline
+  tested.
+- **Phase 8 — tool expansion. ✅ Done.** 21 more read-only tools (26 → 47):
+  Kubernetes depth (`k8s_pods`, `k8s_top_pods`, `k8s_top_nodes`, `k8s_hpa`,
+  `k8s_pvc`, `k8s_contexts`), Docker depth (`docker_networks`,
+  `docker_volumes`, `docker_disk_usage`), GitHub Actions (`gh_runs`,
+  `gh_run_view`, `gh_workflows`), cloud identity (`aws_identity`,
+  `gcloud_identity`, `az_account`, `az_groups`), monitoring/logging
+  (`prom_query`, `loki_query`, `grafana_health` — endpoints from env only),
+  and Ansible listing (`ansible_inventory`, `ansible_playbook_tasks`) —
+  offline tested, live-verified where the host has the CLI.
+- **Phase 9 — trending-market tools. ✅ Done.** 11 more read-only tools
+  (47 → 58): New Relic (`newrelic_nrql`, `newrelic_alerts` — NerdGraph over
+  curl, credentials env-only, GraphQL variables so query text can't escape),
+  Trivy (`trivy_image_scan`), Helm (`helm_list`, `helm_status`,
+  `helm_history` — reads only), Argo CD (`argocd_apps`,
+  `argocd_app_status`), Istio (`istioctl_proxy_status`), Docker Compose
+  (`docker_compose_ls`, `docker_compose_ps` — live-verified on this host).
+- **Later — region-scoped cloud resources** (ec2 describe-*, compute
+  instances list, ...) behind the same template pattern; more observability
+  depth (New Relic dashboards/entities, Prometheus range queries); streaming;
+  conversation-history persistence; and a human-approval gate before any
+  mutating action is ever allowed.