healops 0.1.0__tar.gz

healops-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ali Huzaifa
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

healops-0.1.0/MANIFEST.in

@@ -0,0 +1,4 @@
+include LICENSE README.md
+recursive-include docs *.md
+recursive-include examples *.py *.yaml
+recursive-include tests *.py

healops-0.1.0/PKG-INFO

@@ -0,0 +1,195 @@
+Metadata-Version: 2.4
+Name: healops
+Version: 0.1.0
+Summary: Safe, model-agnostic self-healing for machine-learning applications: detect, diagnose, check against policy, act, verify.
+Author: Ali Huzaifa, Muhammad Rafeh, Sufyan Khan, Maaz Tariq Soomro
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/AliHuzaifa-cs/HealOps
+Project-URL: Issues, https://github.com/AliHuzaifa-cs/HealOps/issues
+Keywords: mlops,self-healing,monitoring,llm,agents,machine-learning,reliability
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Monitoring
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pydantic>=2.7
+Requires-Dist: pyyaml>=6
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# HealOps
+
+**Safe, model-agnostic self-healing for machine-learning applications.**
+
+HealOps watches an ML application, detects when it breaks (a crash, a slow service, a
+bad model, drifting data), asks a diagnosis agent what to do, checks the proposal against
+strict policies, carries it out only if allowed, and verifies that the application really
+recovered. People are asked whenever the policy says so.
+
+> **Status: alpha (0.1.0).** The core loop is built and tested, but it has been exercised
+> on a toy application, not yet on production systems. Read
+> [docs/safety-and-limits.md](docs/safety-and-limits.md) before trusting it with anything
+> that matters.
+
+## The one rule
+
+```
+Monitor -> Detect -> Diagnose -> Plan -> Validate -> Act -> Verify -> Record
+```
+
+| Who | May do | May not do |
+| --- | --- | --- |
+| The **LLM** (or the rule-based agent) | propose a diagnosis and one action | change anything |
+| The **policy engine** | allow, deny, or require a person's approval | invent actions |
+| Your **adapter** | execute an allowed action | decide what to run |
+| The **verifier** | confirm recovery from observed evidence | take the agent's word for it |
+
+The language model is never trusted: its output is validated, it can only choose from
+actions your application really supports, its confidence is capped, and if it is down or
+over budget a deterministic rule-based agent takes over. Everything is written to an
+append-only audit log.
+
+## Install
+
+```
+pip install healops
+```
+
+From a source checkout: `pip install -e .` (add `".[dev]"` for the tests and `ruff`).
+Python 3.11 or newer.
+
+## Five-minute quickstart
+
+HealOps ships a tiny fake model service so you can see the whole loop without any setup.
+
+```python
+from healops import HealOps
+from healops.testing.toy_app import COMPONENT, Fault, ToyApp
+
+config = {
+    "dry_run": False,  # let HealOps really act (the default is a safe dry-run)
+    "audit_log_path": "quickstart_audit.jsonl",
+    "state_db_path": "quickstart_state.db",
+    "check_interval_seconds": 1,
+    "min_confidence_for_auto": 0.3,
+    "thresholds": {"latency_ms": 500, "error_rate": 0.05, "consecutive_breaches": 2},
+    "verification": {"settle_seconds": 0, "checks": 2, "interval_seconds": 0},
+    "policies": {"restart_component": {"verdict": "allow", "cooldown_seconds": 0}},
+}
+
+app = ToyApp()
+healops = HealOps(config)
+healops.register("toy-app", app, component=COMPONENT)
+
+app.inject(Fault.CRASH)
+print("healthy before:", app.get_health())
+healops.run(max_ticks=6)
+print("healthy after: ", app.get_health())
+print("actions taken: ", app.actions)
+```
+
+Expected output:
+
+```
+healthy before: False
+healthy after:  True
+actions taken:  ['restart_component:model-server']
+```
+
+The same file is [examples/quickstart.py](examples/quickstart.py). The run leaves
+`quickstart_state.db` and `quickstart_audit.jsonl` in the current folder (delete them any time).
+
+## Connect your own application
+
+You write a small **adapter**: a class with only the methods your application really has
+(`get_health`, `get_metrics`, `restart_component`, ...). HealOps discovers what it can do and
+never calls anything else. See [docs/adapter-guide.md](docs/adapter-guide.md).
+
+```python
+healops = HealOps.from_config("selfhealing.yaml")
+healops.register("my-model-api", MyAdapter(), component="model-server")
+healops.run()   # until healops.stop() is called
+```
+
+All settings live in one YAML file, with safe defaults:
+[docs/configuration.md](docs/configuration.md) and
+[examples/selfhealing.example.yaml](examples/selfhealing.example.yaml).
+
+## Which language model?
+
+None is required: without an `llm:` section a rule-based agent is used. To use a model, add
+a few lines of configuration; no code changes, and any provider that speaks the OpenAI
+chat format works (OpenAI, Ollama, Gemini's compatibility endpoint, ...) plus Anthropic.
+
+```yaml
+llm:
+  provider: ollama           # ollama | openai | anthropic | openai_compatible
+  model: llama3.2:3b
+  # api_key_env: OPENAI_API_KEY   # the NAME of an environment variable, never the key
+```
+
+API keys are read from environment variables only. A per-hour call budget
+(`max_calls_per_hour`, default 20) bounds cost, and when the model fails or the budget is
+used up, the rule-based agent answers with its confidence capped, so a person decides.
+
+## Approving from another terminal
+
+When the policy needs a person, HealOps parks the incident and alerts you (console, the
+state store and, if configured, email). Answer from any terminal:
+
+```
+healops list
+healops approve 3f9a1c2e --by ali
+healops reject  3f9a1c2e --by ali --reason "not now"
+```
+
+The command only records your answer in the state file; the running HealOps carries it out
+through the same policy, audit and verification path. An approval that expired is never
+applied. (`python -m healops ...` does the same.)
+
+## What is in the box
+
+| Area | What it does |
+| --- | --- |
+| Detection | thresholds that must hold for several checks in a row; model-quality degradation against a baseline |
+| Diagnosis | rule-based agent; LLM agent with strict JSON validation; automatic fallback; call budget |
+| Policy | allow / needs approval / deny per action, retry limits, cooldowns, minimum confidence |
+| Approval | human-in-the-loop queue that survives restarts; silence is never approval |
+| Safety | dry-run by default; time limits on every adapter call; write-ahead audit before real actions |
+| Verification | re-checks health and the incident's own metric after the action |
+| State | SQLite file with observations, incidents, approvals and alerts (for dashboards) |
+| Alerts | console, state store, email with rate limits |
+
+## Documentation
+
+- [Adapter guide](docs/adapter-guide.md): connect your application
+- [Configuration reference](docs/configuration.md): every option and its default
+- [Safety and known limits](docs/safety-and-limits.md): what is guaranteed and what is not
+
+## Authors and acknowledgements
+
+HealOps is the final-year project "Agentic Self-Healing MLOps".
+
+- Ali Huzaifa (maintainer)
+- Muhammad Rafeh
+- Sufyan Khan
+- Maaz Tariq Soomro
+- Supervisor: Sir Abdullah Bin Tahir
+
+Thanks to everyone whose feedback shaped the design.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

healops-0.1.0/README.md

@@ -0,0 +1,164 @@
+# HealOps
+
+**Safe, model-agnostic self-healing for machine-learning applications.**
+
+HealOps watches an ML application, detects when it breaks (a crash, a slow service, a
+bad model, drifting data), asks a diagnosis agent what to do, checks the proposal against
+strict policies, carries it out only if allowed, and verifies that the application really
+recovered. People are asked whenever the policy says so.
+
+> **Status: alpha (0.1.0).** The core loop is built and tested, but it has been exercised
+> on a toy application, not yet on production systems. Read
+> [docs/safety-and-limits.md](docs/safety-and-limits.md) before trusting it with anything
+> that matters.
+
+## The one rule
+
+```
+Monitor -> Detect -> Diagnose -> Plan -> Validate -> Act -> Verify -> Record
+```
+
+| Who | May do | May not do |
+| --- | --- | --- |
+| The **LLM** (or the rule-based agent) | propose a diagnosis and one action | change anything |
+| The **policy engine** | allow, deny, or require a person's approval | invent actions |
+| Your **adapter** | execute an allowed action | decide what to run |
+| The **verifier** | confirm recovery from observed evidence | take the agent's word for it |
+
+The language model is never trusted: its output is validated, it can only choose from
+actions your application really supports, its confidence is capped, and if it is down or
+over budget a deterministic rule-based agent takes over. Everything is written to an
+append-only audit log.
+
+## Install
+
+```
+pip install healops
+```
+
+From a source checkout: `pip install -e .` (add `".[dev]"` for the tests and `ruff`).
+Python 3.11 or newer.
+
+## Five-minute quickstart
+
+HealOps ships a tiny fake model service so you can see the whole loop without any setup.
+
+```python
+from healops import HealOps
+from healops.testing.toy_app import COMPONENT, Fault, ToyApp
+
+config = {
+    "dry_run": False,  # let HealOps really act (the default is a safe dry-run)
+    "audit_log_path": "quickstart_audit.jsonl",
+    "state_db_path": "quickstart_state.db",
+    "check_interval_seconds": 1,
+    "min_confidence_for_auto": 0.3,
+    "thresholds": {"latency_ms": 500, "error_rate": 0.05, "consecutive_breaches": 2},
+    "verification": {"settle_seconds": 0, "checks": 2, "interval_seconds": 0},
+    "policies": {"restart_component": {"verdict": "allow", "cooldown_seconds": 0}},
+}
+
+app = ToyApp()
+healops = HealOps(config)
+healops.register("toy-app", app, component=COMPONENT)
+
+app.inject(Fault.CRASH)
+print("healthy before:", app.get_health())
+healops.run(max_ticks=6)
+print("healthy after: ", app.get_health())
+print("actions taken: ", app.actions)
+```
+
+Expected output:
+
+```
+healthy before: False
+healthy after:  True
+actions taken:  ['restart_component:model-server']
+```
+
+The same file is [examples/quickstart.py](examples/quickstart.py). The run leaves
+`quickstart_state.db` and `quickstart_audit.jsonl` in the current folder (delete them any time).
+
+## Connect your own application
+
+You write a small **adapter**: a class with only the methods your application really has
+(`get_health`, `get_metrics`, `restart_component`, ...). HealOps discovers what it can do and
+never calls anything else. See [docs/adapter-guide.md](docs/adapter-guide.md).
+
+```python
+healops = HealOps.from_config("selfhealing.yaml")
+healops.register("my-model-api", MyAdapter(), component="model-server")
+healops.run()   # until healops.stop() is called
+```
+
+All settings live in one YAML file, with safe defaults:
+[docs/configuration.md](docs/configuration.md) and
+[examples/selfhealing.example.yaml](examples/selfhealing.example.yaml).
+
+## Which language model?
+
+None is required: without an `llm:` section a rule-based agent is used. To use a model, add
+a few lines of configuration; no code changes, and any provider that speaks the OpenAI
+chat format works (OpenAI, Ollama, Gemini's compatibility endpoint, ...) plus Anthropic.
+
+```yaml
+llm:
+  provider: ollama           # ollama | openai | anthropic | openai_compatible
+  model: llama3.2:3b
+  # api_key_env: OPENAI_API_KEY   # the NAME of an environment variable, never the key
+```
+
+API keys are read from environment variables only. A per-hour call budget
+(`max_calls_per_hour`, default 20) bounds cost, and when the model fails or the budget is
+used up, the rule-based agent answers with its confidence capped, so a person decides.
+
+## Approving from another terminal
+
+When the policy needs a person, HealOps parks the incident and alerts you (console, the
+state store and, if configured, email). Answer from any terminal:
+
+```
+healops list
+healops approve 3f9a1c2e --by ali
+healops reject  3f9a1c2e --by ali --reason "not now"
+```
+
+The command only records your answer in the state file; the running HealOps carries it out
+through the same policy, audit and verification path. An approval that expired is never
+applied. (`python -m healops ...` does the same.)
+
+## What is in the box
+
+| Area | What it does |
+| --- | --- |
+| Detection | thresholds that must hold for several checks in a row; model-quality degradation against a baseline |
+| Diagnosis | rule-based agent; LLM agent with strict JSON validation; automatic fallback; call budget |
+| Policy | allow / needs approval / deny per action, retry limits, cooldowns, minimum confidence |
+| Approval | human-in-the-loop queue that survives restarts; silence is never approval |
+| Safety | dry-run by default; time limits on every adapter call; write-ahead audit before real actions |
+| Verification | re-checks health and the incident's own metric after the action |
+| State | SQLite file with observations, incidents, approvals and alerts (for dashboards) |
+| Alerts | console, state store, email with rate limits |
+
+## Documentation
+
+- [Adapter guide](docs/adapter-guide.md): connect your application
+- [Configuration reference](docs/configuration.md): every option and its default
+- [Safety and known limits](docs/safety-and-limits.md): what is guaranteed and what is not
+
+## Authors and acknowledgements
+
+HealOps is the final-year project "Agentic Self-Healing MLOps".
+
+- Ali Huzaifa (maintainer)
+- Muhammad Rafeh
+- Sufyan Khan
+- Maaz Tariq Soomro
+- Supervisor: Sir Abdullah Bin Tahir
+
+Thanks to everyone whose feedback shaped the design.
+
+## License
+
+MIT. See [LICENSE](LICENSE).

healops-0.1.0/docs/adapter-guide.md

@@ -0,0 +1,117 @@
+# Connect your application: the adapter guide
+
+HealOps never touches your application directly. You give it an **adapter**: a plain Python
+object whose methods HealOps may call. Implement only what your application really has;
+HealOps checks which methods exist and offers the agent only the actions that are possible.
+
+## The methods
+
+| Method | Returns | What HealOps does with it | Needed for |
+| --- | --- | --- | --- |
+| `get_health()` | `bool` | `False` (or an error or a timeout) means unhealthy | detecting crashes |
+| `get_metrics()` | `dict[str, float]` | compares named numbers with your thresholds and baselines | slow service, errors, model quality |
+| `get_drift_scores()` | `dict[str, float]` | compares drift scores with `thresholds.drift_score` | data drift |
+| `restart_component(component)` | `str` | restarts one named component | action `restart_component` |
+| `reload_model()` | `str` | reloads the current model artifact | action `reload_model` |
+| `rollback_model()` | `str` | returns to a known-good model version | action `rollback_model` |
+| `request_retraining()` | `str` | starts a retraining run (does **not** deploy the result) | action `request_retraining` |
+| `request_redeployment()` | `str` | redeploys (decides what is live) | action `request_redeployment` |
+
+Action methods return a short message describing what happened (it goes in the audit log).
+If something fails, **raise an exception**: HealOps treats it as a failed action, does not
+retry it by itself and asks a person.
+
+## Rules your adapter must follow
+
+1. **Return real numbers.** Metrics and drift scores must be finite numbers (`bool`, `NaN`
+   and infinity are ignored). Never invent a value when you cannot read one: raise instead.
+2. **Use the metric names the config watches.** `latency_ms` and `error_rate` are matched by
+   the `thresholds` section; model-quality metrics are matched by the names you list under
+   `metric_degradation` (for example `f1_score`). If a watched name is never reported,
+   HealOps warns once after `warn_unreported_after_checks` checks.
+3. **Be safe to call from another thread.** Every call runs in a helper thread with a time
+   limit (`adapter_timeouts`). Do not share objects that must stay on one thread (for
+   example a SQLite connection created in the main thread).
+4. **Return quickly or raise.** A read that takes longer than `read_seconds` counts as a
+   failed reading; an action that takes longer than `action_seconds` is escalated to a
+   person and **not** retried, because HealOps cannot know whether it finished.
+5. **Make actions safe to repeat.** A restart that happens twice should do no harm.
+6. **Keep secrets out of messages.** What you return and raise is written to logs (HealOps
+   hides well-known secret shapes, but that is a safety net, not a guarantee).
+
+## Example
+
+A complete, runnable adapter for an in-memory service (swap the bodies for calls to your
+real service):
+
+```python
+# run-in-tests
+from healops import HealOps
+
+
+class MyServiceAdapter:
+    """Pretends to be a model API. Replace the bodies with real calls."""
+
+    def __init__(self):
+        self.up = False
+        self.restarts = 0
+
+    def get_health(self) -> bool:
+        return self.up
+
+    def get_metrics(self) -> dict[str, float]:
+        return {"latency_ms": 120.0, "error_rate": 0.0}
+
+    def restart_component(self, component: str) -> str:
+        self.restarts += 1
+        self.up = True
+        return f"restarted {component}"
+
+
+config = {
+    "dry_run": False,
+    "audit_log_path": None,   # keep the audit log in memory for this example
+    "state_db_path": None,    # and the state too
+    "min_confidence_for_auto": 0.3,
+    "thresholds": {"latency_ms": 500, "consecutive_breaches": 2},
+    "verification": {"settle_seconds": 0, "checks": 2, "interval_seconds": 0},
+    "policies": {"restart_component": {"verdict": "allow", "cooldown_seconds": 0}},
+}
+
+adapter = MyServiceAdapter()
+healops = HealOps(config)
+healops.register("my-service", adapter, component="api-server")
+for _ in range(5):
+    healops.tick()
+assert adapter.up and adapter.restarts == 1
+print("recovered after", adapter.restarts, "restart")
+```
+
+`register(name, adapter, component=...)` needs `component` whenever the adapter can restart
+components: it is the name passed to `restart_component`. Several applications can be
+registered under different names; each keeps its own component.
+
+## Running it
+
+```python
+healops.run()          # monitor until healops.stop() is called (for example from a signal handler)
+healops.run(max_ticks=10)   # or a fixed number of checks (handy in tests)
+healops.tick()         # a single check
+```
+
+Useful calls while it runs: `healops.pending_approvals()`, `healops.incidents()`,
+`healops.approve(...)`, `healops.reject(...)`, `healops.acknowledge(incident_id)` and
+`healops.stuck_calls()`. People can also answer from another terminal with the `healops`
+command (see the README).
+
+## Start in dry-run
+
+`dry_run` is **on by default**: proposed actions are logged and alerted, never executed.
+Run like that first, read the audit log (`healops_audit.jsonl`), and only then set
+`dry_run: false` and give the actions you trust the verdict `allow`.
+
+## Turning off the time limits
+
+`healops.register(name, adapter, component=..., guard=False)` calls the adapter directly in
+the monitoring thread. Only do this if the adapter cannot be called from another thread, and
+accept that a hanging call then hangs the loop.

healops-0.1.0/docs/configuration.md

@@ -0,0 +1,130 @@
+# Configuration reference
+
+HealOps reads one YAML file (`HealOps.from_config("selfhealing.yaml")`) or a Python
+dictionary (`HealOps({...})`). Unknown keys are **rejected**, so a typo cannot silently
+switch a check off. Every option has a safe default. A commented, ready-to-edit file is
+[examples/selfhealing.example.yaml](../examples/selfhealing.example.yaml).
+
+Secrets never go in this file: where a key or password is needed you write the **name of an
+environment variable** (`api_key_env`, `password_env`) and set the variable in your shell.
+
+## Top level
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `dry_run` | `true` | Log and alert proposed actions but never execute them. Turn off only after reading the audit log. |
+| `check_interval_seconds` | `30` | Seconds between checks. |
+| `min_confidence_for_auto` | `0.6` | An action the policy would allow automatically is downgraded to "needs approval" if the diagnosis confidence is below this (0 to 1). The gate only ever makes things stricter. |
+| `approval_timeout_seconds` | `3600` | How long a person has to answer. After that the incident is escalated; silence is never approval. |
+| `audit_log_path` | `healops_audit.jsonl` | Append-only record of every step. `null` keeps it in memory. |
+| `state_db_path` | `healops_state.db` | SQLite file with observations, incidents, approvals and alerts. `null` keeps it in memory (nothing survives a restart). |
+| `state_retention_days` | `30` | Observations older than this are deleted. Incidents are kept. |
+| `max_recovery_attempts` | `3` | Recovery actions tried for one incident (1 to 10) before a person takes over. |
+| `warn_unreported_after_checks` | `5` | Warn once about anything the config watches that the application never reported. `0` turns the warning off. |
+
+## `thresholds`
+
+A threshold left out is simply not monitored. A breach must repeat for
+`consecutive_breaches` checks in a row before an incident is raised, so one odd reading does
+not trigger anything.
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `latency_ms` | none | Incident `high_latency` when the metric `latency_ms` is above this. |
+| `error_rate` | none | Incident `high_error_rate` when the metric `error_rate` is above this (0 to 1). |
+| `drift_score` | none | Incident `data_drift` when any drift score is above this. |
+| `consecutive_breaches` | `3` | Breaches in a row needed (1 to 20). |
+
+A failing health check always raises a `health_failure` incident.
+
+## `metric_degradation`
+
+Model-quality metrics your application reports through `get_metrics()`, for example:
+
+```yaml
+metric_degradation:
+  f1_score:
+    baseline: 0.90      # the normal level
+    max_change: 0.10    # how far it may move in the bad direction
+    higher_is_better: true
+```
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `baseline` | required | Normal value of the metric. |
+| `max_change` | required | Allowed movement in the bad direction (greater than 0). |
+| `higher_is_better` | `true` | `false` for metrics where lower is better (such as a loss). |
+
+## `verification`
+
+After an action HealOps watches the application before calling it recovered.
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `settle_seconds` | `10` | Wait before the first check. |
+| `checks` | `3` | Number of checks (1 to 20). All must pass for "verified". |
+| `interval_seconds` | `10` | Pause between checks. |
+| `max_window_seconds` | `300` | The whole wait must fit inside this. |
+
+## `default_policy` and `policies`
+
+`policies` maps an action name to its rules; `default_policy` applies to every action not
+listed. Actions: `restart_component`, `reload_model`, `rollback_model`, `request_retraining`,
+`request_redeployment` (`run_diagnostic`, `escalate` and `no_action` are handled by HealOps).
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `verdict` | `needs_approval` | `allow` (run without asking), `needs_approval` (a person decides) or `deny` (never). |
+| `max_retries` | `1` | Retries after the first run, per incident. |
+| `cooldown_seconds` | `300` | Minimum time between runs of the same action on the same application. |
+
+## `adapter_timeouts`
+
+Every call into your adapter has a time limit. See the
+[adapter guide](adapter-guide.md) for what happens on a timeout.
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `read_seconds` | `10` | Limit for `get_health`, `get_metrics`, `get_drift_scores`. |
+| `action_seconds` | `120` | Limit for restart, reload, rollback and retraining calls. |
+| `max_stuck_calls` | `3` | After this many calls to one application are stuck, no new call is made until one finishes. |
+
+## `llm` (optional)
+
+Leave the section out to use the rule-based agent only.
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `provider` | required | `ollama`, `openai`, `anthropic` or `openai_compatible`. |
+| `model` | required | Model name as the provider knows it. |
+| `base_url` | provider default | Required for `openai_compatible`. Ollama defaults to `http://localhost:11434`. |
+| `api_key_env` | none | **Name** of the environment variable holding the key. Not needed for local Ollama. |
+| `chat_path` | provider default | Path of the chat endpoint, for services with a different one (must start with `/`). |
+| `json_mode` | `true` | Ask for a JSON reply; set `false` for services that reject it. |
+| `timeout_seconds` | provider default | Per-call time limit. |
+| `max_tokens` | `800` | Reply size limit. |
+| `max_confidence` | `0.9` | Cap on the confidence the model may claim. |
+| `max_calls_per_hour` | `20` | Budget for model calls. `null` removes the limit. |
+| `use_fallback` | `true` | When the model fails or the budget is used up, the rule-based agent answers instead. |
+| `fallback_confidence_cap` | `0.2` | Confidence cap for those fallback answers, so a person decides. |
+
+## `email` (optional)
+
+Alerts by SMTP when an action needs approval or an incident needs a person. The email only
+informs; it contains the command to run, never a link. Held-back alerts are summarised in the
+next email and always kept in the state store.
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `host` | required | SMTP server. |
+| `port` | `587` | SMTP port. |
+| `security` | `starttls` | `starttls`, `ssl` or `none` (`none` only for localhost). |
+| `username` | none | Login name; needs `password_env`. |
+| `password_env` | none | **Name** of the environment variable holding the password. |
+| `sender` | required | From address. |
+| `recipients` | required | One to twenty addresses. |
+| `timeout_seconds` | `15` | SMTP connection limit. |
+| `max_emails_per_hour` | `6` | Hourly limit. |
+| `cooldown_minutes` | `15` | One email per application and incident type in this time. |
+| `email_in_dry_run` | `false` | Email alerts that come from dry-run mode too. |
+| `subject_prefix` | `[HealOps]` | Start of every subject line. |