@aion0/forge 0.10.23 → 0.10.26

package/docs/tp-automation-api-v2.md

@@ -0,0 +1,482 @@
+# TP Automation API — 3-Step Workflow for Dev/QA
+
+Minimum HTTP contract for driving a `build → upgrade → run tests →
+collect results` cycle against an automation testbed on TP. All
+endpoints are stateless except for the durable `PytestExecution` /
+`SingleDevice` rows the celery workers update — callers just POST then
+poll, no websockets/sse/callbacks.
+
+```
+<TP-base-url>/<endpoint>
+```
+
+`<TP-base-url>`:
+- Production: `https://nac-tp.fortinet-us.com`
+- Test: `http://10.15.33.25:8000`
+- Dev (.11): `http://10.15.33.11:8000`
+
+## Authentication
+
+Every endpoint requires a JWT in the `Authorization` header:
+
+```
+Authorization: JWT <token>
+```
+
+Mint a token:
+
+```bash
+T=$(curl -s -X POST <TP-base-url>/token-auth/ \
+       -H 'Content-Type: application/json' \
+       --data-binary @- <<JSON | jq -r .token
+{"username":"<user>","password":"<pw>"}
+JSON
+)
+```
+
+In examples below `$T` stands for the JWT.
+
+> **Note on prod:** `/token-auth/` is **not** exposed at the public
+> reverse proxy on prod (SSO-only). Automation scripts on prod-facing
+> endpoints need to be run from a host that can reach the internal
+> port, or they need an SSO-issued token. On the dev `.11` server,
+> `/token-auth/` works directly with username + password.
+
+---
+
+## Step 1 — Upgrade the testbed
+
+```
+POST /adc/automation/upgrade/                — kick off (HTTP 202)
+GET  /adc/automation/upgrade/<testbed>/      — poll status
+```
+
+The dev team has three modes available; pick whichever matches your
+build pipeline:
+
+| `mode`     | When to use it | Required extra fields |
+|---|---|---|
+| `command`  | You have a Jenkins-built image and Jenkins already produces the full `execute restore image scp ...` command string. | `command` |
+| `build`    | You have a build number from the official build server. | `build_number` |
+| `ga`       | You want the latest GA build for a specific FortiNAC version. The system resolves the build number via `BuildHistory.GA=True`. | `version` |
+
+### Request
+
+Content-Type: `application/json`.
+
+```json
+{
+  "testbed": "AT16_Combined_FSW",
+  "mode":    "command",
+  "command": "execute restore image scp /var/lib/jenkins/jobs/fortinac-build-7.6-1/builds/3983/archive/nacos/FNAC_ESX-v7-build6956-FORTINET_job1_3983.out 10.15.33.5 jenkins fortinet"
+}
+```
+
+```json
+{
+  "testbed": "AT16_Combined_FSW",
+  "mode":    "build",
+  "build_number": "0815"
+}
+```
+
+```json
+{
+  "testbed": "AT16_Combined_FSW",
+  "mode":    "ga",
+  "version": "7.6.5"
+}
+```
+
+Internal behavior:
+
+1. Looks up the testbed's `deployinfo`, extracts every device whose
+   name contains `nac` or `ncm`. A multi-NAC testbed gets one celery
+   task per device, running in parallel.
+2. `mode=ga` resolves `version` → `build_number` via `BuildHistory`,
+   then dispatches the same path as `mode=build`.
+3. Concurrency guard: HTTP **409** if any target IP is already
+   `PROGRESS`.
+4. `mode=command` validates the command starts with
+   `execute restore image` (security guard — keeps arbitrary CLI from
+   riding through this endpoint).
+
+### Response (HTTP 202)
+
+```json
+{
+  "testbed": "AT16_Combined_FSW",
+  "mode": "command",
+  "build_number": "",
+  "target_ips": ["10.15.52.152"],
+  "tasks": [
+    {"ip": "10.15.52.152", "celery_id": "f62a4cb7-d435-41de-adb6-12cca405d507"}
+  ]
+}
+```
+
+Returns in <500 ms. The actual upgrade work runs asynchronously in
+celery workers — the HTTP call returns the moment the task is queued.
+
+### Poll for status
+
+```
+GET /adc/automation/upgrade/<testbed>/
+```
+
+```json
+{
+  "testbed": "AT16_Combined_FSW",
+  "status":  "PROGRESS",
+  "target_ips": ["10.15.52.152"],
+  "per_device": [
+    {
+      "ip": "10.15.52.152",
+      "status": "PROGRESS",
+      "last_task_type": "command",
+      "last_build_number": null,
+      "updated_at": "2026-05-28T18:14:19+00:00",
+      "log_tail": "...last 2 KB of the NAC's SSH output..."
+    }
+  ]
+}
+```
+
+Aggregate `status` values:
+
+| Value | Meaning |
+|---|---|
+| `UNKNOWN`  | No upgrade has been dispatched against this testbed (yet). |
+| `PROGRESS` | At least one device is still mid-upgrade. |
+| `SUCCESS`  | Every device finished with `SUCCESS`. |
+| `FAILURE`  | At least one device finished with `FAILURE` / `TIMEOUT` (and no device is still in flight). |
+
+`per_device[].log_tail` is the last ~2 KB of the device's SSH session
+output, updated every ~3 s by the worker. Useful for live debugging
+without flooding the response.
+
+### Concrete polling loop
+
+```bash
+while :; do
+  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/upgrade/AT16_Combined_FSW/)
+  STATUS=$(echo "$R" | jq -r .status)
+  echo "$(date +%H:%M:%S) $STATUS"
+  case "$STATUS" in
+    PROGRESS) sleep 15 ;;
+    SUCCESS)  echo "upgrade complete"; break ;;
+    FAILURE)  echo "upgrade FAILED"; echo "$R" | jq .per_device; exit 1 ;;
+    *)        echo "$R"; exit 1 ;;
+  esac
+done
+```
+
+A real upgrade takes 2-15 minutes depending on the image and how many
+devices the testbed has.
+
+---
+
+## Step 2 — Run pytest cases on that testbed
+
+```
+POST /adc/automation/pytest/             — kick off (HTTP 202)
+GET  /adc/automation/pytest/<exec_id>/   — poll status + results
+```
+
+### Request
+
+Content-Type: `application/json`.
+
+```json
+{
+  "user":     "<TP username>",
+  "lab":      "AT16_Combined_FSW",
+  "testcase": [
+    "/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"
+  ],
+  "argument": "-vv --tb=short",
+  "extra_pytest_options": ""
+}
+```
+
+| Field | Type | Notes |
+|---|---|---|
+| `user`     | string | TP username of the caller. Must own the lab (be in `users` or `usedby` on the `AutomationTBUser` row). |
+| `lab`      | string | AT lab name. Same `<testbed>` value as Step 1. |
+| `testcase` | **list of strings** | Each entry is a pytest test-id path **relative to the tests repo root** (`/root/fnac_auto/tests` on the controller). Leading slash is OK; both `/Tests_CLI/...` and `Tests_CLI/...` work. The handler iterates the list, joins with spaces. |
+| `argument` | string | Raw pytest CLI args injected between the testcase paths and the framework's `--html`/`--rack-file` flags. `-k`, `-m`, `-vv`, `--tb=short`, `--maxfail=N`, etc. |
+| `extra_pytest_options` | string | (optional) Extra options appended *after* `--html`/`--rack-file` on the real run. Use for env-specific flags that shouldn't apply to the `--collect-only` dry run. |
+
+### Response (HTTP 202)
+
+```json
+{
+  "exec_id": 446,
+  "status":  "Initiating",
+  "lab":     "AT16_Combined_FSW",
+  "testcase_count": 1
+}
+```
+
+Returns in ~300 ms. Save `exec_id` — it's the handle for Step 3.
+
+### What runs on the controller
+
+The celery worker SFTP-uploads this script to the controller VM and
+launches it under `nohup setsid`:
+
+```bash
+#!/bin/bash
+set +e
+cd /root/fnac_auto/tests          && git pull origin main || true
+cd /root/fnac_auto/test-framework && git pull origin main || true
+cd /root/fnac_auto
+source venv/bin/activate
+export PYTHONPATH=/root/fnac_auto/test-framework:/root/fnac_auto/tests
+export DISPLAY=:99
+pytest <expanded testcase paths> <argument> --collect-only
+pytest <expanded testcase paths> <argument> --html=<report> --rack-file <rack> <extra_pytest_options>
+echo $? > /tmp/pytest_<exec_id>.exit
+```
+
+Key properties:
+
+- **Pytest is fully detached** from the launching ssh session via
+  `nohup setsid`. The ssh session can drop, the network can blip, the
+  TP backend can restart — pytest keeps running on the controller.
+- **TP polls via short, separate ssh round-trips** to read the log
+  file and check for the terminal `/tmp/pytest_<exec_id>.exit` marker.
+- **Surviving a TP backend restart** is automatic — the celery worker
+  is a separate process and just keeps polling. Verified end-to-end on
+  `.11` (exec_id 447: killed runserver mid-run, exec completed
+  normally and report was fetched back).
+
+### Concurrency guard
+
+If another `PytestExecution` on the same lab has `status` =
+`Running` or `Initiating`, the POST returns HTTP **409**:
+
+```json
+{"error": "another execution is already in flight on 'AT16_Combined_FSW'"}
+```
+
+---
+
+## Step 3 — Read per-test results
+
+```
+GET /adc/automation/pytest/<exec_id>/
+```
+
+Same endpoint as the Step-2 poll — there's nothing distinct to call.
+
+### Response
+
+```json
+{
+  "exec_id":          446,
+  "status":           "Done",
+  "lab":              "AT16_Combined_FSW",
+  "controller":       "10.15.52.159",
+  "remote_pid":       "177854",
+  "pass_count":       1,
+  "fail_count":       0,
+  "skip_count":       0,
+  "error_count":      0,
+  "total_count":      1,
+  "start_timestamp":  "1780090011",
+  "end_timestamp":    "1780090041",
+  "report_file_path": "data/AutomationTest/test_446_report_2026-05-29_21:26:50.html",
+  "log_file_path":    "/tmp/pytest_446.log",
+  "log_size":         25338,
+  "log_tail":         "...last 4 KB of pytest stdout..."
+}
+```
+
+| Field | Meaning |
+|---|---|
+| `status` | One of `Initiating`, `Running`, `Done`, `Failed`, `Cancelled`. **Treat `Done` as terminal — success/failure is inferred from counts.** |
+| `pass_count`, `fail_count`, `skip_count`, `error_count`, `total_count` | Parsed from pytest's own summary line. Populated **mid-run** (updated every 15 s), so you can see partial progress without waiting for the run to finish. |
+| `start_timestamp` / `end_timestamp` | Unix epoch seconds, written by the worker. |
+| `report_file_path` | Server-relative path to the pytest-html report. Fetch via `<TP>/<report_file_path>` with the JWT for per-test rows + tracebacks. |
+| `log_file_path` | On-controller path to the live log. Mostly informational — use `log_tail` to read the last 4 KB without an extra round-trip. |
+| `log_tail` | Last 4 KB of pytest stdout. |
+| `remote_pid` | The detached pytest tree's pid on the controller. Useful if you ever need to ssh to the controller and kill it manually. |
+
+### Concrete polling loop
+
+```bash
+EXEC=$(curl -sX POST -H "Authorization: JWT $T" \
+     -H 'Content-Type: application/json' \
+     -d '{"user":"alice","lab":"AT16_Combined_FSW","testcase":["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"],"argument":"-vv"}' \
+     $TP/adc/automation/pytest/ | jq -r .exec_id)
+echo "started exec $EXEC"
+
+while :; do
+  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/pytest/$EXEC/)
+  STATUS=$(echo "$R" | jq -r .status)
+  COUNTS=$(echo "$R" | jq -r '"P\(.pass_count)/F\(.fail_count)/S\(.skip_count)/E\(.error_count) of T\(.total_count)"')
+  echo "$(date +%H:%M:%S) $STATUS $COUNTS"
+  case "$STATUS" in
+    Initiating|Running) sleep 15 ;;
+    Done)
+      FAIL=$(echo "$R" | jq -r .fail_count)
+      ERR=$(echo "$R"  | jq -r .error_count)
+      if [ "$FAIL" -gt 0 ] || [ "$ERR" -gt 0 ]; then
+        echo "tests had failures — fetch report at $TP/$(echo "$R" | jq -r .report_file_path)"
+        exit 1
+      fi
+      echo "all pass"; break ;;
+    Failed|Cancelled)
+      echo "execution terminated: $STATUS"
+      echo "$R" | jq -r .log_tail; exit 1 ;;
+  esac
+done
+```
+
+---
+
+## End-to-end script
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+TP=http://10.15.33.11:8000
+USER=admin
+TESTBED=AT16_Combined_FSW
+
+T=$(curl -s -X POST $TP/token-auth/ -H 'Content-Type: application/json' \
+       --data-binary @- <<JSON | jq -r .token
+{"username":"$USER","password":"<your-pw>"}
+JSON
+)
+
+# ---- Step 1: upgrade testbed ----
+COMMAND='execute restore image scp /var/lib/jenkins/jobs/fortinac-build-7.6-1/builds/3983/archive/nacos/FNAC_ESX-v7-build6956-FORTINET_job1_3983.out 10.15.33.5 jenkins fortinet'
+
+curl -s -X POST -H "Authorization: JWT $T" \
+     -H 'Content-Type: application/json' \
+     -d "$(jq -n --arg cmd "$COMMAND" --arg tb "$TESTBED" \
+            '{testbed:$tb, mode:"command", command:$cmd}')" \
+     $TP/adc/automation/upgrade/ > /dev/null
+
+while :; do
+  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/upgrade/$TESTBED/)
+  S=$(echo "$R" | jq -r .status)
+  echo "upgrade: $S"
+  [ "$S" = "SUCCESS" ] && break
+  [ "$S" = "FAILURE" ] && { echo "$R" | jq .per_device; exit 1; }
+  sleep 30
+done
+
+# ---- Step 2 + 3: run tests, collect results ----
+EXEC=$(curl -s -X POST -H "Authorization: JWT $T" \
+     -H 'Content-Type: application/json' \
+     -d "$(jq -n --arg tb "$TESTBED" --arg user "$USER" \
+            '{user:$user, lab:$tb,
+              testcase:["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"],
+              argument:"-vv --tb=short"}')" \
+     $TP/adc/automation/pytest/ | jq -r .exec_id)
+echo "started exec $EXEC"
+
+while :; do
+  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/pytest/$EXEC/)
+  S=$(echo "$R" | jq -r .status)
+  echo "pytest: $S $(echo "$R" | jq -r '"P\(.pass_count)/F\(.fail_count) of T\(.total_count)"')"
+  case "$S" in
+    Done)
+      F=$(echo "$R" | jq -r .fail_count)
+      E=$(echo "$R" | jq -r .error_count)
+      [ "$F" -gt 0 ] || [ "$E" -gt 0 ] && {
+        echo "report: $TP/$(echo "$R" | jq -r .report_file_path)"
+        exit 1
+      }
+      echo "PASS"; break ;;
+    Failed|Cancelled)
+      echo "exec terminated: $S"; echo "$R" | jq -r .log_tail; exit 1 ;;
+    *) sleep 15 ;;
+  esac
+done
+```
+
+---
+
+## Behaviors worth knowing
+
+### Concurrency
+
+For each `<testbed>`:
+- **Upgrade**: HTTP 409 if any target IP currently has
+  `SingleDevice.last_task_status='PROGRESS'`.
+- **Pytest**: HTTP 409 if any `PytestExecution` with this `lab` has
+  `status` in `{Running, Initiating}`.
+
+You can have an upgrade AND a pytest running simultaneously on
+*different* testbeds. The guards are per-testbed.
+
+### Survival semantics
+
+| Failure | Upgrade | Pytest v2 |
+|---|---|---|
+| ssh session drops mid-run | N/A — TP only ssh's briefly | Pytest keeps running (`nohup setsid` on controller) |
+| TP backend restart mid-run | Celery worker keeps going, completes the run, writes terminal state | Same — celery `pytest_poll` keeps short-polling the controller |
+| Controller VM reboot mid-run | Upgrade reboots the NAC; this is expected/desired. | Pytest is lost (controller died); polling will eventually fail with the max-runtime safety net (6 h cap). |
+| Celery worker restart | In-flight task may be lost (celery uses ack_late=false by default). Worth a follow-up. | Same. |
+
+### Per-test pass/fail visibility
+
+The `passed_nodes` / `failed_nodes` fields on the legacy `pytest_run`
+response shape don't apply to v2. v2 surfaces:
+
+- **aggregate counts** (`pass_count`, `fail_count`, etc.) in the GET
+  response — populated mid-run as pytest emits them
+- **the HTML report** at `report_file_path` — the standard
+  pytest-html document with per-test rows, durations, tracebacks, and
+  captured logs. Fetch via `<TP>/<report_file_path>` with the JWT.
+
+If you need per-test JSON instead of the HTML report, ask — it's a
+trivial extra endpoint that parses the HTML and returns
+`{passed: [...], failed: [...]}`.
+
+### Test-case path format
+
+Whatever you'd type after `pytest` on the command line, prefixed (or
+not) with `/`. Relative to `/root/fnac_auto/tests/` on the controller.
+
+| Want to | Use |
+|---|---|
+| Run one test | `["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"]` |
+| Run several | `["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status", "/Tests_CLI/test_cli_sanity.py::TestShowFullConfig::test_nacos_show_config"]` |
+| Run a whole file | `["/Tests_CLI/test_cli_sanity.py"]` |
+| Run by marker | `["/Tests_CLI"]` + `argument: "-m smoke"` |
+| Dry-run / list | `["/Tests_CLI"]` + `argument: "--collect-only"` |
+
+Tip: discover paths via `GET /adc/get_testcases` (legacy endpoint;
+returns the full test tree). Note: this endpoint does a `git pull` on
+TP's local mirror of the tests repo as a side effect — fine in normal
+use, just be aware.
+
+---
+
+## Source files
+
+| Concern | File |
+|---|---|
+| URL routes | `backend/adc/urls.py` |
+| Upgrade API | `backend/adc/views/automation/upgrade_api.py` |
+| Upgrade tasks | `backend/adc/tasks.py` (`upgrade_nac_command`, `upgrade_nac_build`) |
+| Pytest v2 API | `backend/adc/views/automation/pytest_api.py` |
+| Pytest v2 tasks | `backend/adc/tasks.py` (`pytest_launch`, `pytest_poll`) |
+| Per-device state | `backend/adc/models.py` (`SingleDevice`, `PytestExecution`) |
+| GA-build resolution | `backend/adc/models.py` (`BuildHistory.GA=True`) |
+
+Legacy endpoints (still live, used by the `/automation` UI page, kept
+for backward compatibility):
+
+- `POST /adc/nac-upgrade-testbed/`
+- `POST /adc/pytest_run`
+- `POST /adc/get_test_execution_by_id`
+
+Avoid these for new automation — they're synchronous-in-the-request,
+don't survive TP restarts, and the upgrade variant blocks the HTTP
+request for the full restore duration.