dsh-test-drive 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/README.es.md +18 -3
- package/README.hi.md +18 -3
- package/README.md +19 -4
- package/README.pt.md +18 -3
- package/README.zh.md +18 -3
- package/cordis.patch.yml +14 -0
- package/lib/index.js +431 -10
- package/lib/types/capability.d.ts +74 -0
- package/lib/types/capability.d.ts.map +1 -0
- package/lib/types/config.d.ts +33 -0
- package/lib/types/config.d.ts.map +1 -1
- package/lib/types/domain.d.ts +19 -7
- package/lib/types/domain.d.ts.map +1 -1
- package/lib/types/drive.d.ts +5 -0
- package/lib/types/drive.d.ts.map +1 -1
- package/lib/types/driver.d.ts +11 -0
- package/lib/types/driver.d.ts.map +1 -1
- package/lib/types/report.d.ts +2 -2
- package/lib/types/report.d.ts.map +1 -1
- package/lib/types/result.d.ts +31 -1
- package/lib/types/result.d.ts.map +1 -1
- package/lib/types/tools.d.ts.map +1 -1
- package/lib/types/version.d.ts +1 -1
- package/package.json +4 -6
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,12 @@ All notable changes to this project are documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [0.2.0] - 2026-08-18
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
|
|
12
|
+
- Capability-assertion stage: after the boot smoke, the pipeline can drive one headless task that calls a named tool (or runs a `/command`) and verify the durable session log recorded the invocation and that the observed output contains `expect` (case-insensitive substring of the serialized event). Status ladder `observed` → `invoked` → `not-registered`, plus `skipped` (disabled or no `DEEPSEEK_API_KEY`) and `failed` (task failure); `not-registered`/`failed` fail the drive verdict. Config block `capability.*` + `capabilityTimeoutMs`, per-drive `capability` tool argument, and the stage record is an OPTIONAL `stages.capability` field (backward-compatible: pre-capability v1 records still validate, no domain-version bump).
|
|
13
|
+
|
|
8
14
|
## [0.1.0] - 2026-08-16
|
|
9
15
|
|
|
10
16
|
### Added
|
package/README.es.md
CHANGED
|
@@ -32,9 +32,10 @@
|
|
|
32
32
|
|
|
33
33
|
## What you get (Qué obtienes)
|
|
34
34
|
|
|
35
|
-
- Herramienta `test_drive` — un objetivo por el flujo completo: `dsh plugin add` → verificación del parche con `--dump-config` → arranque headless (escaneo de marcadores FAILED + tarea opcional de una frase) → `dsh plugin remove` → limpieza en cuarentena. Devuelve el registro estructurado de forma síncrona, o `{ kind: 'background', jobId }` con `background: true`.
|
|
35
|
+
- Herramienta `test_drive` — un objetivo por el flujo completo: `dsh plugin add` → verificación del parche con `--dump-config` → arranque headless (escaneo de marcadores FAILED + tarea opcional de una frase) → aserción de capacidad opcional → `dsh plugin remove` → limpieza en cuarentena. Devuelve el registro estructurado de forma síncrona, o `{ kind: 'background', jobId }` con `background: true`.
|
|
36
36
|
- Comando `/testdrive` — lote de objetivos separados por espacios/comas como tarea en segundo plano `drive-batch` sobre `ctx.jobs`, produciendo un informe matricial (JSON + Markdown).
|
|
37
37
|
- Herramienta `drive_report` — recupera cualquier ejecución (`tdr_...`), matriz (`tdm_...`) o la última matriz; se renderiza en Markdown.
|
|
38
|
+
- Aserción de capacidad — más allá de “arrancó y salió”: la etapa opcional `capability` hace que el agente llame a la herramienta nombrada (o ejecute `/command`) y verifica que el registro durable de sesión contenga la invocación y que la salida observada contenga `expect`. Un arranque limpio es solo una prueba de humo; `observed` demuestra que una capacidad con nombre realmente funciona.
|
|
38
39
|
- Resultados estructurados — cada registro lleva el discriminador `schema: "dsh-test-drive/v1"` con campos de primera clase: `stages.install.status` (`pass`/`fail`), `stages.smoke.status` (`pass`/`fail`/`boot-ok`/`skipped`), `durationMs` por etapa, `summary`/`outputTail` saneados y un `verdict` global (`pass`/`fail`/`partial`/`unknown`). Este es el contrato legible por máquina que consumen los puntuadores (dsh-score).
|
|
39
40
|
- Seguridad por construcción — cada directorio temporal lo crea este complemento bajo un prefijo dedicado `dsh-test-drive-`, se registra en un registro de propiedad activo y solo se elimina mediante la escalera dry-run → renombrado a cuarentena → borrado. El perfil del host nunca se lee ni se escribe.
|
|
40
41
|
|
|
@@ -82,6 +83,12 @@ Todas las claves son opcionales (se muestran los valores por defecto); los valor
|
|
|
82
83
|
| `installTimeoutMs` | `600000` | Plazo de la etapa `dsh plugin add`. |
|
|
83
84
|
| `configTimeoutMs` | `60000` | Plazo de la etapa `--dump-config`. |
|
|
84
85
|
| `smokeTimeoutMs` | `300000` | Plazo de la etapa de arranque headless. |
|
|
86
|
+
| `capabilityTimeoutMs` | `300000` | Plazo de la tarea de aserción de capacidad.
|
|
87
|
+
| `capability.enabled` | `false` | Ejecuta la etapa de aserción de capacidad (registrado → invocado → observado).
|
|
88
|
+
| `capability.kind` | `tool` | Qué afirmar: `tool` o `command`.
|
|
89
|
+
| `capability.name` | `""` | Nombre de herramienta o comando (sin la `/` inicial).
|
|
90
|
+
| `capability.args` | `""` | Texto de invocación: argumentos de herramienta (estilo JSON) o palabras del comando.
|
|
91
|
+
| `capability.expect` | `""` | Literal esperado en la salida observada (subcadena sin distinción de mayúsculas).
|
|
85
92
|
| `uninstallTimeoutMs` | `120000` | Plazo de la etapa `dsh plugin remove`. |
|
|
86
93
|
| `outputTailBytes` | `8000` | Límite de la cola de salida saneada registrada por etapa. |
|
|
87
94
|
| `keepTempDirs` | `false` | Conserva los directorios temporales ante fallos para análisis forense (se abandona la propiedad; tú limpias). |
|
|
@@ -93,10 +100,13 @@ Todas las claves son opcionales (se muestran los valores por defecto); los valor
|
|
|
93
100
|
### `test_drive`
|
|
94
101
|
|
|
95
102
|
```
|
|
96
|
-
test_drive(target: string, headlessTask?: string, background?: boolean
|
|
103
|
+
test_drive(target: string, headlessTask?: string, background?: boolean,
|
|
104
|
+
capability?: { kind: 'tool' | 'command', name: string,
|
|
105
|
+
args: string, expect: string })
|
|
97
106
|
```
|
|
98
107
|
|
|
99
108
|
- `target` — especificación git (`github:owner/repo#sha`, `git+https://...`), nombre npm, ruta local o tarball `.tgz`.
|
|
109
|
+
- `capability` — aserción posterior al arranque: el agente llama a `name` (herramienta) con `args` o ejecuta `/name` (comando); la etapa lee el registro durable de sesión y exige que la salida observada contenga `expect`. Requiere `DEEPSEEK_API_KEY` (entorno del host o `forwardEnv`); sin ella, la etapa queda `skipped`, nunca falla.
|
|
100
110
|
- Devuelve el registro estructurado completo; ejemplo más abajo.
|
|
101
111
|
- `background: true` inicia una tarea `drive-batch` y devuelve su id.
|
|
102
112
|
|
|
@@ -132,6 +142,11 @@ Devuelve un registro de ejecución (`tdr_...`), una matriz (`tdm_...`) o — sin
|
|
|
132
142
|
"smoke": { "status": "boot-ok", "exitCode": 1, "durationMs": 4123, "attempts": 1,
|
|
133
143
|
"summary": "booted without loader failures; headless task did not complete (credentials/model unreachable)",
|
|
134
144
|
"outputTail": "", "bootFailed": false, "taskCompleted": false },
|
|
145
|
+
"capability": { "status": "observed", "exitCode": 0, "durationMs": 8123, "attempts": 1,
|
|
146
|
+
"summary": "tool \"plugin_vet\" called and its result contains the expectation",
|
|
147
|
+
"outputTail": "", "capabilityKind": "tool", "name": "plugin_vet",
|
|
148
|
+
"expectMatched": true,
|
|
149
|
+
"detail": "tool \"plugin_vet\" called and its result contains the expectation" },
|
|
135
150
|
"uninstall": { "status": "pass", "exitCode": 0, "durationMs": 5123, "attempts": 1,
|
|
136
151
|
"summary": "remove ok (exit 0)", "outputTail": "" },
|
|
137
152
|
"cleanup": { "status": "pass", "quarantined": true, "removed": true,
|
|
@@ -142,7 +157,7 @@ Devuelve un registro de ejecución (`tdr_...`), una matriz (`tdm_...`) o — sin
|
|
|
142
157
|
}
|
|
143
158
|
```
|
|
144
159
|
|
|
145
|
-
Reglas de veredicto: fallo de instalación
|
|
160
|
+
Reglas de veredicto: fallo de instalación, de arranque (`smoke.fail`) o una etapa de capacidad que llegó a `not-registered`/`failed` ⇒ `fail`; instalación aprobada + parche efectivo + arranque limpio (`pass`/`boot-ok`) + desinstalación aprobada ⇒ `pass` (con nota de capacidad cuando `observed`); instalado pero sin alguna garantía posterior ⇒ `partial`; en otro caso ⇒ `unknown`.
|
|
146
161
|
|
|
147
162
|
## Permissions & data (Permisos y datos)
|
|
148
163
|
|
package/README.hi.md
CHANGED
|
@@ -32,9 +32,10 @@
|
|
|
32
32
|
|
|
33
33
|
## What you get (आपको क्या मिलता है)
|
|
34
34
|
|
|
35
|
-
- `test_drive` उपकरण — एक लक्ष्य को पूरी पाइपलाइन से गुज़ारता है: `dsh plugin add` → `--dump-config` पैच जाँच → हेडलेस बूट स्मोक (FAILED मार्कर स्कैन + वैकल्पिक एक-वाक्य कार्य) → `dsh plugin remove` → क्वारंटीन सफ़ाई। संरचित रिकॉर्ड समकालिक रूप से लौटाता है, या `background: true` पर `{ kind: 'background', jobId }`।
|
|
35
|
+
- `test_drive` उपकरण — एक लक्ष्य को पूरी पाइपलाइन से गुज़ारता है: `dsh plugin add` → `--dump-config` पैच जाँच → हेडलेस बूट स्मोक (FAILED मार्कर स्कैन + वैकल्पिक एक-वाक्य कार्य) → वैकल्पिक क्षमता-पुष्टि → `dsh plugin remove` → क्वारंटीन सफ़ाई। संरचित रिकॉर्ड समकालिक रूप से लौटाता है, या `background: true` पर `{ kind: 'background', jobId }`।
|
|
36
36
|
- `/testdrive` कमांड — स्पेस/कॉमा से अलग किए गए लक्ष्यों की सूची को `ctx.jobs` पर `drive-batch` पृष्ठभूमि कार्य के रूप में चलाकर मैट्रिक्स रिपोर्ट (JSON + Markdown) बनाता है।
|
|
37
37
|
- `drive_report` उपकरण — किसी भी रन (`tdr_...`), मैट्रिक्स (`tdm_...`) या नवीनतम मैट्रिक्स को लाता है; Markdown में प्रस्तुत होता है।
|
|
38
|
+
- क्षमता-पुष्टि — “बूट हुआ और बाहर निकला” से आगे: वैकल्पिक `capability` चरण एजेंट से नामित उपकरण को बुलवाता है (या `/command` चलवाता है) और टिकाऊ सत्र लॉग में आह्वान तथा `expect` वाला प्रेक्षित आउटपुट सत्यापित करता है। साफ़ बूट केवल स्मोक टेस्ट है; `observed` सिद्ध करता है कि नामित क्षमता सचमुच काम करती है।
|
|
38
39
|
- संरचित परिणाम — हर रिकॉर्ड में विभेदक `schema: "dsh-test-drive/v1"` और प्रथम-श्रेणी फ़ील्ड होते हैं: `stages.install.status` (`pass`/`fail`), `stages.smoke.status` (`pass`/`fail`/`boot-ok`/`skipped`), प्रति-चरण `durationMs`, सैनिटाइज़ किए गए `summary`/`outputTail`, और समग्र `verdict` (`pass`/`fail`/`partial`/`unknown`)। यही मशीन-पठनीय अनुबंध है जिसे स्कोरिंग पाइपलाइनें (dsh-score) उपभोग करती हैं।
|
|
39
40
|
- निर्माण से सुरक्षा — हर अस्थायी निर्देशिका इस प्लगइन द्वारा समर्पित उपसर्ग `dsh-test-drive-` के तहत बनाई जाती है, एक सक्रिय स्वामित्व रजिस्ट्री में दर्ज होती है, और केवल dry-run → क्वारंटीन-नामांतरण → हटाने की सीढ़ी से हटाई जाती है। होस्ट प्रोफ़ाइल कभी पढ़ी या लिखी नहीं जाती।
|
|
40
41
|
|
|
@@ -82,6 +83,12 @@ dsh plugin --profile web remove dsh-test-drive # अनइंस्टॉल
|
|
|
82
83
|
| `installTimeoutMs` | `600000` | `dsh plugin add` चरण की समय-सीमा। |
|
|
83
84
|
| `configTimeoutMs` | `60000` | `--dump-config` चरण की समय-सीमा। |
|
|
84
85
|
| `smokeTimeoutMs` | `300000` | हेडलेस बूट-स्मोक चरण की समय-सीमा। |
|
|
86
|
+
| `capabilityTimeoutMs` | `300000` | क्षमता-पुष्टि कार्य की समय-सीमा।
|
|
87
|
+
| `capability.enabled` | `false` | क्षमता-पुष्टि चरण चलाएँ (पंजीकृत → आहूत → प्रेक्षित)।
|
|
88
|
+
| `capability.kind` | `tool` | क्या जाँचें: `tool` या `command`।
|
|
89
|
+
| `capability.name` | `""` | उपकरण या कमांड नाम (आगे `/` के बिना)।
|
|
90
|
+
| `capability.args` | `""` | आह्वान पाठ: उपकरण तर्क (JSON-शैली) या कमांड शब्द।
|
|
91
|
+
| `capability.expect` | `""` | प्रेक्षित आउटपुट में अपेक्षित अक्षर (केस-असंवेदी उप-स्ट्रिंग)।
|
|
85
92
|
| `uninstallTimeoutMs` | `120000` | `dsh plugin remove` चरण की समय-सीमा। |
|
|
86
93
|
| `outputTailBytes` | `8000` | प्रति चरण दर्ज सैनिटाइज़्ड आउटपुट टेल की सीमा। |
|
|
87
94
|
| `keepTempDirs` | `false` | विफलता पर फ़ॉरेंसिक हेतु अस्थायी निर्देशिकाएँ रखें (स्वामित्व छोड़ा जाता है; आप साफ़ करें)। |
|
|
@@ -93,10 +100,13 @@ dsh plugin --profile web remove dsh-test-drive # अनइंस्टॉल
|
|
|
93
100
|
### `test_drive`
|
|
94
101
|
|
|
95
102
|
```
|
|
96
|
-
test_drive(target: string, headlessTask?: string, background?: boolean
|
|
103
|
+
test_drive(target: string, headlessTask?: string, background?: boolean,
|
|
104
|
+
capability?: { kind: 'tool' | 'command', name: string,
|
|
105
|
+
args: string, expect: string })
|
|
97
106
|
```
|
|
98
107
|
|
|
99
108
|
- `target` — git स्पेक (`github:owner/repo#sha`, `git+https://...`), npm नाम, स्थानीय पथ या `.tgz` टारबॉल।
|
|
109
|
+
- `capability` — स्मोक के बाद की पुष्टि: एजेंट `args` के साथ `name` (उपकरण) बुलाता है या `/name` (कमांड) चलाता है; चरण टिकाऊ सत्र लॉग पढ़कर `expect` वाला प्रेक्षित आउटपुट माँगता है। `DEEPSEEK_API_KEY` चाहिए (होस्ट परिवेश या `forwardEnv`); बिना उसके चरण `skipped` रहता है, कभी विफल नहीं होता।
|
|
100
110
|
- पूरा संरचित रिकॉर्ड लौटाता है; नमूना नीचे।
|
|
101
111
|
- `background: true` एक `drive-batch` कार्य आरंभ कर उसका id लौटाता है।
|
|
102
112
|
|
|
@@ -132,6 +142,11 @@ test_drive(target: string, headlessTask?: string, background?: boolean)
|
|
|
132
142
|
"smoke": { "status": "boot-ok", "exitCode": 1, "durationMs": 4123, "attempts": 1,
|
|
133
143
|
"summary": "booted without loader failures; headless task did not complete (credentials/model unreachable)",
|
|
134
144
|
"outputTail": "", "bootFailed": false, "taskCompleted": false },
|
|
145
|
+
"capability": { "status": "observed", "exitCode": 0, "durationMs": 8123, "attempts": 1,
|
|
146
|
+
"summary": "tool \"plugin_vet\" called and its result contains the expectation",
|
|
147
|
+
"outputTail": "", "capabilityKind": "tool", "name": "plugin_vet",
|
|
148
|
+
"expectMatched": true,
|
|
149
|
+
"detail": "tool \"plugin_vet\" called and its result contains the expectation" },
|
|
135
150
|
"uninstall": { "status": "pass", "exitCode": 0, "durationMs": 5123, "attempts": 1,
|
|
136
151
|
"summary": "remove ok (exit 0)", "outputTail": "" },
|
|
137
152
|
"cleanup": { "status": "pass", "quarantined": true, "removed": true,
|
|
@@ -142,7 +157,7 @@ test_drive(target: string, headlessTask?: string, background?: boolean)
|
|
|
142
157
|
}
|
|
143
158
|
```
|
|
144
159
|
|
|
145
|
-
निर्णय नियम: इंस्टॉल
|
|
160
|
+
निर्णय नियम: इंस्टॉल विफलता, बूट विफलता (`smoke.fail`) या `not-registered`/`failed` तक पहुँचा क्षमता चरण ⇒ `fail`; इंस्टॉल पास + पैच प्रभावी + साफ़ बूट (`pass`/`boot-ok`) + अनइंस्टॉल पास ⇒ `pass` (`observed` पर क्षमता नोट के साथ); इंस्टॉल हुआ पर बाद की कोई पुष्टि अधूरी ⇒ `partial`; अन्यथा ⇒ `unknown`।
|
|
146
161
|
|
|
147
162
|
## Permissions & data (अनुमतियाँ और डेटा)
|
|
148
163
|
|
package/README.md
CHANGED
|
@@ -32,10 +32,11 @@
|
|
|
32
32
|
|
|
33
33
|
## What you get
|
|
34
34
|
|
|
35
|
-
- `test_drive` tool — one target through the complete pipeline: `dsh plugin add` → `--dump-config` patch check → headless boot smoke (FAILED-marker scan + optional one-shot task) → `dsh plugin remove` → quarantined cleanup. Returns the structured record synchronously, or `{ kind: 'background', jobId }` with `background: true`.
|
|
35
|
+
- `test_drive` tool — one target through the complete pipeline: `dsh plugin add` → `--dump-config` patch check → headless boot smoke (FAILED-marker scan + optional one-shot task) → optional capability assertion → `dsh plugin remove` → quarantined cleanup. Returns the structured record synchronously, or `{ kind: 'background', jobId }` with `background: true`.
|
|
36
36
|
- `/testdrive` command — batch drive of a whitespace/comma-separated target list as a `drive-batch` background job over `ctx.jobs`, producing a matrix report (JSON + Markdown).
|
|
37
37
|
- `drive_report` tool — fetch any stored run (`tdr_...`), matrix (`tdm_...`), or the latest matrix; rendered as Markdown.
|
|
38
|
-
-
|
|
38
|
+
- Capability assertion — beyond "booted and exited": the optional `capability` stage drives one headless task that calls a named tool (or runs a `/command`) and verifies the durable session log recorded the invocation and the observed output contains `expect`. A clean boot is only a smoke test; `observed` proves a named capability really works.
|
|
39
|
+
- Structured results — every record carries the discriminator `schema: "dsh-test-drive/v1"` with first-class fields: `stages.install.status` (`pass`/`fail`), `stages.smoke.status` (`pass`/`fail`/`boot-ok`/`skipped`), `stages.capability.status` (`observed`/`invoked`/`not-registered`/`skipped`/`failed`), per-stage `durationMs`, sanitized `summary`/`outputTail`, and an overall `verdict` (`pass`/`fail`/`partial`/`unknown`). This is the machine-readable contract downstream scorers (dsh-score) consume.
|
|
39
40
|
- Safety by construction — every temp directory is created by this plugin under a dedicated `dsh-test-drive-` prefix, tracked in a live ownership registry, and removed only through a dry-run → quarantine-rename → delete ladder. The host profile is never read or written.
|
|
40
41
|
|
|
41
42
|
## Quick start
|
|
@@ -82,6 +83,12 @@ All keys are optional (defaults shown); invalid values fail loudly at load.
|
|
|
82
83
|
| `installTimeoutMs` | `600000` | `dsh plugin add` stage deadline. |
|
|
83
84
|
| `configTimeoutMs` | `60000` | `--dump-config` stage deadline. |
|
|
84
85
|
| `smokeTimeoutMs` | `300000` | Headless boot-smoke stage deadline. |
|
|
86
|
+
| `capabilityTimeoutMs` | `300000` | Capability-assertion task deadline. |
|
|
87
|
+
| `capability.enabled` | `false` | Run the capability-assertion stage (registered → invoked → observed). |
|
|
88
|
+
| `capability.kind` | `tool` | What to assert: `tool` or `command`. |
|
|
89
|
+
| `capability.name` | `""` | Tool or command name (no leading `/`). |
|
|
90
|
+
| `capability.args` | `""` | Invocation text: tool arguments (JSON-ish) or command words. |
|
|
91
|
+
| `capability.expect` | `""` | Literal expected in the observed output (case-insensitive substring). |
|
|
85
92
|
| `uninstallTimeoutMs` | `120000` | `dsh plugin remove` stage deadline. |
|
|
86
93
|
| `outputTailBytes` | `8000` | Cap on the sanitized output tail recorded per stage. |
|
|
87
94
|
| `keepTempDirs` | `false` | Keep temp dirs on failure for forensics (ownership is dropped; you clean up). |
|
|
@@ -93,10 +100,13 @@ All keys are optional (defaults shown); invalid values fail loudly at load.
|
|
|
93
100
|
### `test_drive`
|
|
94
101
|
|
|
95
102
|
```
|
|
96
|
-
test_drive(target: string, headlessTask?: string, background?: boolean
|
|
103
|
+
test_drive(target: string, headlessTask?: string, background?: boolean,
|
|
104
|
+
capability?: { kind: 'tool' | 'command', name: string,
|
|
105
|
+
args: string, expect: string })
|
|
97
106
|
```
|
|
98
107
|
|
|
99
108
|
- `target` — git spec (`github:owner/repo#sha`, `git+https://...`), npm name, local path, or `.tgz` tarball.
|
|
109
|
+
- `capability` — assertion after the boot smoke: the agent calls `name` (tool) or runs `/name` (command) with `args`; the stage reads the durable session log and requires the observed output to contain `expect`. Needs `DEEPSEEK_API_KEY` (host env or `forwardEnv`); without it the stage is `skipped`, never failed.
|
|
100
110
|
- Returns the full structured record; see the sample below.
|
|
101
111
|
- `background: true` starts a `drive-batch` job and returns its id.
|
|
102
112
|
|
|
@@ -132,6 +142,11 @@ Returns a run record (`tdr_...`), a matrix (`tdm_...`), or — with no id — th
|
|
|
132
142
|
"smoke": { "status": "boot-ok", "exitCode": 1, "durationMs": 4123, "attempts": 1,
|
|
133
143
|
"summary": "booted without loader failures; headless task did not complete (credentials/model unreachable)",
|
|
134
144
|
"outputTail": "", "bootFailed": false, "taskCompleted": false },
|
|
145
|
+
"capability": { "status": "observed", "exitCode": 0, "durationMs": 8123, "attempts": 1,
|
|
146
|
+
"summary": "tool \"plugin_vet\" called and its result contains the expectation",
|
|
147
|
+
"outputTail": "", "capabilityKind": "tool", "name": "plugin_vet",
|
|
148
|
+
"expectMatched": true,
|
|
149
|
+
"detail": "tool \"plugin_vet\" called and its result contains the expectation" },
|
|
135
150
|
"uninstall": { "status": "pass", "exitCode": 0, "durationMs": 5123, "attempts": 1,
|
|
136
151
|
"summary": "remove ok (exit 0)", "outputTail": "" },
|
|
137
152
|
"cleanup": { "status": "pass", "quarantined": true, "removed": true,
|
|
@@ -142,7 +157,7 @@ Returns a run record (`tdr_...`), a matrix (`tdm_...`), or — with no id — th
|
|
|
142
157
|
}
|
|
143
158
|
```
|
|
144
159
|
|
|
145
|
-
Verdict rules: install failure
|
|
160
|
+
Verdict rules: install failure, boot failure (`smoke.fail`), or a capability stage that reached `not-registered`/`failed` ⇒ `fail`; install pass + patch effective + clean boot (`pass`/`boot-ok`) + uninstall pass ⇒ `pass` (with a capability note when `observed`); anything installed but missing a later assurance ⇒ `partial`; otherwise `unknown`.
|
|
146
161
|
|
|
147
162
|
## Permissions & data
|
|
148
163
|
|
package/README.pt.md
CHANGED
|
@@ -32,9 +32,10 @@
|
|
|
32
32
|
|
|
33
33
|
## What you get (O que você obtém)
|
|
34
34
|
|
|
35
|
-
- Ferramenta `test_drive` — um alvo pelo pipeline completo: `dsh plugin add` → verificação do patch com `--dump-config` → inicialização headless (varredura de marcadores FAILED + tarefa opcional de uma frase) → `dsh plugin remove` → limpeza em quarentena. Retorna o registro estruturado de forma síncrona, ou `{ kind: 'background', jobId }` com `background: true`.
|
|
35
|
+
- Ferramenta `test_drive` — um alvo pelo pipeline completo: `dsh plugin add` → verificação do patch com `--dump-config` → inicialização headless (varredura de marcadores FAILED + tarefa opcional de uma frase) → asserção de capacidade opcional → `dsh plugin remove` → limpeza em quarentena. Retorna o registro estruturado de forma síncrona, ou `{ kind: 'background', jobId }` com `background: true`.
|
|
36
36
|
- Comando `/testdrive` — lote de alvos separados por espaços/vírgulas como tarefa em segundo plano `drive-batch` sobre `ctx.jobs`, produzindo um relatório matricial (JSON + Markdown).
|
|
37
37
|
- Ferramenta `drive_report` — busca qualquer execução (`tdr_...`), matriz (`tdm_...`) ou a matriz mais recente; renderizada em Markdown.
|
|
38
|
+
- Asserção de capacidade — além de “inicializou e saiu”: a etapa opcional `capability` faz o agente chamar a ferramenta nomeada (ou executar `/command`) e verifica no registro durável de sessão que a invocação ocorreu e que a saída observada contém `expect`. Uma inicialização limpa é só um teste de fumaça; `observed` prova que uma capacidade nomeada realmente funciona.
|
|
38
39
|
- Resultados estruturados — cada registro carrega o discriminador `schema: "dsh-test-drive/v1"` com campos de primeira classe: `stages.install.status` (`pass`/`fail`), `stages.smoke.status` (`pass`/`fail`/`boot-ok`/`skipped`), `durationMs` por etapa, `summary`/`outputTail` saneados e um `verdict` geral (`pass`/`fail`/`partial`/`unknown`). Este é o contrato legível por máquina consumido pelos pontuadores (dsh-score).
|
|
39
40
|
- Segurança por construção — cada diretório temporário é criado por este plugin sob um prefixo dedicado `dsh-test-drive-`, registrado em um registro de propriedade ativo e removido apenas pela escada dry-run → renomear para quarentena → excluir. O perfil do host nunca é lido nem gravado.
|
|
40
41
|
|
|
@@ -82,6 +83,12 @@ Todas as chaves são opcionais (padrões exibidos); valores inválidos falham ru
|
|
|
82
83
|
| `installTimeoutMs` | `600000` | Prazo da etapa `dsh plugin add`. |
|
|
83
84
|
| `configTimeoutMs` | `60000` | Prazo da etapa `--dump-config`. |
|
|
84
85
|
| `smokeTimeoutMs` | `300000` | Prazo da etapa de inicialização headless. |
|
|
86
|
+
| `capabilityTimeoutMs` | `300000` | Prazo da tarefa de asserção de capacidade.
|
|
87
|
+
| `capability.enabled` | `false` | Executa a etapa de asserção de capacidade (registrado → invocado → observado).
|
|
88
|
+
| `capability.kind` | `tool` | O que afirmar: `tool` ou `command`.
|
|
89
|
+
| `capability.name` | `""` | Nome da ferramenta ou comando (sem a `/` inicial).
|
|
90
|
+
| `capability.args` | `""` | Texto de invocação: argumentos da ferramenta (estilo JSON) ou palavras do comando.
|
|
91
|
+
| `capability.expect` | `""` | Literal esperado na saída observada (substring sem distinção de maiúsculas).
|
|
85
92
|
| `uninstallTimeoutMs` | `120000` | Prazo da etapa `dsh plugin remove`. |
|
|
86
93
|
| `outputTailBytes` | `8000` | Limite da cauda de saída saneada registrada por etapa. |
|
|
87
94
|
| `keepTempDirs` | `false` | Mantém os diretórios temporários em caso de falha para análise forense (a propriedade é abandonada; você limpa). |
|
|
@@ -93,10 +100,13 @@ Todas as chaves são opcionais (padrões exibidos); valores inválidos falham ru
|
|
|
93
100
|
### `test_drive`
|
|
94
101
|
|
|
95
102
|
```
|
|
96
|
-
test_drive(target: string, headlessTask?: string, background?: boolean
|
|
103
|
+
test_drive(target: string, headlessTask?: string, background?: boolean,
|
|
104
|
+
capability?: { kind: 'tool' | 'command', name: string,
|
|
105
|
+
args: string, expect: string })
|
|
97
106
|
```
|
|
98
107
|
|
|
99
108
|
- `target` — especificação git (`github:owner/repo#sha`, `git+https://...`), nome npm, caminho local ou tarball `.tgz`.
|
|
109
|
+
- `capability` — asserção após o smoke: o agente chama `name` (ferramenta) com `args` ou executa `/name` (comando); a etapa lê o registro durável de sessão e exige que a saída observada contenha `expect`. Requer `DEEPSEEK_API_KEY` (ambiente do host ou `forwardEnv`); sem ela, a etapa fica `skipped`, nunca falha.
|
|
100
110
|
- Retorna o registro estruturado completo; exemplo abaixo.
|
|
101
111
|
- `background: true` inicia uma tarefa `drive-batch` e retorna seu id.
|
|
102
112
|
|
|
@@ -132,6 +142,11 @@ Retorna um registro de execução (`tdr_...`), uma matriz (`tdm_...`) ou — sem
|
|
|
132
142
|
"smoke": { "status": "boot-ok", "exitCode": 1, "durationMs": 4123, "attempts": 1,
|
|
133
143
|
"summary": "booted without loader failures; headless task did not complete (credentials/model unreachable)",
|
|
134
144
|
"outputTail": "", "bootFailed": false, "taskCompleted": false },
|
|
145
|
+
"capability": { "status": "observed", "exitCode": 0, "durationMs": 8123, "attempts": 1,
|
|
146
|
+
"summary": "tool \"plugin_vet\" called and its result contains the expectation",
|
|
147
|
+
"outputTail": "", "capabilityKind": "tool", "name": "plugin_vet",
|
|
148
|
+
"expectMatched": true,
|
|
149
|
+
"detail": "tool \"plugin_vet\" called and its result contains the expectation" },
|
|
135
150
|
"uninstall": { "status": "pass", "exitCode": 0, "durationMs": 5123, "attempts": 1,
|
|
136
151
|
"summary": "remove ok (exit 0)", "outputTail": "" },
|
|
137
152
|
"cleanup": { "status": "pass", "quarantined": true, "removed": true,
|
|
@@ -142,7 +157,7 @@ Retorna um registro de execução (`tdr_...`), uma matriz (`tdm_...`) ou — sem
|
|
|
142
157
|
}
|
|
143
158
|
```
|
|
144
159
|
|
|
145
|
-
Regras de veredicto: falha de instalação
|
|
160
|
+
Regras de veredicto: falha de instalação, de inicialização (`smoke.fail`) ou uma etapa de capacidade que chegou a `not-registered`/`failed` ⇒ `fail`; instalação aprovada + patch efetivo + inicialização limpa (`pass`/`boot-ok`) + desinstalação aprovada ⇒ `pass` (com nota de capacidade quando `observed`); instalado mas sem alguma garantia posterior ⇒ `partial`; caso contrário ⇒ `unknown`.
|
|
146
161
|
|
|
147
162
|
## Permissions & data (Permissões e dados)
|
|
148
163
|
|
package/README.zh.md
CHANGED
|
@@ -32,9 +32,10 @@
|
|
|
32
32
|
|
|
33
33
|
## What you get(你能得到什么)
|
|
34
34
|
|
|
35
|
-
- `test_drive` 工具 —— 单个目标跑完整流水线:`dsh plugin add` → `--dump-config` patch 校验 → headless 引导冒烟(FAILED 标记扫描 + 可选一句任务)→ `dsh plugin remove` → 隔离清理。同步返回结构化记录;传 `background: true` 则返回 `{ kind: 'background', jobId }`。
|
|
35
|
+
- `test_drive` 工具 —— 单个目标跑完整流水线:`dsh plugin add` → `--dump-config` patch 校验 → headless 引导冒烟(FAILED 标记扫描 + 可选一句任务)→ 可选能力断言 → `dsh plugin remove` → 隔离清理。同步返回结构化记录;传 `background: true` 则返回 `{ kind: 'background', jobId }`。
|
|
36
36
|
- `/testdrive` 命令 —— 把空格/逗号分隔的目标列表作为 `drive-batch` 后台任务(`ctx.jobs`)批量跑,产出矩阵报告(JSON + Markdown)。
|
|
37
37
|
- `drive_report` 工具 —— 按 id 取回任意单次记录(`tdr_...`)、矩阵(`tdm_...`)或最新矩阵;以 Markdown 渲染。
|
|
38
|
+
- 能力断言 —— 不止于“启动并退出”:可选的 `capability` 阶段让 agent 调用指定工具(或运行 `/command`),并核对持久会话日志确实记录了调用、观察输出包含 `expect`。干净启动只是冒烟;`observed` 才证明具名能力真实可用。
|
|
38
39
|
- 结构化结果 —— 每条记录带判别符 `schema: "dsh-test-drive/v1"`,关键字段均为一级字段:`stages.install.status`(`pass`/`fail`)、`stages.smoke.status`(`pass`/`fail`/`boot-ok`/`skipped`)、各阶段 `durationMs`、脱敏后的 `summary`/`outputTail` 以及总判定 `verdict`(`pass`/`fail`/`partial`/`unknown`)。这是下游评分方(dsh-score)消费的机器可读契约。
|
|
39
40
|
- 结构安全 —— 每个临时目录都由本插件以专属前缀 `dsh-test-drive-` 创建、登记在活跃所有权注册表中,且只经「dry-run → 隔离改名 → 删除」阶梯清理。宿主机 profile 永不被读取或写入。
|
|
40
41
|
|
|
@@ -82,6 +83,12 @@ dsh plugin --profile web remove dsh-test-drive # 卸载
|
|
|
82
83
|
| `installTimeoutMs` | `600000` | `dsh plugin add` 阶段时限。 |
|
|
83
84
|
| `configTimeoutMs` | `60000` | `--dump-config` 阶段时限。 |
|
|
84
85
|
| `smokeTimeoutMs` | `300000` | headless 引导冒烟阶段时限。 |
|
|
86
|
+
| `capabilityTimeoutMs` | `300000` | 能力断言任务时限。
|
|
87
|
+
| `capability.enabled` | `false` | 运行能力断言阶段(注册 → 调用 → 观察到)。
|
|
88
|
+
| `capability.kind` | `tool` | 断言对象:`tool` 或 `command`。
|
|
89
|
+
| `capability.name` | `""` | 工具或命令名(不带前导 `/`)。
|
|
90
|
+
| `capability.args` | `""` | 调用文本:工具参数(JSON 风格)或命令参数。
|
|
91
|
+
| `capability.expect` | `""` | 期望出现在观察输出中的字面量(不区分大小写子串)。
|
|
85
92
|
| `uninstallTimeoutMs` | `120000` | `dsh plugin remove` 阶段时限。 |
|
|
86
93
|
| `outputTailBytes` | `8000` | 每阶段记录的脱敏输出尾部上限(字节)。 |
|
|
87
94
|
| `keepTempDirs` | `false` | 失败时保留临时目录供取证(所有权被放弃,由你清理)。 |
|
|
@@ -93,10 +100,13 @@ dsh plugin --profile web remove dsh-test-drive # 卸载
|
|
|
93
100
|
### `test_drive`
|
|
94
101
|
|
|
95
102
|
```
|
|
96
|
-
test_drive(target: string, headlessTask?: string, background?: boolean
|
|
103
|
+
test_drive(target: string, headlessTask?: string, background?: boolean,
|
|
104
|
+
capability?: { kind: 'tool' | 'command', name: string,
|
|
105
|
+
args: string, expect: string })
|
|
97
106
|
```
|
|
98
107
|
|
|
99
108
|
- `target` —— git 规格(`github:owner/repo#sha`、`git+https://...`)、npm 包名、本地路径或 `.tgz` 压缩包。
|
|
109
|
+
- `capability` —— 冒烟后的断言:agent 以 `args` 调用 `name`(tool)或运行 `/name`(command);阶段读取持久会话日志并要求观察输出包含 `expect`。需要 `DEEPSEEK_API_KEY`(宿主环境或 `forwardEnv`);缺失时该阶段为 `skipped`,绝不算失败。
|
|
100
110
|
- 返回完整结构化记录,样例见下。
|
|
101
111
|
- `background: true` 启动一个 `drive-batch` 任务并返回其 id。
|
|
102
112
|
|
|
@@ -132,6 +142,11 @@ test_drive(target: string, headlessTask?: string, background?: boolean)
|
|
|
132
142
|
"smoke": { "status": "boot-ok", "exitCode": 1, "durationMs": 4123, "attempts": 1,
|
|
133
143
|
"summary": "booted without loader failures; headless task did not complete (credentials/model unreachable)",
|
|
134
144
|
"outputTail": "", "bootFailed": false, "taskCompleted": false },
|
|
145
|
+
"capability": { "status": "observed", "exitCode": 0, "durationMs": 8123, "attempts": 1,
|
|
146
|
+
"summary": "tool \"plugin_vet\" called and its result contains the expectation",
|
|
147
|
+
"outputTail": "", "capabilityKind": "tool", "name": "plugin_vet",
|
|
148
|
+
"expectMatched": true,
|
|
149
|
+
"detail": "tool \"plugin_vet\" called and its result contains the expectation" },
|
|
135
150
|
"uninstall": { "status": "pass", "exitCode": 0, "durationMs": 5123, "attempts": 1,
|
|
136
151
|
"summary": "remove ok (exit 0)", "outputTail": "" },
|
|
137
152
|
"cleanup": { "status": "pass", "quarantined": true, "removed": true,
|
|
@@ -142,7 +157,7 @@ test_drive(target: string, headlessTask?: string, background?: boolean)
|
|
|
142
157
|
}
|
|
143
158
|
```
|
|
144
159
|
|
|
145
|
-
|
|
160
|
+
判定规则:安装失败、启动失败(`smoke.fail`)或能力阶段到达 `not-registered`/`failed` ⇒ `fail`;安装通过 + patch 生效 + 启动干净(`pass`/`boot-ok`)+ 卸载通过 ⇒ `pass`(`observed` 时附能力说明);已安装但后续保证缺失 ⇒ `partial`;其余 ⇒ `unknown`。
|
|
146
161
|
|
|
147
162
|
## Permissions & data(权限与数据)
|
|
148
163
|
|
package/cordis.patch.yml
CHANGED
|
@@ -32,8 +32,22 @@
|
|
|
32
32
|
# installTimeoutMs: 600000
|
|
33
33
|
# configTimeoutMs: 60000
|
|
34
34
|
# smokeTimeoutMs: 300000
|
|
35
|
+
# capabilityTimeoutMs: 300000
|
|
35
36
|
# uninstallTimeoutMs: 120000
|
|
36
37
|
|
|
38
|
+
# Capability assertion (registered -> invoked -> observed): after the
|
|
39
|
+
# boot smoke, drive one headless task that calls a named tool (or runs
|
|
40
|
+
# a /command) and verify the durable session log records the invocation
|
|
41
|
+
# and the observed output contains `expect`. Requires DEEPSEEK_API_KEY
|
|
42
|
+
# (host env or forwardEnv); without it the stage is skipped, never
|
|
43
|
+
# failed. Off by default.
|
|
44
|
+
# capability:
|
|
45
|
+
# enabled: false
|
|
46
|
+
# kind: tool # 'tool' or 'command'
|
|
47
|
+
# name: "" # tool or command name (no leading /)
|
|
48
|
+
# args: "" # tool arguments (JSON-ish) or command words
|
|
49
|
+
# expect: "" # literal expected in the observed output
|
|
50
|
+
|
|
37
51
|
# Cap on the sanitized output tail recorded per stage, in bytes.
|
|
38
52
|
# outputTailBytes: 8000
|
|
39
53
|
|