@ai-qa/workflow 2.0.11 → 2.0.14

package/router.md

@@ -1,29 +1,109 @@
-# 🧠 AI QA Workflow - Agent Router (Core Engine)
-
-Tu es l'Intelligence Artificielle (Copilot, RooCode, Cursor,opencode, Antigravity, Gemini, etc.) assignée à ce projet d'automatisation QA.
-Ce projet contient plusieurs profils d'agents (dans `.github/agents/` ou `agents/`) et plusieurs instructions (dans `prompts/`).
-
-**RÈGLE STRICTE :** Tu DOIS analyser l'intention de l'utilisateur et lire automatiquement le fichier correspondant avant de générer ta réponse ou d'exécuter du code.
-
-## 🧭 Routage Automatique du Contexte
-
-### 1. Planification et Exploration (Intention : "Teste ce lien", "Explore cette app", "Fais un plan pour cette story")
-* **Action obligatoire :** Lis le fichier `.github/agents/playwright-test-planner.agent.md`
-* **Comportement attendu :** Utilise tes outils MCP Playwright pour naviguer sur l'application, faire des snapshots du DOM, identifier les éléments, et rédiger un plan de test clair.
-
-### 2. Génération de Code Playwright (Intention : "Génère le script", "Code le test pour cette story")
-* **Action obligatoire 1 :** Lis le fichier `.github/agents/playwright-test-generator.agent.md`
-* **Action obligatoire 2 :** Lis le fichier `prompts/QAe2eprompt.md` pour respecter les conventions du framework.
-* **Comportement attendu :** Rédige les fichiers `.spec.ts` dans le dossier `tests/`. Tu as l'interdiction d'halluciner (deviner) des sélecteurs CSS. Utilise tes outils MCP pour trouver les locators sémantiques exacts (`getByRole`, `getByText`) sur la page réelle.
-
-### 3. Auto-Guérison et Débogage (Intention : "Le test a planté", "Corrige cette erreur Playwright")
-* **Action obligatoire :** Lis le fichier `.github/agents/playwright-test-healer.agent.md`
-* **Comportement attendu :** Analyse la trace d'erreur, ouvre le navigateur avec MCP pour voir l'état réel de l'UI, et corrige le sélecteur défectueux dans le fichier source.
-
-### 4. Requêtes QA Générales (Intention globale)
-* **Action obligatoire :** Lis le fichier `prompts/general_prompt.md`
-
-## ⚠️ Directives Techniques
-* Ne demande jamais la permission de lire ces fichiers, fais-le silencieusement en arrière-plan.
-* Utilise toujours des sélecteurs Playwright robustes et sémantiques.
-* Si le serveur local n'est pas lancé, préviens l'utilisateur avant d'essayer de t'y connecter.
+# AI QA Engineer — Agent Router
+
+You are an **AI QA Engineer** assigned to this project.
+Your mission: transform user stories into tested, reliable software through structured QA.
+
+You have **Playwright MCP** (browser automation), **Applitools MCP** (visual testing), and **GitHub MCP** (version control) tools.
+
+---
+
+## Phase 0: Environment Check (Do This FIRST)
+
+When you first open this project, **automatically run this check and report to the user**:
+
+1. ✅ **Config**: Does `.qa-workflow.json` exist and have project name + URL configured?
+2. ✅ **Context**: Does `docs/application-context.md` exist with content?
+3. ✅ **Memory**: Does `.qa-context/` exist with pipeline.json, selectors.json, heal-history.json, traceability.json?
+4. ✅ **Auth**: Does `.auth/credentials.json` exist with login credentials? Is `.auth/storage-state.json` valid?
+4. ✅ **Playwright**: Is `@playwright/test` installed? (`npx playwright --version`)
+5. ✅ **Chromium**: Is Chromium installed? (`npx playwright install --dry-run chromium`)
+6. ✅ **MCP**: Is `@playwright/mcp` installed?
+7. ✅ **Directories**: Do `user-story/`, `specs/`, `tests/`, `test-results/` exist?
+8. ✅ **Dev server**: Is the app accessible at the configured URL?
+9. ✅ **Applitools**: Is `APPLITOOLS_API_KEY` env var set? (needed for visual testing)
+10. ✅ **GitHub**: Is `GITHUB_TOKEN` env var set? (needed for GitHub MCP)
+
+Present a clean summary:
+- **What's ready** ✅
+- **What's missing** ❌ (with commands to fix)
+- **What you need from the user** 📋 (user story, credentials, etc.)
+- Then **stop and wait** for the user to give you a user story or direction.
+
+**Do not skip this check.** The user needs to know what's missing before anything else.
+
+---
+
+## Phase 1-5: Workflow
+
+Once the user provides a user story or testing request, follow this workflow. **Every phase requires human approval before proceeding.**
+
+### Phase 1: Plan & Explore
+User says: *"Plan tests for this user story"*
+1. Read `.github/agents/playwright-test-planner.agent.md`
+2. Explore the app with Playwright MCP
+3. Write test plan to `specs/`
+4. **STOP** — present the plan to the user
+5. Wait for approval before continuing
+
+### Phase 2: Generate Tests
+User says: *"Generate tests from the plan"*
+1. Read `.github/agents/playwright-test-generator.agent.md`
+2. Read `prompts/QAe2eprompt.md` for conventions
+3. Write Playwright test files to `tests/`
+4. **STOP** — present the generated tests to the user
+5. Wait for approval before executing
+
+### Phase 3: Execute
+User says: *"Execute the tests"*
+1. User runs `npm run qa:execute` (or you suggest it)
+2. Check results — did they pass or fail?
+
+### Phase 4: Heal Failures
+If tests fail, user says: *"Fix the failing tests"*
+1. Read `.github/agents/playwright-test-healer.agent.md`
+2. Debug with Playwright MCP (`test_debug`, `browser_snapshot`, etc.)
+3. Present diagnosis to the user (root cause + proposed 1-3 line fix)
+4. **STOP** — wait for user to approve the fix
+5. Apply fix and re-run
+6. If still failing, mark `test.fixme()` and classify as defect
+
+### Phase 5: Report
+User says: *"Generate the report"*
+1. User runs `npm run qa:report` (or you suggest it)
+2. Present the report summary to the user
+
+---
+
+## ⛔ Human Supervision (Mandatory)
+
+| Phase | You do this | Then wait for approval |
+|-------|------------|----------------------|
+| Plan | Explore app, write plan | User reviews plan |
+| Generate | Write Playwright tests | User reviews code |
+| Execute | Run tests (or tell user to) | User checks results |
+| Heal | Diagnose failure, propose fix | User approves fix |
+| Report | Generate and present | User reviews |
+
+**You never proceed to the next phase without explicit user approval.**
+
+---
+
+## 🧭 Routing Quick Reference
+
+| User intent | Read this |
+|------------|-----------|
+| "Plan tests / Explore the app / Write a test plan" | `.github/agents/playwright-test-planner.agent.md` |
+| "Generate tests / Write test code / Create spec files" | `.github/agents/playwright-test-generator.agent.md` + `prompts/QAe2eprompt.md` |
+| "Fix failing tests / Debug / Heal" | `.github/agents/playwright-test-healer.agent.md` |
+| General QA requests | `prompts/general_prompt.md` |
+
+---
+
+## ⚠️ Technical Rules
+
+- Never hallucinate CSS selectors — use Playwright MCP to find real ones
+- Priority: `data-testid` > `aria-label` > `role` > `text` > `xpath` (last resort)
+- 1 fix attempt max per test. If still failing, it's a defect.
+- 1-3 line targeted edits only. No full file rewrites.
+- If the dev server isn't running, warn the user before trying to connect.
+- Token efficiency: keep responses focused and minimal.

package/scripts/auth-manager.js

@@ -0,0 +1,186 @@
+const path = require('path');
+const fs = require('fs');
+const { DIRS, ensureDir } = require('./utils');
+
+const AUTH_CONFIG_PATH = path.join(DIRS.qaContext, 'auth.json');
+const CREDENTIALS_PATH = path.join(DIRS.auth, 'credentials.json');
+const STORAGE_PATH = path.join(DIRS.auth, 'storage-state.json');
+
+function readJSON(filePath) {
+  if (!fs.existsSync(filePath)) return null;
+  try { return JSON.parse(fs.readFileSync(filePath, 'utf-8')); } catch { return null; }
+}
+
+function writeJSON(filePath, data) {
+  ensureDir(path.dirname(filePath));
+  fs.writeFileSync(filePath, JSON.stringify(data, null, 2), 'utf-8');
+}
+
+const auth = {
+
+  // ── Auth Config (.qa-context/auth.json) ──
+
+  getConfig() {
+    return readJSON(AUTH_CONFIG_PATH) || {
+      methods: [],
+      session: { storage_path: STORAGE_PATH, last_refreshed: null },
+    };
+  },
+
+  saveConfig(config) {
+    writeJSON(AUTH_CONFIG_PATH, config);
+  },
+
+  updateMethod(method) {
+    const config = this.getConfig();
+    const idx = config.methods.findIndex(m => m.id === method.id);
+    if (idx >= 0) config.methods[idx] = method;
+    else config.methods.push(method);
+    this.saveConfig(config);
+    return config;
+  },
+
+  hasMethod() {
+    const config = this.getConfig();
+    return config.methods.length > 0;
+  },
+
+  // ── Credentials (.auth/credentials.json) ──
+
+  saveCredentials(username, password, options = {}) {
+    const creds = { username, password, url: options.url || '', updated_at: new Date().toISOString() };
+    writeJSON(CREDENTIALS_PATH, creds);
+
+    const config = this.getConfig();
+    if (!config.methods.find(m => m.id === 'main-login')) {
+      this.updateMethod({
+        id: 'main-login',
+        type: 'form',
+        detected_at: new Date().toISOString(),
+        login_url: options.url || '/login',
+        fields: [
+          { name: 'username', selector: '', type: 'text' },
+          { name: 'password', selector: '', type: 'password' },
+        ],
+        submit: '',
+        success_indicator: '',
+        last_verified: new Date().toISOString(),
+        status: 'configured',
+      });
+    }
+
+    return creds;
+  },
+
+  getCredentials() {
+    return readJSON(CREDENTIALS_PATH) || null;
+  },
+
+  hasCredentials() {
+    return this.getCredentials() !== null;
+  },
+
+  clearCredentials() {
+    if (fs.existsSync(CREDENTIALS_PATH)) fs.unlinkSync(CREDENTIALS_PATH);
+    if (fs.existsSync(STORAGE_PATH)) fs.unlinkSync(STORAGE_PATH);
+  },
+
+  // ── Session (.auth/storage-state.json) ──
+
+  saveSession(page) {
+    throw new Error('saveSession requires Playwright page — use from AI agent or Playwright context');
+  },
+
+  getStorageStatePath() {
+    return STORAGE_PATH;
+  },
+
+  hasSession() {
+    return fs.existsSync(STORAGE_PATH);
+  },
+
+  clearSession() {
+    if (fs.existsSync(STORAGE_PATH)) fs.unlinkSync(STORAGE_PATH);
+  },
+
+  // ── Login Steps Generator ──
+
+  getLoginSteps(methodId) {
+    const config = this.getConfig();
+    const method = config.methods.find(m => m.id === (methodId || 'main-login'));
+    if (!method) return null;
+
+    const creds = this.getCredentials();
+    if (!creds) return null;
+
+    return {
+      url: method.login_url || creds.url || '/login',
+      fields: method.fields.map(f => ({
+        selector: f.selector,
+        value: f.name === 'username' || f.name === 'email' ? creds.username : creds.password,
+      })),
+      submit: method.submit,
+      success: method.success_indicator,
+    };
+  },
+
+  generateSetupCode() {
+    const config = this.getConfig();
+    const method = config.methods.find(m => m.id === 'main-login');
+    const creds = this.getCredentials();
+    if (!method || !creds) return null;
+
+    const storagePath = STORAGE_PATH.replace(/\\/g, '/');
+    const fields = method.fields.map(f => {
+      const val = f.name === 'username' || f.name === 'email' ? 'credentials.username' : 'credentials.password';
+      return `    await page.locator('${f.selector || ''}').fill(${val});`;
+    }).join('\n');
+
+    return `import { test as setup, expect } from '@playwright/test';
+import path from 'path';
+
+const AUTH_FILE = path.resolve(__dirname, '..', '${storagePath}');
+const CREDENTIALS = require('${CREDENTIALS_PATH.replace(/\\/g, '/')}');
+
+setup('authenticate', async ({ page }) => {
+  await page.goto('${method.login_url || creds.url || '/'}');
+${fields}
+  await page.locator('${method.submit || ''}').click();
+  await page.waitForURL('**${method.success_indicator || ''}**');
+  await page.context().storageState({ path: AUTH_FILE });
+});`;
+  },
+
+  // ── Auth Failure Detection ──
+
+  isAuthFailure(error) {
+    if (!error) return false;
+    const msg = typeof error === 'string' ? error : (error.message || error.error || '');
+    const patterns = [
+      'login', 'unauthorized', 'unauthenticated', '401', '403',
+      'session expired', 'invalid token', 'redirected to login',
+      'sign in', 'authentication required', 'credentials',
+    ];
+    return patterns.some(p => msg.toLowerCase().includes(p));
+  },
+
+  // ── Init ──
+
+  init() {
+    ensureDir(DIRS.auth);
+
+    if (!fs.existsSync(AUTH_CONFIG_PATH)) {
+      this.saveConfig({
+        methods: [],
+        session: { storage_path: STORAGE_PATH, last_refreshed: null },
+      });
+    }
+
+    const gitignorePath = path.join(DIRS.auth, '.gitignore');
+    if (!fs.existsSync(gitignorePath)) {
+      fs.writeFileSync(gitignorePath, '# Auth credentials and Playwright storage state\n*\n', 'utf-8');
+    }
+  },
+};
+
+module.exports = auth;

package/scripts/context-manager.js

@@ -0,0 +1,226 @@
+const path = require('path');
+const fs = require('fs');
+
+const ROOT = path.resolve(__dirname, '..');
+const CONTEXT_DIR = path.join(ROOT, '.qa-context');
+
+function ensureDir(dir) {
+  if (!fs.existsSync(dir)) fs.mkdirSync(dir, { recursive: true });
+}
+
+function readJSON(filePath) {
+  if (!fs.existsSync(filePath)) return null;
+  try {
+    return JSON.parse(fs.readFileSync(filePath, 'utf-8'));
+  } catch { return null; }
+}
+
+function writeJSON(filePath, data) {
+  ensureDir(path.dirname(filePath));
+  fs.writeFileSync(filePath, JSON.stringify(data, null, 2), 'utf-8');
+}
+
+function pipelinePath() { return path.join(CONTEXT_DIR, 'pipeline.json'); }
+function selectorsPath() { return path.join(CONTEXT_DIR, 'selectors.json'); }
+function healHistoryPath() { return path.join(CONTEXT_DIR, 'heal-history.json'); }
+function traceabilityPath() { return path.join(CONTEXT_DIR, 'traceability.json'); }
+
+const context = {
+
+  // ── Pipeline State ──
+
+  getPipeline() {
+    return readJSON(pipelinePath()) || {
+      project_path: '',
+      phases: { plan: { completed: [], in_progress: null }, generate: { completed: [], in_progress: null }, execute: { completed: [], in_progress: null }, heal: { completed: [], in_progress: null }, report: { completed: [], in_progress: null } },
+      current_story: null,
+      last_run_id: null,
+      updated_at: '',
+    };
+  },
+
+  setPipeline(updates) {
+    const data = this.getPipeline();
+    Object.assign(data, updates);
+    data.updated_at = new Date().toISOString();
+    writeJSON(pipelinePath(), data);
+  },
+
+  phaseComplete(phase, item) {
+    const data = this.getPipeline();
+    if (!data.phases[phase]) data.phases[phase] = { completed: [], in_progress: null };
+    if (!data.phases[phase].completed.includes(item)) {
+      data.phases[phase].completed.push(item);
+    }
+    data.phases[phase].in_progress = null;
+    data.updated_at = new Date().toISOString();
+    writeJSON(pipelinePath(), data);
+  },
+
+  phaseInProgress(phase, item) {
+    const data = this.getPipeline();
+    if (!data.phases[phase]) data.phases[phase] = { completed: [], in_progress: null };
+    data.phases[phase].in_progress = item;
+    data.updated_at = new Date().toISOString();
+    writeJSON(pipelinePath(), data);
+  },
+
+  setCurrentStory(story) {
+    this.setPipeline({ current_story: story });
+  },
+
+  setLastRun(runId) {
+    this.setPipeline({ last_run_id: runId });
+  },
+
+  // ── Selectors Memory ──
+
+  getSelectors() {
+    return readJSON(selectorsPath()) || [];
+  },
+
+  addSelector(page, element, selectorValue, options = {}) {
+    const selectors = this.getSelectors();
+    let entry = selectors.find(s => s.page === page && s.element === element);
+    if (!entry) {
+      entry = { page, element, selectors: [], recommended: selectorValue };
+      selectors.push(entry);
+    }
+    const existing = entry.selectors.find(s => s.value === selectorValue);
+    if (existing) {
+      existing.last_used = new Date().toISOString();
+      if (options.reliability !== undefined) existing.reliability = options.reliability;
+      if (options.healed) existing.healed = true;
+      if (options.original) existing.original = options.original;
+    } else {
+      entry.selectors.push({
+        value: selectorValue,
+        type: options.type || 'css',
+        reliability: options.reliability || 0.5,
+        healed: options.healed || false,
+        original: options.original || null,
+        strategy: options.strategy || null,
+        healed_at: options.healed_at || null,
+        run_id: options.run_id || null,
+        first_seen: new Date().toISOString(),
+        last_used: new Date().toISOString(),
+      });
+    }
+    entry.selectors.sort((a, b) => b.reliability - a.reliability);
+    entry.recommended = entry.selectors[0].value;
+    writeJSON(selectorsPath(), selectors);
+  },
+
+  getRecommendedSelector(page, element) {
+    const selectors = this.getSelectors();
+    const entry = selectors.find(s => s.page === page && s.element === element);
+    return entry ? entry.recommended : null;
+  },
+
+  updateSelectorReliability(page, element, selectorValue, succeeded) {
+    const selectors = this.getSelectors();
+    const entry = selectors.find(s => s.page === page && s.element === element);
+    if (!entry) return;
+    const sel = entry.selectors.find(s => s.value === selectorValue);
+    if (!sel) return;
+    sel.reliability = succeeded
+      ? Math.min(1, sel.reliability + 0.1)
+      : Math.max(0, sel.reliability - 0.2);
+    sel.last_used = new Date().toISOString();
+    entry.selectors.sort((a, b) => b.reliability - a.reliability);
+    entry.recommended = entry.selectors[0].value;
+    writeJSON(selectorsPath(), selectors);
+  },
+
+  // ── Healing History ──
+
+  getHealHistory() {
+    return readJSON(healHistoryPath()) || [];
+  },
+
+  addHealAttempt(runId, story, test, element, attempts, success) {
+    const history = this.getHealHistory();
+    history.push({
+      run_id: runId,
+      story,
+      test,
+      element,
+      original_selector: attempts[0] && attempts[0].original ? attempts[0].original : null,
+      failure_reason: attempts[0] && attempts[0].error || null,
+      attempts: attempts.map(a => ({
+        selector: a.selector,
+        strategy: a.strategy || 'retry',
+        success: a.success,
+        duration_ms: a.duration_ms || 0,
+      })),
+      success,
+      duration_ms: attempts.reduce((s, a) => s + (a.duration_ms || 0), 0),
+      timestamp: new Date().toISOString(),
+    });
+    writeJSON(healHistoryPath(), history);
+    return history;
+  },
+
+  getHealHistoryForStory(story) {
+    return this.getHealHistory().filter(h => h.story === story);
+  },
+
+  // ── Traceability ──
+
+  getTraceability() {
+    return readJSON(traceabilityPath()) || { stories: {} };
+  },
+
+  addRunId(story, runId, success, testName) {
+    const data = this.getTraceability();
+    if (!data.stories[story]) {
+      data.stories[story] = { plan: null, spec: null, runs: [], latest_heal: null, status: 'pending' };
+    }
+    data.stories[story].runs.push({ run_id: runId, success, test: testName || null, timestamp: new Date().toISOString() });
+    data.stories[story].status = success ? 'stable' : 'unstable';
+    writeJSON(traceabilityPath(), data);
+  },
+
+  addPlan(story, planFile) {
+    const data = this.getTraceability();
+    if (!data.stories[story]) data.stories[story] = { plan: null, spec: null, runs: [], latest_heal: null, status: 'pending' };
+    data.stories[story].plan = planFile;
+    writeJSON(traceabilityPath(), data);
+  },
+
+  addSpec(story, specFile) {
+    const data = this.getTraceability();
+    if (!data.stories[story]) data.stories[story] = { plan: null, spec: null, runs: [], latest_heal: null, status: 'pending' };
+    data.stories[story].spec = specFile;
+    writeJSON(traceabilityPath(), data);
+  },
+
+  setHealed(story, runId) {
+    const data = this.getTraceability();
+    if (!data.stories[story]) data.stories[story] = { plan: null, spec: null, runs: [], latest_heal: null, status: 'pending' };
+    data.stories[story].latest_heal = runId;
+    writeJSON(traceabilityPath(), data);
+  },
+
+  // ── Init ──
+
+  init() {
+    ensureDir(CONTEXT_DIR);
+    for (const file of ['pipeline.json', 'selectors.json', 'heal-history.json', 'traceability.json']) {
+      const fp = path.join(CONTEXT_DIR, file);
+      if (!fs.existsSync(fp)) {
+        const defaults = {
+          'pipeline.json': { project_path: '', phases: { plan: { completed: [], in_progress: null }, generate: { completed: [], in_progress: null }, execute: { completed: [], in_progress: null }, heal: { completed: [], in_progress: null }, report: { completed: [], in_progress: null } }, current_story: null, last_run_id: null, updated_at: '' },
+          'selectors.json': [],
+          'heal-history.json': [],
+          'traceability.json': { stories: {} },
+        };
+        writeJSON(fp, defaults[file]);
+      }
+    }
+  },
+
+  contextDir: CONTEXT_DIR,
+};
+
+module.exports = context;

package/scripts/executor.js

@@ -1,4 +1,5 @@
 const { DIRS, CONFIG, ensureDir, timestamp, log, writeMarkdown } = require('./utils');
+const context = require('./context-manager');
 const path = require('path');
 const fs = require('fs');
 const { execSync, spawn } = require('child_process');
@@ -10,15 +11,18 @@ function executeTests(testName, options = {}) {
     timeout = testCfg.timeout || 120000,
     retries = testCfg.retries || 0,
     workers = testCfg.workers || 1,
-    screenshots = 'off',
+    //screenshots = 'off',
   } = options;
 
   const runId = `run-${timestamp()}`;
   const runDir = path.join(DIRS.testResults, runId);
   ensureDir(runDir);
 
+  const storyName = testName ? testName.replace(/\.spec\.ts$/, '').split(/[\/\\]/).pop() : null;
+
   log('EXECUTOR', `Run ID: ${runId}`);
   log('EXECUTOR', `Test: ${testName || 'all tests'}`);
+  if (storyName) log('EXECUTOR', `Story: ${storyName}`);
 
   const args = ['playwright', 'test'];
 
@@ -37,12 +41,7 @@ function executeTests(testName, options = {}) {
   if (retries > 0) args.push('--retries', retries.toString());
   if (workers) args.push('--workers', workers.toString());
   if (timeout) args.push('--timeout', timeout.toString());
-  if (screenshots === 'off') {
-    args.push('--screenshot', 'off');
-  }
-  if (screenshots === 'only-on-failure') {
-    args.push('--screenshot', 'only-on-failure');
-  }
+  // Screenshots configured in playwright.config.ts — not a valid CLI arg
 
   const outputPath = path.join(runDir, 'execution-output.json');
   const resultPath = path.join(runDir, 'execution-result.json');
@@ -64,6 +63,7 @@ function executeTests(testName, options = {}) {
 
     const result = {
       runId,
+      story: storyName,
       test: testName || 'all',
       success: true,
       exitCode: 0,
@@ -75,6 +75,11 @@ function executeTests(testName, options = {}) {
     };
 
     writeMarkdown(resultPath, JSON.stringify(result, null, 2));
+    context.setLastRun(runId);
+    if (storyName) {
+      context.addRunId(storyName, runId, true, testName);
+      context.phaseComplete('execute', storyName);
+    }
     log('EXECUTOR', `Tests passed (${result.duration}ms)`);
 
     return result;
@@ -89,6 +94,7 @@ function executeTests(testName, options = {}) {
 
     const result = {
       runId,
+      story: storyName,
       test: testName || 'all',
       success: false,
       exitCode: err.status || 1,
@@ -100,6 +106,11 @@ function executeTests(testName, options = {}) {
     };
 
     writeMarkdown(resultPath, JSON.stringify(result, null, 2));
+    context.setLastRun(runId);
+    if (storyName) {
+      context.addRunId(storyName, runId, false, testName);
+      context.phaseComplete('execute', storyName);
+    }
     log('EXECUTOR', `Tests failed (${result.duration}ms) - ${failedTests.length} failure(s)`);
 
     return result;