@jiroamato/pstack 0.0.0-stage → 0.15.15

package/pstack/automations/benny/skills/reproduce-and-fix-issues/SKILL.md

@@ -0,0 +1,313 @@
+---
+name: reproduce-and-fix-issues
+description: Reproduce triaged Slack bugs through a configured app-control adapter, verify existing fixes, and open a bounded draft pull request only after before-and-after proof. Use only from the configured Benny repro automation.
+disable-model-invocation: true
+---
+
+# Reproduce and fix issues
+
+Wait for a trusted triage marker in the source thread. Reproduce the exact symptom through the target app's real UI. Verify an existing fix when one exists. Attempt a bounded fix only after a confirmed repro.
+
+Load the external Benny configuration supplied by the automation. If the config, required actions, control adapter, or completed feature map is missing, fail closed.
+
+## Hard safety rules
+
+- Freeze the source channel and root thread coordinates before doing any work.
+- Never post a root message in the source channel.
+- Preflight the source parent before every source-thread post.
+- The coordinator is the only Slack poster.
+- Delegated analysis workers are read-only and return findings or media notes.
+- A fix-phase code worker may edit only when its environment provably excludes Slack credentials and every Slack write action. Otherwise the coordinator edits.
+- Every child prompt must explicitly forbid `SendSlackMessage`, `PostToSlack`, `chat.postMessage`, and all other Slack writes.
+- Never give a child a Slack token, posting instructions, source coordinates for posting, or permission to report externally.
+- If a child needs Slack write access to run, do not launch it.
+- Utility bots are evidence sources. They do not own the fix unless a person explicitly delegated the fix to them.
+- The exact discriminating symptom must appear twice through real UI interaction.
+- State inspection may confirm an observation. It must not inject or force the symptom.
+- No confirmed repro means no authored fix.
+- Existing pull requests or commits switch the run to verify mode. Do not author over them.
+- Use `github.com` pull request links.
+- Keep captures, recordings, logs, and tokens out of source control.
+- Use pstack's `principle-guard-the-context-window` for delegated analysis.
+- Apply pstack's `principle-sequence-verifiable-units`, `principle-fix-root-causes`, and `principle-prove-it-works` through repro, fix, and verification.
+
+## 1. Freeze source coordinates
+
+Before making a work list or delegating:
+
+1. Require the trigger channel to equal the configured source channel.
+2. Set `SOURCE_THREAD_TS` to `trigger.thread_ts` when present. Otherwise use `trigger.ts`.
+3. Require a nonempty `SOURCE_THREAD_TS`.
+4. Store `SOURCE_CHANNEL_ID` and `SOURCE_THREAD_TS` as immutable values.
+5. Read the source thread and verify its root has those exact coordinates.
+6. Fetch the source permalink.
+
+Never replace these values with a reply timestamp, operations timestamp, or status-message timestamp.
+
+Before every source-channel post:
+
+1. Read the thread by the immutable coordinates.
+2. Confirm the parent exists, is not deleted, and still belongs to the source channel.
+3. Send only with `channel=SOURCE_CHANNEL_ID` and `thread_ts=SOURCE_THREAD_TS`.
+4. Read the thread again and verify the new message is a reply.
+
+If any check fails, post nothing. Never retry at the root or in a fallback channel.
+
+## 2. Wait for the triage contract
+
+Watch the source thread for the configured verdict budget. Stay silent while waiting.
+
+Accept a verdict only when:
+
+- Its author matches `slack.triage_identity_user_id`.
+- It is a reply under `SOURCE_THREAD_TS`.
+- It contains exactly one configured marker.
+
+Public marker forms:
+
+```text
+[benny:bug]
+[benny:bug] tracker=https://tracker.example/issue/123
+[benny:performance]
+[benny:performance] tracker=https://tracker.example/issue/123
+[benny:other]
+```
+
+Proceed only for `bug` or `performance`. Capture the optional tracker URL. Stop silently for `other`, a missing verdict, an untrusted author, conflicting markers, or a timeout.
+
+This marker replaces private bot identities and free-form verdict matching.
+
+## 3. Apply ownership and fix-artifact gates
+
+Re-read the thread immediately before starting work.
+
+### Someone is explicitly fixing it
+
+Stop when a person clearly claims the fix, gives a concrete implementation plan, or asks another agent to implement, patch, fix, or open a pull request.
+
+Do not treat these as fix ownership:
+
+- A bot summarizes evidence.
+- A tool looks up logs or tickets.
+- Someone asks a bot to diagnose, explain, inspect, or reproduce.
+- A bot posts a cause hypothesis without agreeing to implement it.
+
+Judge the requested action, not the presence of a bot.
+
+### A fix artifact already exists
+
+If an open pull request or merged commit plausibly fixes this report, switch to `references/verify-existing-fix.md`.
+
+An artifact may come from the thread, tracker issue, repository history, or pull request search. A claim without a commit or pull request is not a fix artifact.
+
+If a person owns the work but has not produced an artifact, stop. Do not race them.
+
+## 4. Open an optional operations thread
+
+If `slack.operations_channel_id` is configured, the coordinator may create one root status message there. This is the only allowed root post in the repro workflow.
+
+Store its coordinates as `OPERATIONS_CHANNEL_ID` and `OPERATIONS_THREAD_TS`. Never confuse them with the source coordinates.
+
+Use the configured plain Unicode status strings. Keep status text short:
+
+- Reproducing
+- Could not reproduce
+- Blocked
+- Reproduced
+- Verifying existing fix
+- Attempting bounded fix
+- Draft pull request opened
+- Fix did not land
+
+Prefer the Slack actions your harness provides (a Slack MCP server or channel). Use `BENNY_SLACK_BOT_TOKEN` only when the user configured it for a narrow missing capability such as editing this one status message. Never expose the token to a worker.
+
+If no operations channel is configured, keep detailed status in the automation run output. Do not substitute a source-channel root message.
+
+## 5. Load and check the control adapter
+
+Read `references/control-adapter.md` and the completed map at `control.feature_map_path`, then invoke the skill named by `control.skill_name`.
+
+For web, IDE, or Electron, load bundled pstack `control-ui` via the `pstack-harness` **skill** row before driving the app. The configured adapter supplies the app-specific launch steps, selectors, and feature map; it is not a separate generic `control` skill. Other UI platforms need their configured platform driver.
+
+Find the feature-map section that matches the reported user path. Read it before driving the app. If no section covers the feature, mark the run blocked instead of inventing a path or selector.
+
+Require all seven capabilities:
+
+1. Bring up the configured target app and test environment.
+2. Navigate the mapped feature and exercise its documented states.
+3. Drive the real UI with clicks, typing, keys, scrolling, drag, resize, or navigation.
+4. Inspect state without mutating it.
+5. Capture screenshots.
+6. Start and stop a screen recording.
+7. Clean up processes, sessions, profiles, and temporary data.
+
+If the adapter is absent or any required capability is missing, mark the operations status as blocked and stop. Do not pretend a screenshot, unit test, state mutation, or source reading is a UI repro.
+
+## 6. Study the report
+
+Read the full source thread and tracker issue when present.
+
+Collect:
+
+- Exact action path
+- Expected behavior
+- Observed behavior
+- Discriminating state where they diverge
+- Frequency
+- Version, environment, and platform
+- Attachments and error signatures
+- Candidate code area
+
+Inspect screenshots and video. Use read-only parallel workers for code history, test ideas, blast-radius mapping, and media review when useful. Each worker gets a narrow question and the Slack-write prohibition.
+
+Use pstack's `how` skill to trace the action through the repository. Use `why` for regression history and defensive code. Form competing cause hypotheses and identify evidence that would separate them.
+
+## 7. Reproduce
+
+Bring up the target app through the control adapter.
+
+Confirm the correct app, workspace, account, data set, and feature state before acting. Use stable app markers. Do not rely on window order or a familiar title alone.
+
+Drive the reported path through real UI actions.
+
+Before calling it reproduced:
+
+1. Name the correct final state.
+2. Name the broken final state.
+3. Reach the point where they diverge.
+4. Observe the broken state.
+5. Reset enough state to make the second attempt independent.
+6. Repeat the same path and observe the same broken state again.
+7. Cross-check a real state value when possible.
+
+An expected dialog, loading state, or setup step is not the bug. Capture the final state that distinguishes correct from broken behavior.
+
+Use the configured repro budget. If the symptom does not reproduce within it, report a clean `Could not reproduce` outcome. If the environment cannot provide a required capability, report `Blocked` and state what was missing.
+
+## 8. Capture and review evidence
+
+For a successful repro:
+
+- Record the full path through the symptom.
+- Capture a screenshot of the broken final state.
+- Save a short note with the exact steps and observed state.
+- Keep artifacts in the configured temporary artifact directory.
+
+Have a read-only media reviewer answer one question: does the evidence visibly show the discriminating broken state?
+
+If the answer is no or uncertain, the repro is not confirmed. Capture better evidence or use `Could not reproduce`.
+
+Post detailed evidence only in the operations thread when configured. Keep the source update concise.
+
+## 9. Report the repro outcome
+
+Update the operations status first.
+
+For `Could not reproduce` or `Blocked`, post nothing in the source thread. The operations thread or run output carries the result.
+
+For a confirmed repro, run the source preflight and post at most one unprompted source reply:
+
+- Say the issue reproduced.
+- Link the operations evidence thread when one exists.
+- Include at most three short findings.
+- Link the tracker issue when one exists.
+- Do not ping an owner by default.
+
+Attach evidence only when the configured Slack action keeps it inside the same source thread and the organization's retention policy allows it.
+
+Wait for the configured rejection window. If a person shows that the setup or interpretation was wrong, correct the repro once. Do not start the fix phase until the window closes without a valid rejection.
+
+## 10. Verify an existing fix
+
+When a fix artifact exists, follow `references/verify-existing-fix.md`.
+
+Verification must show the symptom on the baseline and its absence on the patched build. Both paths use the real UI twice.
+
+Do not edit the existing fix, add a competing patch, or open a replacement pull request.
+
+## 11. Qualify a bounded fix
+
+Attempt a fix only when all of these hold:
+
+- The outcome is a plain confirmed repro.
+- Media review confirmed the broken final state.
+- No existing fix artifact appeared.
+- No person claimed the fix during the rejection window.
+- Runtime evidence identifies the root cause.
+- The likely change fits the configured fix budget and repository scope.
+- The control adapter can run both baseline and patched builds.
+
+If any condition fails, keep the repro report and stop without a pull request.
+
+When the gate passes, update operations status to `Attempting bounded fix`.
+
+## 12. Root-cause and implement
+
+The coordinator owns every Slack post, the final diff review, commits, and the pull request.
+
+Read-only workers may:
+
+- Trace code and history
+- Propose tests
+- Map blast radius
+- Review a diff
+- Review media
+
+They do not edit, run external writes, post status, or own the fix.
+
+A tightly scoped code edit may be delegated during this phase only when tool isolation removes Slack credentials and every Slack write action from that worker. Its prompt must still carry the explicit Slack-write ban. The coordinator reviews the edit and runs or verifies the required tests. If tool isolation is uncertain, keep the edit in the coordinator.
+
+Confirm the mechanism with runtime evidence. Eliminate competing hypotheses before editing.
+
+Fix the root cause with the smallest justified change.
+
+- Invoke pstack's `tdd` skill when there is a cheap local test target, and write the failing test before the fix.
+- State why TDD was skipped when the path is expensive, unclear, or integration-heavy.
+- Keep unrelated cleanup out.
+- Stop if the change grows beyond the configured effort or risk budget.
+
+## 13. Prove the fix
+
+Keep the original baseline evidence.
+
+On the patched build:
+
+1. Run the same real UI path.
+2. Repeat it twice.
+3. Show that the broken state is gone.
+4. Show the expected state in its place.
+5. Capture an after recording and screenshot.
+6. Cross-check the same real state value used for the baseline.
+
+A compile, unit test, code review, or plausible diff is not after evidence.
+
+Run focused tests, then smoke the blast radius around the changed behavior. Cover nearby states, inputs, permissions, platforms, and failure paths that the change could affect. Stop without a pull request if a regression remains.
+
+## 14. Open a draft pull request
+
+Only after before-and-after proof:
+
+- Review the final diff for unrelated changes and secrets.
+- Load bundled pstack `deslop` per the `pstack-harness` **deslop** row before any fix commit. If cleanup changes code, rerun affected tests and the mapped UI repro through the configured adapter, with `control-ui` for web/IDE/Electron, before opening the pull request.
+- Run the repository's required checks.
+- Create small ordered commits when the repository workflow allows it.
+- Open a draft pull request. Never merge or deploy from this workflow.
+- Link the configured tracker issue using the tracker's supported pull request syntax.
+- Use the configured public URL form, normally `https://github.com/{owner}/{repo}/pull/{number}`.
+- Include the repro steps, root cause, test result, before and after evidence, and blast-radius checks.
+- Run the pull request text and all Slack updates through pstack's `unslop` skill.
+
+If pull request creation fails, do not claim success. Keep the commit or branch state in the run output and mark operations status `Fix did not land`.
+
+On success, mark operations status `Draft pull request opened` and post one concise reply in the operations thread with the linked pull request. Do not create a second source-channel root or unprompted source reply.
+
+## 15. Follow-ups and cleanup
+
+Watch the configured operations thread for one follow-up window.
+
+- Answer a direct question from evidence already gathered.
+- Apply one concrete correction and rerun the repro once when it invalidates the setup.
+- Stay out of human coordination and side chatter.
+- Stop when asked.
+
+Always call the control adapter's cleanup capability. Keep artifacts only as long as the configured retention policy allows.

package/pstack/automations/benny/skills/reproduce-and-fix-issues/references/control-adapter.md

@@ -0,0 +1,169 @@
+# Control-adapter contract
+
+Benny does not know how to start or drive every app. Configure an app-specific verification skill or adapter that implements this contract. For web, IDE, or Electron driving, that skill loads bundled pstack `control-ui` and supplies the app-specific launch commands, selectors, and feature map. There is no separate generic `control` skill. The generic `control-ui` skill alone does not supply all of Benny's app-specific capabilities.
+
+Set its skill name in `control.skill_name`.
+
+Set the completed user-facing feature map path in `control.feature_map_path`. Copy and fill [`feature-map.example.md`](./feature-map.example.md) outside `.pstack/automations/benny/` instead of editing the copied example.
+
+If the skill, feature map, or a required capability is absent, ambiguous, or incomplete, repro and fix work must fail closed.
+
+## Required capabilities
+
+### Bring up
+
+Start the requested app revision in the requested test environment.
+
+Input:
+
+- Repository and revision
+- Build or start mode
+- Workspace, account, fixture, and feature-state requirements
+- Artifact directory
+- Completed feature-map path
+
+Return:
+
+- Session identifier
+- How the adapter confirmed the correct app and environment
+- Stable app markers
+- Running process or target details needed by later calls
+- Any missing capability
+
+The adapter must distinguish the target app from a similar window, shell, or production instance.
+
+### Drive UI
+
+Perform real user actions:
+
+- Click
+- Type
+- Press keys
+- Scroll
+- Drag
+- Resize
+- Navigate through app controls
+
+Prefer roles, labels, and stable selectors. Use coordinates only after a fresh screenshot.
+
+Return each action and the observed state change.
+
+Do not set internal state, call hidden app methods, write directly to storage, or inject DOM changes to create the symptom.
+
+### Drive mapped features and states
+
+Read the relevant feature-map section before driving the app.
+
+The adapter must expose ways to:
+
+- Navigate every mapped feature through the user-visible path.
+- Invoke the adapter action names listed for that feature.
+- Interact with default, hover, focus-visible, active, disabled, loading, empty, error, selected, open, expanded, and feature-specific states when they apply.
+- Arrange a state through safe fixture data, permissions, flags, service responses, or supported test controls.
+- Reset the feature for a second independent repro attempt.
+- Capture the screenshot, video, and read-only cross-check named by the feature map.
+
+Use roles, accessible names, ARIA relationships, stable component markers, and purpose-named data attributes. Never use generated CSS or StyleX classes, dynamic hashes, child indexes, or brittle DOM position.
+
+Arranging a precondition is not permission to inject the reported symptom. The repro itself must still come from real user interaction.
+
+### Inspect state
+
+Read state to confirm what the UI shows.
+
+Examples:
+
+- Accessibility tree
+- DOM or view hierarchy
+- Process state
+- Local logs
+- Network request status
+- App-exposed debug state
+
+Inspection is read-only. If a query changes state, it belongs in `drive UI` and must represent a real user action.
+
+### Screenshot
+
+Capture the current app state to a requested path.
+
+Return:
+
+- File path
+- Capture time
+- App marker or window title
+- Short description of what should be visible
+
+The screenshot must show enough app chrome to prove that the correct app is under test.
+
+### Recording
+
+Start and stop a screen recording around the full repro path.
+
+Return:
+
+- File path
+- Start and stop times
+- Captured window or region
+- Whether audio or sensitive overlays were omitted
+
+The recording must show the discriminating final state, not only setup or a loading screen.
+
+### Cleanup
+
+Stop processes and sessions created by the adapter.
+
+Remove disposable:
+
+- Browser or app profiles
+- Temporary workspaces
+- Test accounts or fixtures when the adapter created them
+- Debug ports and tunnels
+- Captures past their retention window
+
+Return what was stopped, removed, retained, or left for a person.
+
+Cleanup must not delete user work.
+
+## Adapter behavior
+
+The adapter must:
+
+- Report capabilities before the repro starts.
+- Report which feature-map sections it can drive and which are blocked.
+- Use the same environment inputs for baseline and patched builds.
+- Surface startup failures as failures.
+- Bound retries.
+- Keep secrets out of logs and artifacts.
+- Keep captures outside the repository.
+- Support a fresh or reset state between the two repro attempts.
+- Avoid production changes unless the user explicitly configured a safe test action.
+
+## Environment translation
+
+Before declaring an environment block, restate the defect without platform-specific nouns and ask whether the same behavior can be tested safely in the available environment.
+
+Examples:
+
+- A named browser may mean any external browser.
+- A named key may mean the configured shortcut.
+- A named remote host may mean a delayed or disconnected remote target.
+
+Use a translated attempt only when it tests the same underlying behavior. Label it as translated evidence. Do not call it an exact repro when the missing environment is part of the defect.
+
+Hardware prompts, operating-system permission dialogs, device-only APIs, and unavailable account states may be real blocks.
+
+## Setup check
+
+Before enabling the repro automation, run one harmless adapter check:
+
+1. Bring up the app.
+2. Confirm the stable app marker.
+3. Load one completed feature-map section.
+4. Navigate to that feature through its user path.
+5. Exercise one disposable state through mapped adapter actions.
+6. Inspect the resulting state.
+7. Capture a screenshot.
+8. Record a short clip.
+9. Clean up.
+
+Enable repro work only when all nine steps succeed and no source-channel Slack post is involved.

package/pstack/automations/benny/skills/reproduce-and-fix-issues/references/feature-map.example.md

@@ -0,0 +1,205 @@
+# Feature-map example
+
+Map every user-facing feature Benny may reproduce. Read the relevant section before driving the app. Keep this map at the user point of view. Discover internals and current code paths at runtime instead of freezing them here.
+
+Copy this file outside `.pstack/automations/benny/`, for example to `.pstack/benny/feature-map.md`, and set `control.feature_map_path` to the copy. Pack refreshes must not overwrite it.
+
+## Per-feature template
+
+### `<feature name>`
+
+`<one-line user-visible purpose>`
+
+#### How a user gets there
+
+- Click path: `<screen> -> <menu, tab, or panel> -> <control>`
+- Keyboard shortcut: `<shortcut or none>`
+
+#### How the control adapter drives it
+
+- `<adapter action>` with `<inputs>` should `<visible result>`.
+- Reset: `<how the adapter returns to a fresh state>`.
+
+#### Stable selectors
+
+- `<role and accessible name>`
+- `<ARIA relationship>`
+- `<data-component or purpose-named data attribute>`
+
+Never use generated CSS or StyleX classes, dynamic hashes, child indexes, or brittle DOM position.
+
+#### States to exercise
+
+- Default, hover, focus-visible, active, disabled
+- Loading, empty, error
+- Selected, open, expanded
+- `<relevant feature-specific variants>`
+
+Mark states that do not apply.
+
+#### Preconditions and setup
+
+- Auth: `<account state>`
+- Data: `<fixture>`
+- Permissions: `<role>`
+- Flags: `<flag or none>`
+- Services: `<required availability>`
+
+#### Evidence and cross-check
+
+- Screenshot: `<app identity, feature, and discriminating state>`
+- Video: `<entry path, interaction, and final state>`
+- Cross-check: `<read-only state or value that confirms the UI>`
+
+#### Gotchas
+
+- `<known dead end or wrong surface>`
+- `<safe environment translation>`
+
+## Fictional example
+
+These features belong to a fictional task app. They are examples, not required Benny features.
+
+### Sign in
+
+Lets a user enter the task app.
+
+#### How a user gets there
+
+- Open the app and choose `Sign in`. No shortcut.
+
+#### How the control adapter drives it
+
+- `open_app`, `click Sign in`, `fill credentials`, and `click Continue` should open the item list.
+- Reset by signing out and clearing the disposable session.
+
+#### Stable selectors
+
+- Button `Sign in`, textboxes `Email` and `Password`, `data-component="sign-in-form"`
+
+#### States to exercise
+
+- Default, focus-visible, submitting, disabled, loading, error
+
+#### Preconditions and setup
+
+- Disposable account and available authentication service
+
+#### Evidence and cross-check
+
+- Record landing page through item list. Check read-only session state.
+
+#### Gotchas
+
+- A marketing page is the wrong surface. A missing auth service is a block.
+
+### Item list and detail
+
+Lets a user browse items and open one.
+
+#### How a user gets there
+
+- Open the `Items` tab, then choose a row.
+
+#### How the control adapter drives it
+
+- `select_tab Items` and `click <fixture item>` should open its detail.
+- Reset by closing the detail and clearing selection.
+
+#### Stable selectors
+
+- Tab and list named `Items`, fixture-named row, `data-component="item-detail"`
+
+#### States to exercise
+
+- Loading, empty, error, selected, open, expanded
+
+#### Preconditions and setup
+
+- Named fixture items, read permission, available item service
+
+#### Evidence and cross-check
+
+- Show selection and matching detail title. Check selected-item ID.
+
+#### Gotchas
+
+- Search results may look similar but use a different path.
+
+### Item editor
+
+Lets a user create or edit an item.
+
+#### How a user gets there
+
+- Choose `Edit` from detail or `New item` from the list.
+
+#### How the control adapter drives it
+
+- `click Edit`, `fill <field>`, and `click Save` should update detail.
+- Reset by restoring the fixture.
+
+#### Stable selectors
+
+- Buttons `Edit`, `New item`, `Save`, form `Item editor`, label-linked fields
+
+#### States to exercise
+
+- Default, focus-visible, dirty, validating, disabled, saving, error, success
+
+#### Preconditions and setup
+
+- Editable fixture, write permission, available save service
+
+#### Evidence and cross-check
+
+- Show field change through updated detail. Check the stored item value read-only.
+
+#### Gotchas
+
+- Do not inject form state. A read-only detail field is not the editor.
+
+### Settings
+
+Lets a user change personal preferences.
+
+#### How a user gets there
+
+- Open the profile menu, then choose `Settings`.
+
+#### How the control adapter drives it
+
+- `open_menu Profile`, `click Settings`, and `toggle <preference>` should update the control.
+- Reset by restoring the starting preference.
+
+#### Stable selectors
+
+- Button `Profile`, menu item `Settings`, region `Settings`, purpose-named preference attribute
+
+#### States to exercise
+
+- Closed, open, selected, focus-visible, disabled, loading, error
+
+#### Preconditions and setup
+
+- Signed-in test account, known preferences, available preference service
+
+#### Evidence and cross-check
+
+- Show the menu path and final control state. Check the preference value read-only.
+
+#### Gotchas
+
+- Operating-system settings are a different surface.
+
+## Completeness checklist
+
+- Every reproducible user-facing feature has a section.
+- Every section names a user path, adapter actions, and reset.
+- Selectors use roles, names, ARIA, stable component markers, or purpose-named attributes.
+- No selector uses generated classes or DOM position.
+- Relevant interaction, loading, empty, error, selected, and expanded states are covered.
+- Auth, fixtures, permissions, flags, and services are explicit.
+- Screenshot, video, and underlying cross-check requirements are explicit.
+- Wrong surfaces, dead ends, and safe environment translations are listed.
+- Implementation details remain runtime discoveries.