opendevbrowser 0.0.27 → 0.0.29

package/skills/opendevbrowser-research/artifacts/research-workflows.md

@@ -1,29 +1,66 @@
 # Research Workflows
 
-## Deterministic baseline workflow
+Use these workflows to plan evidence first, run provider-constrained collection second, and publish only claims that pass review.
 
-1. Define topic and strict time window
-2. Resolve source selection (`auto|all|explicit`)
-3. Run research workflow command
-4. Persist context/report artifacts
-5. Validate source diversity and freshness
+## Evidence-gated baseline workflow
 
-## Cross-source corroboration workflow
+1. Define topic, scope, and strict time window.
+2. Choose explicit source families before invoking the CLI primitive. Prefer `--sources web,community` for broad public research when both destination pages and community corroboration are useful.
+3. Record why each source family is in scope and which families are intentionally excluded.
+4. Run `opendevbrowser research run` as a low-level best-effort primitive.
+5. Inspect candidates and follow useful destination pages before synthesis. Never treat account, privacy, cookie, login, consent, search shell, not-found, or JavaScript shell pages as final evidence.
+6. If a selected destination is blocked, record the blocker and continue to the next relevant candidate before escalating to cookies or authenticated browser recovery.
+7. Persist `summary.md`, `report.md`, `records.json`, `context.json`, `meta.json`, and `bundle-manifest.json`.
+8. Review `records.json`, `context.json`, and `meta.json` before using `report.md`.
+9. If the selected evidence page itself proves auth, token, challenge, or cookie gating after public candidate triage, rerun through user-authorized browser recovery only after relevant public destination pages are exhausted: `--browser-mode extension` for an existing signed-in relay session, `--use-cookies` only when legitimate cookies are available, and `--challenge-automation-mode browser_with_helper` for browser-scoped assistance.
+10. Publish supported claims, mark weak claims as tentative, and exclude unsupported claims.
 
-1. Identify claims from first-pass output
-2. Require at least two independent source records for critical claims
-3. Mark unsupported claims as tentative
-4. Escalate when corroboration is missing
+## Iterative destination-follow workflow
 
-## Backoff/retry workflow
+Use this when the first pass returns shells, login pages, privacy/cookie pages, or search result pages.
 
-1. Detect repeated 429/upstream throttling
-2. Honor retry windows and bounded retries
-3. Resume from persisted context artifact
-4. Report partial coverage when limits persist
+1. Build a candidate queue from provider output or search-engine discovery: URL, rank, source family, title, and blocker notes.
+2. Remove dead-end routes from the queue: `/login`, `/account`, `/submit`, privacy preference pages, cookie preference pages, search result pages, consent-only pages, static assets, and not-found URLs.
+3. Open the highest-value remaining destination page with OpenDevBrowser.
+4. Extract text, source URL, title, timestamp, and visible limitations.
+5. If extraction returns a shell or blocker, mark that candidate rejected and continue with the next candidate.
+6. If extraction succeeds, add it to the claim map and decide whether more evidence is needed for corroboration.
+7. Stop only when accepted evidence is sufficient or every relevant candidate has a recorded rejection reason.
+
+## Claim-to-source review workflow
+
+1. Extract the claims that the final answer would make.
+2. Map each claim to accepted destination evidence in `records.json`.
+3. Check source date, fetch date, provider, extraction quality, and source independence.
+4. Require corroboration for critical claims when the topic allows it.
+5. Record evidence gaps, stale pages, login walls, challenge pages, rate limits, and extraction limits.
+6. Do not use shell-only, stale-only, login-only, not-found-only, or zero-source-evidence runs as final support.
+
+## Search Engine Discovery Lane
+
+This optional lane is skill-guided, provider-constrained, and discovery-only. It is for richer candidate discovery, not for runtime source-family expansion.
+
+1. Choose up to five engines based on topic and availability: Google, Bing, Brave, DuckDuckGo or Yahoo, Yandex, Baidu, and Kagi only with user account access.
+2. Record search_engine_passes with engine, query, region, language, rationale, cookie or auth needs, and blockers.
+3. Collect up to 10 SERP candidate URLs per engine with engine, query, rank, URL, title if available, and retrieval notes.
+4. Dedupe canonical URLs across engines.
+5. Select the strongest 5 to 10 destination pages for extraction.
+6. Fetch selected destination pages with OpenDevBrowser browsing primitives when useful, including DOM interaction and screenshots. Use cookies or authenticated browsing only after destination-candidate triage shows the selected evidence page itself requires legitimate user-authorized access.
+7. Stand down on robots restrictions, login walls, consent gates, CAPTCHAs, rate limits, anti-bot controls, and access controls. Record the limitation instead.
+8. Keep SERPs discovery-only. SERP snippets, search result pages, shells, and blocked pages cannot be final evidence.
+9. Cite destination-page evidence or other fetched evidence that passed review.
+
+## Backoff and blocker workflow
+
+1. Detect repeated 429 responses, provider throttling, challenge pages, login walls, token gates, consent gates, cookie diagnostics, or extraction failures.
+2. Honor retry windows and bounded retries.
+3. For gated providers such as Reddit, prefer a skill-first recovery rerun with an existing signed-in relay session, legitimate cookies only when available, and browser-scoped challenge assistance only after concrete result URLs from the candidate queue have been attempted or rejected with reasons.
+4. Resume from persisted artifacts only when the next run can add evidence without violating controls.
+5. Report partial coverage, provider constraints, and skipped lanes when limits persist.
 
 ## Compact handoff workflow
 
-1. Produce compact summary with key findings
-2. Include unresolved risks and missing evidence
-3. Attach artifact paths for replay
+1. Produce a compact summary with accepted claims only.
+2. Include evidence gaps, provider constraints, and limitations.
+3. Attach artifact paths for replay and review.
+4. Name unsupported claims that were excluded or left tentative.

package/skills/opendevbrowser-research/assets/templates/compact.md

@@ -1,7 +1,33 @@
 # Compact Research Template
 
-- Top findings:
-- Source diversity:
-- Key risks:
-- Rate-limit/backoff notes:
-- Open evidence gaps:
+- Topic and timebox:
+- Source families used:
+- Evidence-gated status:
+- Accepted claims:
+- Tentative or excluded claims:
+- Evidence gaps:
+- Provider constraints:
+- Staleness checks:
+- Search engine discovery, if used:
+  - direction from providers/search:
+  - candidate_triage:
+  - rejected_candidates:
+  - rejected privacy/login/shell pages:
+  - replacement destination followed:
+  - remaining candidate queue:
+  - deep-dive pages:
+  - next query or evidence gap:
+  - search_engine_passes:
+  - serp_candidates reviewed:
+  - selected_destination_pages:
+  - engine_failures:
+  - provenance:
+- synthesis_feedback:
+- Limitations:
+- Artifact paths:
+  - summary.md:
+  - report.md:
+  - records.json:
+  - context.json:
+  - meta.json:
+  - bundle-manifest.json:

package/skills/opendevbrowser-research/assets/templates/context.json

@@ -1,8 +1,59 @@
 {
   "topic": "",
   "timebox": {
-    "days": 30
+    "days": 30,
+    "from": null,
+    "to": null
   },
+  "source_families": [],
+  "evidence_gate": {
+    "status": "pending_review",
+    "reviewed_artifacts": [],
+    "accepted_claims": [],
+    "tentative_claims": [],
+    "unsupported_claims": [],
+    "blockers": []
+  },
+  "artifact_files": [
+    "summary.md",
+    "report.md",
+    "records.json",
+    "context.json",
+    "meta.json",
+    "bundle-manifest.json"
+  ],
+  "source_ledger": [],
+  "search_direction_notes": [],
+  "candidate_triage_schema": {
+    "url": "",
+    "rank": 0,
+    "engine": "",
+    "query": "",
+    "source_family": "",
+    "title": "",
+    "status": "pending|accepted|rejected",
+    "blocker_notes": "",
+    "rejection_reason": "",
+    "replacement_url": "",
+    "retrieval_notes": "",
+    "extraction_status": "pending|fetched|blocked|shell|stale|irrelevant"
+  },
+  "candidate_triage": {
+    "accepted_destination_records": 0,
+    "rejected_shell_or_dead_end_candidates": 0
+  },
+  "rejected_candidates": [],
+  "deep_dive_pages": [],
+  "iteration_log": [],
+  "synthesis_feedback": "",
+  "staleness_checks": [],
+  "provider_constraints": [],
+  "search_engine_passes": [],
+  "serp_candidates": [],
+  "selected_destination_pages": [],
+  "engine_failures": [],
+  "provenance": [],
+  "limitations": [],
   "issue_ids": [
     "ISSUE-06",
     "ISSUE-09",

package/skills/opendevbrowser-research/assets/templates/report.md

@@ -1,9 +1,62 @@
 # Research Report
 
-## Executive Summary
+## Topic and Scope
 
-## Findings
+## Evidence Gate Status
 
-## Confidence and Gaps
+State whether the run passed evidence review. Do not publish shell-only, stale-only, login-only, not-found-only, or zero-source-evidence claims.
 
-## Sources
+## Claim Map
+
+| Claim | Status | Evidence | Notes |
+|---|---|---|---|
+|  | accepted, tentative, or excluded | destination URL or artifact record id |  |
+
+## Evidence
+
+Summarize accepted destination-page evidence from `records.json`, with source date, fetch date, provider, and extraction quality.
+
+## Confidence
+
+Explain confidence by claim, including corroboration quality and independence.
+
+## Provider Constraints and Limitations
+
+Record provider blockers, rate limits, login walls, challenge pages, stale pages, extraction limits, and unsupported claims.
+
+## Search Engine Discovery
+
+If used, summarize search_engine_passes, serp_candidates, selected_destination_pages, engine_failures, provenance, and limitations. SERPs are discovery-only and cannot be final evidence.
+
+## Search Direction
+
+Summarize what the provider or search-engine pass suggested and which follow-up direction was chosen.
+
+## Candidate Triage
+
+List useful destinations, duplicates, privacy or terms pages, login or account pages, blocked pages, shells, stale pages, and off-topic pages.
+
+## Rejected Candidates
+
+Record rejected_candidates with URL, source, reason, and whether another candidate replaced it.
+
+## Deep Dives
+
+Summarize deep_dive_pages opened after triage, including destination URL, accepted evidence, and remaining uncertainty.
+
+## Synthesis Feedback
+
+Explain how the evidence changed the research direction, what gaps remain, and what the next query or source should be if more work is needed.
+
+## Final Answer
+
+Provide only claims supported by reviewed destination evidence. Label unresolved points as tentative or omit them.
+
+## Artifact Paths
+
+- `summary.md`:
+- `report.md`:
+- `records.json`:
+- `context.json`:
+- `meta.json`:
+- `bundle-manifest.json`:

package/skills/opendevbrowser-research/examples/sample-input.json

@@ -1,6 +1,6 @@
 {
   "topic": "ai browser automation",
   "days": 30,
-  "sourceSelection": "auto",
+  "sources": ["web", "community"],
   "mode": "context"
 }

package/skills/opendevbrowser-research/examples/sample-output.md

@@ -1,4 +1,29 @@
 # Sample Output
 
-1. Example finding one
-2. Example finding two
+## Search Direction
+
+- Provider pass produced destination candidates; accepted evidence came from following usable destination pages before considering browser recovery.
+
+## Candidate Triage
+
+- Accepted: `https://example.com/browser-automation-evidence`
+- Rejected: `https://example.com/login` as login_or_account; replacement destination followed: `https://example.com/browser-automation-evidence`
+- Rejected: `https://example.com/privacy/choices` as privacy_or_terms; replacement destination followed: `https://example.com/browser-automation-evidence`
+- Remaining candidate queue: `https://example.org/challenge-recovery-field-notes`
+
+## Deep Dives
+
+- Opened `https://example.com/browser-automation-evidence`; extracted destination-page evidence, blocker notes, and artifact review guidance.
+
+## Synthesis Feedback
+
+- The first accepted page shifted the next query toward challenge recovery handoff evidence.
+- Next query or evidence gap: find corroborating field notes from an independent browser automation source.
+
+## Evidence
+
+- Accepted claim: Browser automation research needs destination-page evidence, blocker notes, and artifact review before synthesis.
+
+## Limitations
+
+- Reddit evidence was excluded when only login or shell pages were available.

package/skills/opendevbrowser-research/scripts/run-research.sh

@@ -2,24 +2,20 @@
 set -euo pipefail
 
 if [[ $# -lt 1 ]]; then
-  echo "Usage: run-research.sh <topic> [days] [mode] [source-selection] [sources]"
+  echo "Usage: run-research.sh <topic> [days] [mode] [sources]"
   exit 1
 fi
 
 TOPIC="$1"
 DAYS="${2:-30}"
 MODE="${3:-context}"
-SOURCE_SELECTION="${4:-}"
-SOURCES="${5:-}"
+SOURCES="${4:-}"
 
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 # shellcheck source=../../opendevbrowser-best-practices/scripts/resolve-odb-cli.sh
 source "$script_dir/../../opendevbrowser-best-practices/scripts/resolve-odb-cli.sh"
 
 cmd=("${ODB_CLI[@]}" research run --topic "$TOPIC" --days "$DAYS" --mode "$MODE")
-if [[ -n "$SOURCE_SELECTION" ]]; then
-  cmd+=(--source-selection "$SOURCE_SELECTION")
-fi
 if [[ -n "$SOURCES" ]]; then
   cmd+=(--sources "$SOURCES")
 fi

package/skills/opendevbrowser-research/scripts/validate-skill-assets.sh

@@ -15,6 +15,65 @@ required=(
   "scripts/validate-skill-assets.sh"
 )
 
+posture_files=(
+  "SKILL.md"
+  "artifacts/research-workflows.md"
+  "assets/templates/compact.md"
+  "assets/templates/context.json"
+  "assets/templates/report.md"
+)
+
+required_markers=(
+  "evidence-gated"
+  "provider-constrained"
+  "discovery-only"
+  "candidate_triage"
+  "rejected_candidates"
+  "deep_dive_pages"
+  "synthesis_feedback"
+  "search_engine_passes"
+  "summary.md"
+  "report.md"
+  "records.json"
+  "context.json"
+  "meta.json"
+  "bundle-manifest.json"
+)
+
+required_file_markers=(
+  "SKILL.md|Search Engine Discovery Lane"
+  "SKILL.md|Guided Research Loop"
+  "SKILL.md|privacy"
+  "SKILL.md|login"
+  "SKILL.md|Keep SERPs discovery-only"
+  "SKILL.md|bundle-manifest.json"
+  "artifacts/research-workflows.md|Search Engine Discovery Lane"
+  "artifacts/research-workflows.md|Iterative destination-follow workflow"
+  "artifacts/research-workflows.md|candidate queue"
+  "artifacts/research-workflows.md|Keep SERPs discovery-only"
+  "artifacts/research-workflows.md|search_engine_passes"
+  "assets/templates/context.json|bundle-manifest.json"
+  "assets/templates/context.json|rejected_candidates"
+  "assets/templates/context.json|deep_dive_pages"
+  "assets/templates/context.json|search_engine_passes"
+  "assets/templates/context.json|blocker_notes"
+  "assets/templates/context.json|retrieval_notes"
+  "assets/templates/report.md|SERPs are discovery-only"
+  "assets/templates/report.md|Deep Dives"
+  "assets/templates/report.md|Synthesis Feedback"
+  "assets/templates/report.md|bundle-manifest.json"
+  "assets/templates/compact.md|deep-dive pages"
+)
+
+forbidden_markers=(
+  "research_reliable"
+  "auto is the recommended default"
+  "generic topical research is currently safest"
+  "privacy pages can support final claims"
+  "login pages can support final claims"
+  "SERP snippets are evidence"
+)
+
 status=0
 validator_cli="$root/../opendevbrowser-best-practices/scripts/validator-fixture-cli.sh"
 export ODB_CLI_VALIDATOR_OVERRIDE="$validator_cli"
@@ -28,6 +87,44 @@ require_marker() {
     status=1
   fi
 }
+
+require_file_marker() {
+  local marker="$1"
+  local found=0
+  local rel
+  for rel in "${posture_files[@]}"; do
+    if [[ -f "$root/$rel" ]] && grep -Fq "$marker" "$root/$rel"; then
+      found=1
+      break
+    fi
+  done
+  if [[ $found -eq 0 ]]; then
+    echo "Research assets missing required marker: $marker" >&2
+    status=1
+  fi
+}
+
+require_marker_in_file() {
+  local entry="$1"
+  local rel="${entry%%|*}"
+  local marker="${entry#*|}"
+  if [[ ! -f "$root/$rel" ]] || ! grep -Fq "$marker" "$root/$rel"; then
+    echo "Research asset $rel missing required marker: $marker" >&2
+    status=1
+  fi
+}
+
+reject_forbidden_marker() {
+  local marker="$1"
+  local rel
+  for rel in "${posture_files[@]}"; do
+    if [[ -f "$root/$rel" ]] && tr -d '`' < "$root/$rel" | grep -Fq "$marker"; then
+      echo "Research assets contain forbidden marker in $rel: $marker" >&2
+      status=1
+    fi
+  done
+}
+
 for rel in "${required[@]}"; do
   if [[ ! -f "$root/$rel" ]]; then
     echo "Missing required asset: $rel" >&2
@@ -59,6 +156,18 @@ for rel in scripts/run-research.sh scripts/render-output.sh scripts/write-artifa
   fi
 done
 
+for marker in "${required_markers[@]}"; do
+  require_file_marker "$marker"
+done
+
+for marker in "${required_file_markers[@]}"; do
+  require_marker_in_file "$marker"
+done
+
+for marker in "${forbidden_markers[@]}"; do
+  reject_forbidden_marker "$marker"
+done
+
 if [[ -f "$root/assets/templates/context.json" ]]; then
   if ! node -e 'const fs=require("fs"); JSON.parse(fs.readFileSync(process.argv[1], "utf8"));' "$root/assets/templates/context.json" >/dev/null 2>&1; then
     echo "Invalid JSON template: assets/templates/context.json" >&2
@@ -66,13 +175,18 @@ if [[ -f "$root/assets/templates/context.json" ]]; then
   fi
 fi
 
-context_output="$("$root/scripts/run-research.sh" "ai browser automation" 30 context auto "web,docs")"
+context_output="$("$root/scripts/run-research.sh" "ai browser automation" 30 context "web,docs")"
 require_marker "run-research context" "$context_output" "# Research Context"
 require_marker "run-research context" "$context_output" "ISSUE-09"
 
 report_output="$("$root/scripts/render-output.sh" "ai browser automation" report)"
 require_marker "render-output report" "$report_output" "# Research Report"
 require_marker "render-output report" "$report_output" "Source selection"
+require_marker "render-output report" "$report_output" "Search Direction"
+require_marker "render-output report" "$report_output" "Candidate Triage"
+require_marker "render-output report" "$report_output" "Rejected Candidates"
+require_marker "render-output report" "$report_output" "Deep Dives"
+require_marker "render-output report" "$report_output" "Synthesis Feedback"
 
 tmpdir="$(mktemp -d)"
 trap 'rm -rf "$tmpdir"' EXIT