oh-my-opencode 4.19.4 → 5.0.0-beta.10

package/packages/shared-skills/skills/ultimate-browsing/SKILL.md

@@ -37,7 +37,7 @@ Read the matching reference before acting: [`references/insane-search/README.md`
 ## Tier 1 — insane-search (headless extraction)
 
 **When**: content extraction, blocked-URL bypass, media metadata — no browser UI needed.
-**Why first**: ~10x faster than a browser, no process spin-up; handles most "fetch this blocked page" requests via curl_cffi TLS impersonation, yt-dlp (1858 sites), Jina Reader, official public APIs, mobile URL transforms, and a Playwright real-Chrome fallback. The engine lives **inside this skill** at `engine/` and is invoked as a module.
+**Why first**: ~10x faster than a browser, no process spin-up; handles most "fetch this blocked page" requests via curl_cffi TLS impersonation, yt-dlp (1858 sites), official public APIs, mobile URL transforms, **Phase-2.5 surrogate archives** (Wayback / archive.today snapshots, provenance-tagged — see [`references/insane-search/cache-archive.md`](references/insane-search/cache-archive.md)), a key-gated Jina Reader (`JINA_API_KEY`), and a Playwright real-Chrome fallback. The engine lives **inside this skill** at `engine/` and is invoked as a module. Surrogate results are dated COPIES: a result whose `provenance` is `snapshot` must be reported with its `snapshot_timestamp`, never presented as the live page.
 
 ```bash
 # Core command — auto-detects WAF, runs the full fetch grid (run from the skill dir):
@@ -60,7 +60,7 @@ The full engine harness (rules R1-R7, the Phase 0 official-API index, the no-sit
 
 ## Tier 1.5 — agent-reach (platform-native readers)
 
-**When**: the target is a platform with a first-class API/CLI that beats generic fetching — especially Chinese platforms that stealth browsers still cannot reach cleanly. Several channels are zero-config (Douyin, Weibo via Jina, V2EX, Reddit, Jina Reader, RSS, YouTube); others need a one-time auth you supply via environment variables if you have access.
+**When**: the target is a platform with a first-class API/CLI that beats generic fetching — especially Chinese platforms that stealth browsers still cannot reach cleanly. Several channels are zero-config (Douyin, V2EX, Reddit, RSS, YouTube); others need a one-time auth you supply via environment variables if you have access (`JINA_API_KEY` for Jina Reader — anonymous access is dead, see `references/insane-search/jina.md`; `TWITTER_*` for X; a transcription key for podcasts).
 
 | Category | Platforms | Entry |
 |---|---|---|

package/packages/shared-skills/skills/ultimate-browsing/engine/AGENTS.md

@@ -0,0 +1,179 @@
+# ultimate-browsing/engine — Generic WAF-Profile Fetch Chain (Python)
+
+**Generated:** 2026-08-10 / 38d268995
+
+## UPSTREAM BASELINE AND VERSION POLICY
+
+**READ THIS BEFORE TOUCHING `engine/**` OR PROPOSING AN UPSTREAM SYNC.**
+
+This engine is NOT project-original code. It is a vendored-and-modified snapshot of
+[fivetaku/insane-search](https://github.com/fivetaku/insane-search), and the version
+we run on is a deliberate choice, not an accident of neglect.
+
+### The pin
+
+| Fact | Value |
+|---|---|
+| Upstream project | `https://github.com/fivetaku/insane-search` |
+| Vendoring commit | `a4e4ed797` (2026-06-21) `feat(ultimate-browsing): vendor insane-search engine (junk-excluded)` |
+| De-personalization | `4743199a5` (2026-06-21) |
+| Pinned upstream baseline | upstream state as of 2026-06-21, **pre-0.7.0** (0.7.0 is dated 2026-06-22) |
+| Re-vendors since | none — every later change here is ours |
+
+We intentionally track a **pinned baseline plus local divergence**, not upstream HEAD.
+There is no submodule and no automated drift check for this engine (unlike the
+`frontend` skill's upstream submodules): the vendored files ARE the source of truth,
+and upstream is a reference we port FROM, deliberately, file by file.
+
+### Why we do not blind-rebase onto upstream HEAD
+
+1. **Upstream reset its public history.** On 2026-08-06 upstream published a single
+   squashed commit, `019ee16 refactor: reset public history at 0.14.0`, discarding the
+   prior public history through 0.13.x. There is no upstream commit graph to rebase onto and no
+   way to cherry-pick an individual upstream change by sha — only whole-file diffing
+   against a moving HEAD.
+2. **Upstream 0.14.0 REMOVED capability.** The endpoint-mining / internal-API
+   auto-derivation / site-recipe subsystems that upstream carried publicly between
+   0.12.0 and 0.14.0 are gone from upstream HEAD. Syncing to HEAD is therefore not
+   strictly an upgrade: parts of it are a downgrade relative to the intermediate
+   versions, and none of it is recoverable from the reset history.
+3. **Our tree diverged on purpose.** The KEEP list below is functionality upstream
+   never had. A wholesale overwrite with upstream HEAD would silently delete it.
+4. **Different threat model.** Our engine ships inside a published npm package and a
+   public marketplace mirror, under a CI no-site-name gate and a de-personalization
+   deny-list. Upstream carries neither constraint, so upstream code is not
+   drop-in-shippable here.
+
+### KEEP — our divergences a re-vendor MUST NOT regress
+
+These exist only in our tree. Any upstream sync that removes or bypasses one of them
+is a regression, not an upgrade:
+
+- **Phase 2.5 surrogate retrieval** (`surrogate.py`, `surrogates.yaml`) — archive /
+  reader / proxy routes tried before paying for a browser spin-up, with per-entry
+  `last_verified` staleness handling and `--allow-proxy` gating.
+- **Provenance / trust contract** (`result_schema.py`) — `Provenance` and `Trust`
+  literals on every result, so a snapshot can never be reported as the live page.
+- **Surrogate dead-end validation** (L1.5 in `validators.py`) — interstitial titles and
+  AMP-style redirect stubs rejected instead of returned as content.
+- **The no-site-name rule and its CI gate** (`bias_check.py`) — zero hard-coded site
+  names, brands, or target domains in `engine/**`.
+- **Module split of the fetch chain** — `curl_probe` / `referers` / `url_transforms` /
+  `waf_detector` / `validators` / `executor` / `summary` as separate modules rather than
+  one monolith.
+- **The Python test suite** under `engine/tests/` with its HTML/JSON fixtures.
+- **De-personalization** — no personal absolute paths, no personal auth token literals,
+  no personal browser choice; enforced by `depersonalization-gate.test.ts`.
+- **Skill-level layering** — the engine is Tier 1 under a router that also owns Tier 1.5
+  (agent-reach) and Tier 2 (CloakBrowser + agent-browser). Upstream has no such tiering.
+
+### WANT — upstream improvements worth porting forward
+
+Our snapshot predates these; they are wanted, and each must be ported as a reviewed,
+site-agnostic change that preserves every KEEP item above. Port individually; never as
+a tree overwrite:
+
+- **Content quality**: dedicated markdown conversion of fetched HTML, main-content
+  extraction, PDF text extraction, and JSON-LD rescue when the HTML body is thin.
+- **Transient-failure retry** and **render-merge** of statically fetched HTML with the
+  browser-rendered DOM.
+- **Differential block classification** — distinguishing a bot-detection block from an
+  infrastructure or authentication failure, instead of collapsing both into `challenge`.
+- **Additional stealth fetch backends** beyond the current Playwright templates, and
+  additional WAF vendor profiles.
+- **Per-host route learning** — remembering which route succeeded for a host, with a TTL
+  and a bounded store. Must stay runtime state, never committed site knowledge (R4).
+- **Engine-level Phase 0 routing** — the official-public-API preference is currently only
+  a documented rule (R5) the agent can skip; upstream moved it into code so it cannot be
+  skipped. Worth adopting.
+
+### OUT OF SCOPE
+
+- **The removed upstream endpoint-mining / internal-API auto-derivation / site-recipe
+  subsystems.** They are absent from upstream HEAD and are not reconstructed here. They
+  also sit against R3/R4 and R7's anti-bias rule: discovered internal endpoints are
+  runtime findings, never committed engine knowledge.
+- **Any upstream code carrying site-specific selectors, domains, or brand names.** It
+  fails `bias_check.py` at the door; re-derive it site-agnostically or leave it out.
+- **Automated upstream tracking.** No submodule, no drift check, no auto-bump. Syncing is
+  a deliberate, reviewed, human-initiated act.
+
+### THE SYNC RULE
+
+Any future upstream sync preserves BOTH sides. Concretely:
+
+1. Diff the specific upstream capability you want against our tree — do not overwrite
+   files wholesale, and never `git checkout` upstream over `engine/`.
+2. Port it as its own reviewed change, keeping every KEEP item intact.
+3. Re-run `python3 engine/bias_check.py` and the `engine/tests/` suite; a port that
+   introduces a site name or breaks a fixture does not ship.
+4. Update the pin table above (baseline, date, what was ported) in the same change, plus
+   the provenance section of [`../ATTRIBUTION.md`](../ATTRIBUTION.md).
+5. If a port must drop a KEEP item, say so explicitly in the PR and get it agreed first —
+   silent regressions of the KEEP list are the failure mode this policy exists to prevent.
+
+## OVERVIEW
+
+A 17-module Python package embedded in the `ultimate-browsing` skill: a site-agnostic fetch chain that escalates from a cheap curl probe to a real browser, with declarative WAF and surrogate registries. Not "optional scripts" — it has its own CLI entry (`python3 -m engine URL`), two YAML config schemas, a 4-file test suite, and a standalone CI guard. Package exports (`__init__.py`): `fetch`, `FetchResult`, `Attempt`, `Verdict`, `ValidationResult`, `validate`, `CHALLENGE_MARKERS`, `detect`, `TRANSFORMS`, `apply_transform`.
+
+## THE NO-SITE-NAME RULE (enforced in CI)
+
+`engine/**` must contain **zero** hard-coded site names, brands, or target domains. Site specifics belong to runtime hints or observations, never to code. `bias_check.py` is a standalone scanner enforcing this: a brand denylist, a URL regex scan, an allowlist for genuine infrastructure hosts (archive.org, r.jina.ai, google.com, httpbin.org, relay.invalid), and a `# NOTE-BIAS-OK` comment convention for legitimate exemptions such as test fixtures.
+
+```bash
+python3 engine/bias_check.py       # fails on any site-specific leak
+```
+
+## FETCH CHAIN PHASES
+
+```
+fetch(url, ...)                                   # fetch_chain.py
+  Phase 1  curl_probe.py    — curl_cffi TLS-impersonation probe
+  Phase 2  grid             — referer/transform/device attempt grid
+  Phase 2.5 surrogate.py    — third-party archive/reader/proxy routes
+  Phase 3  executor.py      — capability-matched Playwright fallback
+```
+
+Ordering is **not** hardcoded: each `waf_profiles.yaml` profile carries a `fallback_when_challenge` list that drives the ladder. `surrogate_wayback` precedes browser executors in every profile, so archives are tried before paying for a browser spin-up.
+
+## PROVENANCE / TRUST CONTRACT
+
+`result_schema.py` puts two literals on every `FetchResult`:
+
+- `Provenance = "live" | "snapshot" | "proxy"`
+- `Trust = "origin" | "archive" | "untrusted"`
+
+A `snapshot` result carries `snapshot_timestamp` and **must** be cited with that timestamp — never presented as the live page. `surrogates.yaml` `kind` fixes these values: `archive` -> snapshot/archive, `reader` -> live, `proxy` -> proxy/untrusted.
+
+## SURROGATE REGISTRY (`surrogates.yaml`)
+
+Site-agnostic infrastructure only. Every entry carries `last_verified` (ISO date); entries older than 90 days are deprioritized and flagged, because surrogate routes rot (a 2026-08 probe found 4 of 6 known routes dead or stubbed). `proxy` routes are MITM by construction: they require the explicit `--allow-proxy` flag and never receive `Cookie` or `Authorization` headers. Every surrogate response is re-validated with `target_url` set, so an interstitial or a redirect stub is rejected instead of returned as content.
+
+## VALIDATOR LAYERS (`validators.py`)
+
+```
+L1    challenge markers (CHALLENGE_MARKERS)
+L1.5  surrogate dead ends — interstitial titles + AMP-style redirect stubs
+      (is_redirect_stub(), needs target_url)
+L2    size/shape fingerprints
+L3+   content checks
+```
+
+## CLI
+
+```bash
+python3 -m engine URL [--selector S] [--device auto|desktop|mobile]
+                      [--timeout 25] [--max-attempts 12]
+                      [--no-playwright] [--allow-proxy] [--json] [--trace]
+```
+
+## TESTS
+
+`tests/` — `test_surrogate.py` (staleness, proxy gating, short-circuit), `test_surrogate_validators.py`, `test_fetch_chain.py`, `test_playwright_templates.py`, plus HTML/JSON fixtures under `tests/fixtures/`.
+
+## NOTES
+
+- `summary.py` emits an **R7 API-first hint** after >=3 challenge verdicts against a known WAF profile: look for `/api/`, `/graphql`, or `.json` endpoints, which usually carry weaker WAF protection than the HTML surface.
+- `templates/` holds the Playwright JS templates (`playwright_real_chrome.js`, `playwright_mobile_chrome.js`) the executor drives.
+- `url_transforms.py` transforms stay domain-agnostic (`mobile_subdomain`, `am_prefix`, `drop_www`).
+- Parent: [`packages/shared-skills/AGENTS.md`](../../../AGENTS.md).

package/packages/shared-skills/skills/ultimate-browsing/engine/__main__.py

@@ -39,6 +39,10 @@ def build_parser() -> argparse.ArgumentParser:
                    help="Upper bound across all phases (default 12).")
     p.add_argument("--no-playwright", action="store_true",
                    help="Skip Playwright fallback (curl-only).")
+    p.add_argument("--allow-proxy", action="store_true",
+                   help=("Enable kind=proxy surrogate entries (raw relay routes; "
+                        "MITM by construction — never citable alone, never "
+                        "receives cookies/auth headers). Off by default."))
     p.add_argument("--json", action="store_true",
                    help="Emit FetchResult as JSON to stdout (content omitted).")
     p.add_argument("--trace", action="store_true",
@@ -56,6 +60,7 @@ def main(argv: list[str] | None = None) -> int:
             timeout=args.timeout,
             max_attempts=args.max_attempts,
             enable_playwright=not args.no_playwright,
+            allow_surrogate_proxy=args.allow_proxy,
         )
     except Exception as e:
         print(f"engine fatal: {type(e).__name__}: {e}", file=sys.stderr)
@@ -96,7 +101,9 @@ def main(argv: list[str] | None = None) -> int:
         # Default: HTML to stdout, status to stderr.
         print(result.content, end="")
         print(f"\n[engine] ok={result.ok} verdict={result.verdict} "
-              f"profile={result.profile_used} attempts={len(result.trace)}",
+              f"profile={result.profile_used} attempts={len(result.trace)} "
+              f"provenance={result.provenance}"
+              + (f" snapshot_ts={result.snapshot_timestamp}" if result.snapshot_timestamp else ""),
               file=sys.stderr)
 
     return 0 if result.ok else 1

package/packages/shared-skills/skills/ultimate-browsing/engine/bias_check.py

@@ -54,6 +54,14 @@ URL_ALLOWLIST = {
     "www.google.com", "google.com",
     # Generic HTTP test endpoint for infrastructure / transport tests.
     "httpbin.org",
+    # Surrogate-tier infrastructure (engine/surrogates.yaml): archives/relays
+    # that serve a copy of ANY target URL. Same no-site-preference category as
+    # the google.com Referer — they are not target sites.
+    "archive.org", "web.archive.org",
+    "r.jina.ai",
+    # Placeholder host in the proxy-entry shape example; "canonical non-routable
+    # doc domain" (.invalid TLD), can never be a real target.
+    "relay.invalid",
 }
 
 # Files / dirs that must be clean.
@@ -80,6 +88,9 @@ EXPLICIT_ALLOW_FILES = {
     # None right now — add if needed with justification.
 }
 
+# NOTE-BIAS-OK convention for surrogate tests: fixture URLs (stub/interstitial
+# captures from third-party services) carry the marker on the flagged line.
+
 
 def _line_is_exempt(line: str, ext: str) -> bool:
     markers = COMMENT_OK_MARKERS.get(ext, ())

package/packages/shared-skills/skills/ultimate-browsing/engine/fetch_chain.py

@@ -40,6 +40,7 @@ def fetch(
     timeout: int = 25,
     max_attempts: int = 12,
     enable_playwright: bool = True,   # hook left for executor module
+    allow_surrogate_proxy: bool = False,  # opt-in: kind=proxy registry entries
 ) -> FetchResult:
     """Fetch `url` using the generic grid.
 
@@ -175,73 +176,110 @@ def fetch(
                     if att.verdict in (Verdict.STRONG_OK.value, Verdict.WEAK_OK.value):
                         return _build_result(resp, att, trace, profile_used=profile_id)
 
-    # -------- Phase 3: Playwright fallback (profile-driven order) -----------
-    if enable_playwright:
-        try:
-            from .executor import run_playwright_fallback  # lazy import
-            # Honour profile's `fallback_when_challenge` list — iterate the
-            # caller-declared order instead of capability-inferred single pick.
-            fb_profile = load_profile(profile_used or "unknown_challenge", profiles=profiles)
-            fb_order = fb_profile.get("fallback_when_challenge") or ["playwright_real_chrome"]
-            pw_attempt = None
-            pw_content = ""
-            for fb_name in fb_order:
-                if fb_name == "curl_grid_exhaust":
-                    # Already performed in Phase 2; nothing more to do here.
-                    continue
-                pw_attempt, pw_content = run_playwright_fallback(
+    # -------- Phase 2.5 + 3: fallback ladder (profile-driven order) ---------
+    # Surrogate routes are NOT browser work, so they run even when the caller
+    # disabled Playwright; only browser executors honour `enable_playwright`.
+    try:
+        # Honour profile's `fallback_when_challenge` list — iterate the
+        # caller-declared order instead of capability-inferred single pick.
+        fb_profile = load_profile(profile_used or "unknown_challenge", profiles=profiles)
+        fb_order = fb_profile.get("fallback_when_challenge") or ["playwright_real_chrome"]
+        pw_attempt = None
+        pw_content = ""
+        for fb_name in fb_order:
+            if fb_name == "curl_grid_exhaust":
+                # Already performed in Phase 2; nothing more to do here.
+                continue
+            if fb_name.startswith("surrogate_"):
+                # Phase 2.5: archive/reader/proxy routes serving a copy of
+                # the target. Cheaper than a browser; provenance-labeled.
+                from . import surrogate as surrogate_mod
+                s_atts, s_meta, s_content = surrogate_mod.run_surrogate(
                     url,
-                    profile_id=profile_used or "unknown_challenge",
+                    registry=surrogate_mod.load_surrogates(),
+                    allow_proxy=allow_surrogate_proxy,
+                    timeout=timeout,
                     success_selectors=success_selectors,
-                    device_class=device_class,
-                    force_executor=fb_name,
                 )
-                trace.append(pw_attempt)
-                if pw_attempt.verdict in (Verdict.STRONG_OK.value, Verdict.WEAK_OK.value):
+                trace.extend(s_atts)
+                s_att = s_atts[-1]
+                if s_att.verdict in (Verdict.STRONG_OK.value, Verdict.WEAK_OK.value):
                     return FetchResult(
                         ok=True,
-                        content=pw_content,
-                        final_url=pw_attempt.url,
-                        verdict=pw_attempt.verdict,
+                        content=s_content,
+                        final_url=s_att.url,
+                        verdict=s_att.verdict,
                         profile_used=profile_used,
                         trace=trace,
-                        summary=f"Playwright fallback succeeded via {fb_name}",
+                        summary=(
+                            f"surrogate {s_meta.get('surrogate')} succeeded"
+                            f" (provenance={s_meta.get('provenance')}"
+                            + (f" snapshot_ts={s_meta.get('snapshot_timestamp')})" if s_meta.get('snapshot_timestamp') else ")")
+                        ),
+                        provenance=str(s_meta.get("provenance", "live")),
+                        snapshot_timestamp=s_meta.get("snapshot_timestamp"),
+                        trust=str(s_meta.get("trust", "origin")),
                     )
-            # Synthesize a placeholder if no iteration ran (empty list).
-            if pw_attempt is None:
-                pw_attempt = Attempt(
-                    phase="fallback",
-                    executor="none",
-                    url=url,
-                    url_transform="original",
-                    impersonate=None,
-                    referer="",
-                    verdict=Verdict.UNKNOWN.value,
-                    error="profile has empty fallback_when_challenge",
+                continue
+            if not enable_playwright:
+                continue
+            from .executor import run_playwright_fallback  # lazy import
+            pw_attempt, pw_content = run_playwright_fallback(
+                url,
+                profile_id=profile_used or "unknown_challenge",
+                success_selectors=success_selectors,
+                device_class=device_class,
+                force_executor=fb_name,
+            )
+            trace.append(pw_attempt)
+            if pw_attempt.verdict in (Verdict.STRONG_OK.value, Verdict.WEAK_OK.value):
+                return FetchResult(
+                    ok=True,
+                    content=pw_content,
+                    final_url=pw_attempt.url,
+                    verdict=pw_attempt.verdict,
+                    profile_used=profile_used,
+                    trace=trace,
+                    summary=f"Playwright fallback succeeded via {fb_name}",
                 )
-                trace.append(pw_attempt)
-        except ImportError:
-            trace.append(Attempt(
+        # Synthesize a placeholder only when the profile genuinely offers no
+        # fallback route. Entries skipped because the caller disabled the
+        # browser are a caller choice, not a profile defect — no trace noise.
+        actionable = [n for n in fb_order if n != "curl_grid_exhaust"]
+        if pw_attempt is None and not actionable:
+            pw_attempt = Attempt(
                 phase="fallback",
-                executor="playwright",
+                executor="none",
                 url=url,
                 url_transform="original",
                 impersonate=None,
                 referer="",
                 verdict=Verdict.UNKNOWN.value,
-                error="executor module not available",
-            ))
-        except (RuntimeError, OSError) as e:
-            trace.append(Attempt(
-                phase="fallback",
-                executor="playwright",
-                url=url,
-                url_transform="original",
-                impersonate=None,
-                referer="",
-                verdict=Verdict.UNKNOWN.value,
-                error=f"{type(e).__name__}:{str(e)[:200]}",
-            ))
+                error="profile has empty fallback_when_challenge",
+            )
+            trace.append(pw_attempt)
+    except ImportError:
+        trace.append(Attempt(
+            phase="fallback",
+            executor="playwright",
+            url=url,
+            url_transform="original",
+            impersonate=None,
+            referer="",
+            verdict=Verdict.UNKNOWN.value,
+            error="executor module not available",
+        ))
+    except (RuntimeError, OSError) as e:
+        trace.append(Attempt(
+            phase="fallback",
+            executor="playwright",
+            url=url,
+            url_transform="original",
+            impersonate=None,
+            referer="",
+            verdict=Verdict.UNKNOWN.value,
+            error=f"{type(e).__name__}:{str(e)[:200]}",
+        ))
 
     # -------- Give up, return best we have ----------------------------------
     summary = format_summary(trace, profile_used)

package/packages/shared-skills/skills/ultimate-browsing/engine/result_schema.py

@@ -1,7 +1,10 @@
 from __future__ import annotations
 
 from dataclasses import asdict, dataclass, field
-from typing import Optional
+from typing import Literal, Optional
+
+Provenance = Literal["live", "snapshot", "proxy"]
+Trust = Literal["origin", "archive", "untrusted"]
 
 
 @dataclass
@@ -32,6 +35,9 @@ class FetchResult:
     profile_used: Optional[str] = None
     trace: list[Attempt] = field(default_factory=list)
     summary: str = ""
+    provenance: str = "live"                       # live | snapshot | proxy
+    snapshot_timestamp: Optional[str] = None      # archive's own timestamp, when snapshot
+    trust: str = "origin"                         # origin | archive | untrusted
 
     def to_dict(self) -> dict:
         return {
@@ -41,5 +47,8 @@ class FetchResult:
             "profile_used": self.profile_used,
             "trace": [a.to_dict() for a in self.trace],
             "summary": self.summary,
+            "provenance": self.provenance,
+            "snapshot_timestamp": self.snapshot_timestamp,
+            "trust": self.trust,
             "content_length": len(self.content),
         }