oh-my-opencode 4.19.4 → 5.0.0-beta.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.ja.md +1 -1
- package/README.ko.md +1 -1
- package/README.md +6 -5
- package/README.ru.md +1 -1
- package/README.zh-cn.md +1 -1
- package/bin/oh-my-opencode.js +14 -1
- package/bin/oh-my-opencode.test.ts +21 -0
- package/dist/cli/index.js +1452 -703
- package/dist/cli-node/index.js +1452 -703
- package/dist/config/schema/agent-overrides.d.ts +528 -0
- package/dist/config/schema/oh-my-opencode-config.d.ts +495 -0
- package/dist/features/monitor/batcher.d.ts +3 -1
- package/dist/features/monitor/manager-internals.d.ts +1 -0
- package/dist/features/monitor/output-injector-types.d.ts +2 -0
- package/dist/features/monitor/output-injector.d.ts +6 -0
- package/dist/hooks/todo-continuation-enforcer/types.d.ts +1 -0
- package/dist/hooks/todo-continuation-enforcer/unrecoverable-request-error.d.ts +9 -0
- package/dist/hooks/tool-pair-validator/hook.test-support.d.ts +29 -0
- package/dist/hooks/tool-pair-validator/tool-part-ids.d.ts +14 -5
- package/dist/hooks/tool-pair-validator/tool-result-repair.d.ts +4 -3
- package/dist/hooks/tool-pair-validator/types.d.ts +5 -22
- package/dist/index.js +2776 -2240
- package/dist/mcp/lsp.d.ts +1 -0
- package/dist/oh-my-opencode.schema.json +1466 -131
- package/dist/shared/tmux/constants.d.ts +1 -1
- package/dist/skills/ast-grep/SOURCE +1 -1
- package/dist/skills/ast-grep/install.ps1 +2 -2
- package/dist/skills/ast-grep/install.sh +1 -1
- package/dist/skills/ast-grep/references/install.md +2 -2
- package/dist/skills/ast-grep/tests/smoke.sh +1 -1
- package/dist/skills/start-work/SKILL.md +54 -9
- package/dist/skills/ultimate-browsing/SKILL.md +2 -2
- package/dist/skills/ultimate-browsing/engine/__main__.py +8 -1
- package/dist/skills/ultimate-browsing/engine/bias_check.py +11 -0
- package/dist/skills/ultimate-browsing/engine/fetch_chain.py +90 -52
- package/dist/skills/ultimate-browsing/engine/result_schema.py +10 -1
- package/dist/skills/ultimate-browsing/engine/surrogate.py +214 -0
- package/dist/skills/ultimate-browsing/engine/surrogates.yaml +60 -0
- package/dist/skills/ultimate-browsing/engine/tests/fixtures/amp_redirect_stub.html +7 -0
- package/dist/skills/ultimate-browsing/engine/tests/fixtures/search_interstitial.html +19 -0
- package/dist/skills/ultimate-browsing/engine/tests/fixtures/wayback_available.json +1 -0
- package/dist/skills/ultimate-browsing/engine/tests/fixtures/wayback_snapshot.html +1128 -0
- package/dist/skills/ultimate-browsing/engine/tests/test_surrogate.py +252 -0
- package/dist/skills/ultimate-browsing/engine/tests/test_surrogate_validators.py +78 -0
- package/dist/skills/ultimate-browsing/engine/validators.py +46 -0
- package/dist/skills/ultimate-browsing/engine/waf_detector.py +1 -1
- package/dist/skills/ultimate-browsing/engine/waf_profiles.yaml +10 -5
- package/dist/skills/ultimate-browsing/references/agent-reach/social.md +1 -1
- package/dist/skills/ultimate-browsing/references/chrome-stealth.md +13 -11
- package/dist/skills/ultimate-browsing/references/insane-search/README.md +4 -4
- package/dist/skills/ultimate-browsing/references/insane-search/cache-archive.md +51 -50
- package/dist/skills/ultimate-browsing/references/insane-search/fallback.md +1 -1
- package/dist/skills/ultimate-browsing/references/insane-search/jina.md +8 -2
- package/dist/skills/ultimate-browsing/references/insane-search/naver.md +1 -1
- package/dist/skills/ultimate-browsing/references/insane-search/twitter.md +3 -3
- package/dist/skills/ulw-plan/SKILL.md +1 -1
- package/dist/skills/ulw-plan/references/full-workflow.md +3 -3
- package/dist/skills/ulw-plan/scripts/scaffold-plan.mjs +1 -1
- package/dist/skills/ulw-research/SKILL.md +15 -10
- package/dist/tui.js +307 -28
- package/docs/reference/web-terminal-visual-qa.md +1 -1
- package/package.json +27 -19
- package/packages/lsp-core/src/lsp/connection.ts +1 -1
- package/packages/lsp-daemon/dist/cli.js +27 -13
- package/packages/lsp-daemon/dist/client.js +49 -35
- package/packages/lsp-daemon/dist/ensure-daemon.d.ts +1 -0
- package/packages/lsp-daemon/dist/ensure-daemon.js +18 -5
- package/packages/lsp-daemon/dist/index.js +34 -20
- package/packages/lsp-tools-mcp/dist/cli.js +1 -1
- package/packages/lsp-tools-mcp/dist/lsp/manager.js +1 -1
- package/packages/lsp-tools-mcp/dist/mcp.js +1 -1
- package/packages/lsp-tools-mcp/dist/tools.js +1 -1
- package/packages/omo-codex/plugin/.codex-plugin/plugin.json +1 -1
- package/packages/omo-codex/plugin/components/bootstrap/dist/cli.js +268 -92
- package/packages/omo-codex/plugin/components/bootstrap/hooks/hooks.json +1 -1
- package/packages/omo-codex/plugin/components/bootstrap/package.json +1 -1
- package/packages/omo-codex/plugin/components/bootstrap/src/setup.ts +7 -7
- package/packages/omo-codex/plugin/components/bootstrap/src/worker.ts +3 -0
- package/packages/omo-codex/plugin/components/codegraph/dist/cli.js +106 -6
- package/packages/omo-codex/plugin/components/codegraph/dist/serve.js +106 -6
- package/packages/omo-codex/plugin/components/codegraph/package.json +1 -1
- package/packages/omo-codex/plugin/components/comment-checker/hooks/hooks.json +1 -1
- package/packages/omo-codex/plugin/components/comment-checker/package.json +1 -1
- package/packages/omo-codex/plugin/components/git-bash/hooks/hooks.json +2 -2
- package/packages/omo-codex/plugin/components/git-bash/package.json +1 -1
- package/packages/omo-codex/plugin/components/lazycodex-executor-verify/hooks/hooks.json +1 -1
- package/packages/omo-codex/plugin/components/lazycodex-executor-verify/package.json +1 -1
- package/packages/omo-codex/plugin/components/lsp/dist/.omo-runtime-manifest.json +3 -3
- package/packages/omo-codex/plugin/components/lsp/dist/cli.js +56 -42
- package/packages/omo-codex/plugin/components/lsp/hooks/hooks.json +2 -2
- package/packages/omo-codex/plugin/components/lsp/package.json +1 -1
- package/packages/omo-codex/plugin/components/rules/hooks/hooks.json +4 -4
- package/packages/omo-codex/plugin/components/rules/package.json +1 -1
- package/packages/omo-codex/plugin/components/start-work-continuation/AGENTS.md +1 -0
- package/packages/omo-codex/plugin/components/start-work-continuation/README.md +5 -1
- package/packages/omo-codex/plugin/components/start-work-continuation/directive.md +2 -1
- package/packages/omo-codex/plugin/components/start-work-continuation/dist/cli.js +18 -0
- package/packages/omo-codex/plugin/components/start-work-continuation/hooks/hooks.json +2 -2
- package/packages/omo-codex/plugin/components/start-work-continuation/package.json +1 -1
- package/packages/omo-codex/plugin/components/start-work-continuation/src/codex-hook.ts +21 -0
- package/packages/omo-codex/plugin/components/start-work-continuation/test/codex-hook.test.ts +105 -0
- package/packages/omo-codex/plugin/components/teammode/hooks/hooks.json +1 -1
- package/packages/omo-codex/plugin/components/teammode/package.json +1 -1
- package/packages/omo-codex/plugin/components/telemetry/hooks/hooks.json +1 -1
- package/packages/omo-codex/plugin/components/telemetry/package.json +1 -1
- package/packages/omo-codex/plugin/components/ultrawork/CHANGELOG.md +2 -0
- package/packages/omo-codex/plugin/components/ultrawork/agents/lazycodex-code-reviewer.toml +1 -1
- package/packages/omo-codex/plugin/components/ultrawork/agents/lazycodex-gate-reviewer.toml +1 -1
- package/packages/omo-codex/plugin/components/ultrawork/agents/lazycodex-qa-executor.toml +1 -1
- package/packages/omo-codex/plugin/components/ultrawork/agents/lazycodex-worker-high.toml +1 -1
- package/packages/omo-codex/plugin/components/ultrawork/agents/lazycodex-worker-low.toml +1 -1
- package/packages/omo-codex/plugin/components/ultrawork/agents/lazycodex-worker-medium.toml +1 -1
- package/packages/omo-codex/plugin/components/ultrawork/agents/plan.toml +2 -2
- package/packages/omo-codex/plugin/components/ultrawork/directive.md +3 -2
- package/packages/omo-codex/plugin/components/ultrawork/hooks/hooks.json +1 -1
- package/packages/omo-codex/plugin/components/ultrawork/package.json +1 -1
- package/packages/omo-codex/plugin/components/ultrawork/skills/ultrawork/SKILL.md +3 -2
- package/packages/omo-codex/plugin/components/ultrawork/skills/ulw-plan/SKILL.md +1 -1
- package/packages/omo-codex/plugin/components/ultrawork/skills/ulw-plan/references/full-workflow.md +3 -3
- package/packages/omo-codex/plugin/components/ultrawork/skills/ulw-plan/scripts/scaffold-plan.mjs +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/AGENTS.md +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/CHANGELOG.md +2 -0
- package/packages/omo-codex/plugin/components/ulw-loop/README.md +11 -11
- package/packages/omo-codex/plugin/components/ulw-loop/directive.md +3 -2
- package/packages/omo-codex/plugin/components/ulw-loop/dist/checkpoint-reconciliation.js +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/dist/cli-output.d.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/dist/cli-output.js +9 -9
- package/packages/omo-codex/plugin/components/ulw-loop/dist/cli-steering.js +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/dist/cli.js +66 -66
- package/packages/omo-codex/plugin/components/ulw-loop/dist/codex-goal-instruction.js +4 -4
- package/packages/omo-codex/plugin/components/ulw-loop/dist/codex-hook.js +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/dist/plan-crud.js +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/dist/plan-io.js +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/dist/steering.js +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/dist/stop-resume-hook.js +2 -2
- package/packages/omo-codex/plugin/components/ulw-loop/hooks/hooks.json +4 -4
- package/packages/omo-codex/plugin/components/ulw-loop/package.json +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/skills/ulw-loop/SKILL.md +3 -3
- package/packages/omo-codex/plugin/components/ulw-loop/skills/ulw-loop/references/full-workflow.md +22 -25
- package/packages/omo-codex/plugin/components/ulw-loop/src/checkpoint-reconciliation.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/src/cli-output.ts +9 -9
- package/packages/omo-codex/plugin/components/ulw-loop/src/cli-steering.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/src/cli.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/src/codex-goal-instruction.ts +4 -4
- package/packages/omo-codex/plugin/components/ulw-loop/src/codex-hook.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/src/plan-crud.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/src/plan-io.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/src/steering.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/src/stop-resume-hook.ts +2 -2
- package/packages/omo-codex/plugin/components/ulw-loop/test/cli-commands.test.ts +2 -2
- package/packages/omo-codex/plugin/components/ulw-loop/test/cli-entrypoint.test.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/test/cli-helpers.test.ts +2 -2
- package/packages/omo-codex/plugin/components/ulw-loop/test/cli-steering-kind-guidance.test.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/test/codex-hook.test.ts +2 -2
- package/packages/omo-codex/plugin/components/ulw-loop/test/fixtures/quality-gate-builder.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/test/package-smoke.test.ts +5 -5
- package/packages/omo-codex/plugin/components/ulw-loop/test/plan-io.test.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/test/quality-gate-roles.test.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/test/steering.test.ts +1 -1
- package/packages/omo-codex/plugin/components/ulw-loop/test/stop-resume-hook.test.ts +1 -1
- package/packages/omo-codex/plugin/hooks/post-compact-resetting-git-bash-mcp-reminder.json +1 -1
- package/packages/omo-codex/plugin/hooks/post-compact-resetting-lsp-diagnostics-cache.json +1 -1
- package/packages/omo-codex/plugin/hooks/post-compact-resetting-project-rule-cache.json +1 -1
- package/packages/omo-codex/plugin/hooks/post-tool-use-checking-codegraph-init-guidance.json +1 -1
- package/packages/omo-codex/plugin/hooks/post-tool-use-checking-comments.json +1 -1
- package/packages/omo-codex/plugin/hooks/post-tool-use-checking-lsp-diagnostics.json +1 -1
- package/packages/omo-codex/plugin/hooks/post-tool-use-checking-thread-title-hygiene.json +1 -1
- package/packages/omo-codex/plugin/hooks/post-tool-use-matching-project-rules.json +1 -1
- package/packages/omo-codex/plugin/hooks/pre-tool-use-enforcing-unlimited-goal-budget.json +1 -1
- package/packages/omo-codex/plugin/hooks/pre-tool-use-guarding-ulw-loop-spawns.json +1 -1
- package/packages/omo-codex/plugin/hooks/pre-tool-use-recommending-git-bash-mcp.json +1 -1
- package/packages/omo-codex/plugin/hooks/session-start-checking-auto-update.json +1 -1
- package/packages/omo-codex/plugin/hooks/session-start-checking-bootstrap-provisioning.json +1 -1
- package/packages/omo-codex/plugin/hooks/session-start-checking-codegraph-bootstrap.json +1 -1
- package/packages/omo-codex/plugin/hooks/session-start-loading-project-rules.json +1 -1
- package/packages/omo-codex/plugin/hooks/session-start-recording-session-telemetry.json +1 -1
- package/packages/omo-codex/plugin/hooks/stop-checking-start-work-continuation.json +1 -1
- package/packages/omo-codex/plugin/hooks/stop-checking-ulw-loop-resume.json +1 -1
- package/packages/omo-codex/plugin/hooks/subagent-stop-checking-start-work-continuation.json +1 -1
- package/packages/omo-codex/plugin/hooks/subagent-stop-verifying-lazycodex-executor-evidence.json +1 -1
- package/packages/omo-codex/plugin/hooks/user-prompt-submit-checking-ultrawork-trigger.json +1 -1
- package/packages/omo-codex/plugin/hooks/user-prompt-submit-checking-ulw-loop-steering.json +1 -1
- package/packages/omo-codex/plugin/hooks/user-prompt-submit-loading-project-rules.json +1 -1
- package/packages/omo-codex/plugin/package-lock.json +13 -13
- package/packages/omo-codex/plugin/package.json +1 -1
- package/packages/omo-codex/plugin/scripts/sync-skills.mjs +4 -4
- package/packages/omo-codex/plugin/skills/ast-grep/SOURCE +1 -1
- package/packages/omo-codex/plugin/skills/ast-grep/install.ps1 +2 -2
- package/packages/omo-codex/plugin/skills/ast-grep/install.sh +1 -1
- package/packages/omo-codex/plugin/skills/ast-grep/references/install.md +2 -2
- package/packages/omo-codex/plugin/skills/ast-grep/tests/smoke.sh +1 -1
- package/packages/omo-codex/plugin/skills/start-work/SKILL.md +54 -9
- package/packages/omo-codex/plugin/skills/ultimate-browsing/SKILL.md +2 -2
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/__main__.py +8 -1
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/bias_check.py +11 -0
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/fetch_chain.py +90 -52
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/result_schema.py +10 -1
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/surrogate.py +214 -0
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/surrogates.yaml +60 -0
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/tests/fixtures/amp_redirect_stub.html +7 -0
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/tests/fixtures/search_interstitial.html +19 -0
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/tests/fixtures/wayback_available.json +1 -0
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/tests/fixtures/wayback_snapshot.html +1128 -0
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/tests/test_surrogate.py +252 -0
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/tests/test_surrogate_validators.py +78 -0
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/validators.py +46 -0
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/waf_detector.py +1 -1
- package/packages/omo-codex/plugin/skills/ultimate-browsing/engine/waf_profiles.yaml +10 -5
- package/packages/omo-codex/plugin/skills/ultimate-browsing/references/agent-reach/social.md +1 -1
- package/packages/omo-codex/plugin/skills/ultimate-browsing/references/chrome-stealth.md +13 -11
- package/packages/omo-codex/plugin/skills/ultimate-browsing/references/insane-search/README.md +4 -4
- package/packages/omo-codex/plugin/skills/ultimate-browsing/references/insane-search/cache-archive.md +51 -50
- package/packages/omo-codex/plugin/skills/ultimate-browsing/references/insane-search/fallback.md +1 -1
- package/packages/omo-codex/plugin/skills/ultimate-browsing/references/insane-search/jina.md +8 -2
- package/packages/omo-codex/plugin/skills/ultimate-browsing/references/insane-search/naver.md +1 -1
- package/packages/omo-codex/plugin/skills/ultimate-browsing/references/insane-search/twitter.md +3 -3
- package/packages/omo-codex/plugin/skills/ultrawork/SKILL.md +3 -2
- package/packages/omo-codex/plugin/skills/ulw-loop/SKILL.md +3 -3
- package/packages/omo-codex/plugin/skills/ulw-loop/references/full-workflow.md +22 -25
- package/packages/omo-codex/plugin/skills/ulw-plan/SKILL.md +1 -1
- package/packages/omo-codex/plugin/skills/ulw-plan/references/full-workflow.md +3 -3
- package/packages/omo-codex/plugin/skills/ulw-plan/scripts/scaffold-plan.mjs +1 -1
- package/packages/omo-codex/plugin/skills/ulw-research/SKILL.md +15 -10
- package/packages/omo-codex/plugin/test/bootstrap-binlinks.test.mjs +12 -12
- package/packages/omo-codex/plugin/test/bootstrap-orchestration.test.mjs +36 -4
- package/packages/omo-codex/plugin/test/sync-skills-orchestration.test.mjs +1 -1
- package/packages/omo-codex/plugin/test/sync-skills-test-support.mjs +4 -4
- package/packages/omo-codex/scripts/install-bin-links.test.mjs +56 -2
- package/packages/omo-codex/scripts/install-delegated-command.test.mjs +6 -6
- package/packages/omo-codex/scripts/install-dist/install-local.mjs +115 -61
- package/packages/omo-codex/scripts/install-local-entrypoint.test.mjs +4 -4
- package/packages/omo-codex/scripts/install-local.test.mjs +5 -2
- package/packages/shared-skills/index.mjs +19 -1
- package/packages/shared-skills/skills/ast-grep/SOURCE +1 -1
- package/packages/shared-skills/skills/ast-grep/install.ps1 +2 -2
- package/packages/shared-skills/skills/ast-grep/install.sh +1 -1
- package/packages/shared-skills/skills/ast-grep/references/install.md +2 -2
- package/packages/shared-skills/skills/ast-grep/tests/smoke.sh +1 -1
- package/packages/shared-skills/skills/start-work/SKILL.md +54 -9
- package/packages/shared-skills/skills/ultimate-browsing/SKILL.md +2 -2
- package/packages/shared-skills/skills/ultimate-browsing/engine/__main__.py +8 -1
- package/packages/shared-skills/skills/ultimate-browsing/engine/bias_check.py +11 -0
- package/packages/shared-skills/skills/ultimate-browsing/engine/fetch_chain.py +90 -52
- package/packages/shared-skills/skills/ultimate-browsing/engine/result_schema.py +10 -1
- package/packages/shared-skills/skills/ultimate-browsing/engine/surrogate.py +214 -0
- package/packages/shared-skills/skills/ultimate-browsing/engine/surrogates.yaml +60 -0
- package/packages/shared-skills/skills/ultimate-browsing/engine/tests/fixtures/amp_redirect_stub.html +7 -0
- package/packages/shared-skills/skills/ultimate-browsing/engine/tests/fixtures/search_interstitial.html +19 -0
- package/packages/shared-skills/skills/ultimate-browsing/engine/tests/fixtures/wayback_available.json +1 -0
- package/packages/shared-skills/skills/ultimate-browsing/engine/tests/fixtures/wayback_snapshot.html +1128 -0
- package/packages/shared-skills/skills/ultimate-browsing/engine/tests/test_surrogate.py +252 -0
- package/packages/shared-skills/skills/ultimate-browsing/engine/tests/test_surrogate_validators.py +78 -0
- package/packages/shared-skills/skills/ultimate-browsing/engine/validators.py +46 -0
- package/packages/shared-skills/skills/ultimate-browsing/engine/waf_detector.py +1 -1
- package/packages/shared-skills/skills/ultimate-browsing/engine/waf_profiles.yaml +10 -5
- package/packages/shared-skills/skills/ultimate-browsing/references/agent-reach/social.md +1 -1
- package/packages/shared-skills/skills/ultimate-browsing/references/chrome-stealth.md +13 -11
- package/packages/shared-skills/skills/ultimate-browsing/references/insane-search/README.md +4 -4
- package/packages/shared-skills/skills/ultimate-browsing/references/insane-search/cache-archive.md +51 -50
- package/packages/shared-skills/skills/ultimate-browsing/references/insane-search/fallback.md +1 -1
- package/packages/shared-skills/skills/ultimate-browsing/references/insane-search/jina.md +8 -2
- package/packages/shared-skills/skills/ultimate-browsing/references/insane-search/naver.md +1 -1
- package/packages/shared-skills/skills/ultimate-browsing/references/insane-search/twitter.md +3 -3
- package/packages/shared-skills/skills/ulw-plan/SKILL.md +1 -1
- package/packages/shared-skills/skills/ulw-plan/references/full-workflow.md +3 -3
- package/packages/shared-skills/skills/ulw-plan/scripts/scaffold-plan.mjs +1 -1
- package/packages/shared-skills/skills/ulw-research/SKILL.md +15 -10
- package/postinstall.mjs +6 -0
|
@@ -0,0 +1,252 @@
|
|
|
1
|
+
"""Surrogate registry/fetch/ordering tests; registry entries and responses are faked, never live network."""
|
|
2
|
+
from __future__ import annotations
|
|
3
|
+
|
|
4
|
+
import json
|
|
5
|
+
import os
|
|
6
|
+
import sys
|
|
7
|
+
import unittest
|
|
8
|
+
from pathlib import Path
|
|
9
|
+
from unittest.mock import patch
|
|
10
|
+
|
|
11
|
+
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
|
12
|
+
|
|
13
|
+
from engine.result_schema import Attempt # noqa: E402
|
|
14
|
+
from engine.validators import Verdict # noqa: E402
|
|
15
|
+
|
|
16
|
+
FIXTURES = Path(__file__).resolve().parent / "fixtures"
|
|
17
|
+
TARGET = "https://en.wikipedia.org/wiki/Web_archiving" # NOTE-BIAS-OK — fixture URL from a third-party surrogate service, not a target site
|
|
18
|
+
|
|
19
|
+
|
|
20
|
+
def _fixture(name: str) -> str:
|
|
21
|
+
return (FIXTURES / name).read_text(encoding="utf-8", errors="replace")
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
class _Resp:
|
|
25
|
+
def __init__(self, text: str = "", status: int = 200, url: str = "", payload: dict | None = None):
|
|
26
|
+
self.text = text
|
|
27
|
+
self.status_code = status
|
|
28
|
+
self.url = url
|
|
29
|
+
self.cookies = {}
|
|
30
|
+
self.headers = {}
|
|
31
|
+
self._payload = payload
|
|
32
|
+
|
|
33
|
+
def json(self):
|
|
34
|
+
if self._payload is None:
|
|
35
|
+
raise ValueError("no json payload")
|
|
36
|
+
return self._payload
|
|
37
|
+
|
|
38
|
+
|
|
39
|
+
def _fail_attempt(phase: str) -> Attempt:
|
|
40
|
+
return Attempt(
|
|
41
|
+
phase=phase,
|
|
42
|
+
executor="curl_cffi",
|
|
43
|
+
url=TARGET,
|
|
44
|
+
url_transform="original",
|
|
45
|
+
impersonate="safari",
|
|
46
|
+
referer="self_root",
|
|
47
|
+
verdict=Verdict.CHALLENGE.value,
|
|
48
|
+
)
|
|
49
|
+
|
|
50
|
+
|
|
51
|
+
class SurrogateRegistry(unittest.TestCase):
|
|
52
|
+
def test_default_registry_contains_wayback(self) -> None:
|
|
53
|
+
from engine import surrogate
|
|
54
|
+
|
|
55
|
+
reg = surrogate.load_surrogates()
|
|
56
|
+
self.assertIn("wayback", reg)
|
|
57
|
+
self.assertEqual(reg["wayback"]["kind"], "archive")
|
|
58
|
+
self.assertEqual(reg["wayback"]["trust"], "archive")
|
|
59
|
+
|
|
60
|
+
def test_stale_entries_detected(self) -> None:
|
|
61
|
+
from engine import surrogate
|
|
62
|
+
|
|
63
|
+
self.assertTrue(surrogate.is_stale({"last_verified": "2025-01-01"}, max_age_days=90))
|
|
64
|
+
self.assertFalse(surrogate.is_stale({"last_verified": "2999-01-01"}, max_age_days=90))
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
class SurrogateFetch(unittest.TestCase):
|
|
68
|
+
def test_wayback_success_sets_snapshot_provenance(self) -> None:
|
|
69
|
+
from engine import surrogate
|
|
70
|
+
|
|
71
|
+
discovery = json.loads(_fixture("wayback_available.json"))
|
|
72
|
+
|
|
73
|
+
def fake_http(u: str, **kw):
|
|
74
|
+
if "wayback/available" in u:
|
|
75
|
+
return _Resp(json.dumps(discovery), 200, u, payload=discovery)
|
|
76
|
+
return _Resp(_fixture("wayback_snapshot.html"), 200, u)
|
|
77
|
+
|
|
78
|
+
with patch.object(surrogate, "_http_get", side_effect=fake_http):
|
|
79
|
+
atts, meta, content = surrogate.run_surrogate(TARGET, registry=surrogate.load_surrogates(), timeout=5)
|
|
80
|
+
|
|
81
|
+
att = atts[-1]
|
|
82
|
+
self.assertEqual(att.phase, "surrogate")
|
|
83
|
+
self.assertEqual(att.executor, "surrogate_wayback")
|
|
84
|
+
self.assertEqual(att.verdict, Verdict.WEAK_OK.value)
|
|
85
|
+
self.assertEqual(meta["provenance"], "snapshot")
|
|
86
|
+
self.assertTrue(meta["snapshot_timestamp"])
|
|
87
|
+
self.assertEqual(meta["trust"], "archive")
|
|
88
|
+
self.assertGreater(len(content), 3000)
|
|
89
|
+
|
|
90
|
+
def test_stub_snapshot_is_not_accepted(self) -> None:
|
|
91
|
+
from engine import surrogate
|
|
92
|
+
|
|
93
|
+
disc = {"archived_snapshots": {"closest": {"available": True, "url": "https://web.archive.org/web/2024/x", "timestamp": "20240101000000"}}} # NOTE-BIAS-OK — fixture URL from a third-party surrogate service, not a target site
|
|
94
|
+
|
|
95
|
+
def fake_http(u: str, **kw):
|
|
96
|
+
if "wayback/available" in u:
|
|
97
|
+
return _Resp(json.dumps(disc), 200, u, payload=disc)
|
|
98
|
+
return _Resp(_fixture("amp_redirect_stub.html"), 200, u)
|
|
99
|
+
|
|
100
|
+
with patch.object(surrogate, "_http_get", side_effect=fake_http):
|
|
101
|
+
atts, _, _ = surrogate.run_surrogate(TARGET, registry={"wayback": surrogate.DEFAULT_SURROGATES["wayback"]}, timeout=5)
|
|
102
|
+
att = atts[-1]
|
|
103
|
+
self.assertNotIn(att.verdict, (Verdict.STRONG_OK.value, Verdict.WEAK_OK.value))
|
|
104
|
+
|
|
105
|
+
def test_proxy_skips_without_allow_flag(self) -> None:
|
|
106
|
+
from engine import surrogate
|
|
107
|
+
|
|
108
|
+
reg = {"anon_relay": {"kind": "proxy", "trust": "untrusted", "enabled": False,
|
|
109
|
+
"last_verified": "2999-01-01", "fetch": "https://relay.invalid/{target}"}} # NOTE-BIAS-OK — fixture URL from a third-party surrogate service, not a target site
|
|
110
|
+
called: list[str] = []
|
|
111
|
+
|
|
112
|
+
with patch.object(surrogate, "_http_get", side_effect=lambda u, **kw: (called.append(u), _Resp("<html>" + "y" * 4000 + "</html>", 200, u))[1]):
|
|
113
|
+
atts, _, _ = surrogate.run_surrogate(TARGET, registry=reg, allow_proxy=False, timeout=5)
|
|
114
|
+
self.assertEqual(len(called), 0)
|
|
115
|
+
|
|
116
|
+
with patch.object(surrogate, "_http_get", side_effect=lambda u, **kw: (called.append(u), _Resp("<html>" + "y" * 4000 + "</html>", 200, u))[1]):
|
|
117
|
+
atts, meta, _ = surrogate.run_surrogate(TARGET, registry=reg, allow_proxy=True, timeout=5)
|
|
118
|
+
self.assertEqual(len(called), 1)
|
|
119
|
+
att = atts[-1]
|
|
120
|
+
self.assertEqual(att.executor, "surrogate_anon_relay")
|
|
121
|
+
self.assertEqual(meta["trust"], "untrusted")
|
|
122
|
+
|
|
123
|
+
def test_proxy_never_carries_credentials(self) -> None:
|
|
124
|
+
from engine import surrogate
|
|
125
|
+
|
|
126
|
+
seen: list[dict] = []
|
|
127
|
+
|
|
128
|
+
def fake_http(u: str, **kw):
|
|
129
|
+
seen.append(kw.get("headers", {}))
|
|
130
|
+
return _Resp("<html>" + "z" * 4000 + "</html>", 200, u)
|
|
131
|
+
|
|
132
|
+
reg = {"anon_relay": {"kind": "proxy", "trust": "untrusted", "enabled": False,
|
|
133
|
+
"last_verified": "2999-01-01", "fetch": "https://relay.invalid/{target}"}} # NOTE-BIAS-OK — fixture URL from a third-party surrogate service, not a target site
|
|
134
|
+
os.environ["OMOB_TEST_TOKEN"] = "secret"
|
|
135
|
+
try:
|
|
136
|
+
with patch.object(surrogate, "_http_get", side_effect=fake_http):
|
|
137
|
+
surrogate.run_surrogate(TARGET, registry=reg, allow_proxy=True, timeout=5)
|
|
138
|
+
finally:
|
|
139
|
+
os.environ.pop("OMOB_TEST_TOKEN", None)
|
|
140
|
+
|
|
141
|
+
self.assertEqual(len(seen), 1)
|
|
142
|
+
joined = json.dumps(seen[0]).lower()
|
|
143
|
+
self.assertNotIn("secret", joined)
|
|
144
|
+
self.assertNotIn("authorization", joined)
|
|
145
|
+
self.assertNotIn("cookie", joined)
|
|
146
|
+
|
|
147
|
+
|
|
148
|
+
class ChainOrdering(unittest.TestCase):
|
|
149
|
+
def _profile(self) -> dict:
|
|
150
|
+
return {
|
|
151
|
+
"tls_impersonate_candidates": [[]],
|
|
152
|
+
"referer_strategies": [],
|
|
153
|
+
"url_transform_order": ["original"],
|
|
154
|
+
"fallback_when_challenge": ["surrogate_wayback", "playwright_real_chrome"],
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
def test_surrogate_success_short_circuits_playwright(self) -> None:
|
|
158
|
+
from engine.fetch_chain import fetch
|
|
159
|
+
from engine import surrogate
|
|
160
|
+
from engine import executor as ex
|
|
161
|
+
|
|
162
|
+
def fake_surrogate(*a, **kw):
|
|
163
|
+
att = Attempt(
|
|
164
|
+
phase="surrogate",
|
|
165
|
+
executor="surrogate_wayback",
|
|
166
|
+
url=TARGET,
|
|
167
|
+
url_transform="original",
|
|
168
|
+
impersonate=None,
|
|
169
|
+
referer="",
|
|
170
|
+
verdict=Verdict.WEAK_OK.value,
|
|
171
|
+
body_size=5000,
|
|
172
|
+
)
|
|
173
|
+
return [att], {"provenance": "snapshot", "snapshot_timestamp": "20240101000000", "trust": "archive"}, "<html>content</html>"
|
|
174
|
+
|
|
175
|
+
def fail_playwright(*a, **kw):
|
|
176
|
+
raise AssertionError("playwright must not run after surrogate success")
|
|
177
|
+
|
|
178
|
+
with patch("engine.fetch_chain._load_profiles", return_value={}), \
|
|
179
|
+
patch("engine.fetch_chain.last_load_error", return_value=None), \
|
|
180
|
+
patch("engine.fetch_chain.run_attempt", side_effect=lambda *a, **kw: (_fail_attempt(str(kw.get("phase", "grid"))), None)), \
|
|
181
|
+
patch("engine.fetch_chain.detect", return_value=[]), \
|
|
182
|
+
patch("engine.fetch_chain.load_profile", side_effect=lambda *a, **kw: self._profile()), \
|
|
183
|
+
patch.object(surrogate, "run_surrogate", side_effect=fake_surrogate), \
|
|
184
|
+
patch.object(ex, "run_playwright_fallback", side_effect=fail_playwright):
|
|
185
|
+
result = fetch(TARGET, enable_playwright=True, max_attempts=1)
|
|
186
|
+
|
|
187
|
+
self.assertTrue(result.ok)
|
|
188
|
+
self.assertEqual(result.provenance, "snapshot")
|
|
189
|
+
self.assertEqual(result.snapshot_timestamp, "20240101000000")
|
|
190
|
+
|
|
191
|
+
def test_surrogate_still_runs_when_playwright_disabled(self) -> None:
|
|
192
|
+
from engine.fetch_chain import fetch
|
|
193
|
+
from engine import surrogate
|
|
194
|
+
|
|
195
|
+
calls: list[str] = []
|
|
196
|
+
|
|
197
|
+
def fake_surrogate(*a, **kw):
|
|
198
|
+
calls.append("surrogate")
|
|
199
|
+
att = Attempt(phase="surrogate", executor="surrogate_wayback", url=TARGET,
|
|
200
|
+
url_transform="original", impersonate=None, referer="",
|
|
201
|
+
verdict=Verdict.WEAK_OK.value, body_size=5000)
|
|
202
|
+
return [att], {"provenance": "snapshot", "snapshot_timestamp": "20240101000000", "trust": "archive"}, "<html>snap</html>"
|
|
203
|
+
|
|
204
|
+
with patch("engine.fetch_chain._load_profiles", return_value={}), \
|
|
205
|
+
patch("engine.fetch_chain.last_load_error", return_value=None), \
|
|
206
|
+
patch("engine.fetch_chain.run_attempt", side_effect=lambda *a, **kw: (_fail_attempt(str(kw.get("phase", "grid"))), None)), \
|
|
207
|
+
patch("engine.fetch_chain.detect", return_value=[]), \
|
|
208
|
+
patch("engine.fetch_chain.load_profile", side_effect=lambda *a, **kw: self._profile()), \
|
|
209
|
+
patch.object(surrogate, "run_surrogate", side_effect=fake_surrogate):
|
|
210
|
+
result = fetch(TARGET, enable_playwright=False, max_attempts=1)
|
|
211
|
+
|
|
212
|
+
self.assertEqual(calls, ["surrogate"])
|
|
213
|
+
self.assertTrue(result.ok)
|
|
214
|
+
self.assertEqual(result.provenance, "snapshot")
|
|
215
|
+
|
|
216
|
+
def test_playwright_runs_when_surrogate_fails(self) -> None:
|
|
217
|
+
from engine.fetch_chain import fetch
|
|
218
|
+
from engine import surrogate
|
|
219
|
+
from engine import executor as ex
|
|
220
|
+
|
|
221
|
+
calls: list[str] = []
|
|
222
|
+
|
|
223
|
+
def fake_surrogate(*a, **kw):
|
|
224
|
+
calls.append("surrogate")
|
|
225
|
+
att = Attempt(phase="surrogate", executor="surrogate_wayback", url=TARGET,
|
|
226
|
+
url_transform="original", impersonate=None, referer="",
|
|
227
|
+
verdict=Verdict.UNKNOWN.value)
|
|
228
|
+
return [att], {"provenance": "live", "snapshot_timestamp": None, "trust": "origin"}, ""
|
|
229
|
+
|
|
230
|
+
def fake_playwright(*a, **kw):
|
|
231
|
+
calls.append("playwright")
|
|
232
|
+
att = Attempt(phase="fallback", executor="playwright_real_chrome", url=TARGET,
|
|
233
|
+
url_transform="original", impersonate=None, referer="",
|
|
234
|
+
verdict=Verdict.WEAK_OK.value, body_size=5000)
|
|
235
|
+
return att, "<html>chrome content</html>"
|
|
236
|
+
|
|
237
|
+
with patch("engine.fetch_chain._load_profiles", return_value={}), \
|
|
238
|
+
patch("engine.fetch_chain.last_load_error", return_value=None), \
|
|
239
|
+
patch("engine.fetch_chain.run_attempt", side_effect=lambda *a, **kw: (_fail_attempt(str(kw.get("phase", "grid"))), None)), \
|
|
240
|
+
patch("engine.fetch_chain.detect", return_value=[]), \
|
|
241
|
+
patch("engine.fetch_chain.load_profile", side_effect=lambda *a, **kw: self._profile()), \
|
|
242
|
+
patch.object(surrogate, "run_surrogate", side_effect=fake_surrogate), \
|
|
243
|
+
patch.object(ex, "run_playwright_fallback", side_effect=fake_playwright):
|
|
244
|
+
result = fetch(TARGET, enable_playwright=True, max_attempts=1)
|
|
245
|
+
|
|
246
|
+
self.assertTrue(result.ok)
|
|
247
|
+
self.assertEqual(result.provenance, "live")
|
|
248
|
+
self.assertEqual(calls, ["surrogate", "playwright"])
|
|
249
|
+
|
|
250
|
+
|
|
251
|
+
if __name__ == "__main__":
|
|
252
|
+
unittest.main()
|
package/packages/shared-skills/skills/ultimate-browsing/engine/tests/test_surrogate_validators.py
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
"""Surrogate-tier validation tests; fixtures only, never live network, so the suite cannot flake."""
|
|
2
|
+
from __future__ import annotations
|
|
3
|
+
|
|
4
|
+
import sys
|
|
5
|
+
import unittest
|
|
6
|
+
from pathlib import Path
|
|
7
|
+
|
|
8
|
+
sys.path.insert(0, str(Path(__file__).resolve().parents[2]))
|
|
9
|
+
|
|
10
|
+
from engine.result_schema import FetchResult # noqa: E402
|
|
11
|
+
from engine.validators import Verdict, validate # noqa: E402
|
|
12
|
+
|
|
13
|
+
FIXTURES = Path(__file__).resolve().parent / "fixtures"
|
|
14
|
+
|
|
15
|
+
|
|
16
|
+
def _fixture(name: str) -> str:
|
|
17
|
+
return (FIXTURES / name).read_text(encoding="utf-8", errors="replace")
|
|
18
|
+
|
|
19
|
+
|
|
20
|
+
class _Resp:
|
|
21
|
+
def __init__(self, text: str = "", status: int = 200, url: str = ""):
|
|
22
|
+
self.text = text
|
|
23
|
+
self.status_code = status
|
|
24
|
+
self.url = url
|
|
25
|
+
self.cookies = {}
|
|
26
|
+
self.headers = {}
|
|
27
|
+
|
|
28
|
+
|
|
29
|
+
class RedirectStubRejection(unittest.TestCase):
|
|
30
|
+
def test_amp_redirect_stub_is_not_ok(self) -> None:
|
|
31
|
+
body = _fixture("amp_redirect_stub.html")
|
|
32
|
+
self.assertLess(len(body), 1000)
|
|
33
|
+
resp = _Resp(body, 200, "https://en-wikipedia-org.cdn.ampproject.org/c/s/x") # NOTE-BIAS-OK — fixture URL from a third-party surrogate service, not a target site
|
|
34
|
+
vr = validate(resp, target_url="https://en.wikipedia.org/wiki/Web_archiving") # NOTE-BIAS-OK — fixture URL from a third-party surrogate service, not a target site
|
|
35
|
+
self.assertIn(vr.verdict, (Verdict.CHALLENGE, Verdict.UNKNOWN))
|
|
36
|
+
self.assertFalse(vr.ok)
|
|
37
|
+
|
|
38
|
+
def test_real_snapshot_still_passes(self) -> None:
|
|
39
|
+
resp = _Resp(_fixture("wayback_snapshot.html"), 200, "https://web.archive.org/web/2024/x") # NOTE-BIAS-OK — fixture URL from a third-party surrogate service, not a target site
|
|
40
|
+
vr = validate(resp, target_url="https://en.wikipedia.org/wiki/Web_archiving") # NOTE-BIAS-OK — fixture URL from a third-party surrogate service, not a target site
|
|
41
|
+
self.assertTrue(vr.ok)
|
|
42
|
+
|
|
43
|
+
def test_plain_page_still_weak_ok(self) -> None:
|
|
44
|
+
resp = _Resp("<html><body>" + ("x" * 4000) + "</body></html>", 200, "https://example.com/")
|
|
45
|
+
vr = validate(resp, target_url="https://example.com/")
|
|
46
|
+
self.assertTrue(vr.ok)
|
|
47
|
+
|
|
48
|
+
|
|
49
|
+
class InterstitialRejection(unittest.TestCase):
|
|
50
|
+
def test_search_engine_interstitial_is_not_ok(self) -> None:
|
|
51
|
+
body = _fixture("search_interstitial.html")
|
|
52
|
+
resp = _Resp(body, 200, "https://webcache.googleusercontent.com/search?q=cache:x") # NOTE-BIAS-OK — fixture URL from a third-party surrogate service, not a target site
|
|
53
|
+
vr = validate(resp, target_url="https://en.wikipedia.org/wiki/Web_archiving") # NOTE-BIAS-OK — fixture URL from a third-party surrogate service, not a target site
|
|
54
|
+
self.assertIn(vr.verdict, (Verdict.CHALLENGE, Verdict.UNKNOWN))
|
|
55
|
+
self.assertFalse(vr.ok)
|
|
56
|
+
|
|
57
|
+
|
|
58
|
+
class ResultSchemaProvenance(unittest.TestCase):
|
|
59
|
+
def test_defaults_are_live_origin(self) -> None:
|
|
60
|
+
r = FetchResult(ok=True)
|
|
61
|
+
self.assertEqual(r.provenance, "live")
|
|
62
|
+
self.assertEqual(r.trust, "origin")
|
|
63
|
+
self.assertIsNone(r.snapshot_timestamp)
|
|
64
|
+
d = r.to_dict()
|
|
65
|
+
self.assertEqual(d["provenance"], "live")
|
|
66
|
+
self.assertEqual(d["trust"], "origin")
|
|
67
|
+
self.assertIn("snapshot_timestamp", d)
|
|
68
|
+
|
|
69
|
+
def test_snapshot_fields_roundtrip(self) -> None:
|
|
70
|
+
r = FetchResult(ok=True, provenance="snapshot", snapshot_timestamp="20241227132135", trust="archive")
|
|
71
|
+
d = r.to_dict()
|
|
72
|
+
self.assertEqual(d["provenance"], "snapshot")
|
|
73
|
+
self.assertEqual(d["snapshot_timestamp"], "20241227132135")
|
|
74
|
+
self.assertEqual(d["trust"], "archive")
|
|
75
|
+
|
|
76
|
+
|
|
77
|
+
if __name__ == "__main__":
|
|
78
|
+
unittest.main()
|
|
@@ -43,6 +43,41 @@ CHALLENGE_MARKERS: list[str] = [
|
|
|
43
43
|
# that know their response type should pass success_selectors instead.
|
|
44
44
|
SMALL_BODY_THRESHOLD = 3000
|
|
45
45
|
|
|
46
|
+
# Surrogate-route interstitial: a search-engine front page returned in place
|
|
47
|
+
# of the cached page (dead cache services fall back to their home page).
|
|
48
|
+
INTERSTITIAL_TITLE_MARKERS: list[str] = [
|
|
49
|
+
"<title>google search</title>",
|
|
50
|
+
]
|
|
51
|
+
|
|
52
|
+
|
|
53
|
+
def is_interstitial_title(body_lower: str) -> bool:
|
|
54
|
+
return any(m in body_lower for m in INTERSTITIAL_TITLE_MARKERS)
|
|
55
|
+
|
|
56
|
+
|
|
57
|
+
def is_redirect_stub(text: str, *, target_url: Optional[str] = None) -> bool:
|
|
58
|
+
"""True when the body only exists to send the reader back to `target_url`.
|
|
59
|
+
|
|
60
|
+
AMP-cache style stubs answer HTTP 200 with a few hundred bytes whose sole
|
|
61
|
+
content is a meta-refresh / JS redirect pointing at the origin we failed
|
|
62
|
+
to fetch — accepting one loops the caller straight back into the block.
|
|
63
|
+
"""
|
|
64
|
+
if not text or len(text) >= SMALL_BODY_THRESHOLD:
|
|
65
|
+
return False
|
|
66
|
+
lowered = text.lower()
|
|
67
|
+
has_redirect = (
|
|
68
|
+
"http-equiv=\"refresh\"" in lowered
|
|
69
|
+
or "location.replace" in lowered
|
|
70
|
+
or "window.location" in lowered
|
|
71
|
+
)
|
|
72
|
+
if not has_redirect:
|
|
73
|
+
return False
|
|
74
|
+
if target_url is None:
|
|
75
|
+
return True
|
|
76
|
+
host_match = "".join(c for c in target_url.split("//")[-1].split("/")[0])
|
|
77
|
+
if not host_match:
|
|
78
|
+
return True
|
|
79
|
+
return host_match.lower() in lowered
|
|
80
|
+
|
|
46
81
|
|
|
47
82
|
class Verdict(Enum):
|
|
48
83
|
"""Three-level classification (Codex suggestion — avoid binary)."""
|
|
@@ -115,6 +150,7 @@ def validate(
|
|
|
115
150
|
success_selectors: Optional[list[str]] = None,
|
|
116
151
|
known_bad_sizes: Optional[list[int]] = None,
|
|
117
152
|
size_tolerance: int = 20,
|
|
153
|
+
target_url: Optional[str] = None,
|
|
118
154
|
) -> ValidationResult:
|
|
119
155
|
"""Validate a `curl_cffi` / `requests` response.
|
|
120
156
|
|
|
@@ -153,6 +189,16 @@ def validate(
|
|
|
153
189
|
r.reasons.extend(f"marker:{m}" for m in markers[:3])
|
|
154
190
|
return r
|
|
155
191
|
|
|
192
|
+
# --- Layer 1.5: surrogate-route dead ends (interstitial / redirect stub) --
|
|
193
|
+
if is_interstitial_title(lowered):
|
|
194
|
+
r.verdict = Verdict.CHALLENGE
|
|
195
|
+
r.reasons.append("interstitial_title")
|
|
196
|
+
return r
|
|
197
|
+
if is_redirect_stub(text, target_url=target_url):
|
|
198
|
+
r.verdict = Verdict.CHALLENGE
|
|
199
|
+
r.reasons.append("redirect_stub")
|
|
200
|
+
return r
|
|
201
|
+
|
|
156
202
|
# --- Layer 2: size fingerprints (caller hint, tolerant match) ---
|
|
157
203
|
# Fingerprint match is a strong negative signal — override even selectors.
|
|
158
204
|
if known_bad_sizes:
|
|
@@ -39,7 +39,7 @@ _DEFAULT_PROFILES: dict = {
|
|
|
39
39
|
],
|
|
40
40
|
"referer_strategies": ["self_root", "google_search", "none"],
|
|
41
41
|
"url_transform_order": ["original", "mobile_subdomain"],
|
|
42
|
-
"fallback_when_challenge": ["playwright_mcp", "playwright_real_chrome"],
|
|
42
|
+
"fallback_when_challenge": ["surrogate_wayback", "playwright_mcp", "playwright_real_chrome"],
|
|
43
43
|
"notes": "in-code default — waf_profiles.yaml unavailable",
|
|
44
44
|
},
|
|
45
45
|
}
|
|
@@ -34,21 +34,21 @@ akamai_bot_manager:
|
|
|
34
34
|
# at least a challenge page (i.e. IP still alive) rather than an outright
|
|
35
35
|
# TLS reject. Grouped by family; planner tries top groups first.
|
|
36
36
|
# Refresh quarterly — vendor WAFs shift which TLS fingerprints they trust.
|
|
37
|
-
- [safari, safari15_3, safari15_5, safari17_0, safari260]
|
|
37
|
+
- [safari, safari15_3, safari15_5, safari17_0, safari260, safari2601]
|
|
38
38
|
- [safari_ios, safari17_2_ios, safari260_ios]
|
|
39
|
-
- [chrome99, chrome100, chrome101, chrome104, chrome110, chrome116, chrome119, chrome124, chrome131, chrome133a, chrome136, chrome145, chrome146]
|
|
39
|
+
- [chrome99, chrome100, chrome101, chrome104, chrome110, chrome116, chrome119, chrome124, chrome131, chrome133a, chrome136, chrome142, chrome145, chrome146]
|
|
40
40
|
- [chrome_android, chrome131_android]
|
|
41
41
|
- [edge99, edge101]
|
|
42
42
|
tls_impersonate_avoid:
|
|
43
43
|
# Empirically observed to 403 immediately (TLS fingerprint blacklisted).
|
|
44
44
|
# DO NOT hard-block — planner deprioritizes only. Refresh quarterly.
|
|
45
|
+
# NOTE: requires curl_cffi >= 0.15.0 for chrome142/145/146 + safari260/2601 +
|
|
46
|
+
# firefox144/147 targets (added 0.14.0-0.15.0; default aliases resolve to
|
|
47
|
+
# chrome146/safari2601/firefox147 in 0.16.0+). Older installs raise ImpersonateError.
|
|
45
48
|
- safari18_0
|
|
46
49
|
- chrome107
|
|
47
50
|
- chrome120
|
|
48
51
|
- chrome123
|
|
49
|
-
- chrome145 # curl_cffi 0.11+ needed — mark avoid for older installs
|
|
50
|
-
- chrome146
|
|
51
|
-
- firefox
|
|
52
52
|
- firefox133
|
|
53
53
|
- firefox135
|
|
54
54
|
referer_strategies:
|
|
@@ -57,6 +57,7 @@ akamai_bot_manager:
|
|
|
57
57
|
- original
|
|
58
58
|
- mobile_subdomain # www.* → m.* — strong observational win in SSR sites
|
|
59
59
|
fallback_when_challenge:
|
|
60
|
+
- surrogate_wayback # Phase 2.5: archive/reader copy before a browser spin-up
|
|
60
61
|
- curl_grid_exhaust # try more impersonate × referer × url combos
|
|
61
62
|
- playwright_real_chrome
|
|
62
63
|
notes: |
|
|
@@ -77,6 +78,7 @@ cloudflare_turnstile:
|
|
|
77
78
|
- google_search
|
|
78
79
|
- self_root
|
|
79
80
|
fallback_when_challenge:
|
|
81
|
+
- surrogate_wayback # Phase 2.5: archive/reader copy before a browser spin-up
|
|
80
82
|
- playwright_mcp # MCP sufficient; Chromium TLS passes CF baseline
|
|
81
83
|
- playwright_real_chrome
|
|
82
84
|
|
|
@@ -112,6 +114,7 @@ datadome_probable:
|
|
|
112
114
|
tls_impersonate_candidates:
|
|
113
115
|
- [safari, chrome]
|
|
114
116
|
fallback_when_challenge:
|
|
117
|
+
- surrogate_wayback # Phase 2.5: archive/reader copy before a browser spin-up
|
|
115
118
|
- playwright_real_chrome
|
|
116
119
|
notes: |
|
|
117
120
|
"_probable" suffix reminds us this is a growing attack surface — mark
|
|
@@ -127,6 +130,7 @@ perimeterx_human:
|
|
|
127
130
|
tls_impersonate_candidates:
|
|
128
131
|
- [safari, chrome]
|
|
129
132
|
fallback_when_challenge:
|
|
133
|
+
- surrogate_wayback # Phase 2.5: archive/reader copy before a browser spin-up
|
|
130
134
|
- playwright_real_chrome
|
|
131
135
|
notes: |
|
|
132
136
|
PerimeterX (now HUMAN Bot Defender). Distinct cookie family from
|
|
@@ -154,6 +158,7 @@ unknown_challenge:
|
|
|
154
158
|
- original
|
|
155
159
|
- mobile_subdomain
|
|
156
160
|
fallback_when_challenge:
|
|
161
|
+
- surrogate_wayback # Phase 2.5: archive/reader copy before a browser spin-up
|
|
157
162
|
- playwright_mcp
|
|
158
163
|
- playwright_real_chrome
|
|
159
164
|
notes: |
|
|
@@ -40,7 +40,7 @@ xhs favorites # 可能返回 API error
|
|
|
40
40
|
>
|
|
41
41
|
> **频率控制**: 高频请求(批量搜索、深翻评论)会触发验证码,这是平台限制无法绕过。建议每次操作间隔 2-3 秒。
|
|
42
42
|
>
|
|
43
|
-
> **POST
|
|
43
|
+
> **POST 操作**: 发帖(post)、评论(comment)、点赞(like) 等写操作在 v0.6.4 已修复签名问题 (PR [#19](https://github.com/jackwener/xiaohongshu-cli/pull/19)),可正常使用。
|
|
44
44
|
|
|
45
45
|
## 抖音 / Douyin
|
|
46
46
|
|
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
Real interaction (clicks, forms, screenshots, video, persistent login) for pages that defeat Tier 1/1.5. Two runtime tools, both installed on demand — neither is vendored in this skill:
|
|
4
4
|
|
|
5
|
-
- **CloakBrowser** (`pip`) — stealth Chromium with source-level C++ fingerprint patches. The Python wrapper source is MIT; the downloaded Chromium binary is covered by CloakBrowser's separate binary license and is not redistributed by this package. Passes Cloudflare Turnstile, FingerprintJS, BrowserScan, and 30+ detectors. Pin **0.
|
|
6
|
-
- **agent-browser** (`npm`, Apache-2.0) — native CDP automation CLI that drives CloakBrowser. AX-tree snapshots, `@eN` refs, click/fill/type/scroll, screenshots, video, cookie/state/session management. Pin **0.
|
|
5
|
+
- **CloakBrowser** (`pip`) — stealth Chromium with source-level C++ fingerprint patches. The Python wrapper source is MIT; the downloaded Chromium binary is covered by CloakBrowser's separate binary license and is not redistributed by this package. Passes Cloudflare Turnstile, FingerprintJS, BrowserScan, and 30+ detectors. Pin **0.5.5**.
|
|
6
|
+
- **agent-browser** (`npm`, Apache-2.0) — native CDP automation CLI that drives CloakBrowser. AX-tree snapshots, `@eN` refs, click/fill/type/scroll, screenshots, video, cookie/state/session management. Pin **0.33.2**.
|
|
7
7
|
|
|
8
8
|
```
|
|
9
9
|
CloakBrowser (stealth Chromium) <- CDP port 9242 -> agent-browser CLI
|
|
@@ -18,22 +18,22 @@ CloakBrowser (stealth Chromium) <- CDP port 9242 -> agent-browser CLI
|
|
|
18
18
|
CloakBrowser runs in a dedicated Python venv. Cross-platform: macOS, Linux, and Windows all supported by both tools (use the venv path convention for your OS).
|
|
19
19
|
|
|
20
20
|
```bash
|
|
21
|
-
# CloakBrowser (MIT wrapper source; separate binary license, pin 0.
|
|
21
|
+
# CloakBrowser (MIT wrapper source; separate binary license, pin 0.5.5):
|
|
22
22
|
uv venv .cloak-venv --python 3.13
|
|
23
23
|
# macOS/Linux: source .cloak-venv/bin/activate Windows: .cloak-venv\Scripts\activate
|
|
24
|
-
uv pip install "cloakbrowser==0.
|
|
24
|
+
uv pip install "cloakbrowser==0.5.5"
|
|
25
25
|
python -c "import cloakbrowser; cloakbrowser.ensure_binary()" # downloads stealth Chromium on first import
|
|
26
26
|
|
|
27
|
-
# agent-browser (Apache-2.0, pin 0.
|
|
28
|
-
npm i -g agent-browser@0.
|
|
29
|
-
agent-browser --version # 0.
|
|
27
|
+
# agent-browser (Apache-2.0, pin 0.33.2):
|
|
28
|
+
npm i -g agent-browser@0.33.2 && agent-browser install
|
|
29
|
+
agent-browser --version # 0.33.2
|
|
30
30
|
```
|
|
31
31
|
|
|
32
32
|
Verify CloakBrowser:
|
|
33
33
|
|
|
34
34
|
```bash
|
|
35
35
|
python -c "import cloakbrowser; print(cloakbrowser.__version__, cloakbrowser.CHROMIUM_VERSION, cloakbrowser.binary_info()['installed'])"
|
|
36
|
-
# -> 0.
|
|
36
|
+
# -> 0.5.5 <chromium-version> True
|
|
37
37
|
```
|
|
38
38
|
|
|
39
39
|
## Launch + drive
|
|
@@ -76,7 +76,9 @@ agent-browser skills list # everything available on the installed
|
|
|
76
76
|
agent-browser --cdp 9242 eval 'navigator.webdriver' # must print false
|
|
77
77
|
```
|
|
78
78
|
|
|
79
|
-
Verified 2026-07 with CloakBrowser 0.
|
|
79
|
+
Verified 2026-07 with CloakBrowser 0.5.5 + agent-browser 0.33.2: `navigator.webdriver` reads the boolean false with no init-script, bot.sannysoft.com all-green, browserscan.net "Normal" (15/15), nowsecure.nl Turnstile bypassed.
|
|
80
|
+
|
|
81
|
+
> **agent-browser 0.33.x behavior note:** the daemon now defaults to a 1-hour idle timeout (saves restore state, closes the browser, exits after 1 h of no commands). Set `AGENT_BROWSER_IDLE_TIMEOUT_MS=0` to restore the old always-persist behavior. External WebSocket stream consumers see latest-wins frame delivery; `record` (CDP) and the dashboard are unaffected.
|
|
80
82
|
|
|
81
83
|
## Cookie login (cross-platform)
|
|
82
84
|
|
|
@@ -115,6 +117,6 @@ lsof -ti:9242 | xargs kill -9
|
|
|
115
117
|
# agent-browser can't connect:
|
|
116
118
|
curl -s http://127.0.0.1:9242/json/version | head -5 # empty -> CloakBrowser not running
|
|
117
119
|
# Update either tool:
|
|
118
|
-
uv pip install --upgrade "cloakbrowser==0.
|
|
119
|
-
npm i -g agent-browser@0.
|
|
120
|
+
uv pip install --upgrade "cloakbrowser==0.5.5" && python -c "import cloakbrowser; cloakbrowser.ensure_binary()"
|
|
121
|
+
npm i -g agent-browser@0.33.2
|
|
120
122
|
```
|
|
@@ -71,9 +71,9 @@
|
|
|
71
71
|
|------------|------|
|
|
72
72
|
| URL 제공 (`https://...`) | → Phase 0 검사 후 없으면 Phase 1 (generic fetch chain) |
|
|
73
73
|
| 핸들 제공 (`@username`) | → Phase 0 syndication/API |
|
|
74
|
-
| 키워드만 ("X에서 AI 검색") | →
|
|
74
|
+
| 키워드만 ("X에서 AI 검색") | → 웹 검색 도구(`site:{domain} {keyword}`) 먼저 → URL 확보 후 재진입 |
|
|
75
75
|
|
|
76
|
-
> **한국어 신규 콘텐츠 한계**: 네이버/다음/한국 커뮤니티의 키워드 검색은
|
|
76
|
+
> **한국어 신규 콘텐츠 한계**: 네이버/다음/한국 커뮤니티의 키워드 검색은 웹 검색 도구 경유가 유일하며, 신규 콘텐츠 인덱싱이 지연될 수 있다.
|
|
77
77
|
|
|
78
78
|
## Phase 0 — 플랫폼 공식 API 인덱스
|
|
79
79
|
|
|
@@ -83,7 +83,7 @@
|
|
|
83
83
|
|
|
84
84
|
| 플랫폼 | 방법 | 상세 |
|
|
85
85
|
|--------|------|------|
|
|
86
|
-
| X/Twitter | syndication (타임라인) + oEmbed (개별 트윗) + 키워드 검색:
|
|
86
|
+
| X/Twitter | syndication (타임라인) + oEmbed (개별 트윗) + 키워드 검색: 웹 검색 도구 → oEmbed | [twitter.md](twitter.md) |
|
|
87
87
|
| Reddit | URL + `.json` + Mobile UA | [json-api.md](json-api.md) |
|
|
88
88
|
| Bluesky | AT Protocol (`public.api.bsky.app/xrpc/...`) | [public-api.md](public-api.md) |
|
|
89
89
|
| Mastodon | 인스턴스별 공개 API | [public-api.md](public-api.md) |
|
|
@@ -301,7 +301,7 @@ yt-dlp --write-sub --write-auto-sub --sub-lang "en,ko" --skip-download -o "/tmp/
|
|
|
301
301
|
|------|-------------|-----------------|
|
|
302
302
|
| [`json-api.md`](json-api.md) | Reddit/Wikipedia/HN/npm/PyPI 등 **URL 변형만으로** JSON을 주는 사이트 | Reddit `/json` suffix + Mobile UA, HN Firebase, Algolia Search, Wikipedia REST, npm/PyPI Registry API |
|
|
303
303
|
| [`public-api.md`](public-api.md) | Bluesky/Mastodon/arXiv/Stack Overflow/CrossRef/GitHub/OpenLibrary/Wayback 공식 API 사용 시 | 인증 없이 쓰는 공식 공개 REST/AT/Atom API 엔드포인트, 요청 형식, 공통 파라미터 |
|
|
304
|
-
| [`twitter.md`](twitter.md) | X/Twitter 접근 — 프로필 타임라인, 특정 트윗, 키워드 검색 | `syndication.twitter.com` 타임라인, oEmbed 개별 트윗, 검색은
|
|
304
|
+
| [`twitter.md`](twitter.md) | X/Twitter 접근 — 프로필 타임라인, 특정 트윗, 키워드 검색 | `syndication.twitter.com` 타임라인, oEmbed 개별 트윗, 검색은 웹 검색 도구로 URL 확보 후 oEmbed |
|
|
305
305
|
| [`naver.md`](naver.md) | 네이버 블로그·뉴스·증권·검색 접근 | 서비스별 우회(블로그는 `m.blog.naver.com` 변환, 증권은 비공식 JSON, 검색은 `search.naver.com`), 한글 검색 쿼리 패턴 |
|
|
306
306
|
| [`media.md`](media.md) | YouTube/Vimeo/Twitch/TikTok/SoundCloud 등 미디어 메타·자막·오디오 필요 시 | `yt-dlp --dump-json` 기반 1,858개 사이트 커버, 자막 다운로드(`--write-sub`), 포맷 선택, 라이브/팟캐스트 |
|
|
307
307
|
|