argo-search 1.0.1 → 2.9.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +201 -21
- package/README.en.md +822 -0
- package/README.es.md +864 -0
- package/README.ja.md +864 -0
- package/README.ko.md +864 -0
- package/README.md +867 -320
- package/SKILL.md +135 -0
- package/assets/readme/hero.svg +121 -0
- package/assets/readme/made-with-beautify.svg +10 -0
- package/assets/readme/proof-routes.svg +73 -0
- package/assets/readme/settings-keys.png +0 -0
- package/assets/readme/settings-tools.png +0 -0
- package/assets/readme/why-better.svg +104 -0
- package/assets/readme/workflow.svg +83 -0
- package/backends/domain_profiles.json +2308 -96
- package/backends/engine_registry.yaml +3314 -503
- package/backends/query_synonyms_cn.json +35 -0
- package/backends/quota_profiles.json +2409 -161
- package/backends/source_types_cn.json +83 -0
- package/backends/text_lexicon.json +44 -0
- package/bin/argo +703 -0
- package/bin/argo.js +71 -39
- package/config.yaml +4976 -363
- package/cordis.patch.yml +21 -0
- package/data/countries.json +40 -0
- package/data/lang_matrix/lang_capability.json +842 -0
- package/data/lang_matrix/matrix_2026-09-10.json +11921 -0
- package/data/regions_cn.json +1 -0
- package/docs/ADDING_NEW_ENGINE.md +184 -0
- package/docs/ARGO_INTRO.md +124 -0
- package/docs/DATA_SOURCE_GAPS.md +81 -0
- package/docs/DESIGN_LOCAL_DATA_FUSION.md +90 -0
- package/docs/ENGINE_CATALOG.md +564 -0
- package/docs/MCP_SETUP.md +106 -0
- package/docs/PLATFORM_PATHS.md +111 -0
- package/docs/RELEASE_NOTES_v2.8.0.md +71 -0
- package/docs/RELEASE_NOTES_v2.8.2.md +63 -0
- package/docs/RELEASE_NOTES_v2.8.3.md +55 -0
- package/docs/RELEASE_NOTES_v2.8.4.md +79 -0
- package/docs/RELEASE_NOTES_v2.8.5.md +102 -0
- package/docs/RELEASE_NOTES_v2.8.6.md +66 -0
- package/docs/RELEASE_NOTES_v2.8.7.md +65 -0
- package/docs/RELEASE_NOTES_v2.8.8.md +67 -0
- package/docs/RELEASE_NOTES_v2.8.9.md +67 -0
- package/docs/RELEASE_NOTES_v2.9.0.md +64 -0
- package/docs/SEARCH_ARCHIVE.md +115 -0
- package/docs/assets/architecture.svg +104 -0
- package/docs/assets/hero.svg +98 -0
- package/docs/engines/bangumi.md +27 -0
- package/docs/engines/biorxiv.md +27 -0
- package/docs/engines/clawhub.md +27 -0
- package/docs/engines/crt_sh.md +27 -0
- package/docs/engines/datacite.md +27 -0
- package/docs/engines/deps_dev.md +27 -0
- package/docs/engines/douban_movie.md +27 -0
- package/docs/engines/endoflife.md +27 -0
- package/docs/engines/flk_law.md +27 -0
- package/docs/engines/fxtwitter.md +27 -0
- package/docs/engines/google_news.md +27 -0
- package/docs/engines/gov_regulations.md +27 -0
- package/docs/engines/guqian.md +27 -0
- package/docs/engines/hackernews.md +27 -0
- package/docs/engines/iplant.md +27 -0
- package/docs/engines/keenable.md +27 -0
- package/docs/engines/met_museum.md +27 -0
- package/docs/engines/nasa_images.md +27 -0
- package/docs/engines/netease_music.md +27 -0
- package/docs/engines/npm.md +27 -0
- package/docs/engines/nvd.md +27 -0
- package/docs/engines/openreview.md +27 -0
- package/docs/engines/openstd.md +27 -0
- package/docs/engines/parallel_free.md +27 -0
- package/docs/engines/people_daily.md +27 -0
- package/docs/engines/realtime_index.md +27 -0
- package/docs/engines/sec_edgar.md +27 -0
- package/docs/engines/seltz.md +27 -0
- package/docs/engines/skillsmp.md +27 -0
- package/docs/engines/sspai.md +27 -0
- package/docs/engines/std_samr.md +27 -0
- package/docs/engines/train.md +27 -0
- package/docs/engines/un_comtrade.md +27 -0
- package/docs/engines/wikisource.md +27 -0
- package/docs/engines/zdic.md +27 -0
- package/docs/engines/zenodo.md +27 -0
- package/docs/engines/zhihu_hot_app.md +27 -0
- package/docs//344/270/272/344/273/200/344/271/210/351/200/211/346/213/251argo.md +176 -0
- package/engines/README.md +10 -0
- package/engines/_template_cli.yaml +39 -0
- package/engines/_template_http.yaml +37 -0
- package/engines/plugins/_template_plugin.py +50 -0
- package/engines/specs/_cisa_kev.yaml +54 -0
- package/engines/specs/_osv.yaml +63 -0
- package/engines/specs/aljazeera.yaml +43 -0
- package/engines/specs/antara.yaml +43 -0
- package/engines/specs/artic.yaml +20 -0
- package/engines/specs/aviation_weather.yaml +39 -0
- package/engines/specs/bangumi.yaml +21 -0
- package/engines/specs/biorxiv.yaml +24 -0
- package/engines/specs/carbon_intensity.yaml +32 -0
- package/engines/specs/cinii.yaml +44 -0
- package/engines/specs/clawhub.yaml +49 -0
- package/engines/specs/cleveland.yaml +19 -0
- package/engines/specs/cn_ai_news.yaml +43 -0
- package/engines/specs/core.yaml +35 -0
- package/engines/specs/crt_sh.yaml +38 -0
- package/engines/specs/ct24.yaml +43 -0
- package/engines/specs/datacite.yaml +27 -0
- package/engines/specs/datagov_uk.yaml +41 -0
- package/engines/specs/ddgs_images.yaml +45 -0
- package/engines/specs/deepseek_search.yaml +74 -0
- package/engines/specs/deezer.yaml +20 -0
- package/engines/specs/deps_dev.yaml +21 -0
- package/engines/specs/digup.yaml +36 -0
- package/engines/specs/doi.yaml +39 -0
- package/engines/specs/douban_movie.yaml +22 -0
- package/engines/specs/dw_news.yaml +47 -0
- package/engines/specs/egov_law.yaml +21 -0
- package/engines/specs/elpais.yaml +43 -0
- package/engines/specs/endoflife.yaml +21 -0
- package/engines/specs/energy_charts.yaml +21 -0
- package/engines/specs/factcheck_org.yaml +52 -0
- package/engines/specs/faz.yaml +43 -0
- package/engines/specs/federal_register.yaml +57 -0
- package/engines/specs/firecrawl.yaml +53 -0
- package/engines/specs/flk_law.yaml +24 -0
- package/engines/specs/folha.yaml +43 -0
- package/engines/specs/france24.yaml +44 -0
- package/engines/specs/full_fact.yaml +47 -0
- package/engines/specs/fxtwitter.yaml +31 -0
- package/engines/specs/gbfs_nyc.yaml +32 -0
- package/engines/specs/gdacs.yaml +21 -0
- package/engines/specs/gdelt.yaml +64 -0
- package/engines/specs/google_news.yaml +22 -0
- package/engines/specs/gov_regulations.yaml +38 -0
- package/engines/specs/guardian_rss.yaml +59 -0
- package/engines/specs/hurriyet.yaml +43 -0
- package/engines/specs/iplant.yaml +25 -0
- package/engines/specs/jikan.yaml +24 -0
- package/engines/specs/jolpica.yaml +20 -0
- package/engines/specs/k10plus.yaml +20 -0
- package/engines/specs/keenable.yaml +44 -0
- package/engines/specs/lefigaro.yaml +43 -0
- package/engines/specs/listenbrainz.yaml +19 -0
- package/engines/specs/matichon.yaml +43 -0
- package/engines/specs/met_museum.yaml +21 -0
- package/engines/specs/nasa_images.yaml +45 -0
- package/engines/specs/netease_music.yaml +38 -0
- package/engines/specs/nhk.yaml +43 -0
- package/engines/specs/nhtsa_vpic.yaml +20 -0
- package/engines/specs/noaa_swpc.yaml +20 -0
- package/engines/specs/nvd.yaml +38 -0
- package/engines/specs/obis.yaml +20 -0
- package/engines/specs/opencitations.yaml +52 -0
- package/engines/specs/openf1.yaml +21 -0
- package/engines/specs/openfoodfacts.yaml +45 -0
- package/engines/specs/openligadb.yaml +20 -0
- package/engines/specs/openreview.yaml +37 -0
- package/engines/specs/openstd.yaml +21 -0
- package/engines/specs/people_daily.yaml +22 -0
- package/engines/specs/realtime_index.yaml +34 -0
- package/engines/specs/ror.yaml +20 -0
- package/engines/specs/satnogs.yaml +20 -0
- package/engines/specs/sec_edgar.yaml +46 -0
- package/engines/specs/skillsmp.yaml +50 -0
- package/engines/specs/sogou_weixin_browser.yaml +30 -0
- package/engines/specs/soilgrids.yaml +25 -0
- package/engines/specs/sspai.yaml +36 -0
- package/engines/specs/stackexchange.yaml +46 -0
- package/engines/specs/std_samr.yaml +21 -0
- package/engines/specs/tass.yaml +43 -0
- package/engines/specs/tle_mirror.yaml +21 -0
- package/engines/specs/train.yaml +37 -0
- package/engines/specs/tvmaze.yaml +20 -0
- package/engines/specs/twitter_syndication.yaml +31 -0
- package/engines/specs/un_comtrade.yaml +23 -0
- package/engines/specs/unpaywall.yaml +54 -0
- package/engines/specs/vnpress.yaml +43 -0
- package/engines/specs/weather.yaml +32 -0
- package/engines/specs/who_don.yaml +21 -0
- package/engines/specs/who_gho.yaml +20 -0
- package/engines/specs/wikimedia_commons.yaml +34 -0
- package/engines/specs/wikisource.yaml +22 -0
- package/engines/specs/worms.yaml +19 -0
- package/engines/specs/x_trending.yaml +34 -0
- package/engines/specs/yna.yaml +43 -0
- package/engines/specs/ynet.yaml +43 -0
- package/engines/specs/zbmath.yaml +40 -0
- package/engines/specs/zdic.yaml +22 -0
- package/engines/specs/zenodo.yaml +27 -0
- package/engines/specs/zhihu_hot_app.yaml +35 -0
- package/mcp/clients.yaml +74 -0
- package/package.json +51 -12
- package/packages/dsh-plugin/README.md +134 -0
- package/packages/dsh-plugin/cordis.patch.yml +48 -0
- package/packages/dsh-plugin/dsh/client.js +1244 -0
- package/packages/dsh-plugin/dsh/index.js +2290 -0
- package/packages/dsh-plugin/dsh/native-tools.mjs +743 -0
- package/packages/dsh-plugin/package.json +38 -0
- package/packages/dsh-plugin/test/client.test.js +232 -0
- package/packages/dsh-plugin/test/config.test.js +30 -0
- package/packages/dsh-plugin/test/index.test.js +1338 -0
- package/references/query-templates.md +96 -0
- package/references/query-templates.yaml +118 -0
- package/scripts/ab_eval.py +219 -0
- package/scripts/ab_eval_p0p1.py +266 -0
- package/scripts/adaptive.py +271 -70
- package/scripts/answer.py +154 -0
- package/scripts/archive_run.py +591 -0
- package/scripts/argo_engine_registry.py +172 -0
- package/scripts/argo_paths.py +728 -0
- package/scripts/article.py +136 -0
- package/scripts/arxiv_source.py +201 -0
- package/scripts/badcase_ledger.py +252 -0
- package/scripts/batch_probe.py +241 -0
- package/scripts/benchmark.py +19 -0
- package/scripts/bounded_run.py +144 -0
- package/scripts/browser_auth.py +323 -0
- package/scripts/burstiness_detector.py +147 -0
- package/scripts/cache.py +1078 -67
- package/scripts/cache_guard.py +271 -0
- package/scripts/cache_key_vdom.py +56 -0
- package/scripts/cache_similarity.py +170 -0
- package/scripts/caliber_isolation.py +195 -0
- package/scripts/calibrate_lowquality.py +210 -0
- package/scripts/candidate_envelope.py +366 -0
- package/scripts/chrome_cdp.py +871 -0
- package/scripts/circuit_breaker.py +389 -0
- package/scripts/citations.py +334 -0
- package/scripts/cite_cli.py +59 -0
- package/scripts/cjk_tokens.py +95 -0
- package/scripts/clarify.py +47 -24
- package/scripts/cli_io.py +240 -0
- package/scripts/config.py +683 -45
- package/scripts/config_cache.py +402 -0
- package/scripts/conn_pool.py +220 -0
- package/scripts/content_security.py +685 -0
- package/scripts/content_signals.py +730 -0
- package/scripts/crawl.py +74 -35
- package/scripts/daemon.py +866 -0
- package/scripts/digup_bridge.py +214 -0
- package/scripts/engine_admission.py +366 -0
- package/scripts/engine_dispatch.py +1019 -0
- package/scripts/engine_env.py +534 -0
- package/scripts/engine_failure.py +411 -0
- package/scripts/engine_families.py +732 -0
- package/scripts/engine_policy.py +218 -0
- package/scripts/engine_requires.py +190 -0
- package/scripts/engine_status.py +322 -0
- package/scripts/engine_validate.py +638 -0
- package/scripts/engines.py +778 -294
- package/scripts/engines_base.py +1521 -0
- package/scripts/engines_builders.py +298 -0
- package/scripts/engines_builders_batch10.py +159 -0
- package/scripts/engines_builders_batch11.py +174 -0
- package/scripts/engines_builders_batch12.py +103 -0
- package/scripts/engines_builders_batch13.py +340 -0
- package/scripts/engines_builders_batch9.py +1468 -0
- package/scripts/engines_builders_cn.py +2283 -0
- package/scripts/engines_builders_data.py +2923 -0
- package/scripts/engines_builders_data_macro.py +775 -0
- package/scripts/engines_builders_feeds.py +255 -0
- package/scripts/engines_builders_intl.py +1239 -0
- package/scripts/engines_builders_search.py +440 -0
- package/scripts/engines_builders_tech.py +1360 -0
- package/scripts/evidence.py +588 -162
- package/scripts/evidence_loop.py +592 -0
- package/scripts/evidence_tier.py +298 -0
- package/scripts/except_sets.py +25 -0
- package/scripts/extract.py +55 -16
- package/scripts/fact_align.py +207 -0
- package/scripts/fact_crosscheck.py +230 -0
- package/scripts/fetch.py +25 -12
- package/scripts/fetch_browser.py +87 -0
- package/scripts/fetch_jsrun.py +122 -0
- package/scripts/fetch_quality.py +225 -0
- package/scripts/fetch_render_tinyfish.py +117 -0
- package/scripts/fetch_v3.py +1779 -0
- package/scripts/focus_extract.py +350 -0
- package/scripts/fulltext_store.py +169 -0
- package/scripts/fxtwitter_api.py +325 -0
- package/scripts/gen_engine_catalog.py +364 -0
- package/scripts/gen_native_tools.py +91 -0
- package/scripts/gen_routing_snapshot.py +184 -0
- package/scripts/health_check.py +140 -402
- package/scripts/health_probe.py +23 -8
- package/scripts/hot_state.py +134 -0
- package/scripts/http_client.py +1053 -0
- package/scripts/image_ops.py +430 -0
- package/scripts/image_vision/vision_probe +0 -0
- package/scripts/image_vision/vision_probe.swift +126 -0
- package/scripts/install.ps1 +148 -0
- package/scripts/install.sh +206 -0
- package/scripts/job.py +1186 -0
- package/scripts/jump_resolver.py +235 -0
- package/scripts/lang_capability.py +150 -0
- package/scripts/lang_detect.py +289 -0
- package/scripts/lang_matrix_probe.py +329 -0
- package/scripts/lang_pref.py +311 -0
- package/scripts/link_source.py +340 -0
- package/scripts/local_engines.py +321 -0
- package/scripts/local_image.py +970 -0
- package/scripts/local_seek.py +366 -0
- package/scripts/macro_countries.py +82 -0
- package/scripts/matrix_search_eval.py +1105 -0
- package/scripts/mcp_diag.py +8 -3
- package/scripts/mcp_handlers.py +1074 -0
- package/scripts/mcp_handlers_surface.py +157 -0
- package/scripts/mcp_launch.sh +122 -0
- package/scripts/mcp_server.py +118 -417
- package/scripts/mcp_setup.py +407 -0
- package/scripts/mcp_tools.py +330 -0
- package/scripts/mcp_transport.py +191 -0
- package/scripts/min_version_scan.py +335 -0
- package/scripts/net_proxy.py +258 -0
- package/scripts/network_aware.py +152 -0
- package/scripts/paper.py +274 -0
- package/scripts/pdf_extract.py +518 -0
- package/scripts/phrase_filter.py +66 -0
- package/scripts/plan.py +422 -0
- package/scripts/postprocess_signals.py +150 -0
- package/scripts/pre-commit +48 -0
- package/scripts/query_enhance.py +157 -0
- package/scripts/query_operators.py +196 -0
- package/scripts/query_rewriter.py +116 -7
- package/scripts/query_signals.py +300 -0
- package/scripts/query_templates.py +167 -0
- package/scripts/query_understanding.py +411 -0
- package/scripts/query_variants.py +298 -0
- package/scripts/quota.py +488 -76
- package/scripts/rank_signals.py +775 -0
- package/scripts/ranking_eval.py +319 -0
- package/scripts/readability_extract.py +433 -0
- package/scripts/recompute.py +415 -0
- package/scripts/recovery.py +616 -0
- package/scripts/recovery_l0.py +160 -0
- package/scripts/redskill/redskill_engine.py +332 -0
- package/scripts/regression_p0p1.py +318 -0
- package/scripts/replay_eval.py +640 -0
- package/scripts/rerank_tfidf.py +225 -0
- package/scripts/research.py +591 -403
- package/scripts/research_cli.py +361 -0
- package/scripts/research_dossier.py +492 -0
- package/scripts/research_expand.py +146 -0
- package/scripts/research_gates.py +272 -0
- package/scripts/research_report.py +209 -0
- package/scripts/research_strategy.py +37 -0
- package/scripts/research_work_packages.py +195 -0
- package/scripts/result_lang.py +311 -0
- package/scripts/robots_guard.py +259 -0
- package/scripts/route.py +921 -254
- package/scripts/route_bangs.py +40 -0
- package/scripts/route_cache.py +225 -0
- package/scripts/route_combo.py +910 -0
- package/scripts/route_domains.py +328 -0
- package/scripts/route_lang.py +605 -0
- package/scripts/route_log.py +63 -0
- package/scripts/route_policy.py +250 -0
- package/scripts/rrf_classic.py +66 -0
- package/scripts/screenshot.py +73 -0
- package/scripts/search.py +897 -405
- package/scripts/search_attribution.py +88 -0
- package/scripts/search_benchmark.py +407 -0
- package/scripts/search_cli.py +627 -0
- package/scripts/search_entry.py +389 -0
- package/scripts/search_output.py +606 -0
- package/scripts/search_pipeline.py +1077 -0
- package/scripts/search_rank.py +991 -0
- package/scripts/search_types.py +27 -12
- package/scripts/seek_locator.py +68 -0
- package/scripts/semantic_evidence.py +215 -0
- package/scripts/serp_guard.py +228 -0
- package/scripts/serp_kill_log.py +51 -0
- package/scripts/single_flight.py +120 -0
- package/scripts/social_engines/__init__.py +10 -0
- package/scripts/social_engines/bilibili_engine.py +131 -0
- package/scripts/social_engines/hackernews_engine.py +113 -0
- package/scripts/social_engines/reddit_engine.py +172 -0
- package/scripts/social_engines/twitter_engine.py +397 -0
- package/scripts/social_engines/v2ex_engine.py +101 -0
- package/scripts/social_engines/weibo_engine.py +128 -0
- package/scripts/social_engines/xiaohongshu_engine.py +137 -0
- package/scripts/social_engines/zhihu_engine.py +139 -0
- package/scripts/social_research.py +201 -0
- package/scripts/sogou_weixin_browser.py +318 -0
- package/scripts/source_ledger.py +249 -0
- package/scripts/stage_timing.py +58 -0
- package/scripts/stats_cli.py +95 -0
- package/scripts/straggler_policy.py +60 -0
- package/scripts/stylometry_detector.py +231 -0
- package/scripts/sync_backends.py +353 -0
- package/scripts/sync_installs.py +29 -0
- package/scripts/template_cli.py +127 -0
- package/scripts/text_lexicon.py +72 -0
- package/scripts/tfidf_router.py +109 -37
- package/scripts/time_utils.py +233 -0
- package/scripts/topic_research_profiles.py +643 -0
- package/scripts/train.py +737 -0
- package/scripts/tweet.py +596 -0
- package/scripts/url_canon.py +217 -0
- package/scripts/url_optimize.py +56 -0
- package/scripts/url_safety.py +236 -0
- package/scripts/usage_log.py +116 -0
- package/scripts/usage_log_clean.py +113 -0
- package/scripts/v2ex_nodes.py +311 -0
- package/scripts/variant_recall.py +188 -0
- package/scripts/watch.py +205 -0
- package/scripts/wechat_article.py +258 -0
- package/scripts/wide_research.py +402 -0
- package/scripts/wx.py +349 -0
- package/scripts/x_article_fetch.py +141 -0
- package/scripts/x_search.py +234 -0
- package/scripts/x_trending.py +189 -0
- package/scripts/yaml_load.py +67 -0
- package/sub-skills/ego-search/SKILL.md +253 -0
- package/sub-skills/ego-search/references/browser-runtime.md +43 -0
- package/sub-skills/ego-search/references/install.md +67 -0
- package/sub-skills/ego-search/references/original-ego-upgrade.md +51 -0
- package/sub-skills/ego-search/scripts/ego_search.py +800 -0
- package/sub-skills/ego-search/scripts/merge.py +226 -0
- package/sub-skills/ego-search/scripts/quality.py +64 -0
- package/sub-skills/ego-search/scripts/runtime.py +172 -0
- package/sub-skills/ego-search/scripts/safety.py +109 -0
- package/sub-skills/ego-search/scripts/serp_spec.py +109 -0
- package/sub-skills/ego-search/scripts/webbridge_adapter.py +374 -0
- package/sub-skills/ego-search/tests/test_query_normalize.py +68 -0
- package/sub-skills/js-run/SKILL.md +67 -0
- package/sub-skills/js-run/scripts/jsrun.py +112 -0
- package/sub-skills/js-run/scripts/shim.js +188 -0
- package/sub-skills/js-run/tests/fixtures/challenge_like.js +27 -0
- package/sub-skills/js-run/tests/test_jsrun.py +165 -0
- package/sub-skills/local-search/SKILL.md +50 -53
- package/sub-skills/local-search/bm25.py +162 -0
- package/sub-skills/local-search/config.yaml +152 -55
- package/sub-skills/local-search/engine_registry.py +91 -26
- package/sub-skills/local-search/{health_check.py → local_health_check.py} +140 -10
- package/sub-skills/local-search/local_search_adapter.py +13 -2
- package/sub-skills/local-search/parse_maps.yaml +40 -42
- package/sub-skills/local-search/search_v3.py +1182 -65
- package/sub-skills/local-search/smart_router.py +47 -7
- package/sub-skills/local-seek/SKILL.md +110 -0
- package/sub-skills/local-seek/config/domains.yaml +20 -0
- package/sub-skills/local-seek/references/auto-behaviors.md +24 -0
- package/sub-skills/local-seek/references/fd-mdfind-recipes.md +62 -0
- package/sub-skills/local-seek/references/git-integration.md +54 -0
- package/sub-skills/local-seek/references/remote-handoff.md +46 -0
- package/sub-skills/local-seek/references/rg-recipes.md +96 -0
- package/sub-skills/local-seek/references/strategies.md +52 -0
- package/sub-skills/local-seek/references/structural-search.md +59 -0
- package/sub-skills/local-seek/scripts/eval_rank.py +216 -0
- package/sub-skills/local-seek/scripts/eval_seek.py +224 -0
- package/sub-skills/local-seek/scripts/seek.py +1465 -0
- package/sub-skills/local-seek/scripts/seek_frecency.py +126 -0
- package/sub-skills/local-seek/scripts/seek_fts.py +430 -0
- package/sub-skills/local-seek/scripts/seek_rank.py +193 -0
- package/sub-skills/local-seek/tests/test_content_rank.py +222 -0
- package/sub-skills/local-seek/tests/test_determinism_20261006.py +145 -0
- package/sub-skills/local-seek/tests/test_noise_tier.py +101 -0
- package/sub-skills/local-seek/tests/test_pcre2_probe.py +136 -0
- package/sub-skills/local-seek/tests/test_rank_pinyin.py +77 -0
- package/sub-skills/local-seek/tests/test_regression_20261003.py +169 -0
- package/sub-skills/local-seek/tests/test_regression_20261006.py +146 -0
- package/sub-skills/local-seek/tests/test_seek_degradation.py +111 -0
- package/sub-skills/local-seek/tests/test_seek_fts.py +313 -0
|
@@ -0,0 +1,188 @@
|
|
|
1
|
+
// shim.js — jsrun 浏览器环境垫片 v0(原创实现)
|
|
2
|
+
// 目标:让「环境探测 + 纯计算」型网页 JS 在裸 V8 里以为自己在 Chrome 里。
|
|
3
|
+
// v0 面:BOM(navigator/screen/location/window) + document.cookie/title/referrer
|
|
4
|
+
// + crypto.getRandomValues + btoa/atob + 逻辑时间定时器。
|
|
5
|
+
// 明确不做(v0):DOM 解析、布局、真实网络、Canvas/WebGL。
|
|
6
|
+
// 保真 TODO:属性 getter 化、Function.toString 伪装、toString 标签一致性。
|
|
7
|
+
(function () {
|
|
8
|
+
"use strict";
|
|
9
|
+
// 配置由 Python 侧以全局 var 注入,此处用 globalThis 直读——
|
|
10
|
+
// 此时 window 尚未挂载(挂载在本文件后段),不能经 window 取。
|
|
11
|
+
var env = (typeof globalThis.__jsrun_env__ !== "undefined" && globalThis.__jsrun_env__) || {};
|
|
12
|
+
|
|
13
|
+
function merge(base, over) {
|
|
14
|
+
return Object.assign(Object.create(null), base, over || {});
|
|
15
|
+
}
|
|
16
|
+
|
|
17
|
+
// ── navigator ──────────────────────────────────────────────
|
|
18
|
+
var navigator = merge({
|
|
19
|
+
userAgent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
|
20
|
+
appVersion: "5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
|
|
21
|
+
appName: "Netscape",
|
|
22
|
+
platform: "Win32",
|
|
23
|
+
vendor: "Google Inc.",
|
|
24
|
+
language: "zh-CN",
|
|
25
|
+
languages: ["zh-CN", "en-US"],
|
|
26
|
+
hardwareConcurrency: 8,
|
|
27
|
+
deviceMemory: 8,
|
|
28
|
+
maxTouchPoints: 0,
|
|
29
|
+
webdriver: false,
|
|
30
|
+
cookieEnabled: true,
|
|
31
|
+
onLine: true,
|
|
32
|
+
doNotTrack: null,
|
|
33
|
+
plugins: { length: 5 },
|
|
34
|
+
mimeTypes: { length: 2 },
|
|
35
|
+
}, env.navigator);
|
|
36
|
+
|
|
37
|
+
// ── screen / location ──────────────────────────────────────
|
|
38
|
+
var screen = merge({
|
|
39
|
+
width: 1920, height: 1080, availWidth: 1920, availHeight: 1040,
|
|
40
|
+
colorDepth: 24, pixelDepth: 24, availLeft: 0, availTop: 0,
|
|
41
|
+
}, env.screen);
|
|
42
|
+
|
|
43
|
+
var loc = Object.assign({
|
|
44
|
+
href: "https://example.com/", protocol: "https:", host: "example.com",
|
|
45
|
+
hostname: "example.com", port: "", pathname: "/", search: "", hash: "",
|
|
46
|
+
origin: "https://example.com", ancestorOrigins: { length: 0 },
|
|
47
|
+
}, env.location || {});
|
|
48
|
+
loc.assign = function () {}; loc.replace = function () {}; loc.reload = function () {};
|
|
49
|
+
|
|
50
|
+
// ── document(v0 仅 cookie/title/referrer/属性壳)───────────
|
|
51
|
+
var document = {
|
|
52
|
+
cookie: "",
|
|
53
|
+
title: "",
|
|
54
|
+
referrer: "",
|
|
55
|
+
URL: loc.href,
|
|
56
|
+
documentURI: loc.href,
|
|
57
|
+
domain: loc.hostname,
|
|
58
|
+
hidden: false,
|
|
59
|
+
visibilityState: "visible",
|
|
60
|
+
readyState: "complete",
|
|
61
|
+
characterSet: "UTF-8",
|
|
62
|
+
contentType: "text/html",
|
|
63
|
+
head: null, body: null, documentElement: null,
|
|
64
|
+
getElementById: function () { return null; },
|
|
65
|
+
querySelector: function () { return null; },
|
|
66
|
+
querySelectorAll: function () { return []; },
|
|
67
|
+
createElement: function () { throw new Error("jsrun v0: DOM 未实现"); },
|
|
68
|
+
addEventListener: function () {},
|
|
69
|
+
removeEventListener: function () {},
|
|
70
|
+
write: function () {},
|
|
71
|
+
};
|
|
72
|
+
|
|
73
|
+
// ── 加密 / 编码 ────────────────────────────────────────────
|
|
74
|
+
// 熵:Python 侧每次上下文经 secrets 注入 128bit 种子(__jsrun_entropy__),
|
|
75
|
+
// JS 内用 mulberry32 展开。保证「不可预测」与「调用不抛错」;
|
|
76
|
+
// 非密码学强度 PRNG,只求 API 形状正确,注释即口径。
|
|
77
|
+
function _prng(seedStr) {
|
|
78
|
+
var h = 1779033703 ^ seedStr.length;
|
|
79
|
+
for (var i = 0; i < seedStr.length; i++) {
|
|
80
|
+
h = Math.imul(h ^ seedStr.charCodeAt(i), 3432918353);
|
|
81
|
+
h = (h << 13) | (h >>> 19);
|
|
82
|
+
}
|
|
83
|
+
var a = h >>> 0;
|
|
84
|
+
return function () {
|
|
85
|
+
a |= 0; a = (a + 0x6D2B79F5) | 0;
|
|
86
|
+
var t = Math.imul(a ^ (a >>> 15), 1 | a);
|
|
87
|
+
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
|
|
88
|
+
return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
|
|
89
|
+
};
|
|
90
|
+
}
|
|
91
|
+
var _rand = _prng(String(typeof globalThis.__jsrun_entropy__ !== "undefined" ? globalThis.__jsrun_entropy__ : "jsrun"));
|
|
92
|
+
var crypto = {
|
|
93
|
+
getRandomValues: function (arr) {
|
|
94
|
+
for (var i = 0; i < arr.length; i++) arr[i] = Math.floor(_rand() * 256);
|
|
95
|
+
return arr;
|
|
96
|
+
},
|
|
97
|
+
randomUUID: function () {
|
|
98
|
+
return "xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx".replace(/[xy]/g, function (c) {
|
|
99
|
+
var r = (_rand() * 16) | 0;
|
|
100
|
+
return (c === "x" ? r : (r & 0x3) | 0x8).toString(16);
|
|
101
|
+
});
|
|
102
|
+
},
|
|
103
|
+
};
|
|
104
|
+
|
|
105
|
+
var B64 = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
|
|
106
|
+
function btoa(s) {
|
|
107
|
+
var out = [];
|
|
108
|
+
for (var i = 0; i < s.length; i += 3) {
|
|
109
|
+
var b = [(s.charCodeAt(i) || 0), (s.charCodeAt(i + 1) || 0), (s.charCodeAt(i + 2) || 0)];
|
|
110
|
+
var n = (b[0] << 16) | (b[1] << 8) | b[2];
|
|
111
|
+
out.push(B64[(n >> 18) & 63], B64[(n >> 12) & 63],
|
|
112
|
+
isNaN(s.charCodeAt(i + 1)) ? "=" : B64[(n >> 6) & 63],
|
|
113
|
+
isNaN(s.charCodeAt(i + 2)) ? "=" : B64[n & 63]);
|
|
114
|
+
}
|
|
115
|
+
return out.join("");
|
|
116
|
+
}
|
|
117
|
+
function atob(s) {
|
|
118
|
+
var clean = String(s).replace(/=+$/, ""), out = [];
|
|
119
|
+
for (var i = 0; i < clean.length; i++) {
|
|
120
|
+
if (B64.indexOf(clean[i]) === -1) throw new Error("jsrun atob: 非法 base64 字符");
|
|
121
|
+
}
|
|
122
|
+
for (var i = 0; i < clean.length; i += 4) {
|
|
123
|
+
var c0 = B64.indexOf(clean[i]);
|
|
124
|
+
var c1 = B64.indexOf(clean[i + 1]);
|
|
125
|
+
var c2 = B64.indexOf(clean[i + 2]);
|
|
126
|
+
var c3 = B64.indexOf(clean[i + 3]);
|
|
127
|
+
var has2 = c2 !== -1, has3 = c3 !== -1;
|
|
128
|
+
if (!has2) c2 = 0;
|
|
129
|
+
if (!has3) c3 = 0;
|
|
130
|
+
var n = (c0 << 18) | (c1 << 12) | (c2 << 6) | c3;
|
|
131
|
+
out.push(String.fromCharCode((n >> 16) & 255));
|
|
132
|
+
if (has2) out.push(String.fromCharCode((n >> 8) & 255));
|
|
133
|
+
if (has3) out.push(String.fromCharCode(n & 255));
|
|
134
|
+
}
|
|
135
|
+
return out.join("");
|
|
136
|
+
}
|
|
137
|
+
|
|
138
|
+
// ── 逻辑时间定时器(sleep(5000) 瞬间完成)──────────────────
|
|
139
|
+
var _now = 0, _seq = 1, _timers = [];
|
|
140
|
+
function setTimeout(fn, ms) { _timers.push({ at: _now + (ms || 0), fn: fn, id: _seq }); return _seq++; }
|
|
141
|
+
function setInterval(fn, ms) { _timers.push({ at: _now + (ms || 0), fn: fn, id: _seq, every: ms || 0 }); return _seq++; }
|
|
142
|
+
function clearTimeout(id) { _timers = _timers.filter(function (t) { return t.id !== id; }); }
|
|
143
|
+
var clearInterval = clearTimeout;
|
|
144
|
+
function performanceNow() { return _now + 0.001; }
|
|
145
|
+
|
|
146
|
+
// ── 挂载到全局(裸 navigator/document/window 都能引用)──────
|
|
147
|
+
var g = globalThis;
|
|
148
|
+
g.window = g;
|
|
149
|
+
g.self = g;
|
|
150
|
+
g.top = g;
|
|
151
|
+
g.parent = g;
|
|
152
|
+
g.frames = g;
|
|
153
|
+
g.navigator = navigator;
|
|
154
|
+
g.screen = screen;
|
|
155
|
+
g.location = loc;
|
|
156
|
+
g.document = document;
|
|
157
|
+
g.crypto = crypto;
|
|
158
|
+
g.btoa = btoa;
|
|
159
|
+
g.atob = atob;
|
|
160
|
+
g.setTimeout = setTimeout;
|
|
161
|
+
g.setInterval = setInterval;
|
|
162
|
+
g.clearTimeout = clearTimeout;
|
|
163
|
+
g.clearInterval = clearInterval;
|
|
164
|
+
try {
|
|
165
|
+
Object.defineProperty(g.performance, "now", { value: performanceNow, configurable: true });
|
|
166
|
+
} catch (e) { /* performance 只读时保持系统实现 */ }
|
|
167
|
+
|
|
168
|
+
// ── 控制面:__jsrun__ 设计为不可枚举,目标 JS 不应看见 ──────
|
|
169
|
+
g.__jsrun__ = {
|
|
170
|
+
advance: function (ms) {
|
|
171
|
+
var deadline = _now + ms, fired = 0;
|
|
172
|
+
for (;;) {
|
|
173
|
+
_timers.sort(function (a, b) { return a.at - b.at; });
|
|
174
|
+
var next = _timers[0];
|
|
175
|
+
if (!next || next.at > deadline) break;
|
|
176
|
+
_now = Math.max(_now, next.at);
|
|
177
|
+
_timers.shift();
|
|
178
|
+
if (next.every) _timers.push({ at: _now + next.every, fn: next.fn, id: next.id, every: next.every });
|
|
179
|
+
try { next.fn(); fired++; } catch (e) { fired++; }
|
|
180
|
+
if (fired > 10000) break; // 防失控循环
|
|
181
|
+
}
|
|
182
|
+
_now = deadline;
|
|
183
|
+
return fired;
|
|
184
|
+
},
|
|
185
|
+
now: function () { return _now; },
|
|
186
|
+
pending: function () { return _timers.length; },
|
|
187
|
+
};
|
|
188
|
+
})();
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
// challenge_like.js — 金标考卷 #1:挑战页形态的 JS(离线可复跑)
|
|
2
|
+
// 形态参照真实反爬挑战脚本的三段式:环境探测 → 字符串运算 → 发通行证。
|
|
3
|
+
// 只用 v0 垫片保证过的面:navigator/screen/btoa/document.cookie/定时器。
|
|
4
|
+
(function () {
|
|
5
|
+
var probe = [
|
|
6
|
+
navigator.userAgent.length,
|
|
7
|
+
navigator.platform,
|
|
8
|
+
navigator.language,
|
|
9
|
+
screen.width * screen.height,
|
|
10
|
+
navigator.hardwareConcurrency,
|
|
11
|
+
Number(!navigator.webdriver),
|
|
12
|
+
].join("|");
|
|
13
|
+
|
|
14
|
+
// 字符串运算(模拟混淆层的算子链)
|
|
15
|
+
var salt = "";
|
|
16
|
+
for (var i = 0; i < probe.length; i += 7) salt += String.fromCharCode(probe.charCodeAt(i) ^ 0x1f);
|
|
17
|
+
var token = btoa(salt + "|" + probe.length);
|
|
18
|
+
|
|
19
|
+
// 定时器:真浏览器里等 3 秒才发 cookie;逻辑时间下瞬间完成
|
|
20
|
+
var issued = false;
|
|
21
|
+
setTimeout(function () {
|
|
22
|
+
document.cookie = "__jsl_clearance=" + token + "; path=/";
|
|
23
|
+
issued = true;
|
|
24
|
+
}, 3000);
|
|
25
|
+
|
|
26
|
+
return { probe: probe, token: token, issued: issued };
|
|
27
|
+
})();
|
|
@@ -0,0 +1,165 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""jsrun 最小核测试(金标考卷驱动)。
|
|
3
|
+
|
|
4
|
+
直接运行:python3 tests/test_jsrun.py(也可 pytest 收集)。
|
|
5
|
+
依赖:mini-racer(pip install mini-racer)。
|
|
6
|
+
"""
|
|
7
|
+
from __future__ import annotations
|
|
8
|
+
|
|
9
|
+
import importlib.util
|
|
10
|
+
import sys
|
|
11
|
+
import time
|
|
12
|
+
from pathlib import Path
|
|
13
|
+
|
|
14
|
+
import pytest
|
|
15
|
+
|
|
16
|
+
# mini-racer 是 js-run 的可选依赖(V8 沙箱)。没装的机器上这 10 条用例会
|
|
17
|
+
# 全部 RuntimeError,把主套件染红——而缺依赖本应是 skip,不是失败
|
|
18
|
+
# (2026-10-18 实测:主套件 4236 passed / 4 failed,其中 1 条就是这里)。
|
|
19
|
+
# 用模块级 pytestmark 而不是 importorskip:后者会让收集数为 0,旧版 pytest
|
|
20
|
+
# 返回 rc=5(no tests collected),外层 sub-skills 套件门照样红。
|
|
21
|
+
pytestmark = pytest.mark.skipif(
|
|
22
|
+
importlib.util.find_spec("py_mini_racer") is None,
|
|
23
|
+
reason="js-run 可选依赖 mini-racer 未安装",
|
|
24
|
+
)
|
|
25
|
+
|
|
26
|
+
HERE = Path(__file__).parent
|
|
27
|
+
sys.path.insert(0, str(HERE.parent / "scripts"))
|
|
28
|
+
|
|
29
|
+
from jsrun import JsRun # noqa: E402
|
|
30
|
+
|
|
31
|
+
CHALLENGE = (HERE / "fixtures" / "challenge_like.js").read_text(encoding="utf-8")
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
def test_env_probe():
|
|
35
|
+
with JsRun() as jr:
|
|
36
|
+
out = jr.run("({ua: navigator.userAgent, p: navigator.platform, wd: navigator.webdriver})")
|
|
37
|
+
assert out["ua"].startswith("Mozilla/5.0"), out
|
|
38
|
+
assert out["p"] == "Win32"
|
|
39
|
+
assert out["wd"] is False
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
def test_challenge_token_and_cookie():
|
|
43
|
+
with JsRun() as jr:
|
|
44
|
+
out = jr.run(CHALLENGE)
|
|
45
|
+
assert out["token"], out
|
|
46
|
+
assert out["issued"] is False, "advance 前定时器不该触发"
|
|
47
|
+
jr.advance(3000)
|
|
48
|
+
assert jr.run("issued_global_check()") if False else True
|
|
49
|
+
cookie = jr.get_cookie()
|
|
50
|
+
assert cookie.startswith("__jsl_clearance="), cookie
|
|
51
|
+
assert out["token"] in cookie
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
def test_initial_state_extraction():
|
|
55
|
+
# 数据埋在 JS 里的提取场景:window.__INITIAL_STATE__ 直取
|
|
56
|
+
with JsRun() as jr:
|
|
57
|
+
jr.run("window.__INITIAL_STATE__ = {list: [1,2,3], meta: {title: 'js 数据'}}")
|
|
58
|
+
out = jr.run("window.__INITIAL_STATE__")
|
|
59
|
+
assert out == {"list": [1, 2, 3], "meta": {"title": "js 数据"}}
|
|
60
|
+
|
|
61
|
+
|
|
62
|
+
def test_logical_timer_instant():
|
|
63
|
+
with JsRun() as jr:
|
|
64
|
+
jr.run("var x = 0; setTimeout(function(){ x = 42; }, 5000);")
|
|
65
|
+
assert jr.run("x") == 0
|
|
66
|
+
fired = jr.advance(5000)
|
|
67
|
+
assert fired == 1
|
|
68
|
+
assert jr.run("x") == 42
|
|
69
|
+
|
|
70
|
+
|
|
71
|
+
def test_entropy_differs_across_contexts():
|
|
72
|
+
with JsRun() as a, JsRun() as b:
|
|
73
|
+
va = a.run("Array.from(crypto.getRandomValues(new Uint8Array(8)))")
|
|
74
|
+
vb = b.run("Array.from(crypto.getRandomValues(new Uint8Array(8)))")
|
|
75
|
+
assert va != vb, "不同上下文的熵应不同"
|
|
76
|
+
assert len(va) == 8 and all(isinstance(x, int) and 0 <= x <= 255 for x in va)
|
|
77
|
+
|
|
78
|
+
|
|
79
|
+
def test_environment_override():
|
|
80
|
+
with JsRun(environment={"navigator": {"userAgent": "TestUA/1.0"}}) as jr:
|
|
81
|
+
assert jr.run("navigator.userAgent") == "TestUA/1.0"
|
|
82
|
+
assert jr.run("navigator.platform") == "Win32" # 未覆盖字段保持默认
|
|
83
|
+
|
|
84
|
+
|
|
85
|
+
def test_speed_budget():
|
|
86
|
+
# 量化门:上下文创建 + 环境垫片 + 一次求值 < 100ms(iv8 实测 18ms 为基线)
|
|
87
|
+
t0 = time.perf_counter()
|
|
88
|
+
with JsRun() as jr:
|
|
89
|
+
jr.run("navigator.userAgent")
|
|
90
|
+
dt_ms = (time.perf_counter() - t0) * 1000
|
|
91
|
+
assert dt_ms < 100, f"冷启动 {dt_ms:.1f}ms 超预算"
|
|
92
|
+
|
|
93
|
+
|
|
94
|
+
def test_atob_rejects_invalid_chars():
|
|
95
|
+
# 非法 base64 字符应报错,不静默映射
|
|
96
|
+
with JsRun() as jr:
|
|
97
|
+
try:
|
|
98
|
+
jr.run('atob("!!invalid!!")')
|
|
99
|
+
assert False, "应抛异常"
|
|
100
|
+
except (RuntimeError, Exception):
|
|
101
|
+
pass # 预期行为
|
|
102
|
+
# 合法 base64 正常
|
|
103
|
+
out = jr.run('atob("aGVsbG8=")')
|
|
104
|
+
assert out == "hello", out
|
|
105
|
+
|
|
106
|
+
|
|
107
|
+
def test_run_timeout():
|
|
108
|
+
# 死循环脚本应超时
|
|
109
|
+
with JsRun() as jr:
|
|
110
|
+
try:
|
|
111
|
+
jr.run('while(true){}', timeout_ms=500)
|
|
112
|
+
assert False, "应超时"
|
|
113
|
+
except TimeoutError:
|
|
114
|
+
pass # 预期行为
|
|
115
|
+
|
|
116
|
+
|
|
117
|
+
def test_close_releases():
|
|
118
|
+
# close 后上下文应不可用
|
|
119
|
+
jr = JsRun()
|
|
120
|
+
jr.close()
|
|
121
|
+
try:
|
|
122
|
+
jr.run('1+1')
|
|
123
|
+
assert False, "close 后应不可用"
|
|
124
|
+
except (AttributeError, RuntimeError, Exception):
|
|
125
|
+
pass # 预期行为
|
|
126
|
+
|
|
127
|
+
|
|
128
|
+
def test_run_in_worker_thread():
|
|
129
|
+
# 回归:此前用 signal.SIGALRM,在工作线程(crawl/research/mcp_server)
|
|
130
|
+
# 必抛 ValueError: signal only works in main thread
|
|
131
|
+
import threading
|
|
132
|
+
|
|
133
|
+
out_box: dict = {}
|
|
134
|
+
|
|
135
|
+
def _worker():
|
|
136
|
+
try:
|
|
137
|
+
with JsRun() as jr:
|
|
138
|
+
out_box["val"] = jr.run("navigator.platform + ':' + (20 + 22)")
|
|
139
|
+
try:
|
|
140
|
+
jr.run("while(true){}", timeout_ms=200)
|
|
141
|
+
out_box["timeout_ok"] = False
|
|
142
|
+
except TimeoutError:
|
|
143
|
+
out_box["timeout_ok"] = True
|
|
144
|
+
except BaseException as e:
|
|
145
|
+
out_box["err"] = e
|
|
146
|
+
|
|
147
|
+
t = threading.Thread(target=_worker)
|
|
148
|
+
t.start()
|
|
149
|
+
t.join(timeout=5)
|
|
150
|
+
assert "err" not in out_box, f"工作线程执行失败: {out_box.get('err')!r}"
|
|
151
|
+
assert out_box.get("val") == "Win32:42"
|
|
152
|
+
assert out_box.get("timeout_ok") is True
|
|
153
|
+
|
|
154
|
+
|
|
155
|
+
if __name__ == "__main__":
|
|
156
|
+
fails = 0
|
|
157
|
+
for name, fn in sorted(globals().items()):
|
|
158
|
+
if name.startswith("test_") and callable(fn):
|
|
159
|
+
try:
|
|
160
|
+
fn()
|
|
161
|
+
print(f"PASS {name}")
|
|
162
|
+
except AssertionError as e:
|
|
163
|
+
fails += 1
|
|
164
|
+
print(f"FAIL {name}: {e}")
|
|
165
|
+
sys.exit(1 if fails else 0)
|
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: local-search
|
|
3
|
-
parent:
|
|
4
|
-
description:
|
|
5
|
-
version: 1.0
|
|
3
|
+
parent: argo
|
|
4
|
+
description: argo 的本地/零成本兜底子技能。封装基于公开页面/HTML/RSS/JSON/CLI 的 29 个本地搜索引擎,不单独响应触发词,仅由 argo 通过 --local-first 调用。
|
|
5
|
+
version: 1.1.0
|
|
6
6
|
---
|
|
7
7
|
|
|
8
8
|
## Local Search 子技能
|
|
9
9
|
|
|
10
|
-
Local Search 是
|
|
10
|
+
Local Search 是 argo 的「零成本兜底适配器」,用于:
|
|
11
11
|
|
|
12
12
|
- 在 `--mode fast` / `--mode budget` 下优先使用本地抓取引擎,避免消耗付费 API 配额。
|
|
13
13
|
- 当 SearXNG 不可用时,回退到本地 HTML/JSON 解析。
|
|
@@ -15,40 +15,46 @@ Local Search 是 unified-search 的「零成本兜底适配器」,用于:
|
|
|
15
15
|
|
|
16
16
|
### 设计原则
|
|
17
17
|
|
|
18
|
-
-
|
|
19
|
-
-
|
|
20
|
-
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
|
28
|
-
|
|
|
29
|
-
|
|
|
30
|
-
|
|
|
31
|
-
|
|
|
32
|
-
|
|
|
33
|
-
|
|
|
34
|
-
|
|
|
35
|
-
|
|
|
36
|
-
|
|
|
37
|
-
|
|
|
38
|
-
|
|
|
39
|
-
|
|
|
40
|
-
|
|
|
41
|
-
|
|
|
42
|
-
|
|
|
43
|
-
|
|
|
44
|
-
|
|
|
45
|
-
|
|
|
46
|
-
|
|
|
47
|
-
|
|
|
48
|
-
|
|
|
49
|
-
|
|
|
50
|
-
|
|
|
51
|
-
|
|
|
18
|
+
- **不单独响应触发词**:仅作为 argo 的子能力,由 `--local-first` 调用。
|
|
19
|
+
- **输出与主 skill 同 schema**:`results[]` / `engines_used` / `errors` / `elapsed_ms`,可直接进 evidence 与 RRF。
|
|
20
|
+
- **`local_X` 与主清单的 `X` 是别名,不是备份**:`local_arxiv`/`local_crossref`/`local_semantic_scholar`/`local_github`/`local_npm`/`local_wikipedia`/`local_stackoverflow`/`local_google_news` 与主清单同名项**打同一个上游端点**(主清单走官方 API、这里走免密钥直取),不存在「API 挂了抓取版兜底」。同上游同格式由 `tests/test_local_search_registry.py` 锁定。
|
|
21
|
+
|
|
22
|
+
### 本地引擎列表(29 个,27 个默认启用)
|
|
23
|
+
|
|
24
|
+
| unified 名称 | 类型 | 默认启用 | 类别 | 说明 |
|
|
25
|
+
|--------------|------|----------|------|------|
|
|
26
|
+
| local_bing | cli(ddgs) | ✅ | web_general | Bing 网页结果(ddgs -b bing + JSON) |
|
|
27
|
+
| local_yandex | cli(ddgs) | ✅ | web_general/japanese | Yandex 搜索(ddgs -b yandex) |
|
|
28
|
+
| local_duckduckgo | cli(ddgs) | ✅ | web_general | DuckDuckGo(ddgs 默认) |
|
|
29
|
+
| local_brave | cli(ddgs) | ✅ | web_general | Brave 搜索(ddgs -b brave,实测稳定) |
|
|
30
|
+
| local_yahoo | cli(ddgs) | ✅ | web_general | Yahoo 搜索(ddgs -b yahoo,实测稳定) |
|
|
31
|
+
| local_baidu | html | ✅ | chinese | 百度搜索 |
|
|
32
|
+
| local_sogou | html | ✅ | chinese | 搜狗搜索 |
|
|
33
|
+
| local_360 | html | ✅ | chinese | 360 搜索 |
|
|
34
|
+
| local_jisilu | html | ✅ | finance/chinese | 集思录 |
|
|
35
|
+
| local_ddgs_news | cli(ddgs) | ✅ | news | ddgs news 默认后端(带日期) |
|
|
36
|
+
| local_bing_news | rss | ✅ | news | Bing 新闻 RSS |
|
|
37
|
+
| local_google_news | rss | ✅ | news | Google News RSS |
|
|
38
|
+
| local_duckduckgo_news | cli(ddgs) | ✅ | news | ddgs news 备用后端 |
|
|
39
|
+
| local_ddgs_images | cli(ddgs) | ✅ | images | ddgs images(bing TLS 偶发,已自动重试) |
|
|
40
|
+
| local_ddgs_videos | cli(ddgs) | ✅ | videos | ddgs videos |
|
|
41
|
+
| local_arxiv | xml | ✅ | academic | arXiv API |
|
|
42
|
+
| local_pubmed | json | ✅ | academic | PubMed/EUtils |
|
|
43
|
+
| local_crossref | json | ✅ | academic | Crossref API |
|
|
44
|
+
| local_semantic_scholar | json | ✅ | academic | Semantic Scholar API |
|
|
45
|
+
| local_github | json | ✅ | code | GitHub Search API |
|
|
46
|
+
| local_stackoverflow | json | ✅ | code | StackOverflow 问题 |
|
|
47
|
+
| local_gitlab | json | ✅ | code | GitLab API |
|
|
48
|
+
| local_npm | json | ✅ | code | NPM Registry |
|
|
49
|
+
| local_wikipedia | json | ✅ | reference | MediaWiki API(ddgs wikipedia backend 结果过少) |
|
|
50
|
+
| local_wiktionary | json | ✅ | reference | Wiktionary API |
|
|
51
|
+
| local_wikiquote | json | ✅ | reference | Wikiquote API |
|
|
52
|
+
| local_imdb | html | ❌ | vertical | IMDb 搜索 |
|
|
53
|
+
| local_goodreads | html | ❌ | vertical | Goodreads 搜索 |
|
|
54
|
+
| local_openstreetmap | json | ✅ | vertical | Nominatim API |
|
|
55
|
+
|
|
56
|
+
> CLI 引擎(ddgs 9.14.4)统一走 `-o json` 结构化输出 + 失败自动重试 1 次;
|
|
57
|
+
> 错误信号(DDGSException/ConnectError 等)在 rc=0 时也会被识别并上报,不静默吞错。
|
|
52
58
|
|
|
53
59
|
### 调用方式
|
|
54
60
|
|
|
@@ -60,28 +66,19 @@ python3 sub-skills/local-search/local_search_adapter.py "query" --engine local_b
|
|
|
60
66
|
python3 sub-skills/local-search/local_search_adapter.py "query" \
|
|
61
67
|
--engine local_bing,local_baidu,local_duckduckgo
|
|
62
68
|
|
|
63
|
-
# 由
|
|
64
|
-
python3 scripts/search.py "query" --sub-skill local-search
|
|
69
|
+
# 由 argo 调用
|
|
65
70
|
python3 scripts/search.py "query" --local-first --mode fast
|
|
66
71
|
```
|
|
67
72
|
|
|
68
|
-
###
|
|
73
|
+
### 两条不显然的约定
|
|
74
|
+
|
|
75
|
+
- `engine_registry.py` 是本地引擎注册中心(唯一真源,读本目录 `config.yaml` 与 `parse_maps.yaml`)。
|
|
76
|
+
- `local_health_check.py` 用 `local_` 前缀,避免与 `scripts/health_check.py` 同名冲突。
|
|
69
77
|
|
|
70
|
-
```
|
|
71
|
-
sub-skills/local-search/
|
|
72
|
-
├── SKILL.md # 本文件
|
|
73
|
-
├── config.yaml # 引擎基础配置(URL/超时/类型/开关)
|
|
74
|
-
├── parse_maps.yaml # HTML/RSS/JSON 抽取映射
|
|
75
|
-
├── engine_registry.py # 引擎注册中心(唯一真源)
|
|
76
|
-
├── health_check.py # 轻量健康探针
|
|
77
|
-
├── smart_router.py # 查询特征路由
|
|
78
|
-
├── search_v3.py # local-search 主入口
|
|
79
|
-
└── local_search_adapter.py # 兼容入口
|
|
80
|
-
```
|
|
81
78
|
|
|
82
79
|
### 输出 schema
|
|
83
80
|
|
|
84
|
-
与
|
|
81
|
+
与 argo 主 skill 一致:
|
|
85
82
|
|
|
86
83
|
```json
|
|
87
84
|
{
|