websearch-skill 0.1.0__tar.gz

websearch_skill-0.1.0/.gitignore

@@ -0,0 +1,42 @@
+# Research store (design of record; kept local, distilled into docs/ARCHITECTURE.md)
+.research/
+
+# Internal agent instruction files (kept local for the build agent, not public)
+CLAUDE.md
+AGENTS.md
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+.eggs/
+build/
+dist/
+*.so
+
+# Virtualenv / uv
+.venv/
+venv/
+.python-version
+
+# Test / coverage / cache
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+htmlcov/
+coverage.xml
+
+# Editor / OS
+.idea/
+.vscode/
+.DS_Store
+
+# Local Claude Code state (settings.local.json etc.); NOT the .claude-plugin/ manifests.
+.claude/
+
+# Local secrets (search-engine API keys, SearXNG URL overrides)
+.env
+.env.*
+!.env.example

websearch_skill-0.1.0/CHANGELOG.md

@@ -0,0 +1,263 @@
+# Changelog
+
+Notable changes to this project. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/); the project will follow
+semantic versioning once it reaches a tagged release.
+
+## [Unreleased]
+
+### Changed
+
+- fastmcp is now a base dependency, not the optional `mcp` extra, so `websearch mcp`,
+  `uvx websearch-skill mcp`, and the MCP-registry runner start the stdio server with one
+  command and no extra. The `[mcp]` extra stays as a no-op alias so older install commands
+  keep resolving. The package also installs a second console script, `websearch-skill`
+  (matching the distribution name), so `uvx websearch-skill <cmd>` resolves with no `--from`.
+- The agent surface is plug-and-play: `web-search` (CLI) and the `web_search` MCP tool no
+  longer take engine-selection flags (`--engines`, `--no-ddgs`, `--ddgs-backends`, the MCP
+  `engines` parameter, or `WEBSEARCH_DDGS_BACKENDS`). The keyless multi-engine default
+  needs no flags. Engine and backend selection stays on the lower-level `search` command
+  for debugging. `web-search --offset` now actually pages results (the ddgs adapter maps
+  offset to a page), instead of silently returning page one.
+- `fetch@1.2.0`: `FetchResult` gains a structured `failure_kind`
+  (`egress_refused` / `redirect_loop` / `transport_error` / `timeout` /
+  `dependency_missing`) so retriability is set from the cause, not by matching error text.
+
+### Fixed
+
+- No command can dump a raw traceback. `web-fetch --page 0` (or negative), invalid
+  `open`/extract parameters, a store-open failure, and any unexpected error now return a
+  clean `invalid_request` / `internal_error` Envelope. `web-search` rejects a
+  whitespace-only query; an unknown `engines` value falls back to the default with a
+  warning instead of failing.
+- `meta.elapsed_ms` is populated on every Layer-3 (`web_search` / `web_fetch` / `web_open`)
+  and format Envelope (it was always `0.0`); `_propagate_error` keeps the upstream
+  `trace_id` and uses a defined error code.
+- Retriability precision: a permanent fetch failure (egress refusal, redirect loop, missing
+  optional dependency) is non-retriable; a transport error or timeout is retriable.
+- SSRF egress guard uses an allowlist (`is_global`), closing CGNAT `100.64.0.0/10`
+  (RFC 6598) and any other non-public range the prior denylist missed.
+- The page store (SQLite FTS5 and the BM25 fallback) serializes its methods with a lock and
+  writes each document in one atomic transaction, so concurrent MCP tool calls cannot race
+  the shared connection or leave a half-written document. `search().total` is the true match
+  count, not the capped top-k pool. MCP singletons build under a lock.
+- curl_cffi response handling is fully guarded (a mid-body decode/reset becomes an
+  escalatable result, not a crash); GitHub/arXiv tolerate a malformed upstream shape with a
+  clean `upstream_error`; a non-positive chunk size no longer hangs; a NaN score no longer
+  corrupts result ordering; the error-title quality veto no longer tanks a long article
+  whose title merely contains a word like "forbidden" or a number like "404".
+- The CLI pins stdout/stderr to UTF-8 (no `UnicodeEncodeError` under a C/POSIX locale), and
+  the `web-fetch` "more pages" hint suggests a command that actually works in context
+  (`web-open ... --persist-path` when persisting, else a re-fetch).
+
+### Added
+
+- Distribution layer: a Claude Code plugin and marketplace (`.claude-plugin/`, with no
+  SKILL duplication, since `source: "./"` auto-discovers the root `skills/web-search/` and
+  root `.mcp.json`), a generic root `.mcp.json`, an MCP-registry `server.json`
+  (`io.github.hec-ovi/web-search`, PyPI/uvx), a PyPI Trusted-Publishing release workflow
+  (`.github/workflows/release.yml`, OIDC, no token), and `docs/INSTALL.md` covering every
+  harness route (`npx skills add`, `uvx`, the Claude plugin, Codex with its network-sandbox
+  caveat, OpenCode, Cursor, Hermes, OpenClaw, and the registry). Install with
+  `npx skills add hec-ovi/websearch-skill`, `/plugin install web-search@websearch-skill`, or
+  `uvx websearch-skill <cmd>`.
+- Keyless multi-engine search out of the box: the `ddgs` adapter is treated as the
+  metasearch it is (Google, Brave, DuckDuckGo, Yandex, Yahoo, Startpage, Mojeek,
+  Wikipedia by default, with Bing and others selectable by name). The lower-level
+  `search` command can force a subset with `--ddgs-backends google,brave,mojeek`, and
+  `build_router` / `build_agent_io` take a `ddgs_backend` parameter. `ddgs` is the keyless
+  default; a self-hosted SearXNG is the optional broader engine. Public SearXNG instances
+  are not used as a default (most disable the JSON API and rate-limit automated clients).
+- `arxiv@1.0.0` contract and a keyless `websearch arxiv` tool (also the MCP
+  `arxiv_search` tool): arXiv paper search over the official Atom API with field-targeted
+  search (`--field`), sorting, GET caching, and exponential backoff on HTTP 429. Returns
+  structured papers (authors, abstract, categories, abstract and PDF links).
+- `github@1.0.0` contract and a keyless `websearch github` tool (also the MCP
+  `github_search` tool): GitHub repository search over the unauthenticated REST API with
+  typed fields (stars, forks, language, topics) and a clean `rate_limited` error on the
+  ~10 req/min limit. Code search needs a token and is left out of the keyless path.
+- New cross-cutting error codes `upstream_error` and `rate_limited` for the extra tools.
+- `docker/searxng/`: a lean one-container SearXNG self-host config (JSON API enabled, bot
+  limiter off, no Valkey needed) with its own README and a before-you-expose-it checklist.
+- `docs/BENCHMARK.md`: a reproducible, same-query head-to-head against the web search
+  built into Claude Code, with the honest verdict (comparable on retrieval; this tool
+  wins on cost, privacy, control, and extraction).
+- Frozen contracts as JSON Schema 2020-12: `envelope@1.0.0` (the cross-cutting wrapper
+  for every inter-layer message and CLI `--json` output) and `search@1.0.0` (the
+  Layer-1 search port). The semver rule (additive is MINOR, removal/rename/retype is
+  MAJOR) and the consumer-driven fixture check are documented in `contracts/README.md`.
+- Layer 1 (search): a multi-engine router with isolated per-engine adapters behind an
+  `EngineAdapter` port. Ships a SearXNG adapter (keyless backbone, over httpx) and a
+  ddgs adapter (zero-config fallback).
+- Provenance-aware weighted Reciprocal Rank Fusion (k=60) with mandatory
+  de-correlation: engines that share a correlation group count as one independent vote,
+  so a consensus bonus cannot amplify the same crawler agreeing with itself. The router
+  records the de-correlation in a warning.
+- URL canonicalization and dedup with provenance merge, site include/exclude filtering,
+  concurrent fan-out, and per-engine fault tolerance (an error Envelope is returned only
+  when every selected engine fails).
+- CLI entry point `websearch search` (`--json` emits the raw Envelope; exit code 1 on an
+  error Envelope).
+- Test suite: an end-to-end test through the real CLI entry point with both external
+  boundaries stubbed (SearXNG via pytest-httpx, ddgs via a fake), plus focused
+  canonicalization, dedup, fusion, adapter, router fault-tolerance, and
+  contract-conformance tests that validate real output against the frozen schemas. CI
+  runs ruff and pytest on Python 3.11 to 3.13 via uv.
+- Layer 2A contracts: `fetch@1.1.0` (`FetchRequest`, `FetchResult`) and `extract@1.0.0`
+  (`ExtractRequest`, `ExtractResult`, `ExtractSource`, `ExtractPayload`), the two
+  decoupled sub-ports of fetch and extract.
+- Layer 2A (fetch + extract): a tiered fetch that starts on plain httpx and escalates to
+  curl_cffi browser TLS/JA3 impersonation only when it detects an anti-bot block (header
+  markers first, then gated body markers for Cloudflare, DataDome, PerimeterX, Akamai,
+  and Imperva), never on a 404 or a terminal block (rate limit, auth, legal). Extraction
+  defaults to Trafilatura, emitting clean Markdown plus plain text and metadata,
+  recovering raw schema.org JSON-LD and `og:type` with lxml, and computing a heuristic
+  `quality_score` and a cheap `page_type`. Browser/stealth fetch tiers and neural extract
+  engines are named in the contract enums but stay opt-in.
+- CLI `websearch fetch` (`--json` emits the Envelope; `--output-format`, `--quiet`,
+  `--tier`, `--proxy`, `--allow-private-hosts`, and the extract options). No
+  output-length cap: `content_markdown` is never truncated; `max_bytes` is a transport
+  guard only (default 10 MB).
+- SSRF egress guard: an http(s) scheme allowlist plus DNS resolution that refuses
+  private, loopback, link-local (the `169.254.169.254` metadata endpoint), reserved, and
+  multicast targets, applied before the first request and on every redirect hop.
+- Test suite is now 188 tests (Layer 1 plus Layer 2A: block detection, quality scoring,
+  fetch tiers and escalation, the egress guard, extraction, and CLI end-to-end).
+- Layer 2B contracts: `format@1.0.0` (`ResultInput`, `FormatRequest`, `FormatPayload`,
+  `FormatSidecar`, and the derived `AnthropicSearchResultBlock`) and `store@1.0.0`
+  (`PageInput`, `Passage`, `SearchPageRequest`, `SearchPageResult`, `PageDocument`,
+  `ResolveIndex`, `StoreConfig`), the two decoupled sub-ports of format and store.
+- Layer 2B format: turns vendor-neutral results into one layout-stable Markdown
+  document plus a parallel JSON sidecar carrying identical data, ordered by descending
+  relevance and paginated. Near-duplicate dedup (byte-exact SHA-256 first, then a
+  pure-Python MinHash over word 4-gram shingles) folds duplicates into the best-scored
+  canonical and records `dropped_duplicates`. Progressive disclosure picks the render
+  mode: `auto` inlines full bodies when the page fits a token budget, otherwise an
+  index (a preview plus a stable id to resolve). The optional
+  `anthropic_search_result_blocks` view maps 1:1 onto Anthropic search_result content
+  blocks (source as a bare string, at least one non-empty text block, citations
+  all-or-nothing); it is off by default and Layer 3 owns the citations toggle.
+- Layer 2B store: an ephemeral page index behind a `PageIndex` port with
+  `add`/`search`/`get`/`resolve_index`. The default adapter is SQLite FTS5 over an
+  in-memory connection (Python stdlib, BM25 ranking), with a runtime FTS5 probe that
+  falls back to a pure-Python BM25 index when FTS5 is not compiled into the local
+  SQLite. Adds are idempotent on url plus content hash; an arbitrary query is escaped so
+  FTS5 operators never raise a syntax error; persistence is the presence of a file path.
+- CLI `websearch open <url> ...`: composes Layer 2A and 2B (fetch, extract, format,
+  index) into one paginated, deduped document, with `--mode`, `--body`,
+  `--body-char-budget`/`--no-truncate`, `--anthropic-blocks`, `--search` (BM25 passage
+  search over the opened pages), and `--persist-path`. Per-URL fetch failures surface as
+  warnings rather than failing the whole request.
+- No output-length cap in Layer 2B either: full bodies are stored and echoed in the JSON
+  sidecar verbatim in both index and full modes; `body_char_budget` only offloads the
+  rendered Markdown view to the resolver, and `--no-truncate` disables even that.
+- Test suite is now 272 tests (adds dedup, chunk-offset, renderer layout-stability,
+  both store adapters, format/store contract conformance, and the `open` end-to-end).
+- Layer 3 contract: `agent-io@1.0.0` (`AgentSearchRequest`/`AgentSearchPayload`,
+  `AgentFetchRequest`/`AgentOpenRequest`/`AgentPage`/`AgentFetchPayload`, `FenceInfo`),
+  the consolidated agent-facing surface over Layers 1/2A/2B.
+- Layer 3 (agent I/O): `web_search` (Layer 1), `web_fetch` (Layer 2A, fenced and
+  paginated), and `web_open` (paginate an already-fetched page from the Layer 2B store
+  by handle, no re-fetch), all over the same `Envelope`. The only cross-layer key is a
+  human-readable `handle` (`site~shorthash`), not an opaque id. Pagination is lossless
+  progressive disclosure, never a content cap.
+- Untrusted-content fence: each fetched page's content is wrapped in delimiters carrying
+  a per-instance 128-bit random nonce (so injected text cannot forge the closing
+  marker), a data-only directive, and neutralization of any in-body copy of the marker,
+  with optional datamarking (`--datamark`). Documented as reducing, not eliminating,
+  indirect prompt injection (it prevents the boundary breakout, not persuasion).
+- Optional FastMCP stdio server (`websearch mcp`, the `mcp` extra) exposing
+  `web_search`/`web_fetch`/`web_open`; the tool returns the same Envelope JSON the CLI
+  emits. New CLI commands `web-search`/`web-fetch`/`web-open`/`mcp`; the lower-level
+  `search`/`fetch`/`open` commands stay as the per-layer surfaces.
+- A portable `SKILL.md` (`skills/web-search/`) to the Agent Skills standard (name plus
+  description), documenting the command grammar, the search/fetch/open decision table,
+  and the untrusted-content rule.
+- Test suite is now 332 tests (adds the fence, token-budget pagination losslessness,
+  the agent-io facade and contract, and the FastMCP server and `web-*` CLI end-to-end).
+
+### Fixed
+
+After an adversarial multi-agent review (eleven confirmed findings) and a fresh-agent
+dogfooding pass of Layer 3:
+
+- A page reached by a redirect is now keyed by the requested URL (and aliased under the
+  final URL), so a `handle` from `web_search` stays resolvable by `web_open` after a
+  redirect instead of diverging to the post-redirect URL.
+- `web_search` no longer advertises a `next_offset` cursor, because the keyless backends
+  do not page results reliably (feeding it back re-showed earlier results); to get
+  different results, refine the query. The `offset` field stays plumbed for a backend
+  that honors it.
+- The fence neutralizes any copy of its marker case-insensitively (a lowercase or
+  mixed-case copy previously survived verbatim in the body).
+- `web_open` fails closed on the astronomically-unlikely same-site handle collision
+  (returns `not_opened`) rather than serving the wrong cached page; the short hash was
+  widened to 48 bits.
+- A single-URL `web-fetch` failure preserves the specific cause in the error message
+  (it previously collapsed to a generic "all 1 url(s) failed").
+- Doc and contract accuracy: `SKILL.md` surfaces the previously-omitted flags
+  (`--tier`, `--quiet`, `--datamark` on `web-open`, and others) and the full output
+  field list; `AgentPage.fence` is now required in the schema (it was always emitted);
+  and the envelope `meta.layer` description lists `agentio`, matching the code.
+
+### Fixed
+
+After an adversarial multi-agent review (nine confirmed findings) and a fresh-agent
+dogfooding pass of Layer 2B:
+
+- Distinct pages with an empty or whitespace-only body are no longer folded as exact
+  duplicates (they all hashed to the empty-string digest); each body-less result, such
+  as a snippet-only or failed-extraction page, now survives as its own entry.
+- The page-index query escaper strips control characters, so a NUL byte in a query no
+  longer raises a SQLite "unterminated string" error, and an arbitrary query stays safe.
+- The pure-Python BM25 fallback now folds diacritics and tokenizes Unicode letters
+  (NFKD plus a Unicode word pattern), so accented and non-Latin queries match the same
+  pages as the SQLite FTS5 adapter instead of silently returning nothing; its IDF and
+  `resolve_index` ordering after a content change were also aligned with FTS5.
+- The rendered Markdown status line never shows an impossible position (for example
+  "page 6 of 3") on a page past the last one, including when dedup shrinks the set.
+- `websearch open --search` degrades a page-index failure to a warning instead of
+  leaking a traceback, so a successful fetch and format is never lost to a search error.
+
+### Fixed
+
+After an adversarial multi-agent review and a fresh-agent dogfooding pass of Layer 1:
+
+- URL canonicalization no longer crashes on a malformed port (e.g. a non-numeric or
+  out-of-range port) or an IPv6 literal. Previously that ValueError propagated through
+  dedup and aborted the entire search, defeating per-engine fault tolerance; now the
+  whole canonicalization body is guarded and IPv6 hosts keep their brackets.
+- SearXNG and ddgs adapters tolerate malformed responses (non-object JSON, non-dict
+  entries), coerce upstream fields (score to float, publishedDate to str), and use a
+  valid ddgs region for BCP-47 language tags.
+- The router bounds hung engines (a slow engine can no longer block the request past
+  its timeout) and reports unknown engine names instead of silently dropping them.
+- The CLI returns a clean error Envelope on invalid input instead of an uncaught
+  traceback, and `--help` names the built-in engines.
+
+After an adversarial multi-agent review (22 confirmed findings) and a fresh-agent
+dogfooding pass of Layer 2A:
+
+- Stopped treating the Imperva `x-iinfo` / `x-cdn` headers as a block (they are on every
+  Imperva-proxied response, not just challenges), scanned high-precision DataDome and
+  PerimeterX markers on any status/size, and wired the previously dead Akamai body
+  markers. Made the markdown-link regexes non-backtracking to remove a ReDoS path.
+- Tuned the quality score so content-typed-but-thin pages (products, forums, listings)
+  no longer clear the 0.80 gate the way real articles do, widened the link-ratio band,
+  and made paragraph counting robust to single-newline markdown.
+- Decode bodies with declared-charset then detection instead of a blind UTF-8 fallback,
+  default a 10 MB streaming transport guard, and skip the HTML extractor on non-HTML
+  responses with a surfaced warning.
+- `--output-format text` now emits plain text (it was a silent no-op), added `--quiet`
+  for piping the body, and `request_id` is present in `meta` on every response path.
+
+### Notes
+
+- `fusion.method: score_convex` is accepted but currently falls back to `weighted_rrf`
+  (a warning is emitted).
+- Layer 2A still returns clean page content unmodified (so piping and composition stay
+  clean); the untrusted-content fence is applied at the Layer 3 agent boundary
+  (`web_fetch`/`web_open`), not in Layer 2A.
+- The FastMCP server ships in the base install (fastmcp is a base dependency). Harness
+  packaging and distribution (npx skills add, the Claude plugin and marketplace, the MCP
+  registry `server.json`, and PyPI/uvx) are built; see `docs/INSTALL.md`.

websearch_skill-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hector Oviedo
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.